Lies, Damned Lies and Project Metrics (Part 3) - Bruce F. Webster: Instrumentation

This is where instrumentation comes in. Instrumentation simply means setting up your IT project so you can track as much information as possible about its different aspects and characteristics, preferably in an objective, automated manner. One place to start is your SCM (software-configuration management) system, which should be tracking, not just source-code changes, but changes and versions for all project deliverables (requirements, specifications, models, diagrams, test plans and results, and so on).

This means that your instrumentation will likely need to be able to (programmatically) go into your SCM system and extract that information. With this access, you want to extract characteristics, including size, changes, attributes and authorship, from various files and deliverables.

Next, you want to extract information from your defect-management system (you do have one, don’t you?). You would gather information about the number of defects reported, closed to date, deferred, marked as duplicate, marked as enhancements and so on. You would track both total numbers and new reports for each unit of time (say, day or week), as well as the severity and priority of reported defects.

In conjunction with your SCM and defect-management systems, you also want to gather information about your change-control process, that is, the meetings you hold to prioritize defects and to approve or defer features. Another area to track is human information such as organizational structure, turnover, task assignments, background and qualifications of individuals, and meeting schedules. Again, as far as possible, this information-gathering should be objective and automated.

At this point, you’re probably throwing up your hands and saying, “Wait! Stop! That’s too much!”

My response to that is: We are willing to spend millions, tens of millions, even hundreds of millions of dollars on projects that fail, so why are we so reluctant to spend a tiny fraction of that to help those projects succeed? (The same short-sightedness applies to software quality assurance, but that’s a subject for another column.)

Beyond that, the advantage of gathering so much information is that it makes it very difficult to fudge that information. A developer or manager might be able to do that with one or two particular characteristics but not with all the characteristics you’re monitoring. What’s more, there’s a good chance that attempts to fudge a particular characteristic will show up due to inconsistencies with the other information being gathered.