Desperately Seeking App Dev Metrics

CIOs, IT managers and industry analysts have spent years trying to devise ways to measure the business value of information systems. While metrics exist for myriad technologies and operations, the one discipline that defies measurement is application development.

?What we found is that organizations really are suffering because there are not a lot of good practices and metrics to manage their application development teams more effectively,? says Marc Brown, vice president of product marketing at Borland Software, sponsor of a Forrester Research survey of application development managers.

?It really is a manual process,? says Carey Schwaber, Forrester senior analyst for application development. ?The project manager walks around and asks every developer how far along they are. The result is that you don?t get very good data when you rely on off-the-cuff information.?

Based on interviews with 20 development managers and executives in charge of application development at $1 billion-plus companies, Forrester concluded that the combination of two factors?the cost and complexity of metrics collection, and the reliance on superficial metrics?deters companies from making significant improvements in the ways they measure the delivery of software.

?There are a number of measures having to do with quality assurance, but we?ve never really addressed the fundamental issue,? says Brian Kilcourse, former CIO at Longs Drug Stores and now managing partner at RSR Research, a Miami firm that focuses on the retail industry?s use of IT. ?It all starts with someone saying, ?I need this,? and then goes all the way through the development process until ultimately it?s implemented as code,? adds Kilcourse, who also serves as an advisor to Borland.

When asked what obstacles they encountered in gathering application development metrics, 40 percent of those interviewed named the manual effort required in the data collection process. Similarly, 30 percent said collecting metrics is not a habit for developers, nor is it a natural part of the application development process.

Another potential metric for application development that scored weak ratings was productivity. Only two out of 20 companies interviewed said they gather productivity data on their developers. Many of those interviewed, Forrester reported, expressed strong interest in productivity metrics but said they?re dissatisfied with their use of the two most common metrics?lines of code and function points.

?The common units to measure developer productivity are variable or inaccurate,? Schwaber says. ?The fundamental problem is that it?s difficult to size software.?

Perhaps most telling, nearly two times as many companies said they want to gather data on the business value of their application development projects as those that say they do so. The most commonly gathered metrics are progress on schedule, cost, defects and milestones met.

The report concluded that companies should refuse to settle for schedule and cost as the only metrics for development groups. Although these tend to be the most accessible metrics, the report found that they may not be the best. ?The real problem may be that companies are looking at the wrong metrics in the first place,? Schwaber says.