Is Software Development Your Weakest Link?

After years of attempts to effectively measure the business value of IT, if there’s one remaining “black box” operation that continues to defy measurement it’s application development. And judging from the results of a survey of development managers at 20 companies, IT departments are hungry for better ways to measure the effectiveness of software development projects.

“What we found is that organizations really are suffering, because there are not a lot of good practices and metrics to manage their application development teams more effectively,” says Marc Brown, vice president of product marketing at Borland Software Corporation, which sponsored the survey by Forrester Consulting.

“It really is a manual process,” says Carey Schwaber, senior analyst for application development at Forrester Research. “The project manager walks around and asks every developer how far along they are. The result is that you don’t get very good data when you rely on off-the-cuff information.”

Based on interviews with 20 development managers and executives in charge of application development at $1 billion-plus companies, Forrester concluded that two factors— the cost and complexity of metrics collection, and the reliance on superficial metrics— combined to deter companies from making significant improvements in the ways they measure the delivery of software.

“There are a number of measures having to do with quality assurance, but we’ve never really addressed the fundamental issue,” says Brian Kilcourse, former CIO at Long’s Drug and now managing partner at RSR Research, a Miami firm that focuses on the retail industry’s use of IT.

“It all starts with someone saying, ‘I need this,’ and then goes all the way through the development process until ultimately it’s implemented as code,” adds Kilcourse, who also serves as an advisor to Borland.

When asked what obstacles they encountered in gathering application development metrics, the biggest obstacle cited by 40 percent of those interviewed was the manual effort required in the data collection process.

Similarly, 30 percent cited as an obstacle the fact that collecting metrics was not a habit for developers, nor was it a natural part of the application development process.

Another potential metric for application development that was noticeably weak was productivity. Only 2 out of 20 companies interviewed said they gathered productivity data on their developers. Many of those interviewed, Forrester reported, expressed strong interest in productivity metrics but were dissatisfied with their use of the two most common metrics— lines of code and function points.

“The common units to measure developer productivity are variable or inaccurate,” Schwaber says. “The fundamental problem is that it’s hard to size software.”Perhaps most telling, nearly two times as many companies wanted to gather data on the business value of their application development projects as those that actually did. The most commonly gathered metrics cited by respondents were:

  • Schedule (cited by 19 out of 20)
  • Cost (17)
  • Defects (16)
  • Milestones Met (15)

    The report concluded that companies should refuse to settle for only schedule and cost as the only metrics for development groups. Although these tend to be the most accessible metrics, the report found that they may not be the best.

    “The real problem may be that companies are looking at the wrong metrics in the first place,” Schwaber concludes.