Scott Rosenberg has written an important and entertaining book about the way software projects work—or don't. Dreaming In Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software (Crown, 2007), chronicles an open-source effort to build a better personal information manager. Published this month, the book also delves into the history and culture of software development in an attempt to answer a fundamental question: Why is software so hard?

Rosenberg is a founding editor of Salon, the seminal online magazine. While writing the book he had close access to the open-source project, run by the nonprofit Open Source Applications Foundation, and its principals, including Mitch Kapor, the head of OSAF and a software-industry legend. Following in the footsteps of Tracy Kidder's classic book about computer design, The Soul of a New Machine, Rosenberg's book limns the deadline-driven drama of the job and the personalities of the people involved. "I was trying to capture the applied creativity of the process—but I also wanted to tell a good story," he says.

Resource Library:

See Also: 5 Software Development Mistakes to Avoid

Kapor's team faced a series of delays and disappointments as they worked to make their product, named Chandler, into something useful and relevant (the project is ongoing). Rosenberg weaves that tale through a much larger saga about software projects and their discontents. Along the way he introduces generations of programming languages and programmers, gives plain-English explanations of coding arcana, and discusses some of the challenges facing developers in the future. Rosenberg spoke recently with Senior Writer Edward Cone about software development. An edited version of his remarks follows.

CIO Insight: Your narrative of the Chandler project weaves through some much larger issues concerning software development. Did you set out to work on such a large canvas?

Rosenberg: I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem.

Clearly, people have succeeded in creating amazing software systems that do work, and do serve people's needs. In many cases they are very large, from the scale of the Internet to the scale of enterprise platforms. So it's doable. But when you pick up the rock and look underneath the surface of every one of those systems, you find stories of delay and difficulty and problems. And it seems the software industry tends not to be introspective. There isn't a lot of history; there isn't a lot of examination of past mistakes. In the physical world, the examination of past mistakes and failures, and the application of principles based on what you have learned, is the classic technique of engineering. There is a thread of that kind of post-project review running through the software field, but overall it doesn't seem to be as closely studied or as widely known as it probably should be.

That's one of the things I tried to do in the book, to unearth that history. It's an exploration of this stuff, most definitely not a set of recommendations. I hope people will come away from the book with a deeper understanding of what happens in the making of a piece of software. I felt that in so many books about making technology, you'd get to the point where people actually start creating the software and then it would be kind of like the sex scene in an old movie: They would just skip it, cut to the next morning, cut to the marketing team getting ready to ship the product. It was like people would avert their eyes from the actual act of making software. Maybe they were afraid readers would be bored, or didn't understand software.

Read the full story on CIOInsight.com: Scott Rosenberg: What Makes Software So Hard