December 23, 2007

Dreaming In Code

Dreaming In Code
Scott Rosenberg


(December 23, 2007)

"Software is hard," Donald Knuth wrote many years ago, when software seemed much simpler than it does today. This brave book tries to explain just how hard software is, to an audience that knows nothing at all about software and believes this ignorance to be a virtue. People don't know exactly how their cars work, but everyone has opened the hood and seen the engine whir and watched wheels move. Lots of people have no real idea what programming is, or why it's hard.

Rosenberg clearly planned this to be the software Soul of a New Machine, Tracy Kidder's memorable exposé of life on the cutting edge of computer design. He chose for his target a sure thing: Chandler, a personal information manager instigated and funded by Lotus founder Mitch Kapor. With ready access to ample funds, management expertise, the power of open source, and some of the best software minds in the industry, Chandler must have seemed a sure thing.

It sure seemed scary to me, since it launched shortly after Eastgate committed to Tinderbox, stems from the same inspirations, and sought to address some of the same problems. Because Chandler overlaps so much with Tinderbox, I find it difficult to examine this book with confidence.

Vexingly, Chandler didn't work out: by the time Rosenberg decided to finish the book, Chandler had bogged down in protracted code thrashing. But we don't know that there are no second acts in software; perhaps Chandler is going to emerge next week and be terrific. Or perhaps someone will fork it and save the project. Or perhaps Tinderbox made better technical tradeoffs and created a useful modern agenda and that's really what was needed.

Rosenberg makes clear that Chandler suffered from a baked-in incoherence: it was a successor to Kapor's agenda and so it was about agents and lexical properties of notes, but it was also supposed to leverage ubiquitous email and calendaring, and so wound up being about dates and places. That's too much to fit into a product, and there's a Microsoft blockage astride the calendar/email track. Not recognizing this was probably an irremediable mistake. The hybrid Python/C++ architecture always looked dicey to me, as did the premature commitment to be cross platform at launch. But hindsight is easy (and that hybrid architecture looks better now than it did in 2001).

On this framework, Rosenberg hangs a masterful and engaging survey of the thinking that underlies contemporary software engineering. This overview will have lasting importance, as I think it's destined be the textbook that introduces a generation of students throughout the world to the professional practice of software and to its founding voices — Brooks and the Mythical Man Month, Parnas, Joy, the postmoderns, the agilists.

What Rosenberg doesn't capture — because Chandler seldom captured it — is the way software actually gets written: in slow, steady segments, in dashing sprints, in long nights of inspiration, in weeks of staring at the screen, but always — in the end — by one or two people working to get something to work. In practice, this usually means one or two people imagining how it might work, and then making it happen. There wasn't enough of this in Chandler, and when it did happen, it too often happened to infrastructure, deep in foundations that were expected to underpin grand structures that were never built.

Business writers tend to attribute corporate success to the genius of a CEO, and to blame losses on that executive's personal foolishness and depravity. This is sentimentality, not journalism, and its prevalence is a symptom of the corruption of our business press.

Rosenberg set out to replicate a terrific book about hardware in order to explain software. He chose wisely. He had terrific access. He was spectacularly unlucky; instead of covering the inside story of a revolutionary movement, he has the inside story of a project that fizzled. He brilliantly avoids focusing on the failure, the consequences, the recriminations; instead, he explores the underlying needs and explains why failures like this happen all the time. To have carved such a fine, generous, and useful book out of the debris is a very fine accomplishment indeed.