2011-02-15

Estimating Software Development

Are We There Yet?

As an engineer's career progresses, it's more or less inevitable for her to have to do increasingly complex project management and interact with business folks . Much of the time, what this means is that somebody (usually selling something, either to customers for money or to the board of directors for approval and/or money) needs to know when Flanginator 2.0 will be released. Perhaps just as often, it means that somebody already sold Flanginator 2.0 (in beta, if you're lucky) on a contract starting six weeks from yesterday and they need to know what can be crammed into the next six weeks so they can tell the client what they actually bought. In non-dysfunctional organizations, an engineering manager is eventually called upon to answer a few simple questions:

  • How long will Flanginator 2.0 take to build?
  • How much of Flanginator 2.0 can you do to get something out by date X?
  • What resources do you need to build Flanginator 2.0 by event X?
  • Why is Flanginator 2.0 taking so long?
Ideally the engineer is summoned early on, before dates are set and products are sold, but in practice that's not always the case. This is where the art and science of software development estimation comes in. And it's a bit of a paradox, because engineers are both the best and the worst people to do this.

The Reluctant Leading The Blind

Simplifying grossly, it's helpful to see the typical web shop as a place where software engineers, the Makers' Guild, build things business people sell, and business people, the Merchants' Guild, sell things software engineers build.

Time (to market) and timing are everything to the Merchant, while quality, cleverness and/or craftsmanship are everything to the Maker; and both are completely reasonable. The sooner the Merchant gets to sell a product, and the more closely the product availability can be tied to some important externality (e.g. a trade show, a conference, your customers' budget calendar), the sooner the revenue rolls in, and the harder it is for the competition to catch up. Complementarily, the higher-quality the product, the more satisfied the Maker and her customers are, the more revenue rolls in, and the harder it is for the competition to catch up.

Note each guild's goals are exactly the same as the other guild's--make money and beat the competition. But the two camps are going about it from two very different perspectives: the Merchant is very concerned with time, shrinking it or molding it to fit into a complex web of external calendars and deadlines over which she has very little control. By contrast, time is the least of the Maker's concerns, because she has to worry about building a product that works with no splinters or dead-ends or a shaky foundation--it'll be ready when it's done, and we can't compromise code quality / scalability / employee morale / features / database structures / code reviews / our finely chiseled system architecture just to get the product out faster.

In the best-case scenario, this tension leads to a very brisk product release schedule with constant improvements to products that sell themselves because they're finely tuned to what the market wants. In the worst-case scenario, engineers hate the slimy loose-cannon idiots from sales, and sales people hate the scruffy maladjusted untucked nerds from engineering who couldn't sell a warm jacket to a naked Eskimo.

The Thanksgiving Dinner Parable

Taking my cues from a long-dead craftsman from the Mediterranean region, who used to get a ton of mileage out of ostensibly non-à-propos at all yet profoundly relevant stories about people and things unrelated to the question at hand, I'd like to step back from engineering and sales for a moment to try to get at the crux of the biscuit of software estimates, by considering the commonly-shared experience of the Thanksgiving dinner (at least here in the US; feel free to substitute any other magniprandial calendricality of your choice).

In very simple terms, a Thanksgiving dinner is a family gathering on the last Thursday in November during which a large quantity of home-cooked food is consumed, also at home, by nuclear families and sometimes Others (usually not until they have reached the Significant stage). The meal normally includes a very large turkey, cranberry sauce, mashed potatoes or yams, some non-starchy vegetable, stuffing, gravy, some manner of bread, pumpkin pie, and beverages. It is widely accepted that the average Thanksgiving convive will describe with the most fondness and enjoyment those events having involved the maximum amount of home-cooked items, especially when made entirely from scratch, as against, say, canned cranberry sauce or frozen Sara Lee pumpkin pie, which isn't half bad, if you ask me.

Now, if you ask a Thanksgiving dinner maker how long the whole shebang is going to take, you'll probably get a wide range of answers. A young, first-time cook might say "oh golly, I have no idea, it'll take at least a week, and it's already Tuesday!" as s/he rushes to the grocery store. A Thanksgiving veteran might say "I'd say a morning of shopping, thaw the bird overnight, a day of cooking, so let's call it three days." I'd bet some serious Thanksgiving dinner makers put a lot more time into it, while Thanksgiving dilettantes may devote just a few hours (or none at all and just go to the restaurant or order in). I'd also bet not a single one of those guesses will be near the time it actually takes when all is said and done. People are simply not very good at guesstimating how long things take, unless said things are very few, very well-defined and well-rehearsed, and immune to externalities like a flat tire on the way to the store, a gas leak, frigid temperatures adding five hours to the bird's usual thawing time, etc.

Of course nobody would ask the Thanksgiving dinner maker for a very, very specific estimate of how long it will take, or a very, very detailed breakdown of every little task with a time and a name attached to each. If you requested a Thanksgiving task list from a naive cook you'd probably get something like this:

  • go shopping (2 hours)
  • thaw the frozen turkey (36 hours)
  • cook turkey (6 hours)
  • make pumpkin pie (2 hours)
  • etc.
In reality, though, what actually happens is this:
  • drive to Trader Joe's (10 minutes)
  • find a place to park and walk to store (10 minutes)
  • shop (25 minutes)
  • stand in at the register in the crowded store (10 minutes)
  • go back to car, load trunk, return cart (5 minutes)
  • drive to Safeway for the rest of the stuff (10 minutes)
  • find a place to park and walk to store (10 minutes)
  • shop (25 minutes)
  • stand in at the register in the crowded store (10 minutes)
  • go back to car, load trunk, return cart (5 minutes)
  • go to the farmer's market because the asparagus was just ugly (10 minutes)
  • shop at farmer's market (1 hour)
  • get gas (10 minutes)
  • go home (10 minutes)
  • unload (10 minutes)
We've already blown the "go shopping" bullet point estimate (2 hours) by 75% (3 1/2 hours). I'll spare you the expansion of the rest of the list, as I trust the point is made--as absurdly detailed as the longer breakdown is, it's the only way you can get close to how long any single piece of functionality of your Thanksgiving dinner is going to take. The larger/fuzzier the piece ("cook turkey" is a lot more than just sticking a bird in an oven--there's stuffing, basting, gravy, turkey skin hydration and temperature monitoring, etc), the less accurate the ballparks inevitably are.

Many software engineers are Thanksgiving dinner makers, thinking in terms of blocks of functionality that often require a lot of work. And because writing software usually means building something that you haven't built before, there's very little directly applicable hard evidence in your past experience to guide you in your time estimate for the new feature. Some well-rehearsed tasks can be estimated very accurately assuming everything works (e.g. kickstart a new server on standard hardware), but that assumption itself is often faulty since, well, nothing actually works all the time--we're talking about computer hardware and software, and we're barely out of the Ford Model-T era of sophistication and reliability.

So one key to better estimates is to learn and practice the skill of breaking those chunks down into well-defined tasks that can likely accomplished by one or two people in a day or less. If that's too hard, break up your larger features into smaller feature subsets, and break those down into tasks. Use Post-It Notes, with names and time estimates on each of them, line them up on a wall, then add up the times--there's your time estimate. Of course it'll be rough, and three times as long as anyone expected, so that's where you start cutting features and grouping tasks into releasable phases that can be rolled out in daily or weekly increments, or major versions of your software over a longer period of time.

There are diminishing returns the deeper you go into details for your estimate; splitting up a day's work into eight one-hour segments (or smaller) requires a huge amount of upfront analysis and thinking that will most likely be inaccurate or irrelevant by the time you start, since you haven't started writing the software yet and today's guesses about which individual classes and member variables you'll need two weeks from now are mostly a waste of time as the features and implementation details will have changed three times between now and then. So save the very detailed (sub-1-day) breakdowns until you've broken up your feature set into phases, and each phase into bits of functionality, and don't tackle too much in advance--a couple of release cycles at the most, because the work you do today for phase 1 may turn out to invalidate the guesses you make about phase 4.

All that means is that software estimates are just that: estimates. The only time you can precisely say how long a project is going to take is after it's done, and even then it's rarely practical to be exact, because being exact requires either sneaky or overbearingly bureaucratic time-tracking shenanigans, and who has the time?

Note this doesn't absolve the Makers of the responsibility to produce decent estimates, to consciously work at improving them, and to make every reasonable effort to build quality products in accordance with what often seems like a completely fanciful set of arbitrary external deadlines. On the other hand, frank conversations need to take place with the Merchants so that truly arbitrary deadlines that only exist in a Merchant's mind are objectively and clearly separated from the handful of hard deadlines the blowing of which can have truly serious consequences, so that pressure, potential disappointment and ultimate recriminations are kept to a bare minimum.

Real and Fake Deadlines

In addition to estimates, a software engineer's job also requires becoming creative and conversant in the art of compromise. Do you really need the Flanginator to have an electronic stereo klaxon with 127 foot-switchable sounds, or will a hand-activated horn suffice? Does the Flanginator positively, absolutely need to be out and tested by February 12th, or is there flexibility?

Some compromises can't be done: a big turkey will not thaw satisfactorily in two hours. You also can't just eat cranberry sauce and nothing else. Often they can: maybe Whole Foods mashed potatoes are just as good as the ones you were going to make before you fell in the shower and sprained your potato-mashing wrist. Sometimes they're not worth it: do you really want to go to Whole Foods on Thanksgiving Thursday morning, or call in a caterer with 12 hours' notice?

So another key is to be honest with the whole team (Merchants and Makers) and distinguish between real and fake deadlines. Real deadlines are events or signed contracts where having some product is a requirement, like a trade show or a meeting to pitch investors. Blowing those is like having Thanksgiving on Friday--ça ne se fait pas. Sacrifices will likely need to be made all round if the timing gets tight (cut features, hacks, throw-away code). The responsibility is best shared across the team, and your Merchants need to be on board with (and shoulder some of) the sacrifices made to make the deadline. Fake deadlines are those features that just "need to go out tomorrow so this reporter who told me she might write a piece about us for next month's issue can see we're on the ball." Those are more like dinner with friends: no one will die if you skip a weekend, or hang on Saturday rather than Friday, or order in instead of home-baking your famous prime rib cheesecake.

No Eye-Rolling

It's hard to find organizations where engineers don't roll their eyes and complains "Jeez, sales sold something we can't possibly build again. Jill must be trying to max out her quarterly bonus / they have no idea what it takes to build that thing." Those organizations' sales folks are also more likely to wonder "How can we be expected to bring in revenue if engineering won't build products we can sell? Our customers close their budgets by November, so if we don't have Flanginator 2.0 ready to demo by October 17 at 2:17pm EST, then it's an entire year's work down the toilet." This is bad, but not very difficult to address.

Educate your developers and product folks by getting the sales folks' perspective on what the market wants. The sales force is on the ground, talking to customers every day, and has a unique perspective on what kinds of products can be sold, so sales should be involved in the product development. Even a small UI tweak being considered to make something fit more gracefully into a page can have a huge impact on the sellability of a product (maybe it means your paying partner's logo will look smaller, or be their name will be moved off to a less desirable position). The only worse thing than having to scramble to hotfix a roll-back of this change is getting a call from sales right after they hear about the tweak from their very angry paying customers. It's not fair to sales, it's not fair to engineers, and worst of all it's not fair to your paying customers. Get sales involved.

On the engineering side, invite your engineers to industry events, focus groups, and user testing; heck, encourage them to run some of those user-testing sessions. They're inexpensive and eye-opening, and nothing beats hearing the same three wishes or complaints directly from real customers.

Ultimately nothing beats product and business leaders with some technical sense, engineering leaders with some business sense, and engineers who are realistic and honest about what can and can't be done. Ideally the skill sets in a product development organization overlap a bit like this:



If you don't have that today, get started on it now, because your organization is a vital organism that must--and can--change, adapt, learn and improve.