The Planning Dilemma

After a few years in the tech industry in Silicon Valley, I've noticed a recurring theme in growing startups: planning is a problem. Now this isn't a novel insight--books have been written (partially) on the topic, successful consultancies have been built (partially) to solve this problem, and lots of very smart people have been working on it for a long time.

Ultimately, however, every organization needs to figure out what sets of practices work best for its own unique circumstances: how big and experienced their team is, what kind of product they're building, how risk-averse and regulated their sector or customers are, and many other considerations. Much like learning to read, dance or play the piano, it's up to the learner to put in the work so they can become good at it. Planning is no different.

The Growth of Planning

The particular dilemma that led me to write this post is a common trajectory I've seen in quite a few growing startups grappling with the question of "how do we plan". It tends to go like this:

1. No planning. Feature / task prioritization is obvious, the number of people doing the work is small, and you don't have time to do anything but build as fast as you can.

2. You have your beta out, enough to get a little funding (maybe a big seed or even an A round) so you can actually hire a couple of engineers, a product manager, maybe a designer, beyond the founder(s) and initial hire(s). You may have traffic to your app, or actual customers using your service, maybe a product-focused board member, a trade show or hard date you want to hit in a few months to demo your app to secure more funding. Whatever the reason, you start putting some planning in place, maybe with daily stand-ups to coordinate between teams, or weekly sprint planning. Still fairly lightweight. Your leads/execs get together once every 2-4 weeks for higher-level feature prioritization and roadmapping, more or less often depending on your product and industry.

3. You've secured funding and hired more people. You now have 2-3 distinct teams working on separate features (web and mobile, Android and iOS, e-commerce and logistics, design and manufacturing, whatever). You now have a fully-fledged marketing team with "rolling thunder" tactics, media tours to plan, conference appearances to organize, trade show booths to buy, ad spend budgets to propose, etc. You may have a manufacturing pipeline with very real lead times. The point is that people outside of engineering need to know what's happening when, sometimes pretty far in advance.

Yes, we've that software estimates are lies, that "it'll ship when it's ready," and other enlightened developer-friendly manifestos. I won't discuss the merits of that point in this post, and simply stipulate someone somewhere needs to know what's happening when beyond the next few days.

Planning For Others

At this stage of growth, in most (all?) of the startups I've observed or been a part of, the typical setup from step 2 expands to:

a) 2-3 teams, each with dedicated product, design and engineering resources; then

b) management overhead to coordinate between those teams and make sure the company's overall goals and priorities are being reflected correctly; then

c) meta-management overhead to make sure the meta-product management from b) is communicated effectively to people outside the product/engineering org (e.g. marketing, sales, execs, board, etc)

In practice, that often looks like this:

  • execs meet, often "off-site", once a quarter or so, to define the Big Goals or Roadmap or whatever it's called for the company; then
  • division heads from b) (VP product, VP engineering) have regular planning sessions to define and prioritize the next batch of work for some period (1-4 weeks) (the "pre-sprint planning")
  • each division head meets with "their" leads (PM, lead eng) from each team in a) to clarify and prioritize stories for the upcoming sprint (the "sprint planning", per team)
  • the division heads communicate any relevant timing updates to their peers (e.g. sales or marketing)
  • each team has their own internal touch points (daily standups are still common)
  • each team may elect to do a post-sprint retrospective to look back at the sprint, see what was good and what could be improved, to feed into the next sprint

Given that sprints tend to be brief in order to be able to respond quickly to changing priorities (sometimes as short as one week), and that any sufficiently meaty story needs a fair bit of time to write, read and understand before it can be implemented, I'd estimate the above planning duties combined make up at least a whole day's work per week per developer, and probably about three days' work per week per division head.

If you want to check my math:

For developers:

  • sprint planning = 2-4 hours
  • retrospective = 2 hours
  • daily standup = 5 x 12 min = 1 hour
  • context switching costs = 1-2 hours over the course of the week
  • ancillary negotiation, hemming, hawing, clarification, etc. over stories = 1-2 hours

Twenty percent of a developer's time spent on process management in the name of agility seems excessive to me.

Process Artifacts

Anecdotally, both as a developer and as a manager, I have heard feedback that daily stand-ups as a regular event aren't terribly useful (they are a costly interruption / context switch and often devolve into status reports that aren't useful to anybody or tangents that take too long and engage far too many people who are too polite to just leave a conversation they're not involved in; dependencies and blockers are best dealt with one on one, not as a group).

I have also participated in and/or led retrospectives and have mixed feelings about them; as blameless post-mortems (not necessarily after an outage: they could be held after a good or a bad/stressful release) I find them to be good vehicles for venting and constructive criticism that can lead to improvements, but as meta-exercises about the process, I find them self-serving and unnecessary.

Sprint planning is tricky. Writing stories is difficult; estimating them arguably harder; story points, planning poker and other artifacts are questionable; evidence-based scheduling seems more promising, but I feel you need a dedicated technical but non-coding manager to deal with it reliably and dispassionately, and that's hard to come by in a small startup.

What's The Upshot?

I don't have a good answer. At this point I'm pretty familiar with the new software development orthodoxy (most things under the "agile" umbrella), and while I find it preferable to a waterfall approach in principle (a system that is gleefully mocked by many people who've never actually experienced it; I certainly haven't, and I've been around a while), I feel a lot of its standard assumptions, tools and practices need to be challenged objectively and dispassionately. Otherwise it's just bandwagoning, and it's the opposite of what it's supposed to be--pragmatic.

I'd love to hear your thoughts and experience and learn from people who've been dealing with this. Please comment or get in touch!