Estimating at a project level.

One of the most difficult aspects of the transition to Agile is the confusion over how estimation is done.

Estimation is difficult, the experts suggest that even with a full picture of what is required and with clear detailed and fixed requirements, the best estimators cannot realistically estimate better than within a 25% margin of error. It’s easily possible to do worse than this. But It isn’t possible to be consistently more accurate; it’s only possible to occasionally get lucky.

But in agile we start without clear requirements, we don’t have a fixed scope and chances are the requirements we do have are at a high level and there are many unknowns. I could talk about the cone of uncertainty but I’m not convinced most businesses will accept that level of uncertainty even if it is based on sound reasoning. In my experience they would rather base decisions on a specific guess than an accurate ranged estimate especially a wide range. Sounds daft when I say it like that but I bet you have experienced it.

Nevertheless it is still often necessary for commercial reasons to have a solid estimate before starting a project (Agile or otherwise), in many situations projects need to provide a good ROI or are limited by a budget.  In some situations the ability to estimate reliably could be the difference between the success and failure of a business. These estimates can be crucial.

So how do projects provide reliable and useful estimates?

First of all it is worth noting that estimates are largely misunderstood in general, they are misused and can often be damaging to the project. But still estimates are asked for and used to make important decisions.

In a survey from a few years ago*, a range of IT companies were asked about estimation strategies, the results were both worrying and yet reassuring that difficulties were universal.

* http://www.ambysoft.com/surveys/stateOfITUnion200907.html

Around 44% of the project teams in the survey described themselves as ‘Agile’ so this is a balanced pool of projects and should give an idea of estimation across the board.

When asked to give estimates to the business for project delivery around 65% of teams were asked by the business to provide estimates within the 25% margin of error range that experts in the field say is ‘impossible’. 11% were allowed no margin of error at all they had to specify a single date or a specific cost for the project,  conversely 21% were not asked to give any estimates at all. The rest allowed a margin of up to 50% on their estimates.

So how did that pan out for those companies?

Well 40% never even tracked whether those initial estimates were correct, it is difficult to draw any conclusions from this, but 40% came within that magic 25% of their estimates, which frankly is an incredible statistic, when I first read this I started questioning the validity of the survey. 40% of software project estimates were more accurate than the ‘experts’ say is possible to achieve consistently, 40% is more than just getting lucky it is frankly unbelievable.   At this point I was about to dismiss the survey as nonsense, but I read on…

How is it possible?

In order to achieve the 25% margin of error the projects did the following:

  • 18% admitted they had padded their original estimate
  • 63% de-scoped towards the end of the project to deliver on the estimated schedule.
  • 34% asked for extra funds to complete the projects on the original estimated schedule
  • 72% extended the schedule to deliver the promised scope (effectively revising the estimate and success was then measured on the revised estimate not the original)

It is impossible to tell from this how many of the projects matched the original estimates, but clearly it wasn’t very many, it is not a stretch to conclude that the vast majority of respondents de-scoped and/or extended the original estimates, including those that had already padded the original estimates.

Moving goalposts is the key

My reading of this survey is that very few if any delivered what was estimated in the originally estimated time-frame/budget. It makes very bleak reading and regardless of whether the project was or wasn’t Agile the estimates did not deliver what the business asked them to.

If we take the stated purpose as being simply to plan and budget and assume the estimates were not padded or interpreted then they hold very little value based on  the lack of accuracy.

In my opinion if any of the businesses that demanded such specific estimates went on to actually base business decisions on the accuracy of those estimates, then they were just setting themselves up for disappointment and future problems.

There is no way from this survey to conclude what the accuracy of the original estimates actually was other than to say that even with padding, de-scoping and extending schedules they were still unable to meet the original expectations and were overwhelmingly wrong and seemingly nearly always underestimated the true time/cost. This reads like a recipe for disappointed customers and shrinking profit margins.

That is a very long winded way of saying that (according to this survey at least) no one in the industry, Agile or otherwise is producing reliable estimates for software projects, we consistently get it wrong, and more worryingly fudge the figures so we never learn from our mistakes.  So any suggestion that estimating Agile projects is more difficult is not based in fact, estimating for software projects is difficult full stop.

Do estimates have value?

Now that is a different question, if I was running a business and I received a project estimate of 6 months, I would be foolish to consistently believe it will be delivered to the defined scope in that time-frame. But that doesn’t make the estimate useless.  If one project estimates 6 months and another estimates 3 months. I can conclude that the first is likely to take longer than the second, especially if the same person or group has estimated both.  Both estimates are likely wrong but chances are that on average and over time they will be wrong by a consistent margin, which makes them predictable.

If I check historic records I might be able to see that projects estimated at 6m generally take 8-12 months, or better yet I could ask the estimators to compare the current proposed project and identify a previously completed project that is closest in size and scope and use the actual figures from a sensible comparator.  Empirical evidence is so valuable I’m surprised more emphasis is not put into keeping track of past estimates and actual delivery costs and schedules.

Estimates are not commitments

Essentially we need to accept estimates as simply estimates not as a plan or a commitment.  Any PM that transposes an estimate of a software project straight into a plan is nuts, and it happens so often that in my experience developers turn white and have panic attacks when asked for an estimate, painful experience says they will be misused and ultimately the one that gave the estimate gets blamed.  If the business could be trusted to accept that estimates are not an exact science and factor in realistic contingency based on empirical evidence then developers would be less afraid to give estimates.

So how should we do it?

I have two suggestions, the first is to use an extension of the Planning Poker process.  Take a group of people that are experienced with software delivery and relatively knowledgeable about the scope and complexity of what is being asked. E.g. Product Owners, Business analysts, Project managers, representatives from development and testing.  Ask them to give estimates of a variety of projects relative to each other.  I’d use Fibonacci numbers or T-shirt estimates, to keep it at an abstract level.  If possible I’d try to include a benchmark project (or more than one) where the actual time/cost is known.

Blue-11Blue-6If we accept that the best we are going to get is a granular; relative; ball-park estimate of a project then this should give you that and more. In fact for budgeting purposes a reliable granular estimate is of far more value than an unreliable specific figure, and far more valuable than the estimates in the survey. Over time it is likely that the estimation team will learn and improve, they will get better with every completed project. I’d have far more confidence saying a project is either a Medium or Large T-shirt.  The T-shirt sizes could map to high level budgets.

My second suggestion which could be used in conjunction or independently of the first is to set a budget and ask the team to provide the best product they can within that time/cost. A good Scrum team will be able to prioritise stories and features to ensure you get the best value for money. If that budget is based on the poker estimates above it is more likely that the budget chosen is realistic and you will get the product you want.  You will also very quickly be able to tell if the project will be unable to meet the goal and can cut your losses early, rather than having to pour more money into a money-pit project that is over-running but too far down the line to cancel.

Estimation is a difficult skill to master but a group is far better than an individual.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s