Estimating at a project level.

One of the most difficult aspects of the transition to Agile is the confusion over how estimation is done.

Estimation is difficult, the experts suggest that even with a full picture of what is required and with clear detailed and fixed requirements, the best estimators cannot realistically estimate better than within a 25% margin of error. It’s easily possible to do worse than this. But It isn’t possible to be consistently more accurate; it’s only possible to occasionally get lucky.

But in agile we start without clear requirements, we don’t have a fixed scope and chances are the requirements we do have are at a high level and there are many unknowns. I could talk about the cone of uncertainty but I’m not convinced most businesses will accept that level of uncertainty even if it is based on sound reasoning. In my experience they would rather base decisions on a specific guess than an accurate ranged estimate especially a wide range. Sounds daft when I say it like that but I bet you have experienced it.

Nevertheless it is still often necessary for commercial reasons to have a solid estimate before starting a project (Agile or otherwise), in many situations projects need to provide a good ROI or are limited by a budget.  In some situations the ability to estimate reliably could be the difference between the success and failure of a business. These estimates can be crucial.

So how do projects provide reliable and useful estimates?

First of all it is worth noting that estimates are largely misunderstood in general, they are misused and can often be damaging to the project. But still estimates are asked for and used to make important decisions.

In a survey from a few years ago*, a range of IT companies were asked about estimation strategies, the results were both worrying and yet reassuring that difficulties were universal.

* http://www.ambysoft.com/surveys/stateOfITUnion200907.html

Around 44% of the project teams in the survey described themselves as ‘Agile’ so this is a balanced pool of projects and should give an idea of estimation across the board.

When asked to give estimates to the business for project delivery around 65% of teams were asked by the business to provide estimates within the 25% margin of error range that experts in the field say is ‘impossible’. 11% were allowed no margin of error at all they had to specify a single date or a specific cost for the project,  conversely 21% were not asked to give any estimates at all. The rest allowed a margin of up to 50% on their estimates.

So how did that pan out for those companies?

Well 40% never even tracked whether those initial estimates were correct, it is difficult to draw any conclusions from this, but 40% came within that magic 25% of their estimates, which frankly is an incredible statistic, when I first read this I started questioning the validity of the survey. 40% of software project estimates were more accurate than the ‘experts’ say is possible to achieve consistently, 40% is more than just getting lucky it is frankly unbelievable.   At this point I was about to dismiss the survey as nonsense, but I read on…

How is it possible?

In order to achieve the 25% margin of error the projects did the following:

  • 18% admitted they had padded their original estimate
  • 63% de-scoped towards the end of the project to deliver on the estimated schedule.
  • 34% asked for extra funds to complete the projects on the original estimated schedule
  • 72% extended the schedule to deliver the promised scope (effectively revising the estimate and success was then measured on the revised estimate not the original)

It is impossible to tell from this how many of the projects matched the original estimates, but clearly it wasn’t very many, it is not a stretch to conclude that the vast majority of respondents de-scoped and/or extended the original estimates, including those that had already padded the original estimates.

Moving goalposts is the key

My reading of this survey is that very few if any delivered what was estimated in the originally estimated time-frame/budget. It makes very bleak reading and regardless of whether the project was or wasn’t Agile the estimates did not deliver what the business asked them to.

If we take the stated purpose as being simply to plan and budget and assume the estimates were not padded or interpreted then they hold very little value based on  the lack of accuracy.

In my opinion if any of the businesses that demanded such specific estimates went on to actually base business decisions on the accuracy of those estimates, then they were just setting themselves up for disappointment and future problems.

There is no way from this survey to conclude what the accuracy of the original estimates actually was other than to say that even with padding, de-scoping and extending schedules they were still unable to meet the original expectations and were overwhelmingly wrong and seemingly nearly always underestimated the true time/cost. This reads like a recipe for disappointed customers and shrinking profit margins.

That is a very long winded way of saying that (according to this survey at least) no one in the industry, Agile or otherwise is producing reliable estimates for software projects, we consistently get it wrong, and more worryingly fudge the figures so we never learn from our mistakes.  So any suggestion that estimating Agile projects is more difficult is not based in fact, estimating for software projects is difficult full stop.

Do estimates have value?

Now that is a different question, if I was running a business and I received a project estimate of 6 months, I would be foolish to consistently believe it will be delivered to the defined scope in that time-frame. But that doesn’t make the estimate useless.  If one project estimates 6 months and another estimates 3 months. I can conclude that the first is likely to take longer than the second, especially if the same person or group has estimated both.  Both estimates are likely wrong but chances are that on average and over time they will be wrong by a consistent margin, which makes them predictable.

If I check historic records I might be able to see that projects estimated at 6m generally take 8-12 months, or better yet I could ask the estimators to compare the current proposed project and identify a previously completed project that is closest in size and scope and use the actual figures from a sensible comparator.  Empirical evidence is so valuable I’m surprised more emphasis is not put into keeping track of past estimates and actual delivery costs and schedules.

Estimates are not commitments

Essentially we need to accept estimates as simply estimates not as a plan or a commitment.  Any PM that transposes an estimate of a software project straight into a plan is nuts, and it happens so often that in my experience developers turn white and have panic attacks when asked for an estimate, painful experience says they will be misused and ultimately the one that gave the estimate gets blamed.  If the business could be trusted to accept that estimates are not an exact science and factor in realistic contingency based on empirical evidence then developers would be less afraid to give estimates.

So how should we do it?

I have two suggestions, the first is to use an extension of the Planning Poker process.  Take a group of people that are experienced with software delivery and relatively knowledgeable about the scope and complexity of what is being asked. E.g. Product Owners, Business analysts, Project managers, representatives from development and testing.  Ask them to give estimates of a variety of projects relative to each other.  I’d use Fibonacci numbers or T-shirt estimates, to keep it at an abstract level.  If possible I’d try to include a benchmark project (or more than one) where the actual time/cost is known.

Blue-11Blue-6If we accept that the best we are going to get is a granular; relative; ball-park estimate of a project then this should give you that and more. In fact for budgeting purposes a reliable granular estimate is of far more value than an unreliable specific figure, and far more valuable than the estimates in the survey. Over time it is likely that the estimation team will learn and improve, they will get better with every completed project. I’d have far more confidence saying a project is either a Medium or Large T-shirt.  The T-shirt sizes could map to high level budgets.

My second suggestion which could be used in conjunction or independently of the first is to set a budget and ask the team to provide the best product they can within that time/cost. A good Scrum team will be able to prioritise stories and features to ensure you get the best value for money. If that budget is based on the poker estimates above it is more likely that the budget chosen is realistic and you will get the product you want.  You will also very quickly be able to tell if the project will be unable to meet the goal and can cut your losses early, rather than having to pour more money into a money-pit project that is over-running but too far down the line to cancel.

Estimation is a difficult skill to master but a group is far better than an individual.

Is estimating story points waste?

I occasionally participate in some of the discussions on Scrum/Agile where questions are posed and answered by experts. I have noted that there is a pretty strong contingent that see estimating as ‘waste’ and advocate against estimating, there are various techniques proposed that involve not estimating. I will state upfront that I am not one of them, and although I do understand and agree with many of their reasons, I feel that in some situations they may be missing out on a number of subtle and not so subtle benefits that the process of estimation brings.

Do it my way.

My first counter argument is that I object to anyone saying ‘their method is right and yours is wrong’. In my experience every team, every organisation and every project, is different, suggesting a one size fits all is ludicrous. I believe a good Agile Coach will assess each situation and seek to find a suitable framework for that particular situation. Blindly proposing changing a situation to fit your preferred method is hubris – in my opinion.

I will refer to my experience of ‘estimation’ and how and why I feel it is not waste and does actually add both direct and indirect value to a software delivery framework.

How can you assess ROI – without knowing the I?

First and foremost, it is very simply often necessary. Some are fortunate to be in situations where direct ROI (Return on Investment) is not scrutinised. But I have nearly always worked in situations that are commercially driven or commercially motivated. A feature is prioritised based on its value to the customer (the R in ROI) but not in a bubble, if the cost of providing that benefit is high (The I), the priority changes, often the feature can be assessed as not viable. A Product Owner or a business cannot make a judgement on ROI without an estimate of both R and I, if the team isn’t providing that estimate then the Product Owner or BA is doing the estimating themselves on some level and an estimate of work taken outside the team is highly likely to be less accurate and therefore less useful than if provided by the team.

Whose waste? Are you eliminating waste or dumping it on someone else?

So if ROI is a factor and someone IS doing an estimate somewhere, then calling it waste and not doing it within the team is just pushing it out to somewhere else, you are not eliminating waste you are moving the work elsewhere and to somewhere where the results are less accurate or reliable. This is not ‘lean’ it is not eliminating waste, it is actually just pushing a problem upstream and adding risk.

How much waste?

Even if you still believe that the act of estimating is waste – just how much waste are we talking about? I find story refining vital, one of the biggest hindrances to flow of getting stories to delivery is not having them refined in advance, if a story is well prepared the team can pick up and go. So assuming you agree with refining stories then estimating a story is simply a couple of minutes tagged on to the end of a refinement exercise – we are talking minutes – the waste(if you still believe it is waste) is tiny and yet the commercial value of the estimate to those outside the team is high. If you push it back on the PO or BA their effort will be much greater, you may even be creating waste by refusing to estimate.

Subtle benefits.

But even ignoring the commercial need for estimating, there are other benefits too. As part of the refining process a story is discussed and assessed until the team feel they have a common understanding. But how can you be sure they have sufficient understanding, when does the conversation end? In Scrum at some point they are able to estimate the story this is a natural destination of the conversation. In this situation the very act of estimating is a tool for focussing a discussion and reaching a consensus, the readiness to estimate is a measure of the conversation concluding.

Is an estimate a Barometer of consensus?

But it can also be a barometer of a consensus of understanding. If when estimating there is a wide distribution of estimates or a single estimate that differs from the others this is immediately an indicator that common understanding has not been achieved and that the requirements may not have not been universally understood, I say May as we do not know the reason for the difference without further discussion, but without an estimate this discussion would not have occurred. A consensus and narrow distribution of estimates is an indication that there is a common understanding, the fact that there is a relative size estimate that can be used for other purposes is a bonus. In this example the value in measuring that you have a common understanding of requirements and agreement that a story is ‘ready’ is hugely valuable to promoting flow.

Is an estimate a conversation starter?

But there is more, an estimate is a conversation piece, it may throw up suggestions of quicker solutions or ways to combine stories or breakdown stories. Without an estimate there is no focus for these discussion, without an idea of scale or scope there is no incentive to find a smaller solution or a more efficient solution.

An estimate is a conversation not a number.

In essence what I am saying is that an estimate is not just the number, it is the result or the output following a conversation and whilst you may debate the value of the output, the value of the conversation is considerable and is so often underestimated.