Lies; Damn Lies; and Forecasting…

NoEstimates in a Nutshell

NoEstimates has made a lot of traction over the last few years, with good reason, it is primarily about adopting Agile properly, delivering the valuable work in order of priority and in small chunks, and by doing so eliminating the need for a heavy duty estimation process. If we are only planning for the next delivery we can reliably forecast.

But sadly that is generally not good enough and some level of forecasting is often requested. So NoEstimates came up with a very useful and low cost method of forecasting. However, it has brought with it a whole host of misunderstandings, most of which are not from the book. The author must be as frustrated as anyone by the misinterpretation of his proposal. This has led to resistance from many (including me) to adopting this method for forecasting. I am all for delivering value quickly and small chunks or prioritized work, but slogans that are used to excuse bad behaviour are damaging and hard to resolve, especially when they seem so simple.

My biggest bugbear and one I have covered previously is that many have interpreted NoEstimates as an excuse to skip story refining entirely, this was not in the book but nevertheless you can see any number of articles on the internet professing how adopting NoEstimates has saved them from wasteful refining meetings, the misconception is that if you don’t need to estimate the story then the act of understanding the story is no longer required. When actually the author was suggesting that you don’t need to refine all work up front and could defer deeper understanding until it became relevant – the last responsible moment.

Story Writing and being Estimable

I encourage those writing stories to use the INVEST model for assessing the suitability of a story and in that: the ‘E’ is Estimable, but that doesn’t mean you must actually estimate the story, just that you ask yourself whether the story is clear enough and well understood enough to estimate if asked – are there open questions? is it clear what the acceptance criteria are and that these can be met? There may be a subtle distinction there, but NoEstimates does not offer an alternative to writing and refining good stories. It is just a method for simple forecasting and encouraging deferring effort until it is necessary.

How does NoEstimates work?

Caveat aside I will try to give a very high level summary of how NoEstimates forecasting works, and when and where it doesn’t work. I shall do so via the medium of potatoes.

Preparing Dinner

I have a pile of potatoes on the side and I am peeling them ready for a big family dinner. My wife asks me how much longer will it take me? By counting how many potatoes I have peeled in the last 5 minutes (10) and by counting the potatoes I still have left to do (30) I can quickly and simply calculate a forecast of 15 minutes.

That is NoEstimates forecasting in a nutshell, it really is that simple.

Assumptions

However, the mathematics requires a certain set of assumptions,

1. I did not apply any sorting criteria to the potatoes I selected- e.g. I wasn’t picking either small or large potatoes, we assume my selection was random or at least consistent with how I will behave in the future.

2. That the team doing the work doesn’t change, if my son were to take over to finish the job he may very well be faster or slower than me and my forecast would not be useful.

3. We also assume that I will not get faster

4. We assume that all potatoes in the backlog will be peeled, and no others will be added. If my wife asks me to peel more potatoes or to do the carrots too, the forecast will no longer apply and will need revising.
So there we have it, a very simple and surprisingly accurate method for forecasting future work. But do you see any flaws to the system?

Flaws in the system

Flaw 1. Comparing potatoes with potatoes

The first flaw is that I am getting potatoes ready for roasting so I want them to be broadly similar in size, so when I get to peel a potato I am also sometimes slicing it, some potatoes only need peeling others may be sliced once and others more than once. Some potatoes are bad and I throw them away.

If my wife comes along and sees my pile of potatoes and asks how much longer it will take? I can look at my pile of potatoes I have completed in the last 5 minutes (18) and I can count the potatoes I still have left to (30). The problem is I don’t know how many unpeeled potatoes were needed to produce those 18 peeled and sliced potatoes, I am not comparing like for like. To be able to give this estimate I would have needed to count how many unpeeled potatoes I had peeled, information I don’t have. Maybe I could take a guess and then use that guess to extrapolate a forecast, but that sounds like guesswork rather than forecasting.

Flaw 2. Forecasting an unknown

Let’s assume that I am producing 10 peeled potatoes in 5 minutes, and I am asked to give a forecast as to when I will be done, but so far I have been grabbing a handful of potatoes at a time, peeling them and then going back for more, one could say that my backlog of work is not definitive, We have a whole sack of potatoes but I won’t use them all for this one meal. I am simply adding work as I need it. My aim being to judge when I am satisfied I am done and start cooking. It is very difficult for me to judge when the sack will be empty or when I have prepared enough for lunch.

Flaw 3. Changing and evolving work

It is a big family dinner and uncle Freddie has just called to say he will be coming so we need to add more food, Aunt Florance eats like a bird so probably not worth doing a full portion for her. And the table isn’t really big enough for everyone, so maybe we should do an early meal for the kids first. The point here is that simple forecasting only works if you have a reasonably good assessment of what the work is still to be done, if your backlog of work is evolving, work being added or removed then the forecast will be unstable.

Flaw 4. Assuming consistency

When selecting work to do next I have a tendency to choose the work that will bring me the most value for the least effort. The highest ROI, so in this case I may choose the small potatoes first, less peeling and less chopping. But that means that if I count my competed work and use that to forecast my future work I will end up underestimating how much is left, the backlog has some really big awkward shaped potatoes that will take far longer to do. But my forecast is based on only doing small simple potatoes.

Doesn’t this apply to all forms of estimates and forecasts?

Flaws 2 and 3 apply to any form of forecasting, they are not unique to NoEstimates. Flaw 1 and Flaw 4 could potentially be mitigated with the use of T-shirt sizing or story points, but to do so requires a level of upfront effort. Effort that is not spent on peeling potatoes, so may well be considered waste – that is unless you see value in a more reliable forecast.
For me Flaw 1 is my main objection to NoEstimates (beyond the belief that refining is unnecessary) When stories are refined and better understood it is normal to split or discard stories, and often add stories as the subject becomes better understood. So any forecasting tool that uses a metric based on counting refined stories to predict a backlog of unrefined stories is risking over simplification of the problem. But because the maths is so simple it can lead to a confidence level that exceeds the quality of the data. These assumptions based on flawed data gets even worse when you use a tool like Monte Carlo forecasting which applies a further confidence level to the forecast. By giving a date combined with a confidence level adds such a degree of validity and assurance that it is easy to forget that a forecast based on duff data will result in a duff estimate – no matter how prettily we dress it up.

Summary

Forecasting is risky at the best of times, especially in Agile where it is our goal to have the work evolve and change in order to give the customer what they truly want. Forecasting needs to be understood by both parties and accepted that it is an evolving and changing metric. Anyone expecting a forecast to be a commitment or to be static is likely to be disappointed. Just take a look at the weather forecast, the week ahead changes day by day, the further away the forecast the more unreliable it becomes. Understanding the limits of the forecasting method is crucial, a simple tool like NoEstimates is fantastic IF the assumptions can be satisfied, if they cannot then the forecast will be unreliable.

It is probably also true that your forecast will improve if you spend more effort understanding the work. Time spent refining the stories will improve your knowledge. But no forecast can reliably predict work you do not yet know about. The question as always is “What problem are you trying to solve by forecasting?” That will guide you in determining whether the up front effort is worth it.
Related articles:

Why I think estimating isn’t waste
Demystifying story point estimation

I do not think that word means what you think it means…

As an Agile Coach I naturally talk about agile practices very frequently, and participate in a variety of discussions with people of varied levels of understanding of agile principles and practices. But there are certain words that I notice are used with a disproportionate frequency, and used to describe different things. One word in particular gets used very frequently with emphasis and rarely in the right context, that word is Efficiency. And just for the record, I have been know to misuse it and am making a concerted effort to find the right word for the context, to that end I will try to clarify some of the situations where a different word may be more appropriate.

Efficiency seems to be used to convey such things as. Utilization, Productivity, Effectiveness, Velocity, and sometimes even Efficiency. Naturally I am not suggesting that people are not using English correctly, or that they don’t know what ‘efficiency’ means, but when you look at the examples below you will see that whilst they use the word efficiency in a way that is syntactically correct, they are missing the depth of their question or statement, and often the context. The preoccupation with efficiency results in behaviour that reduces the effectiveness of the team.

For example:

Isn’t Pair Programming less efficient than working solo?
Isn’t it less efficient to have a developer testing software?
Isn’t it inefficient to have the whole team doing story writing (or backlog refining, or stand-up, etc)?
Isn’t it inefficient to attend all these meetings, we’d be more productive here?
Wouldn’t it be more efficient if I specialize in this one area and all tasks come to me?
Wouldn’t we be more efficient and increase our velocity if we ignored technical debt or skipped testing?

I previously covered an example where maximizing utilization of resources was seen as being efficient, but that efficiency was at the expense of the profitability of the business. I think you are missing the point…

In most cases these questions are simply missing the bigger picture, or the larger context.

In KanBan there is a saying that any efficiency improvement on anything other than your primary constraint (your bottleneck) is just an illusion.

If something isn’t on your critical path then it shouldn’t be your focus. Of course as you make improvements to flow and address one constraint the critical path may change, but you should stay focused on the critical path. Identify your constraint and put your efforts there.

Isn’t Pair Programming less efficient than working solo?

The Efficiency of Pair Programming is a recurring question, and one that I likely will not quash here, and that is not my goal. The issue is whether the question of efficiency is the right word, or the right focus for your efforts.

The definition of Efficiency is: “achieving a goal with the least effort or least waste.”

Pair Programming assumes two developers at one workstation, and the implication of the question is that more can be achieved with less waste by working independently in parallel.

The problem with Efficiency is understanding what’s it is that you are measuring – what is your goal, and identifying whether this particular activity is your constraint in achieving that goal.

Let us take a car journey as a very simple example: What is the most efficient journey? Is it driving at 50mph so as to use less fuel? Is it to get to the destination in the shortest time to reduce utilization of the vehicle, or could it be driving in the middle of the night when the roads are clear and the vehicle is not needed by anyone else?

In all of those examples we are focused solely on the individual journey itself and even then we still have ambiguity of which efficiency we are concerned with. But very likely the reality in that situation the destination is just a step towards the goal. That isolated Driving journey is unlikely to be entire goal in itself – unless maybe I am a taxi driver, and even then I suspect my constraint is maximizing fares rather than reducing costs, so likely maximizing the number of fares rather than driving efficiently is my goal. Driving efficiently only at 4 am may not be the ideal business model.

The road less travelled…

If for now we assume that we are driving efficiently to use less fuel, because fuel is expensive and money is more of a constraint than our time. Then possibly the most efficient journey would be to combine multiple journeys into one (car sharing, or batch delivery), or make a phone call, in this context a journey not taken(or shared), is far more efficient than the most efficient journey.

I glossed over that fairly quickly, so take a moment to think about it. Efficiently doing an unnecessary task is just waste.

Efficiently doing an unnecessary task is just waste.

Time = Money

But hold on, the most efficient journey for me could simply be one that gets me to my destination on time, regardless of cost or duration. My time and the appointment are more important than the cost efficiency of the journey. Sure I would prefer the journey to be shorter and cost less if that can be achieved without impacting on my goal, but my priority is being there at a particular time. A super efficient journey that gets me to my destination 10 minutes early so I can sit idle for 10 minutes does not help me to my goal, the efficiency gain is just an illusion.

For a while I used to car share, and it saved me quite a lot of money, but I was tied to a schedule and my journey was much longer. After a while I realized my constraint was time and not money. The ability to be flexible over my schedule was a much bigger constraint. My journey became more expensive but I needed flexibility to reach my goal. In other words efficiency in one area may not just be an illusion it may actually cost you more elsewhere as a consequence.

This is a far too common situation in business, a business will often go on an efficiency drive and look to cut costs without any consideration to the larger impact that the efficiency savings are causing. The focus becomes narrow and they delude themselves and often others that they are being efficient, when actually the true constraint is ignored. The cost cutting creates larger costs elsewhere.

Why? What is the goal?

I don’t mean to labour the point, but to be able to understand efficiency we must first understand the true goal, and it is very likely that the true efficiency we are interested in is related to a goal at a higher level.

Let us revisit the Pair Programming. Our goal in programming is to deliver a software solution, likely a quality solution that meets the client’s needs, with minimal support requirements. It is likely that they are also interested in maximizing return on investment. So where does the issue of efficiency for programmers come in?

Pair Programming as an activity shares and grows knowledge, it often enables more creative solutions, the quality is higher, there is an inherent in built code review and QA in to the process. So the question isn’t so much whether Pair Programming is more or less efficient in terms of number of lines of code written; or stories coded; or bugs found; but whether it is producing a better return on investment overall in the context of our true goal: in terms of quality, support and meeting the client’s needs.

I don’t plan on answering the question to the satisfaction of the skeptics but I suspect my opinion is fairly evident. My point is not to directly answer the question, the point is that the real issue is not what it first appears to be when the question was broached.

If your team is dealing with support issues, bugs, deployment problems, getting features to customers quickly and so on. Then it is possible that ‘coding time’ is not your primary constraint.

In this case the response to the question of efficiency is: What are you measuring to determine efficiency? Or… Is efficiency of one singular part of the process our primary goal or is it just an illusion? Would being more efficient help you to be more effective at achieving your goal?

Summary

In this example the word efficiency is taken in a context to justify behaviour that possibly diminishes the effectiveness of the team. Lean principles do look to minimize waste and to be more efficient, but lean looks at the end to end flow and is only concerned with efficiencies when it is applied to the critical path or a bottleneck and ONLY when those efficiencies are removing waste in the context of our overall goal, not simply a small part of the flow.

If we improve efficiency at a point before a bottleneck all we do is pile up work faster, if we improve efficiency after a bottleneck or anywhere not on the critical path we fail to see the benefit because the flow is already limited by an earlier bottleneck, there is no net gain to our efforts. Our efforts are just an illusion.

As a very generalised statement we as individuals find it inconceivable! (Had to include it somewhere) to view our work as part of a larger scope, our desire to be efficient in our own little bubble often leads us to behaviour that is inefficient for the larger scope that we are part of. We should be less concerned with being efficient in our roles and more concerned with doing what is effective for the larger system or team we are part of.

Efficiency is not the same as being Effective

Prefer being Effective over being Efficient. Make sure you are effective at achieving the goal first and foremost before even looking at efficiency, and then only if increasing efficiency does not come at the expense of being effective.

Visualising our work in that larger scope is one of the best ways of understanding where we are struggling to be effective and to highlight where the true efficiencies can be gained. But try to see the whole process not just your part of it.

I think you are missing the point…

A brief story….

A team found that many of their impediments and problems with delivery were caused by a lack of access to IT operations or lack of responsiveness, they requested a member of the IT operations team be co-located with (and become part of) the delivery team. This was approved and trialled as an experiment.

From the perspective of the development team this was a major success, delivery times improved, delays were reduced and the major constraint to delivery was aleiviated. In short software projects were delivered sooner and with less complications, the overall cost of delivery went down dramatically.

However, the management team decided to remove the IT ops team member and retrurn them to their old team on the basis that they were not being fully utilized. The notion of a human resource not being fully utilized was too much for them to bear.

[Please note when I say utilization – that is on IT Ops tasks – they were using ‘slack time’ to support the team in other ways]

When I challenged this decision, I was told that they were unable to increase headcount for IT operations staff unless it could be demonstrated that existing resources were more than 100% utilized on IT Ops tasks over the course of a normal week. So anyone not 100% utilized was considered a problem for them and became a burden on other team members.

This same IT operations group has a huge backlog of work and lead times were extremely long, they were a major bottleneck for almost every aspect of the business operations. Sadly the management team did not see any correlation between the absence of slack time and the long lead times.

Too often the dependency on IT Operations and their perceived hindrance to business operations, leads to unnecessary conflicts and frustrations. Usually felt by the staff on the IT Ops teams. But it is because IT Operations are so important that they get this reputation.

How can you justify extra resource if you are not fully utilizing the ones you have?

I have no doubt this is a common story, “how can you justify extra resource if you are not fully utilizing the ones you have?” In principle this would appear to be a perfectly reasonable and logical statement, and I can understand why many management teams fall into this trap. But it is the wrong question.

What troubled me most was that we had demonstrated that the impact and delays suffered by the delivery team in just this one example were actually costing the organization so much that they could easily pay for 5 or more additional IT operations staff with the benefits gained. And more so that this situation could be repeated all over the business. The response from the IT Ops manager was that I was missing the point, and they couldn’t justify underutilizing staff. Their measure for success was maximizing utilization of staff, and not supporting business goals, like say profitability.

I see this mainly as a failure of the business, for not making clear to management where they contribute and how they impact on the larger business objectives. That combined with a manager that either doesn’t see or doesn’t challenge this omission leads to perpetuation of dysfunctional behaviour.

Managing resources effectively not efficiently

I apologize for comparing people to resources, but in this context it is applicable.

Could you imagine if that same principle was applied to other resources? Your PC would be taken away if you didn’t use it 40 hours a week, you would be forced sell your car because my guess is that you only use it 10% of the time. You would wear the same clothes every day. Cooking would be a nightmare if you could only have food where all ingredients took exactly the same amount of prep time and cooking time. I’m getting silly now but it is the same principle, by totally ignoring the reason why we have a resource and focusing entirely on maximizing it’s utilization we behave in really rather perverse and contrary ways.

In any other context the notion that anything less than 100% utilization prohibits buying a second item (regardless of how beneficial) is plain nuts.

Please do not misunderstand, I am not encouraging waste by this, I am encouraging understanding of how resources are being used. When you are meeting the business goals and your flow is optimized, that is the time to look at whether reducing waste is possible but to do so in a way that ensures that any reduction in waste is not hurting business goals. It may turn out that just like your car, the optimum utilization is less than 100% and availability and responsiveness are valued higher than utilisation.

Summary

In this situation, I am very proud to be “missing the point” and I wish far more managers would take the time to understand how their role impacts the larger business and perhaps they too will miss the point and start asking important questions. IT Operations in particular are the lifeblood of so many businesses and their actions can make or break a business, understanding all the areas of the business that IT operations supports and how the cost of delay is often significantly higher than the individual cost to operations of that task or that ‘resource’. The ability for IT operations to prioritize and react promptly to the needs of a business is a far better measurement than measuring utilization of staff.

I’d highly recommend that anyone in IT Operations reads the book The Phoenix Project by Gene Kim, and ensure that anyone that benefits from the service they provide also read it.

Understanding how you contribute to the goals of the business is so important, and if after that you truly believe that your role should be maximizing the utilization of resources and not supporting the business achieve it’s goals, then I would suggest it may be you that is missing the point.