Monte Carlo Syndrome

Monte Carlo Method

Monte Carlo Forecasting is a method for creating predictive forecasts based on a technique of repeatedly running random simulations of the samples to see a range of possible outcomes, and then to use those outcomes to forecast results of larger (or future situations). The predictive model offers a percentage probability of being in certain ranges.

It can be a very effective tool for statistical analysis of data, and there has been a surge in it’s use in Software delivery Forecasting. It can be a useful tool even with small sample sizes but it relies on three very crucial premises.

The sample data MUST be reflective of the forecast situation,
The data must be also be statistically independent (results of one event does not impact another).
The greater the sample size the more reliable the simulation will be.

Monte Carlo Fallacy

Rather amusingly there is also a psychological condition called the Monte Carlo Fallacy where we assume past results impose a probabilistic bias on future events. “The last 5 spins of a roulette wheel came up Black therefore the next is more likley to be Red as the odds must balance out.”

That is the fallacy. In a fair roulette wheel the odds of being black or red never change no matter how many times you get one result, any combination of outcomes has the same odds as any other. In fact it is far more likely that the wheel has a physical bias towards Black than the odds righting themselves, probability has no memory.

Applying the Method to the Fallacy

Amusing as it is the Monte Carlo Fallacy is the act of using observation to misinterpret probabalistic based statistics assuming the probabilities have memory, and the Monte Carlo Method is using observed events to create probabalistic forecasts by assuming (often dependent) events have independence.

Just as a point of interest. If you used the Monte Carlo Method for gambling on ‘fair’ Roulette Wheels you would likely have no difference to any other ‘system’ – probability cannot be influenced, it would simply be using a ‘method’ to compound your Fallacy.

However, in theory the method could be used to identify faulty Roulette Wheels or other unusual variations from probabilistic results (e.g. to spot cheating), or to predict gamblers’ behaviour.

Applying precision to inaccurate data is dysfunctional behavior. It is using the fog of precision to create an illusion of accuracy.

The Fallacy of the Monte Carlo Method

As you have probably surmised I have some grave concerns about using the Monte Carlo Method for forecasting software delivery. In my opinion the Monte Carlo Method applies a huge degree of precision to an inaccurate forecast. And applying precision to inaccuracy is dysfunctional behavior. It feels to me that we are using the fog of precision to create an illusion of accuracy.

The weaker the data available upon which to base one’s conclusion, the greater the precision which should be quoted in order to give the data authenticity.

Norman Ralph Augustine

Applying to Software Development

In software projects we tend to apply the reverse of the Monte Carlo Fallacy, we make the assumption that the past accurately predicts the future, so accurately we can give a percentage confidence level. By doing so we are making certain assumptions.

The sample data MUST be reflective of the forecast situation.

Future stories are similar in size, scope, complexity, and effort to the sample data. E.g. early stories are similar to later stories.
Future stories are impacted similarly by external events (we don’t learn or fix problems)
Our ability to do the work is constant
The team does not change, either in size or skill set
The team’s ability to work together does not improve or degrade

The sampledata must be also be statistically independent (results of one event does not impact another).

The future work is not made easier by learning from previous work
The future work is not made harder by adding to a growing code base
We are not creating more rework later than we did at the beginning
Testing, feedback or support do not change as product grows or ages.
We do not improve our skills, or knowledge of the domain.
We do not mitigate problems to prevent recurrence.
We do not learn.

Would you feel comfortable giving that list of caveats along with a Monte Carlo forecast?

Where does Monte Carlo Work?

Monte Carlo Method and the simulations have many useful applications, googling brings up a whole bunch of options, but it works best where observations of sample data can be used to predict behavior. As an example: parcel delivery time, whilst it may change over time it will likely be static enough to predict times based on certain volumes. Or to set reasonable SLAs for an IT help desk where the work is likely consistent in variation.

Where does Monte Carlo NOT Work?

Where it does not work very well is in situations where the sample is small or is not likely reflective of the event being forecasted. Or where past events influence future events, if you learn, or improve or grow, or become over loaded or congested

Monte Carlo magnifies the significance of the sample data. Say you took a sample of 10 items and a 1 in a 100 event occurred Monte Carlo would apply a 1 in 10 significance to that event despite it being 1 in a 100. In software terms an unusually large story or small story or abnormal blocking event can throw off the results by magnifying the significance.

Quick example,

I ask 100 people to solve a puzzle and measure the time each takes.

If I use Monte Carlo forecasting based on the results, it will likely give me a pretty good projection of how the next 100 people will take to complete the puzzle.

If I apply Monte Carlo to forecast how long it will take those same 100 people to complete the same puzzle again it will likely get it completely wrong as some of them will likely get quicker learning from the first attempt.

Forecasting is Hard

The problem is that in most cases Software delivery is uncertain, most software products are complex and the complexity varies from story to story. Work varies, early stories differ in composition, size and scope from later stories, we don’t order work in a manner that balances effort or delivery time, we order based on maximising value. Work evolves, we respond to feedback and we update. We learn, the first time we see a particular problem we may struggle but the next time it is a breeze. As a consequence forecasting is very very hard to do.

NoEstimates helped a lot, we discovered that upfront estimating stories only gave us a marginal gain in accuracy for an upfront cost, and that by simply counting stories and measuring throughput we can get an adequate level of predictability, if we accept and understand the limitations.

Whenever I see a forecast written out to two decimal places, I cannot help but wonder if there is a misunderstanding of the limitations of the data, and an illusion of precision.

Barry Ritholtz

Accurate Forecasting of Software Products is Snake Oil

I am being a touch cynical but it is my experience that in most cases of people using Monte Carlo Simulations for software delivery forecasting it is being used by people that do not fully understand the tool, and the results are being presented to people that do not understand the limitations.

I see people repeatedly tweak the settings until they get the answer they want,
others excluding data they don’t like
others making forecasts based on sample sizes of 6 stories
Many more based on backlogs or stories that include work that will be broken down at the point the work starts
or on backlogs excluding the possibility of new work being added (customer requests or bugs)

And those receiving the predictions believe that when the results say that there is a 90% confidence of hitting a particular date that they are completely unaware and uninformed of the assumptions behind that 90% figure or how it is calculated and there will be a great deal of expectation management needed to fix it later.

K.I.S.S.

I much prefer a simple moving average based on story count, nearly anyone can understand the numbers and the expectations and assertions of precision are absent so the expectation of accuracy is reduced. Good healthy conversation is invited about what might impact the product and it is very easy to see what could be done if the resulting forecast is beyond expectations.

Other than for a desire to dazzle someone with smoke and mirrors or to create false confidence, I struggle to see many situations where Monte Carlo adds any value over and above that which can be achieved with simple calculations. For me there is far more value in everyone understanding the calculation and limitations.

Final word

In the right context and for the right audience the Monte Carlo Method and the simulations are hugely valuable and can be used to great effect. But ensure you understand how and when to use them.

A forecast is only useful if it is data that can be used to make an informed decision to take action.

If you do not fully understand how Monte Carlo Simulations work (including the assumptions and limitations), OR if those you are presenting to, do not fully understand how Monte Carlo Simulations work or it’s limitations then be wary that you are not simply baffling your audience with graphs rather than presenting them with valuable information they can act on. They could be making ill-informed decisions.

Forecasting, asking the why? behind the When?

A Product Owner, or development team will often be asked for an estimate or a forecast for a product; a feature; an iteration; or a story. But when you go to a team member and ask them it is not uncommon for the colour to drain from their face, twitching to start, and their pulse to race. Past experience says that the next words they speak will haunt them for the foreseeable future, maybe even for the rest of their career. It is only a rough estimate you say to reassure them, but they just know you have other plans and the fate of mankind lies in the balance. Or so it seems.

dilbert estimate

The difference between a forecast and an estimate.

For most practical purposes there are only subtle differences, the main one being that a forecast deals exclusively with the future. When I think of the two I tend to think of an estimate as information and a forecast as expectation. It is likely that I will use estimates to create my forecast and I may use multiple estimates and even apply other factors to derive my subjective forecast. Whereas an estimate is generally an isolated objective assessment.

Both have huge degrees of variation; accuracy; precision; and risk factors, but in some instances it may very well be that my estimate and my forecast are the same for all practical purposes.

For example, If I was asked to estimate a journey, I may say that it is between 15 and 45 minutes and on average it is 30 minutes. If I am asked to forecast when I will arrive, that requires knowing not only the estimate but the time the journey starts and if there are other factors such as a need to stop to get fuel, or food. I may also add a certain weighting to the journey based on the time of day. Thus my forecast, whilst based on an estimate may not be the same as the estimate. Other times though I may simply take the estimate and it will become my forecast.

In practice though the terms are used interchangeably and likely it really makes very little difference, except when it comes to expectations. If I ask “How long will this take” I am asking for an estimate, if I ask “When will this be done” I am asking for a forecast, both are fine so long as everyone knows what is meant by the question.

Forecasting is misunderstood

Forecasting is guesswork, it may be scientific guesswork and it may be based on past experience and metrics and clever projection tools, but it is a guess. You will be wrong far more often than you are right. The more professional and clever and precise the forecast looks the more confidence you may instill in that guess. But it is still a guess, and when your audience gains undue confidence in your guess they have a tendency to believe it as fact not forecast. It might be that a guess is all that is needed but dressing a guess up and delivering it with confidence may create a perception of commitment.

A forecast is commitment (in the eyes of the one asking)

In normal circumstances by giving a forecast there is an implied commitment, even if you give caveat after caveat, and at any point where you even hint at a commitment to delivery to a fixed scope, fixed cost AND fixed date you are setting yourself up for disappointment. And sadly that is what a forecast is seen as by most people.

By all means work to a budget or a schedule or even a scope (although that is very likely to vary) but ideally ONLY one and never all three.

Understanding why a forecast is needed.

There are many reasons why people ask for forecasts, and the why is the most crucial aspect of the process. Understanding the why is the first step to providing the right information, and hopefully changing the conversation. Forecasting for stories and features is both more reliable and more accurate but the project/product level forecasting is where there is the most confusion and least understood purpose.

If possible, try to change the question of “When?” in to an understanding of “Why?”

Generally speaking we gather information to help us make decisions. If the information will have no bearing on our decisions then the gathering of that information is wasteful. Forecasting all too often falls into this category. We take time and effort to produce a forecast and yet the forecast has no bearing on any decision, that effort was wasted, or more often the level of detail in the forecast was unnecessary for the purpose for which it was used.

Making predictions of unknown work based on incomplete information and a variety of assumptions leads to poor decisions, especially when the questions being asked are not directly reflective of the decisions being made.

In my experience the Why? generally falls into three broad categories:

How long will this take?
How much will this cost? and
simple curiosity/routine.

But most people don’t ask why, they spend time creating a forecast and present it without knowing how it will be used. Some people asking for a forecast/estimate may not even know why they are asking, they just always get a forecast. But let us delve a little deeper.

How long will this take?

Why do you need to know? Some typical answers are:

I need to plan
I have dependencies
I need to prioritize
I have a deadline

How much will this cost?

Why do you need to know? Some typical answers are:

I want to know if I will get a good Return on Investment
I need to budget
I need to prioritize
I have limited available funds

Curiosity/routine

We always ask for a forecast, I need to put something in my report
I want reassurance that you know what you are doing
I want to know if my project is on track

What you will notice about these questions is that when you ask why, the first request for a forecast suddenly doesn’t make sense anymore, they are not really interested in the forecast itself, but in some other factor that they can infer from the forecast. If the intent is clear then the question can be tailored to get the required information in a better way.

e.g.

I need a forecast – Why?… I need to know when I can allocate staff to the next product/project.

In this case would a simple high level guess be sufficient? I feel confident that staff won’t be available for the next 3-6 months, in 3 months let’s review, I’ll have a better idea then…

I could put a lot of effort into a detailed forecast but an instinctive response may give all the information needed, saving us a lot of trouble.

I need a forecast – Why?… I want to ship this to maximize the Christmas shopping period – or I want to time the launch for a trade show etc.

This isn’t a request for a forecast, this is a request for an assurance that there will be something suitable available for a particular event/date. I can give you an assurance and confidence level without a detailed forecast, I may even change the priority of some features to ensure that those needs are factored into the product earlier. Or can de-scope some features to meet a certain date.

I need a forecast – Why?… I have limited funds available, and I want to know when I can start getting a return on this investment.

This isn’t a request for a forecast, it is a request to plan the product delivery so that revenue can be realized sooner and for the least investment. It may be possible to organise delivery so that future development is funded from delivering a reduced functionality product early. Or that development is spread over a longer period to meet your budget.

I need a forecast – Why?… I need to budget, The way this company works I must get approval for my project expenses and staffing in advance so I need to present forecasts of costs and timelines. This answer is twofold, first – can you challenge the process? It might be better to have a fixed staffing pool and prioritize products/projects such that the most important ones are done first and then move on to the next, in which case the forecast for this is irrelevant, it is a question of prioritization. Or if the issue is ensuring staffing for the forthcoming year could I simply say whether this product will not be completed in the next budget year?

I need a forecast – Why?… I want confidence that you know what you are doing. This is not a request for a forecast it is an assessment of trust in the team. There are many more reliable ways to ascertain confidence and trust in a team than asking for a forecast.

I need a forecast – Why?… I have a dependency on an aspect of this project. This may not be a request for a forecast of this project, but more a request to prioritize a dependency higher so it is completed sooner to enable other work to start.

I need a forecast – Why?… I want to know if my project is on track. Essentially what you are saying is that I want to track actual progress of work done, against a guess made in advance that was based on incomplete information and unclear expectations. And I will declare this project to be ahead or behind based on this. I am sure those of you reading this will know that what you are measuring here is the accuracy of the original guess, not the health of the project. But we have been doing that for decades so why stop now?

finally the closest to a genuine need for a forecast

I need a forecast – Why?… I need to prioritize or I want to know if I will get a good Return on Investment.

Both of these are very similar questions, but are really requests for estimates not forecasts a rough estimate helps me gauge the cost and when I evaluate that against my determination of the value expected to be gained from the project it may help me decide whether the project is worth doing at all or if there are other projects that are more important. E.g. If it is a short project it may impact my decision on priorities

But even here it not the estimate that has value it is just information that helps me evaluate and prioritize. If I already know that this project has huge value and will be my top priority does forecasting aid with that decision?

When does it make sense to forecast?

Listing those questions above it seems like I am suggesting that a request for a forecast is always the wrong question and is never really needed. And it is true that I did struggle to come up with a good example of when it makes sense to do a detailed project level forecast that included dates or any type of scheduling expectations.

It may be necessary for a sales contract, to have a common set of expectations. Although I would very much hope that sales contract for agile projects are for time and materials and are flexible in scope and dates, if not cost too. So for the purposes of setting expectations and in negotiations I can see that there is value in an estimate, although I do still wonder if a detailed forecast adds anything here that a reasonable cost estimate doesn’t cover. If possible I would rather work to an agreed date or an agreed budget than suggest a forecast that may lead to false expectations.

But the reality is that sometimes your customer does want one and wont tell you why, or doesn’t know why. Some customers (and managers) are willing to accept that a forecast will cost time and money and the more detailed it is the more it costs, and of course being more detailed may not make it any more accurate.

More detailed investigation for your forecasting is likely to build greater confidence but may not be more accurate and you should ask the question “will more detail have a material impact on your decisions?”, and if the extra effort wouldn’t affect your decision then it is just waste.

I would caution that if there is no other alternative and a forecast is made, that it is revised regularly and transparently, the sooner the forecast is seen as variable the more useful it is. There is so much assumption tied to a forecast that it becomes a ball and chain if there is not an expectation of it changing and so it must be refreshed regularly to prevent the early assumptions being seen as certainty, or they will lead to disappointment later.

Short term forecasting

The real value in forecasts is when the forecast is for a short frame of time. Over the short term we can have much more confidence in our forecasts, especially if we have been working on this project for a while and have historic information we can base our forecast on. There are fewer assumptions and less variables.

Can you forecast which features are likely to be included in the next release?
If I add a new feature now can you estimate a lead time for this?
Can you give me an estimate of how much this feature would cost?

By limiting the scope of the forecast to an areas where we have more confidence in our expectations the forecasts become more meaningful and whilst there is still a danger they are seen as commitments the risk is mitigated by the shorter time frame.

Alternatives to forecasting

Many of the questions above could have been resolved with much less effort than a detailed project level forecast. In most cases we could achieve sufficient accuracy for decision making with a high level estimate. E.g. A product like that is similar to ‘x’ therefore I’d estimate 4-8 months for a small team. Or as a rough estimate 12-18 months for two teams and calculate costs accordingly.

These estimates are certainly broad but if you have confidence in your teams and you believe they will use Agile principles to get value early and are able to communicate progress and be transparent with issues then I see no issue with broad estimates. It is sufficient to allocate staff and resources, to prioritize, to schedule and to determine Return on investment decisions.

For the other questions you may achieve far better answers through the use of Product/feature burndown charts, user story mapping, or even simple high level Road maps. These tools provide useful information which can be used for managing expectations, identifying dependencies and visualising the progress of a product. And crucially – aid in setting priorities and keeping the progress transparent.

The Chart is not the Voyage

Or more conventionally – The map is not the terrain.

This is an old quote attributed to Alfred Korzybski which is highly appropriate to my current client, as we make Charts of the oceans. They will be the first to agree that a chart is not the topography.

Charts can and frequently go out of date and so frequent updates of changes have to be notified to mariners promptly. In response to the Titanic disaster there was an international convention for the Safety of Life at Sea and now all larger ships are legally required to carry the latest charts and to keep them up to date in response to warnings. But even with an up to date chart no mariner would blindly assume it covered all hazards or was completely correct at any given point in time. Sands shift, sea beds may have objects that drift, some hazards may seem too small to note at some scales, some Hazards are temporary, etc.

The expression though is a warning to planners and navigators that the information they are using to plan is a simplified and likely an out-of-date illustration of the terrain. Navigational charts will often come with additional advice on approaching ports or other hazards but none can confidently claim to cover everything accurately.

Have you ever used a Sat Nav and found it taking you a convoluted and often much slower route because the pathfinding thinks a particular route looks shorter but is actually a narrow country lane that has limited passing places, I’m sure we’ve all see photos of trucks stuck in a narrow road as a result of following a map (or Sat Nav) without really understanding the terrain.

Planning

When planning a project you gather information to help you make the plan but here is the challenge of all planners:

If you try to cover every detail, every eventuality and every hazard it would likely take you as long as if you acted out the plan, probably longer as you would be mapping potential avenues that are never followed.
If you over simplify the plan you may omit key information and as in the case of the Sat Nav missing something crucial like a bridge being closed could throw the plan entirely. Making it difficult to stick to a plan.
If you plan from an illustration it is difficult to truly anticipate how long the actual journey will take.

Accurately planning for a team, large or small requires that you understand in detail both the journey and the team.

So how is it possible to get a team to successfully follow a detailed project plan?

Is it possible to successfully follow a detailed plan?

In a rather defeatist way, my answer is that it is not possible, or at least not consistently.

In waterfall projects, the normal solution is often to add huge amounts of contingency, if I think it is a 6 month project, call it 12 months etc. Have milestones to regroup and get back on track, but contingency when not needed is waste and in waterfall contingency is generally consumed. But contingency is in essence a form of planning based on an assumption that our plan is flawed – which it likely is, but this is hardly efficient.

We may also throw more staff at a plan or change scope or move dates the usual reactive crisis management that occurs when a Waterfall project is running late, but all of these are the result of a flawed plan, and in my experience the longer the plan duration the more inaccurate it becomes.

So if it is not possible to successfully plan a complex project why bother?

Essentially what I am saying is that Eisenhower hit the nail on the head when he said

“Plans are useless, But planning is indispensable”

Eisenhower

Planning is Indispensable

In Agile terms, my advice is to set a clear Vision, we absolutely must know where we want to go, and then simplify the plan into high-level objectives, and then prioritize them. Flesh out the detail of the highest priority item, and begin to explore more detail on the next highest priority and so on. Have a plan that has the minimal detail and expand it as you go and be prepared to adjust. Agile projects have the wonderful tendency to separate the wheat from the chaff and you will get a much leaner solution, lower priority features are deferred until later and often dropped altogether.

If we return to a voyage analogy if I were planning a voyage that involved multiple stops, I’d want to know the final destination and the major ports on the way perhaps an idea of key navigational way points, I would eventually want a detailed chart for each stage of my journey but I don’t need to study them yet (the charts may change or perhaps I will need to add a stop or skip one), the detail only matters when I reach that point. Right now all I need are detailed charts for the current stage, so long as I can plan that part of the journey I can proceed.

How long will the project take?

“But I’m a Project Manager and I need to know how long the project will take.”

If you have been nodding along with me as you read this you will be in agreement that the notion of accurately planning a complex project is a flawed concept, and that your previous attempts to predict project duration inevitably result in inflated estimates to cover contingency and you are not competitive. So the question becomes one of “do you really need to know how long this will take?” Or “why do you need to know how long it will take?”

Generally this falls into a number of categories:

1 Dependency:

If you are planning a launch date and it must coincide with another particular event then this may seem to be a reasonable request, but the question isn’t really one of how long the project will take, but one of how much can be completed by that date? Can it be delivered in phases? What is the MVP? How fixed is the date? Could it slip? Could it be deferred? What are the consequences if we are late?

The best advice would be to defer fixing the date as long as possible if content (MVP) is critical, or to ask yourself if you are willing to adjust scope, and or cost to meet the date? It is rare to find a project where 100% of the requested features are truly mandatory on a particular date and when that is the case it is unwise to have a fixed delivery date.

2 Return on investment

More often the usual reasons for wanting a plan are because of cost constraints or price/profit related issues, neither of which are helped by an inaccurate plan. What they mean is “How can I predict whether this project will be profitable?” or “I’d like an estimate for how much is this project going to cost me?”

Both of these are perfectly reasonable requests but neither require a date to be plucked from the air and written on the back of a contract for the soul of the Product Owner.

In all cases where Scope, Time or Cost are a limiting factor Agile planning and Agile delivery should mean that you get the highest Business Value items for the lowest cost, and that you will know at the earliest possible moment when there are issues that may impact on the success of the project. The aim is to deliver value at the earliest opportunity, in theory this is the most efficient and therefore most profitable/lowest cost solution for the given parameters. And if the project fails it will fail quickly for the lowest cost.

Agile doesn’t guarantee success or profitability, but when applied well it maximizes the value of successful projects, and perhaps more crucially in commercially driven projects it enables mitigation of failures at the earliest opportunity.

But you will have spotted that doesn’t answer the question: An estimate is a different matter entirely – see Estimates – Planning and estimation are frequently confused and they really shouldn’t be, but that is another topic.

3 Resource allocation

There may be other projects and you may want to predict when the team will become available, but assuming you are working on the highest priority project then your team is being utilized in the most appropriate way. Programme planning, like project planning is better done at a high level for the future and only becoming more specific as the task draws closer. If there is a dependency then see above, otherwise trust that the project/product will be completed as quickly as possible and the team can move on to the next task at the earliest opportunity.

Summary

There is nothing wrong with planning, planning is vitally important, the mistake is relying on a flawed plan, rigidly sticking to it in the face of reality.

Plan with the expectation that your plan will change.