Is Efficiency the killer of projects?

I have worked in software delivery for over 25 years now and in that time I don’t recall ever hearing of a project being cancelled due to lack of efficiency. By contrast I have seen numerous project cancelled because of misguided decisions in the name of efficiency. The most common is where team decide to build out infrastructure or data layers or business logic that they “know they will need later” and it is more efficient to do them now. The team works really hard but the client sees little or no progress and cancels the project. This sadly is something I have seen far too often.

This thought process is anti-Lean and it is anti-Agile but still it comes up daily. Teams anticipate future work and prepare for it, they over engineer stories or add elements that were not requested, “while I was here I added somethings we will need later”, all the time giving the impression to the customer that progress is slow. The team themselves feel they working hard and are adding value by being efficient. This lack of alignment between team activity and customer priorities and expectations almost always results in customer disappointment and teams feeling underappreciated. Since it is the customer that pays the bills misalignment with them is destructive to a project. That perceived future efficiency is never realized because the project doesn’t last long enough to benefit from it.

So why is it that we see efficiency as such a powerful virtue despite evidence to the contrary?

The sooner begun, the sooner done
proverb

A lot of it stems from misguided definitions of efficiency. From an early age many of us are taught that the sooner something is started the sooner it is finished. Or if you don’t start it, it will never get finished. We have a bulk buying culture and a bulk buy mentality. The belief that over time we will save money or make efficiency gains. What we don’t think about is opportunity cost or what the trade offs are for our actions. These costs in general far out-weigh any efficiency gains.

Cashflow is king

82% of small business closures are due to cashflow rather than profitability. It is not that they are not profitable but that they don’t have the cash available immediately often because it is tied up in unsold inventory or materials waiting to be processed. This is essentially the same manifestation of what is happening in software. We are tying up our time and energy in future potential and we find the project gets cancelled before we get to profit from our efforts. Value to customers is like cash is to businesses. It is no good telling a customer we are accumulating value for later when they need and expect the value now.

So the claim that the best way to get something done is to begin is only half right. The best way to get something done is to finish what you started, before starting something else. If we pay more attention to starting than we do to finishing we end up with many started jobs and very few finished jobs. We need to reverse our sense of efficiency to see value in finishing jobs and efficiency in completing work sooner rather than starting more jobs or extending them.

Stop starting, Start finishing

Whether it be software or manufacturing the efficiency we should be striving for is reducing the number of activities in progress. Be efficient in how many we complete, not how many we start. Starting on something that will not be finished until much later will more often than not create additional work in maintenance and confusion, not to mention the more valuable work that could have been done while you were being ‘efficient.’

Let’s reevaluate our understanding of efficiency and let’s stop starting and start finishing.

Tic Toc – Productivity Experiment

Have you ever noticed that late projects only ever seem to get later?
Why is it that any gains get absorbed and delays get compounded?

Let’s say I have a mini-project where I have a builder adding a kitchen to a house, he will construct the walls and then I’ll have a plumber come in to install the pipes and fittings. The builder estimates 4 days and the plumber 4 days.

If the builder is done sooner, say after 2 days, that gain is lost as the plumber was expecting 4 days so is not available until the originally planned date. If however the builder takes longer – say 6 days then this impacts on the plumber and at best delays him by 2 days, but more likely he has other commitments so it may result in delays beyond the 2 days ‘lost’ by the builder if he has to reschedule.

The only way to avoid this is for the plumber to build in slack before and after the planned job which is not practical unless he significantly inflates his fees to enable that level of flexibility.

That is a very simple example of a single dependency between two variable events. The majority of systems have chains of dependencies, each compounding this problem for every additional link in the chain.

Just think about your Doctor, they make appointments every 15 minutes throughout the day, if the average appointment lasts 15 minutes then it is likely by the end of the day as any appointment that runs over will delay the next appointment but any time they get ahead the patient hasn’t arrived yet. By the end of the day they may be running a long way behind and relying on your slack time to ensure you get seen. These are just a couple of examples of the Theory of Constraints in action.

Team Activity/Experiment

I’d like to share an activity I have been using in training classes to help people understand the impact of variability, constraints and dependencies on a system – including and especially software delivery. The principles are not limited to software or manufacturing but apply to any system where you are dependent on someone or something. It also shows how improvements in the wrong area do nothing to help your team.

I find learning is best cemented when you can take a group and have them look at a problem in a particular way so that they discover the solution and form their own understanding. No matter how respected the teacher offering solutions is they will never have the same impact as when you discover the answer for yourself. This is why I love teaching in a way that involves experiments.

The Experiment

The activity is best played in groups of 4, this gives everyone a role (and a roll of the dice) and enough opinions to have a debate about the implications.

Each team starts with 4 workstations or 4 stages in a workflow.

Each team starts with a backlog of work (a pile of blocks) and each block must flow through each workstation.

In any given iteration, each person in the workflow has the same potential output/productivity and is represented with the roll of a dice. The dice roll simulates a perfectly balanced system and allows for the variability of work: some tasks are easier or harder than others and some days we are more productive than others, but in this simulation we are all – on average – equally capable, we are perfectly balanced.

We run for 10 iterations. Each iteration is started with the first person will perform their work by rolling the dice, moving blocks, and passing the dice on to the next person. Any unmoved blocks remain in progress until the next round, and so on until each person has completed the work for that iteration. Each person in theory is capable of producing an average of 3.5 units of work and after 10 iterations the system at worst case will produce 10 units at best it will have produced 60.

Over 10 iterations, each workstation has an average capability of 35 units, all workstations have identical potential capability. If everyone worked at average productivity the system should be capable of producing 35 units of work.

Planning based on potential capability

This is very similar to the planning process in Scrum where planning is based on estimated times for all tasks. With planning based on the sum of the duration of all tasks being equal to the amount of time available in the sprint. E.g. 10 days worth of team effort in a 10 day sprint.

In the game, we ask the teams to predict the system output, and challenge any team that predicts a system output of less than 35 to justify why they are expecting their workers to be working at below their average potential.

This can be an interesting debate. Some suggest that people are not as variable as dice, or that stories are not as variable, but this usually results in debate and finally agreement that in fact people are MORE variable: tired, hungry, sick, bad day, good day, vacation, level of experience or familiarity with the job at hand etc. all result in huge variation in the person’s output on any given day. Stories can be split to be a smaller size to reduce variation in theory, but the complexity varies. Some need input from 3rd parties, some will go smoothly while others will hit issues, some take longer depending on who is working on it and so on. We take steps to reduce variation but we cannot eliminate it completely. We rarely know everything, so there will always be some surprises.

This conversation is crucial to understanding the the problem that most of us routinely overlook or choose to ignore despite the massive impact on throughput. We allow this conversation to flow to help people understand the fundamental variability in any system.

By the end of the conversation normally there is consensus that variability is the normal situation and a dice roll is an adequate metaphor, albeit real life is far more volatile.

First Iteration

In the first iteration, the teams will be set up so that each player rolls the dice and passes the appropriate number of blocks to the next player, and then that player rolls and passes, and so on until all players have ‘worked’ for that iteration(cycle).

We then repeat for 9 further iterations(cycles) and observe the outcome.

You can run this experiment for yourself and play along here: Tic Toc Game

In general, the result will be an average output of around two-thirds or a little more of what would be expected – e.g. around 23-27 out of an expected 35. Sometimes more, sometimes less, as you would expect when introducing variability but the average for the room is typically in the ~25 range.

We ask the teams to explain why they are running at 70% efficiency and what is going wrong. This usually results in one person being identified as being unlucky or consistently rolling low numbers, but generally there is some understanding that there is a reason for it.

At this point you can dive deeper but usually there is sufficient belief that luck is a factor so you may need them to play again for some to accept that it is not luck.

Conclusion

By running the experiment a few times people generally begin to realise that the nature of variability and dependency have a pretty significant impact on one another and that creating a balanced system is actually pretty futile. We can improve the situation by reducing the dependencies, cross training people or giving them more autonomy – one of the principles of agile is to have a team with everyone needed to get the job done – this is because of this situation. Dependencies have far more impact on productivity than most people are willing to accept. The other factor is variability, whilst there is no way to remove variability completely, we can strive for smaller stories. This is one way of limiting variability. When it comes to humans it is even harder to remove variability, but if we strive for a sustainable pace and a good working environment, this reduces variability even if it doesn’t remove it entirely.

Second Iteration

This experiment shows what will happen with a perfectly balanced system, but what is the impact when you have an unbalanced system or add WiP (Work in Progress) limits to a system? And have you ever wondered what the financial cost of big deployments was, we can experiment to see the how much it costs to have daily deliveries compared to monthly or quarterly deliveries.

In the next iteration we can explore the impact of an unbalanced system, where your system has a bottleneck and how you can deal with it.

Explaining Herbie…

I have been asked by a friend to write an article explaining the concepts of Herbie from the book “The Goal” in a more accessible way. The book itself is amazing but in my opinion the language used in the 5-focusing steps has tried too hard to be scientific and precise that it makes it harder to understand. My annotations simplify the terms but likely lose some of the precision of the wording, I offer this as an introduction to the topic but encourage reading the book to get the full impact, I cannot do it justice here.

Herbie is a character from the book “The Goal” he was used in the story as a metaphor for a system constraint and the impact it has on your delivery. I’d highly encourage you to read the book but hopefully it is not essential to be able to follow the explanation, the purpose of the metaphor was to make the thought processes relate-able, hopefully this helps.

The 5 focusing steps

Identify the system constraint
Exploit the constraint
(Make the best use of what you currently have)
Subordinate everything to the constraint
(Make decisions based on the constraint, the system should run at the pace of the constraint)
Elevate the constraint
(Make improvements to the constraint)
Repeat the steps

Step 1: Identify the constraint

We need to figure out which part of the system is holding us back the most – limiting our ability. We should know that only an improvement at the constraint makes a difference.

Herbie: In the example of Herbie, he was observed as holding everyone back he was slow and the rest of the group had to regularly stop to wait for him to catch up or were racing ahead. In the case of Herbie the constraint was evident through observation.

Reality: There are different ways to identify a system constraint sometimes observation and experience is sufficient, or metrics can help. The simplest way is to look at your work is it piling up somewhere? Do you have a lot of stories in Dev Done, or waiting on PO approval or needing wire-frames? Piles of work are a sign of a constraint.

Another common constraint are blockers, if you are consistently waiting on another team, another person or another resource this is a sign of a constraint, we may use blocker clustering or other root-cause technique to see if there is a common theme to delays.

The third most common technique is to review cycle-time for stories, look for stories that are taking longer than others see if there is a pattern, common aspects, this could be a constraint that may be improved.

Step 2: Exploit the constraint (Optimize)

Before we consider adding capacity, we should make the best use of the capacity we already have. We ‘Exploit’ in this sense means doing everything possible to use the constraint to its fullest capacity. This is also the easiest and generally cheapest improvement you can make to your system.

Herbie: In order to maximise the performance of Herbie, the contents of his backpack were shared out so he could concentrate on the most important task of walking. Making the others slower (less efficient) by carrying his stuff was unimportant as Herbie sets the pace for the entire group.

We could also have ensured that he got his lunch first so could start sooner, and others could have covered his chores so he was more rested – our goal is to get the best out of Herbie on his most important task – walking.

Reality: In our systems we need to identify what the most important task is, typically this is getting a story to done and the software in the hands of the user. So when you identify your constraint look for anything they are doing that is not helping to move stories or for anything they are doing that someone else could do. Are they filling in paperwork that someone else could do? does this task really require their expertise? Could you make them a coffee so they could focus. (This is not encouragement to over-work, breaks socialising and generally a healthy working environment always takes priority).

Step 3: Subordinate the non-constraints

The job of all non-constraints is to subordinate their decisions to the constraint’s needs.

That sounds extreme wording but essentially it means that no mater how fast you go, if you are dependent on someone slower you are wasting your efforts, so work at the pace of the constraint, you have reserves so you can catch-up if necessary. Use the time saved to help optimize the rest of the system.

We do this as step 3 because it is more disruptive and more costly than step 2.

Herbie: In the story Herbie was put at the front and the rest of the group were told not to overtake him. When Herbie was able to go faster there was no one in front to delay him so we got the best from him and as the others were able to go faster overall they were able to make up any delays. The goal was only achieved when everyone was done so there is no benefit in getting ahead.

Reality: Don’t let work pile up, if we limit Work in progress we focus on flow rather than utilization, put effort where it has most impact on the whole system. In the case of dependencies on 3rd parties, could you give them advance notice so they are better prepared and can respond sooner. If you are dependent on a 3rd party could you see if there is a time/day they can respond more quickly, could you tailor your process to be more convenient for them? Tailor your process to maximise the effectiveness of the constraint, and this is the tough bit – even if that means making the process worse for others.

Tailor your process to maximise the effectiveness of the constraint, even if that means making the process worse for others.

Step 4: Elevate the constraint

Only once we’ve completed the previous steps does it make sense to add more constraint capacity, and thereby increase system performance. Because adding capacity is tremendously expensive in terms of time and money, we do it as a last resort, not a first step.

Herbie: In the case of Herbie, we could get him doing fitness training so he is quicker, or buy him a bicycle.

Reality: This is typically where you spend money, hire more people, get better/more equipment. Invest in training or education. Remember improvements here improve your entire system so the investment must be calculated based on system income not local costs. Maybe you ship parts by air, maybe you fly in an expensive expert, maybe you lose money on this aspect of a project so that the project as a whole makes more money. Keep the focus on the gain to the entire system and not get distracted by the cost at one stage.

Step 5: Return to step 1

The inevitable result of the first four steps, and the reason this is a “continuous” improvement method, is that the constraint will inevitably move somewhere else.

This step insists that you start back at the beginning, and don’t let inertia become the constraint.

A second example of the steps being applied…

Identify

Access to overworked Doctors in an emergency – constraint is a shortage of Doctors:

Exploit: Make the best use of what you currently have

Ensure Doctor is not doing any activity that someone else could do. All admin work is removed, prep work done by nurses or other staff.
Vacations/time off is deferred,
Doctors may work overtime so long as it doesn’t risk the patient or cause the Doctor to quit

Subordinate: Make decisions based on the constraint, the system should run at the pace of the constraint

Any non-urgent treatment is stopped until emergency is over.
Cases are prioritized so Doctors are only working on the most urgent cases.
Patients are only ‘prepped’ according to the capacity of the Doctors.
Patients are prepped in advance so that the Doctor is never waiting.
It may be quicker for a Doctor to move between patients rather than them coming to the Doctor.

Elevate: Make improvements to the constraint

Now we look to increase the number of Doctors or their effectiveness

transferring Doctors from other areas withing the organisation
recruiting more Doctors even at inflated rates or short term contracts,
any increase is an increase so we can hire someone that is semi-retired or part-time.
We may automate activities or buy tools to improve the efficiency and effectiveness of the doctor.
We could train Nurse to perform some of the Doctor’s activities

Repeat

We look and see whether the Doctors are still the constraint or whether there is a new constraint.

Summary:

There may be a little ambiguity with some of these improvements, e.g. A Doctor moving between patients may be considered elevating or subordinating, but it doesn’t matter it is an improvement, it is a thinking process and some changes don’t neatly fit into one category, don’t let that distract you from the thinking process itself.

The Goal by Eli Goldratt

This book uses a production plant as a metaphor for delivering value to your customers. The emphasis is on the manufacturing equivalent of delivering working software, doing it regularly, and that delivery is not about development, OR production OR deployment it is about everything – essentially encouraging Dev Ops. It also has a lot to say about how the only Done that matters is to have it in your customers’ hands.

Read it! This is hands down the best book on Agility I have ever read. There are many others by this author all of which are great but this is my favourite.

Estimation is waste?

There is an ongoing debate about whether or not to use Estimates in your software development process, largely fueled by #NoEstimates, but as is often the case the battle has been picked up by many that have become over zealous without fully understanding but with great certainty tell you that you should never estimate.

There are many good explanations as to why estimating may not help you and some great explanations of alternative ways to get the information you need, or to help you understand why the information was not needed. But I want to focus on one of my pet-peeves – Thou shalt not estimate because estimating is ‘waste’ Every time I see that I shudder, and quite often I am left with the feeling that the person writing doesn’t understand either Lean or #NoEstimates.

Thou shalt not estimate because estimating is ‘waste’

What is Waste?

First of all the statement makes it sound like waste is bad, in fact the word does seem to imply that waste is bad. However, Lean is a lot less emotive with the term, Lean is often confused and simplified into simply the reduction/removal of waste. But this is a very lazy and incorrect interpretation.

Lean is about productivity first and foremost, and is about reducing ‘waste’ ONLY when AND if doing so does not impact the system productivity. In other words, in Lean ‘waste reduction’ is far less important than system improvement, but for whatever reason we get hung up on waste – especially when bashing others. We reduce waste to improve the system – waste reduction is not our goal it is just a tool to help us.

Is Estimation waste?

Is Estimation waste? The short answer is yes, but that is only part of the truth. Lean considers waste to be activities that do not directly add value to your product and can be considered either ‘Necessary Waste’ or ‘Pure Waste’.

Waste covers a whole host of things, but waste includes: Planning, testing, reporting, breaks, vacation, sickness, and a great deal of other things far too many to mention.

So calling Estimation ‘waste’ is akin to calling Planning ‘waste’, if we were to eliminate say planning and testing in an effort to reduce waste, we would very likely cause more and worse waste by producing the wrong thing (over production waste), in the wrong order(over production), or poor quality (rework waste).

In other words not all waste is bad and not all waste should be removed – simply calling something ‘waste‘ does not help the conversation.

Sometimes a little waste now can save a lot of waste later.

Is it beneficial?

The real question is whether the activity helps the system? and as a follow-up, is there a better way of achieving the same thing?

Questions to ask when considering waste:

Does this activity help the system to be productive now and in the future? (Would removing it impact our productivity)
Is there a better was to achieve the same outcome?

I can’t answer the question of whether Estimation is beneficial in your system, because every system is unique, if you are using estimation for forecasting purposes then I’d suggest that there may be alternative solutions that are better and #NoEstimates may be a good place to start. But forecasting is only one use of an estimate, your system may find that estimates are beneficial it is for you to decide.

Next time you see someone use blanket statements that eliminating ‘waste’ as an absolute and unqualified justification for not doing something, please challenge them to qualify their statement. Remember that waste is often necessary, our goal is more often to improve or understand the wasteful activity rather than eliminate it entirely.

Let’s consider a world without waste

If you are still unsure think what would happen to your system if you abolished all wasteful activities:

Vacations – typically 10% of your productivity lost.
Coffee breaks – 10 minutes every 2 hours = another 8% productivity lost
Toilet breaks
Stand-ups – yep they are waste too.
Demos, Retrospectives, Planning, all are ‘waste’

Just imagine how productive you would be with no direction, no feedback and no staff?

Feedback is not a passive activity

In Lean manufacturing setting up feedback loops is considered a critical part of the operation, so much so there is a term for this – Andon – a system to notify management, maintenance, and other workers of a quality or process problem.

The principle is that it gives the worker the ability, and moreover the empowerment, to stop production when a defect is found, and immediately call for assistance. Workers are encouraged to use this feedback mechanism freely. Common reasons for manual activation of the Andon are part shortage (dependency), defect created or found, stoppage, or the existence of a safety problem. Work is generally stopped until a solution has been found.

Loosely translated an Andon is a Paper Lantern – To shine a light on a problem.

Sounds great in principle, any worker is empowered to give feedback to management if they have concerns over safety quality or even a weakness in a process, but for it to become culture it needs to be adopted in a no-blame manner and used frequently, lack of utilization of an Andon is a serious problem and is addressed in Lean.

If a feedback mechanism is not triggered regularly then the settings are considered too loose. The threshold for triggering an Andon would continually be made tighter and tighter, quality is expected to be higher, time for a task is squeezed and so on until there is an increase in frequency of Andon being used.

The aim is to get a regular feedback of actionable information, too little and the feedback loop has failed, too much and you cannot see the problem so it needs tuning and adjusting slowly.

What does that mean to us in a non-manufacturing environment?

We have got pretty good at retrospectives and giving feedback locally, but feedback to management is largely absent.

The difficulty in many organizations is that senior management hide behind an open door policy. “Employees can talk to me any time, my door is always open”. It is very easy to pretend you are open to feedback but much harder to actually be open.

“Employees can talk to me at any time, my door is always open”

– the unapproachable manager

In many cases the open door is actually an invisible barricade: fear of retribution, fear of not being supported, fear of being ignored, fear of the messenger being shot. In many cases the fear is justified, but even when it isn’t, it doesn’t make the fear any less real to those with genuine feedback to share.

Creating Feedback Loops

Just like with Andon, this is feedback that should be sought and encouraged and your measure should be how frequently you are given constructive feedback from your employees, if you are challenged regularly and respond to it regularly then it is working, but if you are not getting regular feedback (from those outside your inner circle) then it is likely your “open door” is not that open.

Has someone in the last week given you critical feedback without being asked?

If on the occasions you do explicitly ask for feedback are you bombarded with hostile questions? Do the questions catch you by surprise? Do people seem dis-satisfied with your responses? Do you only ask for feedback when people quit? If yes then perhaps you are not asking for feedback often enough, or are not responding to the feedback you are getting.

Feedback is not passive

Feedback is generally not passive, you need to invite it, create forums where feedback is invited and expected, be open to the feedback and when you are not willing to change be prepared to explain yourself, and be prepared to repeat yourself.

It really comes down to whether you truly are a feedback culture, if you are you have to work for it.

Mindset or method?

Straw poll:

Should we teach practices and methods in the belief that an Agile mindset will evolve from the good practices?

Should we teach an Agile Mindset in the hope that with an Agile Mindset good practices will emerge?

I know of a few Scrum Masters and a few Agile Coaches in each camp, and some that feel very strongly on this topic. And of course the choice lies at the heart of most Agile Transformations and the outcome will therefore be far reaching.

Method over Mindset

Around 10 or 11 years ago I was introduced to a new way of managing projects and a new software tool for doing it. I was trained in how to use the software, but not in the theory behind the tool.

We are what we repeatedly do. Excellence, then is not an act but a habit.

– Aristotle

The tool was amazing, it really blew my mind. As a Project Management tool/method it made more sense than anything I had used previously and I loved it, I happily used the tool and got good results. But despite having the tool I never learned the theory and didn’t even know where to look to find out more. No lack of interest or enthusiasm but I was content to use the methods and it helped in my current situation but I was never able to make the leap to more than a method despite my enthusiasm.

It is a little beside the point but the tool was Critical Chain for Project Management and before I was introduced to Agile I thought this was the best option out there for Waterfall projects (probably still is). Sadly despite it being so great, it was still based on the assumption that the end state was known at the start of the project, which for me is the ultimate failure of Waterfall thinking, and my primary reason for moving to Agile.

Only much later did I discover the theory behind the tool was The Theory of Constraints and the 5 focusing steps, IF only I had been shown that and I feel my enlightenment would have come much sooner. But a method without understanding the theory leaves you unable to adapt and improve, I was limited to the context in which I was shown.

What is more I had a co-worker that struggled with the concept of slack and wanted to follow the method apart from that one aspect – the lack of understanding left him unable to differentiate between a necessary aspect of the method and an arbitrary one, and ultimately for him the tool didn’t work because he rejected the necessity for slack.

A lack of understanding of the theory leaves you unable to differentiate between a necessary aspect of a method and an arbitrary one.

– John Yorke

Method over Mindset, the Return on Investment

Another consideration is that teaching theory takes considerably longer and has a much lower conversion rate. That is to say that I could teach the Scrum framework fairly quickly – a matter of days, and be pretty confident that the instructions are clear and ‘could’ be followed effectively. To take that same group and to teach them the theory to the point of understanding the why behind each of the practices could take weeks or months, and likely longer before they are fully understood. It is also possible that some will never grasp the theory but are still perfectly capable of being good team players.

Continuous improvement is better than delayed perfection

– Mark Twain

Many organizations are interested in the short-term results rather than long-term understanding and so there is a desire to do the least for the most impact. So we see the evolution of phrases like ‘Scrum-but’ as a means to discourage deviation from the defined framework. Our goal becomes to repeat, rather than understand.

Mindset over Method

If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea!

– Antoine de Saint-Exupéry

The flip side of this is the desire to teach/show the impact of having an Agile Mindset and then having the individuals identify a solution using that knowledge.

Clearly this is a much greater challenge, even those with an Agile Mindset will initially lack any practical applications to draw upon, even the most Agile of mindsets is limited to what they have experienced or read about, and contriving a new bespoke process to your environment may ultimately be the most desirable solution. However, it is impeded by the ability to share that vision and understanding with others, and limited by your understanding, ability and creativity. And frankly do we really think that we are able to come up with a better framework than Kanban or Scrum on our first pass at an Agile Transformation?

A middle ground?

As you have probably guessed I am not a fan of either approach. I believe that most people will not become Agile overnight and even the most eager of minds will take time to absorb and understand the implications and possibilities of Agile.

I further believe that teaching Scrum as a closed framework that must not ever be deviated from is not the solution. Instead I would compare it to learning any new skill, we teach good practices and when you have mastered them you are ready to move on, we may teach a variety of good practices so you have some comprehension of the possibilities available, that way you do not limit your thinking so just one solution.

But we learn to master the basics, and we learn to question ‘why?’ when we feel we have mastered the basics and we understand why, only then we are in a position to start to evolve.

Applying Lean and XP and Kanban to the Scrum framework can let you grow in understanding within a safe set of guidelines. And when you understand the mindset enough to comprehend the limitations then maybe you are ready to craft your own solution.

If you can’t explain it to a six year old, you don’t understand it yourself.

― Albert Einstein

Why I recommend Scrum

Personally I love Scrum as a foundation for a transformation to Agile for those undertaking software development projects, not because I believe it is better than Kanban or other tools, but because to succeed with Kanban you need a level of self-discipline and understanding that is often absent for those new to Agile, and it becomes far to easy to gold plate or run long.

Scrum adds some safety nets and feedback loops that counter many of the usual “human nature” problems that arise from teams new to self-organization. Self-organization is a skill we need to learn and develop like any other. Scrum is so simple to learn, and easy to follow (if you are willing) and once you understand the Agile mindset it is a great framework for evolving into a great Agile team. But like any tool if used inappropriately you can make a mess. But any change requires a good guide.

That being said for support or reactive work Kanban is ideal as the discipline generally comes from outside. It is never a one size fits all.

Method without mindset can only take you so far

Teaching the mechanics without teaching the theory is only half the task and whilst it is sufficient for many consultants to get paid, what they leave behind is a culture unable to improve, and without improvement entropy will set in. I believe both some guidance on the mindset and instruction in some methods are both necessary for a successful Agile Transformation, along with a healthy amount of enthusiasm and patience from those leading the transformation.

WiP Limits and Velocity

I recently presented at a conference on the topic of Work in Progress (WiP) limits, and in particular Little’s law. As part of I gave a demonstration and from it drew the conclusion that WiP limits have no direct bearing on velocity.

This caused some controversy so I’ll try to explain again here.

Applying Little’s Law

Mathematically speaking we can increase the speed with which we get served in two ways:

We reduce the number of people in the queue (We limit WiP) or
We increase the speed each person is served (We increase Throughput – in this case by adding more people, or installing faster equipment)

Note: When mathematically applying Little’s law the pre-condition is that the variables are independent, the assumption is that number of people in the queue does not impact on how quickly the server works. This works in mathematics, but maybe not so much with people in reality.

The principle is that WiP and Throughput are two independent variables and changing one does not change the other, only cycle time is impacted.

The wider impact of WiP limits

Of course the goal of my talk was to illustrate that whilst you may feel you are getting more done by not limiting your WiP, and having a whole bunch of work in progress this has no bearing on your actual throughput which is governed by your bottle neck.

When using Little’s Law in a controlled environment we can show that limiting WiP doesn’t impact throughput (unless we limit it to less then our bottleneck and starve it – say two servers and limiting the queue to 1 person). What limiting WiP does do is reduce cycle-time enabling you to be more flexible and more responsive and to have more reliable forecasts.

Work in Progress Limits will directly impact Cycle-time

Contradictory Claims

Unfortunately this contradicted one of my other claims in the presentation where I said that Limiting WiP helps increase throughput of the team.

The confusion of course is understandable, when using a mathematical example there is no cost for context switching and the example was to show the impact of WiP on a stable system isolating variables to demonstrate what happens. The real world is far more complex and there are many more variables.

In the real world

Generally speaking the ability to focus on less things means less multi-tasking, reduced cycle time means you have less things on the go at once, and for less time. These help you be more productive and thus indirectly the throughput will go up, but this is a result of you being able to focus not as a direct impact of the WiP limits.

There are many indirect benefits of using WiP limits, but Cycle-time is the prime beneficiary, the rest is a bonus or an indirect consequence.

New Kanban Game Available

Around two years ago I was introduced to a Kanban board game which we used as part of a training program and I really enjoyed it, it was a great learning tool but it had three major drawbacks.

First it was heavily scripted and so it was only suitable for a single play through, you couldn’t replay it and get more out of it. It was not a game in that sense, it was much more of a training tool/prop.
The second issue was that the exercise was taking 3 or more hours to run and whilst Kanban is a critical aspect of the training the return on investment was not sufficient.
Finally it didn’t effectively address Little’s Law or the Theory of Constraints, there were aspects of that in the game but the heavy scripting meant that the learning was masked or even lost.

A New Kanban Game

It got me thinking and so I created a couple of workshop games that addressed this in more detail. One of those workshop games was such good fun that I expanded it and combined it with training. For the last year or so I have been running workshops using it and continually looking to find improvements and modifications to make it more fun or to emphasize the learning aspects.

I have now presented this at a number of Lean/Agile conferences and have had a lot of great feedback, including a lot of encouragement to publish it as a board game.

And so Herbie* – the affectionate name for the workshop game, became Motor City a Kanban based board game. That is first and foremost fun to play and re-play, but the game is under-pinned with an understanding of Kanban, Little’s Law and the Theory of Constraints. You can play the game without any knowledge of those and it will still be fun, but people that instinctively apply those practices or use them as a result of understanding will fare better playing the game.

*Herbie was named after the character from the book The Goal, by Eli Goldratt.

Game Prototype (Early Access)

The game is now in prototype, with a small number published for evaluation and for use in training and conferences, with the intent to publish the game more widely after getting feedback from the prototype.

To purchase the game…

The game is provisionally priced at $150 but for early access to the prototype it will be charged at $120
I’d love to hear of anyone that uses this as part of a training program or just as a fun game for your team to play.

If you would like a copy of the prototype board game for evaluation and feedback, please contact me.

Similarly if you are interested in training on the Theory of Constraints or Kanban and would like to use this game as part of that training, I would be happy to offer guidance on how to incorporate the training into a workshop.

Thank you

Thank you to those that have supported and encouraged me, I can assure you that board game design is far more complicated, costly and time-consuming than I ever imagined. In particular thank you to Toby Gerber who has helped with the graphic design and turning my ideas into reality, I could not have done it without him.

Identifying Waste in Your Processes

I think as agile practitioners we can learn a lot from Lean thinking and there are a number of concepts of Lean that I think apply very well to Agile Software Development.

This is the second part of a blog attempting to convey the Lean concept of waste and I have introduced some ideas for how we can help mitigate waste in an Agile environment.

The 7 deadly sins

Lean identifies 7 wastes (recently adding an 8th Waste)*

Over-Production
Inventory
Transportation
Over-Processing
Waiting
Motion
Defects
Wasted Potential of People*

Over-Production

explored in more detail here.

Inventory / Work In Progress

In manufacturing the physical properties of inventory are obvious, piles of raw materials, stacks of part built products, warehouses of finished goods are all big and obvious, but in software, inventory is harder to see, it is intangible to the casual observer.

In software, Work in Progress takes many forms, most obviously this is in the form of code, any code not deployed to a customer is Work In Progress, that code cost time and money to create, the raw materials are developer thought and effort.

But Inventory is not just code, it is also knowledge, it is thought and problem solving, unrealized ideas and even experience and learning. Code is the manifestation of thinking, problem solving and planning. WiP/inventory is also the consequence and ideas resulting of conversations with users, usability testing, marketing. All of these are the efforts needed to create a product. But until those efforts are turned into a product it is all potential waste: time and money that are spent but until released to a customer are not providing value.

If we can learn to manage our Inventory/Work In Progress to limit it to only what we need and only what we can benefit from in a reasonable timeframe we can cut back on excess Inventory which in business terms this can have a major beneficial impact on our cashflow. Reducing Work In Progress also has the added benefit of making it easier to see other waste.

Effectively managing your WiP is probably the second biggest opportunity for waste reduction and the act of limiting WiP will help identify other wastes in your system.

Transportation

In manufacturing, there is deemed no value in transportation. Moving a product does not add to it’s value and may even damage it with wear or breakage. reducing the distance between workstations or distance from your raw materials or the distance to customers can reduce the cost of time or effort and reduce the risk of wear or damage.

In software transportation could be best viewed as hand-offs, handing off work to another person or team requires effort, bringing them up to speed, co-ordinating, sharing the work. All this time and effort does not add any value to the software. But worse there is consequential loss of knowledge when there is a hand-off, and each hand-off results in a greater loss as the knowledge becomes fragmented and dispersed. Vital information may not be passed on leading to poor decisions, this is equatable to a breakage during transit.

Limiting hand-offs can reduce waste, having a team that contains all the knowledge and skills necessary to complete the work can help. Steps to improve and enhance communication so that less knowledge is held by a small number of people can mitigate the loss of knowledge during a hand-off. E.g. have all members of the team involved in story refining meetings so everyone is apprised of the intent of a story and can ask questions.

Over-Processing

In manufacturing this is making a product to a higher standard than is required or spending longer on something than is necessary, or in using a tool that is over powered for the job at hand.

This is similar to Over Production in many respects as is it unnecessary work, but in software it is obfuscated. A feature may be needed, but we have over engineered it, spent more time and effort on it than is necessary. This is not an excuse to cut corners, but development and test effort should be reflective of the customer need.

TDD – Test Driven Development can help here, by writing the test first and limiting coding to passing the test can help reduce over-processing.

However, many developers engage in EDD – Ego Driven Development where they write unnecessarily complex code to solve a problem so they appear clever to their peers. Or RDD – Resume Driven Development where they introduce a shiny new: tool, technique or gadget, because… ” it is shiny and I wanted to try it out” or “It will look good on my resume” rather than because the product needs it.

EDD – Ego Driven Development where developers write unnecessarily complex code to solve a problem so they appear clever to their peers.

There is another form of Over-Processing and that is writing software to solve a problem that could be done more easily another way. We have a great desire to automate everything but sometimes the solution costs far more than the problem we are solving.

The waste of this seems obvious when described here but can be difficult to spot in the wild, pairing and TDD can help keep the focus on the goal and limit the opportunities for over processing.

Over-Processing – Re-Learning

Another aspect of over-processing is the act of re-learning, if you have solved a problem once you should not need to solve it a second time, or if someone else has solved a problem can you benefit from their learning.

We can reduce this waste with knowledge sharing within the team or having the whole team in discussions about design or story refining so that opportunities to avoid re-learning are increased.

Avoid long delays between first looking at a product and then coming back to it later where you need to remind yourself of context better to only start work on something when you are ready to see it through.

Waiting

Waiting is any time that the product is not being actively worked on, this could be a story that is blocked waiting on someone or some event. I think most people see that as waiting time, but any card that is not being actively worked at any time is ‘waiting‘.

Any state such as ‘dev done’, or ‘QA done’, or ‘ready for test’ or ‘ready to deploy’ or even stories written but waiting in the backlog are ‘waiting‘. Most stories will spend far more time waiting than they will being worked.

This is often a symptom of having too much work in progress, but items waiting may be a sign of a bottleneck or potential areas for improvement in your workflow. Kanban focuses on flow – minimizing the waiting time, so identifying any area where there is excessive time waiting is a candidate for improvement in the workflow.

Keeping track of repeat blockers, or being aware of areas of your system where stories regularly spend time waiting can help reduce this waste. Challenge any story that is not actively being worked and ask why.

One technique for this is for team members to each have only one avatar and have them place it on the card/story they are currently working on. Any card without an avatar is waiting. Only allow one avatar per person.

Motion

Similar to transport in many ways but refers to the operator or the operation rather than the transport of the product, could the workstation be operated more effectively.

In software this is sometimes interpreted as Context Switching which unnecessarily slows down the team, this is certainly a major part of it, but I prefer a broader interpretation, anything that hinders the team from getting the job done effectively, this could be having the right tools, the right environment, the team working effectively together, communication barriers. But I can understand the desire to focus on context switching such as: support calls, emails, even distractions in the office space are forms of task switching that impair the effectiveness of the team to get the job done.

Identifying ways to improve the work environment and limit the interruptions could cut huge amounts of wasted time from your workflow.

Defects

Defects are another waste that gets compounded as a defect is exposed to all of the other wastes as it goes through the system which magnifies a problem that is already serious.

Defects are waste that grows the longer they go undetected, the problem itself is magnified by time and use, the more time the more frequently the defect occurs and the more users it impacts the more it costs you. That is until reported it is a hidden cost.

The waste is a hidden waste until much later so it can give us a false sense of security until the product reaches end of life and we have a full picture of the total lifetime cost we will not know the true extent of the cost of defects.

Another factor is that the cost goes beyond the impact of the defect itself the longer it goes undetected the less recent knowledge the development team has of the subject and so the learning time goes up. How often have you been asked t fix code of someone that left the company years ago and have no clue what they were writing. Once again one waste impacts another if that developer was using EDD you may have code that is so complex the effort to resolve is much higher.

We can try and combat defects with good development practices, such as pair programming, TDD, and comprehensive regression testing. Any efforts we can find to identify defects sooner or prevent them occurring reduces the waste and the long term impact.

Wasted Potential of People

I have seen this added as an 8th waste of manufacturing, and it goes doubly so for software development. In software development people are your greatest asset, it is a creative knowledge based activity and so providing the right work environment and opportunities is essential to getting the best from your people.

Demotivated people work slower and with less care, but enabling people to reach their full potential could reap rewards for you and your products.

Summary

Any system will have waste, some is necessary and some is not, but being able to identify what is necessary and what is unnecessary or what is waste and what is slack can greatly enhance your chances of making improvements that really help your software development process.

Often what you thought was waste is actually a valuable activity and what you thought was adding value was just adding waste. Understanding this helps you make the right decisions.

However, please remember – It is very important that waste reduction is NOT the Goal of your efforts, Waste Identification is a supporting activity for the Theory of Constraints, any efforts to improve a part of your system that is not the bottleneck is a waste in itself.

Your goal is to improve the entire system and focusing on one area without consideration for the system as a whole leads to the wrong decisions being made. Waste is just one of many considerations.

Weeks of coding can save hours of planning

In the context of Lean manufacturing there have been numerous studies that estimate that the proportion of value adding time in a production cycle is around 5%. The other 95% is deemed to be waste. These studies also conclude that by far the biggest waste is overproduction.

Anecdotally I think it is a very similar story in Software development although my fear is that the proportion of value adding time is even lower.

Waste takes many forms but broadly speaking effort as a team/company fall in to three areas:

1 . Valuable Effort – Value adding activities, these activities cost time and money, and there is a consequential opportunity cost but they add some value. However, the value may be able to be realized more effectively.

2. Obvious Waste – Non-value adding activities that are evident. These activities cost time and money (and opportunity cost) but add no value to the product being created. Examples are vacation, breaks, sickness.

3. Valueless Effort – Non-value adding activities masquerading as Valuable Effort, they cost time, money and have an opportunity cost but add no value to the product being delivered.

Waste Reduction is Not Your Goal

I will be covering the wastes in more detail but it is very important that waste reduction is NOT the focus of your efforts, Waste Identification is a supporting activity for the Theory of Constraints, any efforts to improve a part of your system that is not the bottleneck is a waste in itself. But identifying waste can aid in your efforts to improve your bottleneck so is a great tool to support your other activities.

The 7 deadly sins

Lean identifies 7 wastes (recently adding an 8th Waste)

Overproduction
Inventory
Tranportation
Over-Processing
Waiting
Motion
Defects
Wasted Potential of People

There is nothing so useless as doing efficiently that which should not be done at all.

Peter Drucker

Overproduction

Overproduction is considered the worst of the wastes and ultimately it is this particular waste that is the basis of much of the Agile Manifesto for Software Development. Which is why systems thinking is a topic I keep coming back to.

Agile is founded on the premise that at the start of the product we know the least and lack of flexibility is the biggest constraint in the success of software development. Traditional methods plan, create, test and support endless features that were unnecessary, the cost of this waste is unfathomable and big enough to trigger the Agile movement and challenge the way we work.

Overproduction was also the primary waste identified in the book “The Goal” by Eli Goldratt, and if you haven’t read that book I would heartily recommend it. The book is . great primer for the Theory of Constraints and for understanding Lean thinking.

Planning is Waste?

But Agile has not completely fixed this problem, and in my experience development teams regular and persistently develop unnecessary features. With many teams choosing to work without sufficient time spent planning; story writing; prioritizing; and without consideration for whether the work they are doing adds any value now or more significantly adds the most value next.

Road mapping, planning and even story writing are often incorrectly perceived as ‘waste’ and the avoidance of waste is used as a justification to get back to coding on something that is likely not valuable and almost certainly not the most valuable activity to be worked on next, which is a far more insidious waste than the planning ever could be .

Features “not needed yet” are implemented on the basis that it will save time later, or features are added because “We know we will need it later” – (7 words I dread to hear.)

We work to look or feel busy, in our minds we translate activity as being productivity when there is little correlation between the two in the context of software development. In software, creativity; problem solving and planning are far more valuable efforts than unfocused coding.

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.

Abraham Lincoln

In most cases features have no value until they are in the hands of a user and being used for productive effort. So any activity not spent getting the next most valuable feature into the hands of a user quickly is just waste.

Writing extra features that are not used or rarely used extends the production time but brings little or no value, it also compounds all of the other wastes because by it’s nature production of something of little value needs to go through the system thus exposing an unnecessary feature to all the other wastes.

When you next start work on something ask yourself: “Is this the next most valuable activity for our customer?” If you cannot confidently answer that question then maybe you need to spend more time planning so you can spend less time coding something that is not needed.

The worst case scenario

In software the biggest risk to any project is that it will be cancelled or obsolete prior to being used by your customers. This may sound obvious but if a project is cancelled ALL of the features are/were waste. All that effort produced no value for anyone. That time, effort and money could have been spent elsewhere. So getting your product right is only any good if you get your product delivered.

Delivering value to your customer quickly is the best way to mitigate the risk of your efforts being wasted, and Done is not really done unless it is in the hands of a customer and actually being used for a productive purpose.

Overproduction is a big topic and deserves a little extra focus so I’ll delve deeper into the other wastes another time.