The 4 steps to successful software delivery

After [reading / posting] a recent article on waste, (read it here), someone asked me why we shouldn’t do work that we will need later if it is more efficient to do it now?  You have already dug the hole—doesn’t it make sense to do all the work in that area together?

This is very similar to the argument for splitting stories horizontally along business layers: “We can be more efficient if we optimize and specialize.”

Yes, you might achieve local efficiencies by taking these steps. But local efficiency is not the most important factor in your decisions. Efficiencies and optimizations need to be considered in the context of the whole system.

The primary goal is to Maximize Value Early

For both project and support work, our goal is to maximize value for our organization, and to realize this value as early as possible.  In a consultancy firm, there is a slight abstraction, as work is paid based on time and materials, and these steps are distributed between client and team. Regardless of who owns the following steps, success depends on shared awareness of these priorities.

The 4 Steps

The four steps are designed to focus our energy where it will have the most impact first, and then building upon the previous step by shrinking the focus area through each subsequent step. Hence they are in descending order of impact.

1 – Prioritization
(Selecting Which Project/Product to Build Next?)

The most important decision your organization needs to make is which work will bring the most value NEXT. This has the most impact on your company’s bottom line, so should get the lion’s share of effort and focus.

Program Management

 

Describing the value a project will bring is far and away the most important aspect of software delivery. If possible, create a formula for calculating business value to help with these decisions. If you are unable to quantify or qualify the value to the organization of the work being undertaken (at a high level), then you are likely working on the wrong thing and all other actions you take could be waste.

Some projects/products will obviously bring in huge amounts of revenue. Others may save huge costs in manual entry and work-hours. Some may support business-critical systems that, if not supported, create extensive risk and potential cost. A project might also fulfill a safety or regulatory concern, or business objective.

Naturally this is complicated and can be difficult. Priorities change. Time factors and organizational politics come into play. A lot of the time you will be making informed guesses about value and costs. But this decision dwarfs all of the others in the impact to the bottom line, so is critical to put the right amount of effort here.

I would consider this to be the responsibility of the Program Manager or Project Management Office, but I see the responsibility to be to identify the projects that will bring most value to the organization as the end of their role. “Day-to-day Project Management” responsibility lies elsewhere.

I would however strongly encourage this decision-making process to be transparent and inclusive.  Hold a regular Project review board and invite the stakeholders. Make the process clear and help everyone understand how the decisions are made. I would also suggest a visible roadmap of work planned.

Finally, I would encourage you to not plan beyond your company’s horizon. If new work is likely to emerge in the next 3 months that may significantly change priorities, then only plan for 3 months.  Only start a product or project at a point you feel committed and certain it will be completed. Starting something with the expectation of pausing or pivoting suggests it wasn’t the most important thing to work on next.

2 – Direction
(Product Ownership – Building the right Product)

Whether this a new product or supporting an existing product, it is important to ensure that you build the right product,  Product Ownership is covered in lots of detail in other places so I won’t go in to details but I suggest watching this video:

po nutshell

Like prioritization in Step 1, the most important responsibility in Product Ownership is Maximizing value–Are you working on the most valuable thing next?  You do this through experience and through feedback from users and stakeholders.

Feedback is crucial. To get frequent user feedback, you should be thinking about how you intend to deploy your software and then update it regularly. You should be thinking of this before or with your first story.

Deployment and ongoing updates should not be an after-thought.

Product Ownership is also about maximizing the work not done. The product should meet the needs and goals but only do what is necessary. There will be a point when the returns diminish. All work carries an opportunity cost, and at some point your time is more valuably spent elsewhere. Understanding this is vital.

Finally, I want to stress the importance of getting value to the customer quickly. Value is only realized when the software is used. We should be striving to getting the software in use as soon as possible. This may be in an unfinished state. It may be only a partial solution, but it should be adding value. e.g. it can be used alongside an existing product, or it could be used for this workflow but for another you use another tool etc.

Feedback from people actually using your software is the most valuable feedback you can get.

Just like prioritizing projects, prioritizing stories has far more impact on the success of the project than the following steps.

3 – Quality
(Build the Product Right)

Building the right product is more important than building it right. Building the wrong product is just a waste. But that doesn’t mean that quality is not important.

Quality means doing it right when no one is looking.

Henry Ford

When assessing cost of software, it is easy to focus on the initial development cost alone. But support maintenance and future enhancements all fall into the total lifetime cost of a product. The cost of defects magnifies over time, and the cost of supporting poor quality code is significantly higher and more risky. Re-learning and knowledge transfer are painful and costly to an organization and I can think of numerous examples where companies have been forced to abandon projects or replace software because they lost a key employee who had the knowledge locked in his head–or software so fragile no one dares touch it.

Good practices such as Test Driven Development, Pair Programming, Behaviour Driven Development and Automated Regression Testing may seem costly at first. But when considered in the context of lifetime cost, they are a sound investment. More importantly, they enable the right business decisions to be made because there is a confidence in the software.

Quality code with a solid set of tests enables refactoring and new features to be added with the knowledge you are not causing unintended consequences elsewhere by breaking older functionality. This reduction in risk is a considerable advantage, especially when supporting older applications.

This is not a carte blanche to gold plate or over-engineer. Quality Code is the right code for the story and No More. We write just enough to satisfy the story and we write it well. Over-engineering applies to the GUI too–we don’t add features or over-engineer.

4 – Efficiency and Improvement
(Building it Better and Faster)

By this point we should be working on the most valuable product to our company. With that taken care of, our next focus is to work on the most valuable feature or story for our users/customers. We should be doing it in a manner that enables us to be confident in the work and able to expand it later.

But guess what? You can do it better.  We should reflect on the decisions we made and how we make them. Are we making the right decisions?  What helped us make those decisions?  Are we making the wrong decisions? If so what could we change to avoid repeating those mistakes.

The reflection and improvement should be done at all steps and should incorporate the learning from all stages.  If we continually look for ways to improve we will get better and better.

If you always do what you always did, you’ll always get what you always got

Henry Ford

 

What about efficiency of developer effort?

So what of the original question about being more efficient with horizontal splitting of stories, or the perceived inefficiency of re-working the same area of code later?

It is a question of impact and perspective. In both cases the question is founded on ‘efficiency’ being a measure of the developer’s efforts.  What I would like you to consider is that in terms of delivering value to your users and ultimately your company efficiency should not be measured in terms of developer effort, but should be measured in terms of delivered value.

Efficiency should not be measured in terms of expended effort, but should be measured in terms of delivered value.

The most valuable decision is to work on the right project – this dwarfs all other decisions by an order of magnitude. But it is at a different level of scope than this question.

The next most valuable decision is what to work on. This generally best considered from the perspective of delivering value to the customer quickly and of being able to respond to their changing and emerging needs.

When we split horizontally or when we do extra work, we know we will need later we may give the appearance of making more efficient use of developer effort, but in doing so we reduce our ability to adapt to changing needs and more significantly we delay the time to deliver the next most valuable feature.

This is an example of working with a waterfall mindset. The measure is that while it delays us this week, we will make back the time next month or next year. So if we are not planning to deliver this to our customer until next month or next year, it might appear to make sense. But you may not have considered two factors.

If we are deploying frequently, we are realizing value with every release and in the value of our software isn’t massively more than the cost of developer time to create it, then it is likely we shouldn’t be working this project at all. So ultimately the cost of developer effort for having to rework the same code later for a new feature is insignificant to the value gained by delivering sooner. We should measure ourselves based on value delivered not the effort to deliver it.

The second consideration is that the work you are doing on the assumption of needing it for a future feature may not actually be needed. That feature is by definition lower value and less important as it has been prioritized lower.  There is a pretty reasonable chance that the customer may change their needs or there is the possibility of the project being cancelled or considered good enough as it is.  That work may never actually be needed and you would have delayed the project for work that had no value. The efficiency you are trying to achieve may just be an illusion.

This is the business equivalent of spending $10 today to save $1 next year.

When considering efficiency and improvement, remember your goal is to maximize value to your customer and to realize that value as quickly as possible.  Ask yourself if the efficiency improvement you are proposing fulfills those goals? If not it may be just be an illusion.

 

 

 

 

 

 

Do you know ‘what’ your problem is?

The title is a little tongue in cheek, because in my experience it is the ‘what‘ that is likely your problem. We focus far to much on what we are doing both in our work and in our product, and far too little on why we are doing it.

Failing to understand the why

The obsession with local efficiency and and maximizing utilization of people is crippling customer focus and value.

There is nothing so useless as doing efficiently that which should not be done at all.

Dr Peter F. Drucker

Peter Drucker has some very inspiring quotes but that one is by far my favorite and one that I wish was better understood and adopted in practice

shakespeare-quotes-9-1024x540

We look for work to stay busy, whether it be setting up grand databases or magnificent CI solutions. I am not saying either is unnecessary, but does the solution fit the need? Were they built when needed or in anticipation of a possible future need?

The focus on lead/cycle times can result in stories becoming rushed, the focus becomes on getting work ‘done’ as quickly as possible, rather than a focus on getting the work ‘done right’. We are completing work quickly at the expense of adding value.  It also leads to ‘creative accounting’ where the desire to reduce cycle times results in the definition of done getting corrupted.

Refactoring gets sacrificed when throughput is the priority.

Stories focus on the what not the why.

We are often in such a rush to complete we either skip the why entirely or we take the easy route and look for a ‘why’ that satisfies our ‘what’  rather than actually asking why or giving it proper thought and effort into understanding the real why.

Of course the “Why conversation” can be hard, or time consuming and sometimes it seems obvious but can be hard to articulate. Story writing is a time consuming process but when the process becomes rushed there can be a tendency to prioritize stories according to what is written or what is easy to write rather than what is the next priority based on value or other prioritization methods.  We tend towards keeping people busy and the process flowing rather than ensuring we are delivering value and working on the right thing.

It is far too easy to get into a cycle of business as usual, all the time running to catch up but not really knowing if you are working on the right thing. But because everyone is busy and something is getting done “Asking why?” drops below the radar and the hard question of value does not get asked.

stoneage

Step back and evaluate if you are working on the right thing. Sometimes going slower is actually going faster.

Taking a little while to do it right could well pay dividends.  Our goal should be to give our customer the most value, not maximize how busy we are. And whilst logically these two should be related, in reality the opposite can be true.

Learning to write good stories with an emphasis on understanding the why this story adds value, learning to prioritize with the emphasis on why this story is the one we work on next can make a huge difference to a project delivery.   Incorporating story maps; road maps, story boards and utilizing other prioritization techniques can help your product deliver the right thing.

The customer should be your focus

Your product owner should be able to articulate clearly and confidently why the next story is next and that justification should be customer centric. Your goal is to make the customer happy NOT to keep your team busy.

When lead/cycle times start to become more important than refactoring it leads to a degradation of the system, either in terms of a reduction in the quality of the code, or an increase in technical debt, but more often a reduction in quality of the user experience.

UX is an afterthought

When we teach story writing we teach the INVEST method, the I is independent, and V is valuable, but I wonder if the independent is misunderstood, especially when it comes to UX.  The goal of a story is NOT to minimize Effort, it is to maximize Value. Sometimes we can maximize value by minimizing effort (ROI: Return on investment) but our goal is still the value and NOT the reduction in effort.

The goal of a story is NOT to minimize development Effort, it is to maximize customer Value

A story that adds new functionality could easily be tagged on to the existing system with little redesign of what already exists, e.g. We add a new tab or a new page or a new button, just append it to what is there. In some circumstances this may be the right outcome, but it should be a conscious decision not a default lazy way out.

A new story that is adding functionality should be assumed to be incorporating that extra functionality into the system where it fits best for the user and maintains the user experience and flow NOT where it is the easiest to integrate by the developer.  If we just tag on new functionality without consideration for the user or the existing system it can quickly become ugly and cumbersome.  Sadly the prospect of changing completed work – especially visible work can be daunting and many teams are resistant to make changes to the UI, instead preferring to add new functionality where it is least disruptive.

Each new story especially UI stories should require some reflection and a conscious decision to maintain flow and look and feel and maintaining the user experience, maintaining the user experience may very well be more valuable than the new functionality adds.

Acceptance Criteria is a substitute for thinking and conversation

We can get so focused on delivering quickly and meeting AC for a specific story that we become blinkered, we know we need to maintain quality in the sense of stability, reliability and security, but we can easily forget about the less quantifiable quality of the user experience of the system as a whole. We can seek to address the acceptance criteria without thinking whether what you are doing truly adds to the value of the product (not story) and when A/C is specific we don’t question whether it is right, we skip the conversations.

images

Being busy is not your goal

It can be a very hard adjustment especially when you are paid by the hour, or even bill by the hour, for us to accept and understand that sometimes the most effective use of our time is not keeping busy. It may be to do something inefficiently or to not do anything at all.  Keeping you busy is not the goal, delivering the best product to the customer is.

If I were a painter, I…

Recently I have spent a lot of time talking about story writing, but this week the focus has been on Acceptance Criteria and there have been a variety of comments that have caused me to reflect and have caused me some concern.
Some examples:

  • Unless stated otherwise a story is assumed to be happy path only. If it isn’t in the Acceptance Criteria we don’t have to do it.
    • Acceptance Criteria must contain all details including all edge cases that must be tested, anything not stated doesn’t have to be implemented.
  • If all acceptance criteria are met the PO/Customer MUST accept the story even if they are not satisfied.
  • If acceptance criteria change before the story is done, it should be completed as specified and a new story should be created for the changes. after all, late changes to Acceptance Criteria demoralize Developers.
  • If Acceptance Criteria are unclear we can choose what to do.
  • The Acceptance Criteria should never tell us how to do the job.

A metaphor

To respond to these questions I thought I would use a metaphor – A homeowner wanting to redecorate her house.

img_0374

The home owner hires a firm of decorators because they have a reputation for quality.

Her first request is that the painter decorate the sitting room,  She would like the walls painted blue, the ceiling an off-white, the windows also off-white but the paint needs to be wipe-able to avoid finger prints, and the doors and trim should match the window color.

A story is an expression of a need

Acceptance Criteria are the notes from the discussion to help clarify the need

In this case the homeowner has specified a number of acceptance criteria:

  • Walls – blue
  • Ceiling off-white
  • Windows off-white but wipe-able paint
  • Trim same as windows

The acceptance criteria are details provided by the Product owner that would help the workman understand how to complete the job to the satisfaction of the Product Owner – in this case the home owner.

The acceptance criteria are details provided by the Product owner that would help the workman understand how to complete the job to the satisfaction of the Product Owner

The job can and probably should be broken down into smaller tasks (stories). Having just the windows painted has value, just the ceiling does too, just the trim and potentially each wall independently may have value although that may be harder to qualify with the Product Owner, depending on their needs.

Detail in Acceptance Criteria

What you will note though is that the acceptance criteria did not state that the areas to be painted must be professionally prepared and sanded, that a primer coat be applied and two top coats are applied.  They do not state that the carpets should be protected and that spills should be cleaned up and that there should be no paint on the glass and no wall paint on the ceiling, no ceiling paint on the walls or that the paint should fully cover the walls and be an even drip free finish.  In fact the Acceptance criteria make no mention of how the work should be completed at all.

Who is the audience of your Acceptance Criteria?

The Acceptance Criteria come from a conversation with the Product Owner, but you may also want to take technical notes from conversations with those that will do the work. E.g. if the contractor has an apprentice, it may be necessary to spell out to him/her the technical details mentioned above, what seems common sense to you and the Product Owner may not be as obvious to someone less experienced. But notes on implementation are not acceptance criteria in the same sense. The product Owner should not be expected to understand all the technical details, what is acceptable is always determined by the Product Owner.

If quality is not specified in the acceptance criteria

But that means I only need to worry about the happy path right? So I need to do a good job with the specified work but I don’t need to care about drips or mess or if I am painting the walls I don’t need to care about the ceiling after all I am only doing the happy path. I can ignore walls that are hidden behind furniture, I can do the job quicker if I don’t put down dust sheets and we can worry about cleaning up the mess later right?

Put like that it sounds silly doesn’t it, but that  is what you are suggesting if you don’t consider anything other than the happy path, the minimal amount is done but you are leaving a mess to clean up later.  If this is clearly understood and acceptable to the customer or that you have split a story like this to get feedback sooner there may be an exception but if you think that you can declare a story ‘done’ with a mess left to clear up later it is a recipe for a disappointed customer or a misleading impression that you have achieved more than you really have. Giving a sense of artificial progress may lead to disappointment later.

img_0376

Acceptance Criteria are comprehensive and exhaustive

If all acceptance criteria are met the PO/homeowner MUST accept the story even if the homeowner is not satisfied.   There is some crown moulding between the walls and the ceiling, it was not mentioned in the Acceptance Criteria and the decorator has painted it the wall colour, but the Home owner wanted it the same as the ceiling and thought it was obvious so had not mentioned it.  The decorator insists the job is done because the Acceptance Criteria are met.

This is actually a very common issue, Acceptance Criteria are often not comprehensive and there are ambiguities and omissions, so how do you handle them?  Assuming you are paying for the work on a time an materials basis the customer is the one that suffers, they will have to pay for the rework(or extra work), but it is generally the workman that gets frustrated, they believed they did a good job and so it is frustrating to have to repeat good work.   But the customer is not satisfied, and that should be the only thing that matters. Your goal as a quality workman is to satisfy your customer.  The job is not done until they are satisfied (assuming the request is reasonable)

It might be that the customer is willing to accept the work as it stands and agree to the repainting later. But more likely they will want it fixed before you move on to the next piece of work.

What is not acceptable is to refuse to make the changes because they were not explicit in the Acceptance Criteria.  The story card is an invitation to a conversation, the Acceptance Criteria are reminders of what has been discussed. At no point do Acceptance Criteria or the card become a sealed contract.

img_0373

Changes during a story

The painter is half way through painting the window and the Customer comes to check on progress. She sees the paint colour of the windows and decides it is too white, too stark and clinical and would like the colour changed to an off-white.  She understands that the rework will cost and the new materials will cost and she accepts this, but she is unhappy with the colour.

The painter responds: “But the thing is the A/C were clear about the colour, the shade was specified, I have spent a long time getting through the first half of this job and it is only fair that I finish it according to the A/C”  Then we can have a separate job to repaint it a different colour.  Frankly my metrics will look bad if this job takes 50% longer, it will look like I am slow. Do you know how frustrating it is to do a job and have it rejected..”

I have heard frequently how when a PO changes A/C the story should be completed as it is and a new story created for the revisions.  But any continuation on work that is not wanted is waste. The notion of continuing to work on something that is unwanted, and even worse billing the customer for that time, is just wrong.  If metrics are driving bad behaviour then the metrics are failing. If someone is demoralized because a PO has clarified their needs then they need to evaluate their understanding of Agile.   A full half of the Manifesto is:

Customer collaboration over contract negotiation
Responding to change over following a plan

If you are unwilling to accept and adapt to changes in A/C even late into development, then you really need to go back to basics and ask if you genuinely support Agile software development.  I am sorry if that sounds harsh but the notion that keeping devs happy and metrics pretty is more important than delivering the right thing for your customer is very frustrating for me to hear.

 If Acceptance Criteria are unclear we can choose what to do.

The homeowner has been pretty vague in her choice of colours, blue and off-white. So presumably we can choose whatever we want?

Clearly that is unrealistic.  What might be more reasonable would be to talk to the homeowner and say “We prefer this brand and type of paint and these are the available colours within that range can you pick one”   The homeowner may say, No I want a very specific Blue only available in this brand that needs to be imported from New Zealand.  In which case you should explain that such a choice will increase the cost and duration of the job and adds to the risk – being an unfamiliar material, it may need more coats etc.  If the homeowner understands and accepts those risks then it should be done, but if the homeowner can be presented with a less costly and less risky alternative then they may be willing to modify their request.

Ambiguity should trigger conversation it should not be an excuse for making assumptions.

Ambiguity should trigger conversation it should not be an excuse for making assumptions. The goal is a finish that the homeowner is happy with, the specifics are likely open to negotiation. But whilst you may present alternatives it is for the homeowner to make the decision.

Anything not stated doesn’t have to be done

There are two types of things not stated in the Acceptance Criteria that apply here, how we do the job and some of the specifics.

How: The Acceptance Criteria doesn’t explicitly say that we need a primer coat, but we know that for a quality job it needs to be done. The Acceptance Criteria doesn’t tell us which brushes to use, or how to paint to get a good neat coverage. The contractor may need to specify those to an apprentice, but the homeowner should not need to specify the tools to be used or how to do the job (unless that forms a particular part of the need)  You as the contractor may need to spell out those details to your apprentice, consider those sub-tasks of the story.  Prepare and protect work area; Sand surface, apply primer, apply first coat,  apply 2nd coat, clean up area, clean brushes etc.  But these are NOT Acceptance Criteria to be supplied by the home owner. They are about how to do the job not about the job that needs to be done. Sometimes these do overlap but generally we try to keep the how out of the request.

There are also some specifics that may not be mentioned, for example there is a recess around the fireplace but only 4 walls were mentioned.  Can I ignore the other surfaces?  Likely not, details may have been assumed, if there is doubt clarify with the homeowner. There is something fixed to the wall which is too close to be painted behind, but if not painted it would look odd. The Acceptance Criteria shouldn’t need to specify that you need to remove it and paint behind to do a good job. But again if there is any doubt clarify it with the homeowner.  The Acceptance Criteria will rarely be comprehensive and conversation will be necessary.

Specifics: The Acceptance Criteria should not specify how the job must be done

Normally we try not to specify how the job must be done, but there will be exceptions to this. The homeowner may have a need to decorate using traditional materials, and so may specify that particular tools are used – e.g. No power tools, no lead based paint. No synthetic materials in paint or tools.  Sometimes those requirements are not necessarily clear to you. The Customer may insist on 2 primer coats and 3 top coats.  Two more coats than you feel is necessary for a quality finish, but they insist are how the job should be done.  In those instances you may make recommendations or suggestions, and try to understand why they feel this is a requirement. But ultimately it is for them to decide – so long as they understand the additional cost and risk.

But again there are exceptions to this, for example the customer my request that you skip a primer coat and only apply a single top coat, or insists that you use low quality materials to save on cost. If you believe that this will result in a substandard job, then again you should explain your reasons and try to find an alternative and if necessary refuse to accept the job if you feel that the result would reflect badly on you.  Ideally it should be clear from the start what the expectations around quality of service are, but it may be necessary to re-clarify periodically if there is doubt.

Summary

  • In story terms, the story should express the need, what your goal is. It is the start of the conversation(not the end) for assessing how to satisfy that need.
  • The Acceptance criteria are simply notes to help remember that conversation, they do not form a contract.
  • Acceptance Criteria should never be considered absolute, immutable or comprehensive.  Uncertainty should be clarified with the Product Owner early and often.
  • You should respond to changes in Acceptance Criteria quickly, the quicker you respond the less waste there will be.
  • Acceptance Criteria do not normally express how to do the job, although you may note how you intend to do the job to illustrate to the PO to help clarify any ambiguities.

What about Gherkin?

You can’t have a discussion on Acceptance Criteria without mentioning Gherkin, Gherkin is simply a tool for expressing Acceptance Criteria in a standardised and hopefully clearer format. However, Acceptance Criteria Gherkin and Test scenario Gherkin are very often confused, and the Gherkin constraint may result in the focus shifting from the conversation about the user’s need, to how to use Gherkin and the user’s need may get lost.

Story Acceptance Criteria is also not the place for identifying all of the test scenarios and edge cases, it is about understanding the need of the customer.

Another common issue with Gherkin is that the precise nature of the format and terms implies that the Acceptance Criteria are absolute and non-negotiable which is the exact opposite of how they should be considered.  If you start to fall in to the trap of seeing the Gherkin as a contract (rather than notes on a conversation) then stop using it until you get in the habit of regular conversation with the Product Owner.

By all means practice with Gherkin until you get better, but if you remember that it is about capturing notes from a conversation with the PO and most certainly NOT negotiating the terms of a contract then it is a useful tool.  But personally I prefer not to use Gherkin in the initial conversations about Acceptance Criteria simply because it is not a natural way of thinking for many POs so the format constrains thinking and conversation and the frequency in which it is treated as an explicit contract undermines it’s value.   Once you have discussed the Acceptance Criteria converting them to Gherkin may be beneficial.

However, I prefer to see Gherkin used for Acceptance Testing where those writing the tests are familiar with the syntax and where precision is more important.
If you take nothing else from this, please remember that Accceptance Criteria are notes on a conversation, they are not clauses in a contract.

Accceptance Criteria are notes on a conversation, they are not clauses in a contract.

It’s my way or the highway…

The last few months have been pretty busy for me and so I have dropped out of the online community for a while. Over the Christmas break I had some free time and so I ventured back in, caught up on some reading and tried to get a feel for the common issues of the day.

What I found was post after post condemning just about all things ‘Agile’    

“Ban the use of Story points”

“Agile is dead”

“Scrum/KanBan should be sent to the scrap heap”

“We don’t need: Scrum Master/Product Owner/Agile Coach/QA” 

There were some even suggesting that Agile had run it’s course and we should go back to Waterfall.

I found all this very disheartening, on one hand most of these were from people presenting a new idea or new techniques, and it is sad to say but in our culture it seems the only way we seem to know how to sell something is by presenting everything else as bad or with Click Bait statements, we all want to get noticed.  But when did ‘Agile’ become so black and white?

For me Agile has always been first and foremost about inspecting and improving – finding better ways.  But that doesn’t automatically make anything that differs from your approach Bad.  And it certainly doesn’t make everything New to be Good.

There is not one Right Way.

In almost every case the examples/justifications were examples of poor implementations of Scrum/Kanban; incorrect use of Story Points; or a person being ineffective in a role, or in some cases there were teams that had matured to a point where they were no longer getting benefit from a tool or a role.

But not one of those examples makes the tool or the role Bad.   In each and every case the team should inspect and adapt, and identify improvements.  If one team decides that Story Points do not add value for them, that is great, that team has found a better way that they can deliver software.

But that doesn’t mean that other teams that find them effective are automatically doing it wrong. If you find you can use a tool effectively and it works for you, then use it. Just keep inspecting and improving.  If a tool doesn’t work for you or you find an alternative tool that works that is great too. But try to understand why you are using a tool, what is your desired outcome and is there a better way for you.

As an anecdote, I was using a drill to make holes to hang shelves, I carefully marked and measured and then drilled. A friend was with me and he picked up a cross-head screwdriver lined it up on the wall and hit it hard, punching a hole in the plasterboard wall, far quicker than I was doing with the drill.  The technique was great for him in that situation, but when faced with wood or brick it was not suitable, and in both cases I still needed to measure and mark.  

Tools are context AND user specific, what works for one user in one situation does not automatically work for all users in all situations.

The same goes for the roles argument, I agree that there are some teams that have the versatility, the right mix of people and the capability to work effectively without one or all of the roles, but that doesn’t mean they are an ideal we should all strive for. Many teams find they are far more effective with those roles, and I would consider that to be just as much an ideal state as the other. If your team is effective and improving then you are demonstrating an Agile mindset.  Our goal should be to improve how we deliver software, not strive for someone else’s idea of utopia. Concentrate on improving YOUR team rather than mimicking another team in a different situation.

Declaring your way the only way is not demonstrating an Agile Mindset.

Explain the benefits, do not attack the alternatives

If you find a way that you feel works well and you want to share it, then explain what problem it solved and why you felt it improved the way your team worked, share that example, but please don’t do so by declaring all other tools WRONG.  What is right for you may not be right for everyone. You will just sound like a Snake Oil salesman.

Don’t reinvent the wheel

Having said all that we don’t need to reinvent the wheel and discover everything ourselves. We can benefit from other people’s experience and advice.

Both Scrum and Kanban and any of the tools mentioned can be badly applied, but when implemented with the guidance of someone with a genuine Agile Mindset (rather than a blind by the book interpretation) they can be excellent foundations from which you can evolve, by inspecting and improving, they are not the end point but a safe and structured starting point.

The same applies to the named roles, having someone with an Agile Mindset that can help create a foundation of understanding of the roles and when and where they add value can provide a structure and consistency that allows you to grow from a starting point that has proven to be effective for many.

Do not stand still.

But most of all, whether it be Story Points or the need for a QA, inspect regularly, assess what problem you are trying to solve in terms of outcomes, if you feel a change will show improvement then experiment and review.  Small changes are generally preferred, it is easier to see the impact and easier to undo if the results are not what was expected. Understand what you are changing and why so that you can properly assess whether the change was beneficial.

I have seen teams conclude a Scrum Master/QA/Product Owner is not necessary after all they can cover the role themselves (usually as the result of cost saving measures or because they are presented with a false notion that good teams don’t need help). Sometimes this works, but more often the responsibilities are performed less effectively when distributed among the team and so the net result is a reduction in the effectiveness of the team.

There are certain expertise and mindsets that often come with a role that is not fully appreciated until it is gone. This generally and understandably gets neglected when it isn’t the core responsibility of the team members.

Let the team decide

If a team feels they need a change I’d have far more confidence than if the decision was made by someone with little or no exposure to the team, or on the back of reading a blog suggesting “good teams don’t use Scrum Masters/Story Points/Scrum/etc.”

Team leadership in Agile teams

badleader

Yesterday I had two people both of whom I have a lot of respect for, independently say that having a single person in charge of a team ‘works‘.

I was taken aback by this, partly the surprise that two people would say the same thing on the same day, but mostly because this goes against my experience and my opinions on good team leadership. This caused me to step back and reconsider my opinion and my reasons for it. For me there is a great pleasure in being challenged on my opinions especially ones that I was so sure of and so I have given it a lot of thought.

My experience

I have worked for other people directly or indirectly for more than 25 years, and I have managed teams myself, I have also coached quite a few teams – so was able to witness leadership from the sidelines. By a very quick count of those I can remember I would say that 70-80% of them were (in my personal opinion) poor leaders, there were a few that got the job done by force of will, or by leveraging authority, or by imposing death marches on the team. The organisation sometimes saw them as successful but the teams thought them dictators or bullies.

Many team leads simply were unwilling to see any perspective other than their own. Others who were clearly insecure at accepting other people’s ideas.  But there were a few good or even great leaders that didn’t see management as a tool for control but as a scaffold for building the team and achieving things, if only these could be cloned.

david-brent

So I am probably coloured by my experiences but the notion of one person in charge of the team fills me with dread.  and whilst I wholeheartedly agree that the model of a single leader can work with the right person, that does not mean that it will work in most cases or that those qualities are the norm, and it certainly doesn’t mean it will work as a model in every case. What is more I think those great leaders would thrive in an environment where they didn’t have defined authority- but more on that later.

I can only imagine that just as I have been coloured by my experiences my colleagues have equally been coloured by theirs but they have had the good fortune to see better leaders than I have and I would like to (and will) discuss this further with them to see why they feel this is a good approach and whether my instinctive reaction and poor experiences are in contrast to theirs.

“Leadership should mean giving control rather than taking control and creating leaders rather than forging followers.”

David Marquet

oh Captain, my Captain

The military is often used as an example of one named leader, but there is a distinction in the military and that is they have needs that are very different to a software team. Those differences are a need for independence and a desire for expedience in decision making: Military units will often need to operate independently without contact with their parent structure. So it may be necessary for a local arbiter. In a business environment it is rare that a disagreement is so urgent that it could not be referred up if there was a dispute with an impasse. The other aspect of a military organisation is that life and death decisions need to be made very quickly and so there may not be the luxury of time to debate and reach consensus.

However, even in military structures there are examples of leaders who take the view that they are the Product Owners, they will say this is what I want to achieve, you tell me how… and then the suggestions are made and the leader simply endorses the team’s advice (Make it so..). Naturally there is an element of accountability but the trust that this demonstrates in the team is significant in empowering the team and growing them.

This notion is explored more deeply in this book:  Turn this ship around

turn-ship

 

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

Agile principle No 5.

We want conflict and debate

In software teams it is often the debate that produces the greatest thinking and ideas so stifling the debate is a negative. Having a single person making decisions stifles debate, it thwarts conversation, and it disempowers the team.  If a team is overruled often enough they will stop making suggestions, if one person becomes so myopic in their opinions it can make the team feel powerless and excluded.  Also where there is a defined leader they have a tendency to not be transparent, information is selectively shared (in both directions) and again lack of information impedes debate.  In short I believe that having a defined leader is in conflict with the Agile principles.

The best architectures, requirements, and designs emerge from self-organizing teams.

Agile principle No 11.

Merging the what and the how

Having the same person responsible for both the what (vision) and the how (implementation) stifles innovation. Rather than the team determining the best architecture and the best solution it becomes driven by a single individual. Again this is at conflict with the Agile principles, and whilst you may say that a good leader wouldn’t let this happen, experience and evidence is to the contrary. Power corrupts and if someone has the responsibility and authority it becomes hard for them not to use it especially if they perceive the team is making a mistake, and if teams are prevented from making mistakes they will stop experimenting.

We want balance

And this is where my agenda is.  Software development is a balance of content, quality, cost, value, consistency, team growth and a variety of other factors. It is rare or at least uncommon to have a single individual that is able to understand and balance those conflicting elements effectively on their own. More often a single individual prioritises one above the others, driving to a deadline, or gold plating a solution, or any other single aspect usually becomes prioritised at the expense of the rest.

I believe that a model where there is shared leadership and shared management between multiple roles solves many of these problems. Having someone focused on the what and others focused on the how and someone else focused on team improvement and consistency creates conflict (deliberately) but it also creates balance and it becomes a catalyst for debate.

We don’t want a single point of failure or a silo

If we make one person responsible for direction, implementation and team growth, we are putting all our eggs in one basket, if they are on leave or sick or move on then the impact can be significant. There can be so much knowledge tied up in one person they become indispensable.

leadership
Problems with the shared leadership model

First and foremost there is a cost. For small teams it may be difficult to find team members that have a natural affiliation to the balanced leadership model, with part time POs or coaches/scrum masters or where those roles are not named but the responsibilities are shared it can be tricky. But even then I would say that calling one person ‘Lead’ or ‘Manager’ or anything of that sort is destructive.  And the notion of combining Coach, PO and implementation into just one named role can lead to dysfunction. In many respects I wonder if the cost of additional named roles is worth it just to prevent the dysfunction a single leader creates. Or if the team is too small to warrant the explicit roles then get rid of named roles entirely – if a team is that small naming a leader should also be unnecessary, they ought to be able to self-organise within those boundaries.

This may sound hypocritical, after all I have spent a good proportion of my career with leader or manager in my job title.  But my experiences have been an internal conflict in that role. As a Software Development Manager you become the defacto Product Owner, Project Manager, architect and team coach. But there was always pressure from somewhere and normally that pressure was to the detriment of the team, when I challenged it, I did so a personal risk. Customer and senior management pressure to deliver, cuts costs and drive timelines meant that team growth became secondary, team welfare was deprioritised and making a safe place for teams to learn from mistakes was difficult to justify.  Some of that is company culture but I believe much of that comes from the pressure of responsibility and bestowing leadership carries pressure and expectation (sometimes real, sometimes assumed). And speaking as someone that has experienced it, I’d rather share that responsibility.

Conclusion

So after sleeping on the issue I am even more convinced that a named leader – whether it be Team lead, tech lead, manager, senior or nerd wrangler, goes against the principles of Agile, I believe it undermines self organising teams and leads to dysfunction and imbalance.  There absolutely are exceptions and there are some team leads that are effective, but I wonder if they would be just as effective or more so in a self-organising team structure. But there are more examples of ineffective team leads where the power corrupts and they dominate the team, stifle debate and innovation and disrupt or impede team growth.

In my very personal opinion the best leadership model is a balance of What, How and Team Improvement, and the more people those responsibilities are spread amongst the better. In practical terms I’d like to see a team with a PO to determine the What but a PO who actively engages with the team. A coach that is focused on the Team’s improvement and process improvement, and the rest of the team is responsible for the How.  Within the team there is no need to identify a senior or a leader they can work out amongst themselves how best to make decisions and titles get in the way.  This model may come with a cost and it may be difficult to get the balance right but in my experience this balance leads to the best results.

Ironically the examples of leaders that I have seen as being successful (as measured by both results and team morale) have voluntarily and noticeable made themselves servant leaders, stepping back and inviting the team in, choosing to give away authority and creating healthy debate and healthy conflict. So if that is how they lead effectively why not make that the model to start from?

Why I came to adopt an Agile mindset.

Ultimately it was because of seeing poor leaders disempower the team and abuse teams into death marches and drive poor design decisions that I came into Agile in the first place. I saw Agile as a method for empowering the teams and taking away abusive power from lone leaders. The stimulating of constructive conflict and healthy debate are so essential to the process that I object to any impediment to this on principle.

To my delight in most cases Agile has done that and so much more, in a creative environment like software the gains of self organised teams so massively outweigh the losses that result from a lack of a single clear leader that I am more confident in my opinion that the words ‘lead’ or ‘manager’ have no place in job titles on an agile team. I would love to trial and experiment to compare, but as much of this comes down to the personality of the people involved it is hard to do more than make subjective experience based assessments.

 

Why do we use WiP limits?

I participated in a great discussion this week on the use of WiP limits.  For those that don’t know, WiP stands for Work in Progress, and is a measure of how many activities you have started but not completed.

It is important to note that this is not a measure of how many tasks you are currently actively working on.  If I started a task but have become blocked, so I start another then my Work in Progress is 2 – even though I am only actively working on one. WiP is a measure of how many started but incomplete tasks I have.

For example: call-waiting or being put on hold during a phone call:  You are talking away and get another call so you switch to the second call without hanging up, and when you get another call you take that too, how many people can you have on hold at once?  You are only working on one call at a time, although you must remember the content and context of all those other calls. For the people still on hold while you have all these conversations I expect they are wishing you had a WiP limit.  Your WiP-Work In Progress is the sum of all active(incomplete) phone calls.

Limiting WiP

There are many reasons for limiting Work in Progress not least because I hate being put on hold.  I will go in to some later and it is likely that if you follow any type of Agile framework you already limit WiP although you may not be consciously aware of it.

If you have a backlog, then you are limiting WiP. You are consciously choosing not to start new work immediately but are prioritising it and organising it to work on later. It is likely that you are informally limiting your WiP according to what you feel you or your team can manage at a given time. It is important to realise that this is a WiP limit even though it may not be conscious or scientific.

If you follow Scrum: WiP is consciously and explicitly limited at Sprint Planning, often by estimating effort, or counting stories or calculating story points. The limit is generally set based on Velocity – our average achieved over previous sprints. If on average we finish 10 stories a Sprint, or we average 35 points, then that average is generally taken as the starting point for our WiP limit, we may choose to push for a few more, or if we are expecting to be slower due to vacation time, or known issues we may choose to commit to less. But in Scrum we don’t usually refer to it as a WiP limit. But that is exactly what it is.

KanBan is actually very similar, if we on average are completing 10 stories a week, then it is likely that we will consciously or subconsciously limit ourselves to only take on 10 new stories within a week, although in the case of KanBan this is not done in a Big Bang explicit decision but gradually over the measured period and is more a consequence than a plan (A Pull rather than a Push). We pull stories when we are ready and we pull at the pace we are working at, that just happens to be 10.

Slightly off-topic, but one of the crucial differences between Scrum and KanBan is that Scrum is a ‘Push’ model and KanBan is a ‘Pull’ model.  In Scrum we Push an amount of work to the board and spend the Sprint working to Remove(complete) it.  With KanBan we are aiming for a more steady amount of Work in Progress and will pull new work as current work is completed, which creates a smoother flow, but conversely lacks a unified time based goal or target.  But it is important to understand that both frameworks limit WiP and both focus on ‘completed’ work being the primary objective.

Why do we limit WiP?

I’d like to think it was obvious by now and I think the basic principle is, but there are nuances that may complicate things which may be where the cause for debate comes from.

At it’s simplistic level if I can only complete an average of 10 stories a week/sprint then starting an average of 20 a week without changing anything else is nuts, all that will happen is that on average 10 or more of those stories each week will remain incomplete and stay ‘in progress’ by the start of week 4 we will still have completed 30, but will have 50 now in progress.  Chances are by now we will actually go slower because we will be distracted and falling over ourselves trying to juggle more than we can possibly achieve.

Think of all those people on hold, growing frustrated at being ignored, and you trying to remember all those conversations, the phone rings again, can you cope with another caller on hold?

We limit WiP because we understand that by working only on what we can achieve, we can focus and be more effective.

Limiting WiP and WiP limits

“Ah but you have talked about Limiting WiP but you haven’t mentioned WiP limits” I hear you say…

And this is where I think the conversation really stems from and where we get into nuances.  KanBan began in manufacturing where work was passed from one workstation to another and there was a measurable flow through the system.   Limiting the production on one workstation so that it maintained pace with the entire system limits waste. Having one hyper-performing efficient widget maker that can produce widgets 4 times faster than they can possibly be used is wasted effort. And the same applies to a software process, we use WiP limits to regulate one part of the flow to maintain pace with the flow as a whole, the goal is to visualise bottlenecks and by visualising bottlenecks we can take steps to increase the flow of the whole system.

Other forms of WiP limit

But WiP limits can take many forms and are applicable to almost all aspects of life. The most common example used is a highway, when a road gets busy it slows down, when it gets very busy it grinds to a halt.  Limiting access actually speeds things up for everyone on the road, and ultimately more cars can get through far quicker.

But there are other less obvious WiP limits. A long-distance runner wants to increase her times, she starts fast but she runs out of energy and is slowing down near the end of the distance. By setting herself a slower time per mile early on (Pace is a form of WiP limit) she has a consistent speed and reserves energy so is able to run the full distance faster by slowing down during the early part of the race.

Are you dedicated to a single product/project?  That is a very low WiP limit of 1 project.

Do you use a calendar and try not to book two meetings at the same time, that is a very tight 1 item at a time WiP limit.

What about a budget?  Do you manage your finances as an individual or company. A budget is a WiP limit for your money, by limiting money spent on beer you have more available for clothes. By regulating your spending you can save for a holiday.  By remaining in credit at the bank you do not have to pay interest and so on.

Example of how to choose a WiP limit in a workflow

In the case of the widget maker, if the system can only use 10 widgets a day then we may set his WiP limit to 10, when he gets to 10 he is free to go and help someone else.  Suddenly rather than unused widgets we have an extra pair of hands to do other work.   But hold on! these widgets are easy to make and only take a short while to do, we could probably limit WiP to a lower limit and jump on the workstation as an when needed to produce more.  So how do we set the WiP limit?  My advice is not very scientific at this point.  Start by continue doing what you currently do and just watch, if your workstation/activity columns are sub divided in to doing and done, take a look on a regular basis and see where the biggest stacks are. If one column regularly has more cards on the done column than doing, it ‘may’ be a sign that you should reduce your WiP limit for that activity. Try it and see if it improves your flow.

The largest queues of done work are usually the area that needs limiting, but be aware of ‘bursts’ of activity, for example there may be a workstation that is only available for one day a week, in which case the queue for that workstation may need to be longer – or maybe you should be asking for more regular access to the workstation. But when imposing limits do it gradually and monitor to see if it improves the flow.

Anything in a done column is normally waste, if it is done and not in production that effort cost you money and is just sat there.  Sometimes that makes sense but the car industry learned that complete cars sat in storage amounted to $millions of wasted investment, that components stockpiled in advance was essentially big piles of cash that could be used more effectively, we can learn the same.

What I am saying is that a WiP limit being reached is a conversation trigger, a long queue is a conversation trigger, and so on. With so many things in Agile we create early warning systems to create conversations and make decisions when it is necessary.  We make small changes to the process and watch, we see what changes and if it is better we carry on, but look for another new small improvement.

Summary

WiP limits are not unique to KanBan, and like it or not you are already limiting your work and activities in all aspects of your daily life, you just perhaps didn’t realize it.  In the context of a KanBan board we do not arbitrarily impose a WiP limit because we can or because it is fun. We impose a limit when we feel it adds value, normally where we see a queue of completed work growing faster than it is being consumed. We limit various stages of the work because our goal is the output of the system (the whole team) and not the perceived utilization of individuals.  One team member being 100% occupied all day producing something that will sit untouched for a week is not efficient.

We are not trying to manage a system where our goal is to keep busy a group of individuals, our goal is to create a cooperative and coordinated team working towards a common purpose in the most effective way possible. A WiP limit is just one of many tools that can be used to enhance the prioritisation of work and focus the team on the true goal.

 

 

 

Is structure supportive or restrictive?

I have been engaged in a discussion along these lines with a colleague over the last few days, it is a subject we both are interested in and one that I have been very much on the fence. 

I began from a position of a very strong belief that structure was beneficial and that it provides a scaffolding that enables teams to grow. But my colleague took the view that once in place the structure becomes at best a crutch and at worst a barrier to self-organisation and agility. 

This conversation is the basis for the usual discussion on whether a Scrum Master is necessary or should they make themselves redundant.  Or – Do we need a designated product owner or simply a better understanding of product ownership? 

Even to the point of whether there are any designated roles within the team, do we need to identify Devs, Testers, QAs, UX, BAs etc etc or can we consider all “Team Members” that are ‘T’ shaped, that is to say that whilst they possess specialty skills they are first and foremost team players able to cover most of the core software delivery skills.

Baseball

I am (very)English, so my knowledge of baseball is minimal. But today I had someone explain the finer points of ERA and ERA+ and some of the other joys of the statistics of baseball. I can sense you preparing yawns at this point so I shall go no further. But the conversation progressed to “Designated Hitters” players who do not field, they have a specialist skill, or are able to stand in to save the pitcher from batting (another dedicated role?)  I may have misunderstood this, but it was described as an “abomination to the sport” by someone that clearly believes that it is a sport where the players must participate beyond their specialty.  The converse argument presumably is that if you have exceptionally skilled specialists why should they need to participate in other roles – the team is more effective as a whole by them specializing.  Okay so I am stretching the analogy a little, but the argument holds.

Should we specialize

Specialism may aid the efficiency or the effectiveness of the team, but is it still a team sport and can a team ‘self-organize’ if there a members that can only fulfil a specialty role?

And let us suppose that we have two expert catchers, and they have no batting or fielding skills, there is no benefit to having both of them on the team. And what if our best catcher is also a good outfielder, do we use our second best catcher in the catcher position because he only has the one skill?

My view started with the opinion that the most effective teams I have worked with had both a Coach and a Product Owner, that specialism in those roles is necessary to achieve the focus necessary to be optimally effective, both individually and for the team. In the case of the coach his input magnifies the team at a higher rate than adding another team member and for most medium or larger teams the product ownership tasks are demanding and require one person to have a view on many aspects of it – albeit that the team can and should be involved as much as possible.

I stand by this position, if our only goal was the effectiveness of an average team in average circumstances.

The problem with this situation is that there is an implied hierarchy, and that the appointed people in those roles may not be the best choice, that members of the team if enabled may be able to take on some or even all of those responsibilities.  Essentially those enabling roles also become limiting factors to growth.

So long as there is a coach there, some of the team will use them as a crutch, the implied hierarchy may disguise or suppress dysfunctions, the coach may be more valuable elsewhere or in another role but is constrained.  The same for Product owner, they can easily become a bottleneck as much as a facilitator/enabler. And let’s be frank how often is the workload for a coach or a product owner approximately 1 full-time employee. Chances are one or the other or even both will be grossly over or underworked and consequently stressed or bored.

The teams will never become completely self organising in those situations. Not until enough of the team are skilled and effective in those roles and are able to internally organize themselves. And that can and will only happen if they are allowed to grow into that. 

What to do?

But of course just starting on day one and say “go organize yourselves” would be madness in most circumstances, the skills are not off the shelf skills and generally come more from experience than teaching.  But having guidelines or even common practice of teams of that structure creates an expectation of what ‘good’ looks like, and it becomes a self-perpetuating concept.

My belief and I am conscious I may be a minority here, is that we provide scaffolding, but from the outset make it clear that the end goal is independence from constraint, that the structure presented is to enable learning and will eventually be removed.  That we take our ‘T’ piece team members and we encourage them to diversify further, to become capable and aware of more skills.  We support specialism, specialty skills ARE valuable but all rounders are more valuable.

Say no to designated batters…

We don’t want designated batters, we are a team, and in a team we still function near to 100% if/when we lose a player. If that means we lose some specialists because we favor all rounders, perhaps that is a risk we have to take.  I don’t believe this means that we compromise on quality, it just means being more open to learning other skills.

Coaches should focus as much on teaching and enabling independence as they do on effectiveness.  The pragmatist in me still believes that we are a long way from this ideal, and that in most cases the teams will likely organize themselves with designated product owners, QA, UX and Devs, as inevitably there will be some people that are more skilled than others, but I would like to see us blurring those lines to the point where it is difficult for an outsider to recognize. 

The teams will likely still desire coaching and support in difficult situations, I am still of the opinion that we all benefit from coaching (even/especially the coaches), but if this can be owned by the team and it can be their choice as to who and how performs the roles we will be much closer to the goal of self-organization.

A lofty goal and one that will likely have a few hiccups along the way, but  I am beginning to believe that we can use the scaffolding of structure to be a stepping stone to self-organization.