The 3 Elusive Qualities of a Product Owner

It may sound overly simplistic but in my experience products generally succeed or fail in correlation to the effectiveness of Product Ownership on the team. Mainly because we under-appreciate the importance and the difficulty of the Product Ownership aspects in the product creation process. In my opinion the choice of this role is far more important than the coach or other team members. The role is full spectrum, they go all the way from discovery to delivery and more.

Ironically, we tend to think of software applications as being primarily technical objectives, our focus is too often on the quality and ability of the engineers, the architecture and design. We will spend time on process and optimization and efficiency. All the while forgetting that the most common and most consistent failure of all software products is that no matter how good the quality, how optimized or efficient it is, or how good the engineers are that built it – if you build the wrong thing it is a failure. A perfectly designed but unused product is worthless.

History if full of great products that missed the mark, how many apps have you downloaded to your phone that are no longer used or were never used after the first 30 seconds. There are many many more products that fail to ever get in the hands of customers at all. Internal IT products are often culprits of this, the assumption seems to be that because the users are forced to use the product we don’t have to actually ask them what they want or how they will use it, we build great products that don’t fulfill the actual user need.

I think probably my favorite and most over used quote is:

There is surely nothing quite so useless as doing with great efficiency what should not be done at all.

Peter Drucker

I use the quote far too frequently but in 25 years developing software this remains the most frequent sight I see, more products fail for not meeting the most basic customer needs than for any other reason. And yet when I say this, it generally gets scoffed at, until you start asking how many people have worked on products that have been abandoned or shelved mid development. Teams rarely believe their product is at risk of being cancelled, or are too far removed from the customer to appreciate that the user liking and using the product is infinitely more important than whether they are efficiently re-using a section of code. We have created a mindset that ‘done’ is delivering code. A product/feature/story is only really done when the customer is using it.

Product Ownership does not need to be a single person in that role, but it should be a mindset within the team. Ideally I’d love for this to be a shared responsibility. Unfortunately in my experience a true appreciation of Product Ownership is as scarce as Hen’s Teeth, and generally unappreciated and vastly underpaid, those that do it well make it look easy and doing it badly or being indecisive can really destroy a product so it can be a risk to spread the responsibility beyond one person. I have heard people talk of teams where the teams do it well and they don’t need a PO, but I have yet to see this for myself. Because of this experience I think of the Product Owner as the most important ‘hire’ for the team, getting the right person in this role can be crucial to the success of the product.

A good PO these days is hard to find…

So why is a good Product Owner so hard to find and what makes the role so difficult?

1: Differentiating what is needed from what is wanted

A significant part of the role is identifying what is needed to solve your users problems or to fulfill their needs, to enable them to be better at their job, or to maximize their fun or relaxation. The role is about understanding the problem and creating a vision of how that could be solved – not the technical solution but discerning the true need or true problem.

Henry Ford With 1921 Model T — Henry Ford stood next to his ‘Faster Horse’

If I had asked people what they wanted, they would have said faster horses.

Henry Ford

The reason this skill is so tough is that users rarely know what they need, typically a user’s vision is constrained by what they currently do. As a result many products become re-works of a paper system or re-engineering an old system. A poor product owner gives a customer what they want or what they ask for, a good product owner listens and watches and blends wants and needs to solve the customer’s problem and give them what they need.

I often seen backlogs that are full of stories that are unfiltered customer requests, it is lazy and it is easier to do what is asked. But to find what is needed typically requires experimentation, feedback, conversations and observation. It is far harder to build what is needed than it is to build what is asked for. Similarly this can manifest itself when department managers decide what their team needs or wants without involving them in the discussion, a good Product Owner will spend time with the users of the product and ideally see how they use it.

A Product Owner with the creativity, the initiative and the Vision to create a great Product is worth their weight in gold,

2: Understanding what is required to create a successful application

A second aspect of the role is the one I see most frequently overlooked. Perhaps because of the emphasis on 1, the need to understand the business, I see people choosing someone from the business to become the Product Owner, their knowledge of the business is seen as their primary qualification and in many cases their only qualification for the role. And worse they are often asked to do the role in addition to their normal job.

However, a great Product Owner understands not only what to build but is familiar with the development process, they understand the ebbs and flows, they understand the implications of technical debt, automation and testing. The understand how crucial it is to have working software in the hands of users as soon as possible. I am sorry to say that business POs often lack this knowledge and experience, they will mistakenly perceive a desire for a fully working product rather than appreciate the value of early feedback. Many are plagued with the last bus syndrome, trying to include every imaginable feature rather than working towards a minimum feature set to get feedback and derive value early.

Personally I’d invariably trade an experienced Product Owner with zero business knowledge for an expert in the business with zero software delivery experience. Business knowledge can be gleaned from effective conversation, observation and feedback. The understanding of a good Agile Software Development lifecycle is far harder to learn and there is no substitute for experience.

Typically the best products result from experiments, reviews and feedback, a willingness to be flexible and respond. The other disagreeable quality of a Product Owner taken from the business is that their knowledge is about how the work is done now, and they are often blinkered to the possibilities.

Remember that the Product Owner sets priorities and interprets the Vision, they have more influence on the product creation than anyone. Their choice of priorities can profoundly impact the value, and the quality of the product. If they lack the understanding of the importance of testing, feedback and good software principles it can lead to conflict and disharmony in a team.

3: Communicating effectively with both technical and business stakeholders

Finally there is a need for effective communication. We always talk about effective communication to the point that it becomes background noise. But the importance cannot be overstated.

A great Product Owner, listens, they observe, they probe and they reflect. The ask for feedback and they read between the lines, they watch body language, they see and hear what isn’t said as much as what is said. When it comes to understanding the user they have great attention to detail and they look intently for pain points and points of pleasure, they are looking for a way to serve their users in the best way they can. Most of us think of communication as speaking, but for a Product Owner watching and listening are so very important. Empathy with the user, the customer and the ability to communicate using their terms and understanding their domain and conversing with them in a meaningful way is crucial, and if you don’t understand the terms develop an attitude of attentive student ask questions and show you are interested and are learning.

A great Product Owner learns to say ‘No’ powerfully respectfully and does so in a way that conveys they understanding to the person asking, a great Product Owner should not be seen as an obstacle or a hurdle but a steward of good decisions. The word ‘No’ should be delivered in a way that leaves me feeling confident that the Product Owner will deliver the best product within the parameters and leaving me confident that this decision is right even if it is not what I initially wanted. The Product Owner should be able to convey the implications of saying ‘Yes’ so that I understand why I am being told No. A great Product Owner has any number of tools that can help people see this and yet we regularly see backlogs full of feature requests that will never see the light of day because the PO doesn’t feel capable or sufficiently empowered to say NO!

But the PO must also communicate with the development team, they must understand technical development terms, to have a grasp of the technical implementation, even if they do not have authority over the ‘How’ I’d expect them to have a vested interest in understanding it. They also need to be able to communicate the business needs and the business terms in a meaningful way to the development team, This requires a versatility in speech and communication that is unusual to say the least. Being respected and understood by both the business and the technical team is a major feat, and as we all know any cloistered group speaks in three letter acronyms that only they understand and the audible sign when they have to explain to an unlearned outsider can demoralize the best of us. Do not underestimate this ability.

But there is more, they need the creativity to communicate design ideas or if the ideas come from others the scope to understand that creativity, to imagine what could be and express that to the business and to the technical team. And then we get into the really tricky realm of the stakeholders as opposed to the users, this generally is the group that funds the product, they want to see progress, they want to see good stewardship of their investment, they may ask for forecasts and may even press for commitments.

We have now loaded our Product Owner with the need to communicate with senior people that make financial decisions, to express Risk and Forecasts in a manner that it concise but meaningful, we want to be transparent and informative, but confident. If we are using forecasts especially mathematical forecasts we had better understand them and be able to back them up with explanation and confidence. A Product Owner my stand their ground in the face of authority and speak truth to power. A Product Owner that makes unrealistic commitments or promises, will lose the confidence of her team and her stakeholders in a flash.

If a Product is to be sold it is likely that you will need to be involved in Sales and Marketing, which brings a whole host of other skills and communication issues to bear, understanding the implication to the market, and how you measure that and how you engage with that is a new realm of understanding and confusion.

A great Product Owner is a master communicator, they have an insatiable desire for knowledge, to understand and be understood.

Summary

As you can see the skill set of a great Product Owner is monumental, and yet the good ones make it look easy. If you have a mindset suited to Product Ownership it is likely you already have good communication skills and enjoy learning more domains.

It is likely you are the sort of person that thrives on the delivery of a product and revels in the feedback, enjoying the flexibility and change, is unmoved at the thought of redoing something twice or even more to get it right.

Solving problem is a thrill and being able to see beyond the obvious and draw out information from users and then present something back is a joy.

That doesn’t mean it isn’t incredibly difficult, overwhelming at times and sadly unappreciated, and very often underpaid.

At some point the industry dismissed all these skills as unimportant and we see a flurry of Product Owners that write stories exactly as dictated by users, that input data into forecasting tools without the vaguest notion of what the forecasts mean or how they are calculated.

As I said at the start a great Product Owner is the lynchpin of a great Product, we should be seeking them out and rewarding them appropriately, you will be amazed at what can be achieved with the right person in that role.

Never talk about politics!

I grew up in the UK but I have lived in the USA for a little over 6 years in total, which equates to around a quarter of my working life, so whilst I still feel very British there is an element of the mid-Atlantic creeping in. That is to say I feel like I have grown a level of understanding for some of the quirks of American culture and am a little more sympathetic to the perception of the UK culture from the American side of the pond.

However being so overtly British I get to hear the same questions repeatedly and some of the misinformation is a little troubling at times. Especially when it can be fact checked so easily. Since this is my soapbox I’ll indulge a little before I get on to the topic at hand.

First the USA did not win the war of 1812 please take 5 minutes to read wikipedia. Second, the UK NHS: I have been in the USA for 6 years and in that time I have worked for two employers both of whom tell me that the health insurance they provide is the best there is. My colleagues tell me it is better than anywhere else they have ever worked. I have no reason to doubt that this is great Health Insurance, and apparently it costs around $27,000 per year – mostly paid by my employer and that is in addition to the social security deductions which are approximately 8% of my paycheque.

In my experience the quality of service offered by the ‘best‘ Health Insurance is approximately identical in quality of healthcare to that received from the NHS. Yep you heard me, I find no discernible difference in quality of care. There are certainly differences, I have to pay a co-pay to see my Doctor here in the US which is free in the UK. Doctors in the USA want to push you to take every test under the sun so they can bill the insurance, where in the UK they assess if there is a need first. In the USA if there is a referral it is on you, whereas in the UK your doctor would follow-up. The Doctors’ surgeries here seem to be a bit more lavish and they need an army of staff to administer the insurance bureaucracy but as far as wait times, quality of medical treatment and accessibility of treatment I see no difference.

So I do get a little upset when I say people describing the UK medical system as having poor treatment and long wait times and something to generally be afraid of. Think of it as having the best medical insurance you can buy, and then imagine that it is free to you, I can see why that is so incomprehensible when you are used to spending nearly $30,000 per year on that service. I am sorry to say it so bluntly, but the USA spends on average 4 times as much for what is very much an inferior system for most people.

Okay I’m off-my NHS soap box now.

But the third and most troubling perception that I am regularly confronted with is the notion that the UK is a ‘Socialist‘ country and the USA is a ‘Capitalist‘ country. This notion troubled me a lot and got me thinking, but first let’s clarify the definitions:

Socialism: political and economic theory of social organization that advocates that the means of production, distribution, and exchange should be owned or regulated by the community as a whole.

Capitalism: an economic and political system in which a country’s trade and industry are controlled by private owners for profit, rather than by the state.

Essentially the extent to which you are a capitalist or socialist country is determined by the extent to which the government interferes with the free market, either through spending or through legislation.

Measuring Capitalism

Spending is pretty easy to assess as we can see what proportion of GDP comes from government spending and how much from ‘the free market’ However, legislation is much harder to measure and by definition any intervention in the market is ‘socialist’. E.g. Defence, immigration, policing, taxation, tariffs, subsidies, etc.

I think most people realize that pure socialism is fatally flawed and equally flawed is pure capitalism, so it is really a question of how little or how much government is needed and how much can the market provide on it’s own. We all have a reasonable expectation that infrastructure, defense, police, jails etc cannot be provided by the market and that the whole country benefits from education and healthcare for the poor as the benefits are far reaching to the economy but how much of the rest is open to debate.

Let’s take the USA, the US spends a staggering 35% of it’s GDP on socialist activities, that is more than a third, and is incredibly socialist in it’s government policies and legislation, just recently there have been some very high profile trade tariffs, a trade tariff is direct manipulation of the free market by the government – it is the very definition of socialism. Lobby groups regularly persuade the government to invest in particular parts of the country or legislate for the benefit of certain industries. Both the UK and the USA bailed out banks and the USA bailed out the car industry and corn farmers.

By contrast the UK spends 41% of it’s GDP on socialist activities, and also heavily regulates the free-market with employment laws, they provide tax-breaks for entrepreneurs and subsidies for ‘green’ businesses to name a few, but notably didn’t bail out the car industry or steel industry or mining industry.

In my anecdotal experience and to be clear this is opinion based on only very limited education in Economics and Politics, neither the UK nor the USA has any claim on being capitalist countries, both may claim to be slightly right of center but are both squarely ‘Mixed economies’ and have found a balance that is slightly further from Socialism than Capitalism but not by much. And whilst the UK spends more, my observance is that the USA regulates more and is far more interventionist in investing, bailing-out, propping up and otherwise supporting USA based industries.

TL:DR Summary:

The UK and USA are broadly similar in terms of the degree to which they are Socialist or Capitalist in practical terms, but in my observation the UK is more willing to intervene for the benefit of the people – especially the poorer and the USA is more likely to intervene for the benefit of business.

What has this to do with Agile?

So what does that have to do with Agile? Well it strikes me that there are a great many parallels. If we equate Socialism – a planning heavy form of governance, where plans are made by a few experts:- with ‘Waterfall’ and Capitalism – a responsive form of governance where the market leads and the many respond independently of governance:- with Agile, then maybe you can see where I am headed.

In small scale both Waterfall and Agile succeed, but as we scale both suffer in different ways. In Waterfall as in Socialism the planners become further removed from the market they are serving and are less responsive to particular needs and more and more out of touch. a great deal of overhead is needed to align the vision from the top and unsurprisingly they are often so far removed they get it wrong, corruption sets in and effectiveness (the true productivity) plummets.

By contrast in Agile as in Capitalism works very well in small scale but as we grow the decision makers are not able to see the big picture and whilst they are responsive to what they can see, they fail to see the larger system and optimize for themselves rather than the larger organization and problems set in with misalignment. There are some aspects of a business that simply don’t work by being responsive and need to be planned and organized.

Governance

So where does that leave us? Well at heart I am both a capitalist and an Agilist so I am biased, but the way I see it as we scale with Agile and in business in general we discover that we need governance, some degree of consistency, infrastructure and support and most of all a clear direction. But I believe we should keep that governance as small as practical and only govern that which we cannot govern ourselves.

In the case of big government I believe that includes socialised healthcare because normal market forces do not apply when it comes to our health and so the market cannot effectively provide what we need, I also believe that we should invest heavily in socialised education as better educated workers make everyone wealthier.

When it comes to Agile teams I don’t think the teams should do the hiring or decide pay and they don’t generally want or need to be bothered with administration or infrastructure, I feel that is governance that they should be spared, similarly I think ‘health care’ such as vacation, benefits, and support (HR) is governance. Start-ups will need to deal with all of this but at scale we shouldn’t have to. Nothing controversial so far, but where we may start to see differences of opinion is the level of governance needed for teams.

In my opinion we achieve the best results from small teams, that are given clear direction and are empowered and enabled, that means direction not a director, so team leaders, project managers or any other similar role is bad for teams, it is unnecessary governance and is in my opinion fundamentally anti-agile.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The best architectures, requirements, and designs
emerge from self-organizing teams.

That doesn’t mean we don’t need managers or leaders, but we don’t need them involved in the work. In Scrum we empower Product Owners to set direction and priorities but they stay out of the HOW, we empower Scrum Masters to coach us on process and guide us on improving flow, we even empower them to remove impediments on our behalf but we do not empower them to tell us HOW to do the job. In my opinion Project Managers (when they are deemed necessary) should respond to information provided by the team and not the other way round, they are a consumer of information only! They should convey no authority.

Where we need managers is NOT for doing the work, a suitably cross-functional team should be able to do that without an anointed leader. What we need managers for is the ability to create those cross functional teams, and do this by listening to what is needed, hiring and enabling the forming of those teams. We need leadership to provide clarity of direction, clarity of culture and clarity of values. We need managers to help us by enabling us grow as individuals in our individual careers in parallel to our goals for the team. We need managers for ensuring that we as an organisation are planning for the future so we can focus on delivering in the present.

If you feel you need a manager to manage the team doing the work then in my opinion you have failed elsewhere, it is a sticking plaster to a bigger problem, all you are doing by instilling team leaders/Project managers or team managers on a delivery team is masking a bigger problem elsewhere, possibly in hiring, training, education or communication.

The prime role of leadership is ensuring clarity, they have the big picture, making sure their employees see it too should be their number one goal, any other governance in the day to day job is an unnecessary overhead and a smell that there are problems elsewhere. The corruption inherent in larger systems needs to be cut out at the source not covered up with a layer of management. If you are hiring the wrong people then fix your hiring process, don’t hire people to manage problem staff. If you are struggling with consistency or maintaining values then it is an issue for education and clarity, putting a manager on as team is not the solution: enforcing values is not the same as instilling values.

In Agile, like Capitalism, we want government to be as small as possible and only get involved in things that we either can’t do ourselves or are necessary but unrelated to our day to day work. Unnecessary bureaucracy and governance are well, unnecessary.

Give them the environment and support they need, and trust them to get the job done.

Hire the right people and let us do our job, educate don’t regulate, enable don’t enforce, but most of all trust us, don’t try to control us. Set a direction, then get out of our way.

Goldratt User Stories

As a product owner we should be building products to deliver on the Vision and a set of defined objectives or success criteria. But often our plan is not closely mapped to those criteria or it can be difficult to measure whether we have achieved our Vision or delivered on our objective.

Let’s change the way we write stories

So I wondered if we could expand on a desire for delivering measurable value by changing the way we both plan projects/product and write our user stories?

Tell me how you measure me and I’ll tell you how I will behave.

Eli Goldratt

The notion of this is that rather than defining our User Stories in terms of behaviour, or actions or activities – in which we primarily focus on the ‘What’ we should change, our start point should become us defining the “needle we want to move” Let us start with the measurement of success. Understanding how we will be measured and identifying behaviour that impacts the measurement.

E.g. our goal is to increase the number of items per basket for online sales, our measurement is therefore the average number of items sold per transaction.

Equally it could be to increase the average value of the basket per transaction, or increase the number of units sold of high profit products

Hypothesis rather than Story

Our ‘story’ then would be to ask the Product Owner and Team to identify one or more hypothesis for ways that could impact that measurement. The notion being that a hypothesis must be tested before the story could be considered done.

This could take the form or a brainstorm session to identify the more traditional stories or it could simply be left for the team (in conjunction with the PO) to determine the best way (possibly two or three alternatives) to achieve the desired outcome after the story has been pulled.

This is building on the principle:

The best architectures, requirements, and designs
emerge from self-organizing teams.

Already happening?

You could argue that this is already the role of the Product Owner and that the Product already does this, and in identifying and prioritizing stories they have already considered the impact. In some cases I would agree with you, in a few cases I have seen products or projects with clear objectives in terms of the goal of the product and even sometimes a measure of success in terms of measurable objectives.

I have yet to see the end goals mapped even at a high level to the stories and features, in most cases success criteria are wooly or undefined and more often than not the Product Owner and team will brainstorm stories (story mapping) without reference to the goals, and even Impact Mapping often will not extend to measurable goals.

That is not to say that the Product Owner is not considering this holistically, the measurements are generally part of the consideration but many stories creep in that have no material benefit or impact on the success criteria of the product.

Measurements for Goals as the basis for Story Mapping

For reference I love Story Mapping, I consider it my favourite tool for Product Ownership and have been known to rave a little too much about it on occasion.

The start point in story mapping is typically to identify themes of user activities (you could call them epics but don’t get hung up on terminology they are essentially big rocks and then we will progressively break them down in to smaller rocks arranged underneath the big rocks.

We can then prioritise the work in the context of the whole rather than being constrained by a linear backlog which makes the context difficult to visualise.

It is also a fantastic tool for explaining the product for release planning and forecasting and so on. Like I say one of my favourite tools.

But what if instead of starting with activities or big stories, what if we started with our measurable goals or success criteria. e.g. new subscribers per month, revenue per visit, time to achieve objective using application etc. And we then identify stories that best enable us to achieve that goal.

There may very well be some overlap so there maybe the notion of tagging a secondary objective, but imagine if this helps us weed out stories that are not taking us to our goal or have limited value in the context of our goal.

We would ask which of these stories has the potential to move the needle the most? and prioritise accordingly.

Make it measurable

Focusing our efforts on measurable impact to the product should implicitly be the goal of any Product Owner so why not make it our explicit goal?

Are you working too much?

Would it surprise you to learn that the person in your team or company that regularly works 60 hours a week (and makes sure everyone knows it) is very likely the least productive person on the team?

The person working 60 hours is likely the least productive person on the team.

And I don’t simply mean that productivity diminishes after 40 hours, I mean absolute total productive output is less from someone working 60 hours compared to someone working less than 40. The reality is that you would actually get more done by NOT working that time than you will working those extra hours.

This is a hard pill to swallow in a culture where ‘working hard’ is seen as a virtue, and the more hours I work the more virtuous I am. Long hours are often assumed to show commitment, dedication, loyalty. Maybe they do, but they don’t show productivity. If your company values those things above actual productivity then so be it, but understanding that it is about those attributes rather than productivity is a rather brutal fact to face. However, if you want to produce more and maintain high quality then you need to work less hours.

Parkinson’s Law

I am likely to offend a few hardworking people with this but my assessment of the findings of these studies is that is is a subconscious manifestation of Parkinson’s law. “Work expands so as to fill the time available for its completion” If I expect to work 60 hours then I’ll make my work last 60 hours.

Not just a European thing.

Europeans have a reputation for favoring reduced working hours and I’m sure it is an ongoing frustration to those in America that despite typically working more hours, American companies are consistently similarly or less productive than their European or Japanese counterparts despite the longer working hours and less vacation.
Interestingly, the study I’m quoting was actually a 12 year study by the Ford Motor Company of production in the USA.

Henry Ford… again.

Before I go into the study in more detail I’d like to go back in time and talk about the history of working hours and in particular Henry Ford.

If we go back 150 years there were campaigns all over the world to reduce the working week from 16 hour days, 6 days a week – 96 hours was considered a normal working week for many. Forward thinkers pushed for limiting the working day to 10 hours with breaks for food – a radical concept at the time.

Over time much of this became law despite fierce objection from industry, and by 1914 a normal working week was 9 hour days, 6 days a week. The pressure until then mostly came from humanitarian motives and agendas, and perhaps because of that, businesses took steps to prevent it taking hold, reducing pay in line with reduced hours so a reduction of 20% of hours came with a reduction of 20% in wages, this undermined the whole movement.

But then came Henry Ford. Ford created the 5 day working week, and a limit of 40 hours, but best of all he did it for profit and not for social good. At the time his factories worked 6 days and 9 hours a day (54 hours per week) just like everyone else, but he reduced working hours by 25% and instead of cutting wages he doubled them.

The productivity increases resulting from reducing hours were so significant that Henry Ford was attacked by the Wall Street Journal “To double the minimum wage, without regard to length of service, is to apply Biblical or spiritual principles where they do not belong.” going on to say “in his social endeavor he has committed blunders, if not crimes. They may return to plague him and the industry he represents, as well as organized society.”

51098603-f186b080bc53f64bbbd46208f7dec61a538631e4-s1200 — Henry Ford may have paid his workers a good wage, but it wasn’t out of charity — it was a good business decision that some say helped the middle class take off.

As you can imagine his competition was not happy either, paying more for “less hours” was maddening, but what really made them mad was that productivity jumped up massively. People simply couldn’t get their heads around the idea that less hours increased overall productivity. Slowly the competition copied Ford and got the same results, productivity went up for less hours, and eventually this carried across to office workers too. But even with those startling results and seemingly unquestionable evidence we stubbornly cling to the notion that more work is more productive.

I have written more about Henry Ford here.

Back to the present.

Last time the proposed reduction in hours was met with hostility but eventually it was seen as beneficial to everyone, especially the business owners that fought so hard against it. Once again studies show that reducing hours will increase overall productivity, and once again industry is pushing back on it in the face of the evidence. Challenging established culture with only facts is apparently not an easy battle.

Studies repeatedly show that long hours (40+ hour) results in reduced productivity, reduced quality and a less healthy lifestyle, and yet society still perceives the number of hours worked as a measure of hard work.
It is as if we see more value in the hours worked than the output of their labour. Someone that works 30 hours and produces 30 widgets is seen as less valuable than someone that works 60 hours and produces 25 widgets, because he works so much harder.

In our culture we see more value in the hours worked than we do in the output of our labour.

*In our culture we see more value in the hours worked than we do in the output of our labour.*

How did we get into such a mindset and how do we challenge that way of thinking. Surely we should reward the more productive and effective person? But we don’t, we reward the inefficient, we lift up Jane for being here at 7:30 PM finishing up her work and we criticize Anne who got all her work done and left at 4.

Part of the problem, especially in knowledge work, is that productivity is not easy to measure, just like quality and mistakes are hard to quantify. So rather than focusing on what is important we focus on what is easiest to measure – number of hours in the office, it is ironic that in our laziness in finding effective measures we end up working longer hours.

So what about overtime?

There is more bad news in the 12 year study by Ford they found that overtime (over 40 hours) results in reduced productivity in the long term. 40 hours is the very limit of maximum total productivity for a manual worker, this applies in both the short and long term.

Short term (less than 3 weeks) there are productivity gains for working beyond 40 hours, but after 3 weeks those gains disappear and actually reduce productivity afterwards. Even just 2 weeks of overtime and then stopping results in more loss in recovery than the gain made in the short term boost.

In other words, yes you can make a short-term gain using overtime to meet a deadline, but it will cost you more than you gain over the following weeks. So use it wisely and expect a recovery period.

Bad news for knowledge workers

For knowledge workers the news is worse still, the point of peak productivity is under 35 hours, and any hours above that are actually damaging, quality is markedly worse, mistakes and resulting corrections mean that productivity reduces dramatically when knowledge workers work beyond 35 hours, primarily because fatigue, stress and concentration have a profound impact on the quality rather than the quantity of work. We simply cannot be creative when tired, we struggle to solve problems come up with ideas or compose constructive discourse and debate, we are actually very likely making things worse not better and the clean up cost for knowledge work – bad decisions, introduced bugs, stifled creativity is far more damaging than not being there.

Worse news for Lawyers

Lawyers are typically in an unfortunately situation where they bill by the hour rather than by quality or even quantity of work. They are essentially encouraged by their business model to be unproductive (not consciously, it is just a product of the business model). Essentially a 60 hours week for them is the result of rewarding bad behavior, if you pay for time then they will take more time, pay for productivity and we become more productive. We also reward with status the hard working person that is willing to sacrifice his weekend for the appearance of working harder. More encouragement of bad behavior.

In ‘bill by the hour‘ work environments it is a challenge to change the model, productivity is very hard to measure – even if we see stress, mistakes and problems taking longer to solve. So it is hard to convince clients that working less is more productive, especially when it means charging more, and it becomes even harder when we can’t even convince ourselves because this notion is so culturally ingrained in us.

What now?

It is pretty simple really if you want to maximize the productivity of your teams, then reduce their maximum working hours to less than 32 hours (without reducing pay). They will work harder and more effectively for five 6-hours days than they will for 8 or 10-hour days. Your overall productivity will be higher.

Or work 8-hours, 4-days a week and you will find your teams will be more focused, more energized and and noticeably more creative. They will be happier, harder working and make less mistakes. They WILL get more done and it will be higher quality and more creative. They will also be refreshed have a happier home life.

Or we can ignore the brutal facts and continue to pretend that working long hours is a good thing, and praising the guy that works 60 hours a week and yet produces less than a part-time worker.

Note: my wife finds this particularly amusing as I work long hours and when I’m not working I’m blogging and when I’m not blogging I’m working on my game or running meet ups – there is a significant degree of hypocrisy and self-denial at play here.

Summary

Let’s not underestimate the difficulty here, this is a complex topic, and it is unlikely to be accepted any time soon, there is a lot of resistance – and that resistance is not based on facts or data. There are numerous studies on this topic, all with the same findings. Oddly enough despite the strong resistance to this idea, I couldn’t find a single study that didn’t find that overall productivity went down when hours went above 35-40 hours. There were even some studies that showed that regularly working 60 hours resulted in the same number of mistakes and level of performance as being drunk.

(If you see a study that contradicts this please share it, I spent quite a few hours on this but couldn’t fine one – and no, the contradiction has not escaped me.)

Decision Makers

Decision makers typically work longer hours and it is effectively their money, asking them to pay more for the perception of ‘less‘ work is never going to be an easy decision. We come from a puritan culture where we make a cultural assessment of someone based on effort not outcome. Deep-down we know this is not objective, I think we know this is wrong, but culture is not based in logic and changing culture will take more than facts and figures. It needs someone like Henry Ford to lead the way again.

References:

https://cs.stanford.edu/people/eroberts/cs201/projects/crunchmode/econ-crunch-mode.html

https://www.igda.org/page/crunchsixlessons

The 4 steps to successful software delivery

After [reading / posting] a recent article on waste, (read it here), someone asked me why we shouldn’t do work that we will need later if it is more efficient to do it now? You have already dug the hole—doesn’t it make sense to do all the work in that area together?

This is very similar to the argument for splitting stories horizontally along business layers: “We can be more efficient if we optimize and specialize.”

Yes, you might achieve local efficiencies by taking these steps. But local efficiency is not the most important factor in your decisions. Efficiencies and optimizations need to be considered in the context of the whole system.

The primary goal is to Maximize Value Early

For both project and support work, our goal is to maximize value for our organization, and to realize this value as early as possible. In a consultancy firm, there is a slight abstraction, as work is paid based on time and materials, and these steps are distributed between client and team. Regardless of who owns the following steps, success depends on shared awareness of these priorities.

The 4 Steps

The four steps are designed to focus our energy where it will have the most impact first, and then building upon the previous step by shrinking the focus area through each subsequent step. Hence they are in descending order of impact.

1 – Prioritization
(Selecting Which Project/Product to Build Next?)

The most important decision your organization needs to make is which work will bring the most value NEXT. This has the most impact on your company’s bottom line, so should get the lion’s share of effort and focus.

Describing the value a project will bring is far and away the most important aspect of software delivery. If possible, create a formula for calculating business value to help with these decisions. If you are unable to quantify or qualify the value to the organization of the work being undertaken (at a high level), then you are likely working on the wrong thing and all other actions you take could be waste.

Some projects/products will obviously bring in huge amounts of revenue. Others may save huge costs in manual entry and work-hours. Some may support business-critical systems that, if not supported, create extensive risk and potential cost. A project might also fulfill a safety or regulatory concern, or business objective.

Naturally this is complicated and can be difficult. Priorities change. Time factors and organizational politics come into play. A lot of the time you will be making informed guesses about value and costs. But this decision dwarfs all of the others in the impact to the bottom line, so is critical to put the right amount of effort here.

I would consider this to be the responsibility of the Program Manager or Project Management Office, but I see the responsibility to be to identify the projects that will bring most value to the organization as the end of their role. “Day-to-day Project Management” responsibility lies elsewhere.

I would however strongly encourage this decision-making process to be transparent and inclusive. Hold a regular Project review board and invite the stakeholders. Make the process clear and help everyone understand how the decisions are made. I would also suggest a visible roadmap of work planned.

Finally, I would encourage you to not plan beyond your company’s horizon. If new work is likely to emerge in the next 3 months that may significantly change priorities, then only plan for 3 months. Only start a product or project at a point you feel committed and certain it will be completed. Starting something with the expectation of pausing or pivoting suggests it wasn’t the most important thing to work on next.

2 – Direction
(Product Ownership – Building the right Product)

Whether this a new product or supporting an existing product, it is important to ensure that you build the right product, Product Ownership is covered in lots of detail in other places so I won’t go in to details but I suggest watching this video:

Like prioritization in Step 1, the most important responsibility in Product Ownership is Maximizing value–Are you working on the most valuable thing next? You do this through experience and through feedback from users and stakeholders.

Feedback is crucial. To get frequent user feedback, you should be thinking about how you intend to deploy your software and then update it regularly. You should be thinking of this before or with your first story.

Deployment and ongoing updates should not be an after-thought.

Product Ownership is also about maximizing the work not done. The product should meet the needs and goals but only do what is necessary. There will be a point when the returns diminish. All work carries an opportunity cost, and at some point your time is more valuably spent elsewhere. Understanding this is vital.

Finally, I want to stress the importance of getting value to the customer quickly. Value is only realized when the software is used. We should be striving to getting the software in use as soon as possible. This may be in an unfinished state. It may be only a partial solution, but it should be adding value. e.g. it can be used alongside an existing product, or it could be used for this workflow but for another you use another tool etc.

Feedback from people actually using your software is the most valuable feedback you can get.

Just like prioritizing projects, prioritizing stories has far more impact on the success of the project than the following steps.

3 – Quality
(Build the Product Right)

Building the right product is more important than building it right. Building the wrong product is just a waste. But that doesn’t mean that quality is not important.

Quality means doing it right when no one is looking.

Henry Ford

When assessing cost of software, it is easy to focus on the initial development cost alone. But support maintenance and future enhancements all fall into the total lifetime cost of a product. The cost of defects magnifies over time, and the cost of supporting poor quality code is significantly higher and more risky. Re-learning and knowledge transfer are painful and costly to an organization and I can think of numerous examples where companies have been forced to abandon projects or replace software because they lost a key employee who had the knowledge locked in his head–or software so fragile no one dares touch it.

Good practices such as Test Driven Development, Pair Programming, Behaviour Driven Development and Automated Regression Testing may seem costly at first. But when considered in the context of lifetime cost, they are a sound investment. More importantly, they enable the right business decisions to be made because there is a confidence in the software.

Quality code with a solid set of tests enables refactoring and new features to be added with the knowledge you are not causing unintended consequences elsewhere by breaking older functionality. This reduction in risk is a considerable advantage, especially when supporting older applications.

This is not a carte blanche to gold plate or over-engineer. Quality Code is the right code for the story and No More. We write just enough to satisfy the story and we write it well. Over-engineering applies to the GUI too–we don’t add features or over-engineer.

4 – Efficiency and Improvement
(Building it Better and Faster)

By this point we should be working on the most valuable product to our company. With that taken care of, our next focus is to work on the most valuable feature or story for our users/customers. We should be doing it in a manner that enables us to be confident in the work and able to expand it later.

But guess what? You can do it better. We should reflect on the decisions we made and how we make them. Are we making the right decisions? What helped us make those decisions? Are we making the wrong decisions? If so what could we change to avoid repeating those mistakes.

The reflection and improvement should be done at all steps and should incorporate the learning from all stages. If we continually look for ways to improve we will get better and better.

If you always do what you always did, you’ll always get what you always got

Henry Ford

What about efficiency of developer effort?

So what of the original question about being more efficient with horizontal splitting of stories, or the perceived inefficiency of re-working the same area of code later?

It is a question of impact and perspective. In both cases the question is founded on ‘efficiency’ being a measure of the developer’s efforts. What I would like you to consider is that in terms of delivering value to your users and ultimately your company efficiency should not be measured in terms of developer effort, but should be measured in terms of delivered value.

Efficiency should not be measured in terms of expended effort, but should be measured in terms of delivered value.

The most valuable decision is to work on the right project – this dwarfs all other decisions by an order of magnitude. But it is at a different level of scope than this question.

The next most valuable decision is what to work on. This generally best considered from the perspective of delivering value to the customer quickly and of being able to respond to their changing and emerging needs.

When we split horizontally or when we do extra work, we know we will need later we may give the appearance of making more efficient use of developer effort, but in doing so we reduce our ability to adapt to changing needs and more significantly we delay the time to deliver the next most valuable feature.

This is an example of working with a waterfall mindset. The measure is that while it delays us this week, we will make back the time next month or next year. So if we are not planning to deliver this to our customer until next month or next year, it might appear to make sense. But you may not have considered two factors.

If we are deploying frequently, we are realizing value with every release and in the value of our software isn’t massively more than the cost of developer time to create it, then it is likely we shouldn’t be working this project at all. So ultimately the cost of developer effort for having to rework the same code later for a new feature is insignificant to the value gained by delivering sooner. We should measure ourselves based on value delivered not the effort to deliver it.

The second consideration is that the work you are doing on the assumption of needing it for a future feature may not actually be needed. That feature is by definition lower value and less important as it has been prioritized lower. There is a pretty reasonable chance that the customer may change their needs or there is the possibility of the project being cancelled or considered good enough as it is. That work may never actually be needed and you would have delayed the project for work that had no value. The efficiency you are trying to achieve may just be an illusion.

This is the business equivalent of spending $10 today to save $1 next year.

When considering efficiency and improvement, remember your goal is to maximize value to your customer and to realize that value as quickly as possible. Ask yourself if the efficiency improvement you are proposing fulfills those goals? If not it may be just be an illusion.

It is all in the follow through…

Accountability of others

Holding another person accountable can be a scary thing, even just saying that sounds a big deal. And whilst it is a big deal, it doesn’t need to be scary. Feedback is a skill, and it one that you get much better at the more you practice.

In theory feedback in the context of accountability should be the easiest kind. Your work colleague or partner has made a commitment to you or to the team and in doing so they are implicitly inviting you to give them feedback if they stray from this commitment. Learning to give that feedback constructively and supportively comes with practice, and frequency. If you are giving feedback regularly it is easier to give and easier to receive, eventually it will become a normal and natural interaction on a team.

If they took an action item to do something by the end of the week a polite enquiry on their progress, may serve as a reminder that the task is important and that they have committed to it. There is no need for judgement or pressure and certainly no need to micro-manage. An offer to help or an enquiry if there are any impediments you should be aware of, or can assist with, should be sufficient to reassert the importance of the task and a reminder of the commitment made.

In most cases if someone has taken responsibility and committed to something, they will be happy to be held accountable, especially if you have established trust on the team and have decided as a group this is something important, a good team player understands that group decisions impact all of us and are in all our interests to get it done, so understand and expect you to have an interest in getting it done.

Holding yourself accountable

Holding yourself accountable is much harder, after all if we could see we were slipping we’d do something about it. Which is why it is so important to explicitly ask others to help you with this or to find techniques to help you stay focused on what is important.

Accountability techniques

There are a couple of very easy techniques for creating opportunities for holding each other accountable. If done right these should lead to more frequent feedback, if they become a constraint and feedback becomes limited to these opportunities then try something new.

A regular stand-up meeting

Get together with your team on a regular basis, a lot of teams find once a day is a good cadence, review priorities, objectives and actions, by sharing what you are working on and where you need help you are inviting input from others, if an action item is being neglected this is an opportunity to ask why or to remind others of the importance of it. Be careful this does not become reporting to someone, this is about sharing information and looking for opportunities to assist each other.

Make your work visible

Something like a Kanban board or a todo list is fantastic for sharing with your team mates what you are working on, you and they can immediately see if an action is not being given priority and can see why or what is. If Kanban is used then priorities of work are clear and your policies are explicit, everyone is clear just by looking, what is going on. The key to maximising the value of this is by ensuring it is used correctly, all tasks are on the board and you follow the rules, ambiguity can ruin this technique.

If combined with a stand-up it can enhance the effectiveness of both techniques. Also whilst Kanban boards are great for team working, they also work well for an individual, many people find a personal Kanban board is a great way of keeping focused on what is important and avoids getting distracted by less important interruptions. and having it visible in a public space too will aid you holding yourself accountable to it.

Always review previous actions

At the start of a team meeting it is important to review the previous action items, and to not lose sight of any that remain outstanding. First the team should be focused on the most important issues, and actions should be the team’s collective view on how best to proceed. If as a team you don’t care enough to want to know how those actions turned out, you have to question your buy-in to those desicions or if your team is actually focused on the right priorities.

Was the purpose achieved?

Remember the reason for the Action item. I would also suggest that an action is the result of a discussion about resolving a problem, satisfying a need, or is progress towards a goal.
Taking some time not only to review whether the action was taken, but also to review whether it achieved it’s purpose would be valuable. It is quite common for an action to not have the expected results, and in that case the underlying issue still needs addressing.

Summary

Accountability should become routine, and we can all benefit from being held accountable. If it doesn’t become easier or routine then perhaps there are other underlying problems either with trust on the team or a lack of genuine commitment to decisions.

Forecasting, asking the why? behind the When?

A Product Owner, or development team will often be asked for an estimate or a forecast for a product; a feature; an iteration; or a story. But when you go to a team member and ask them it is not uncommon for the colour to drain from their face, twitching to start, and their pulse to race. Past experience says that the next words they speak will haunt them for the foreseeable future, maybe even for the rest of their career. It is only a rough estimate you say to reassure them, but they just know you have other plans and the fate of mankind lies in the balance. Or so it seems.

dilbert estimate

The difference between a forecast and an estimate.

For most practical purposes there are only subtle differences, the main one being that a forecast deals exclusively with the future. When I think of the two I tend to think of an estimate as information and a forecast as expectation. It is likely that I will use estimates to create my forecast and I may use multiple estimates and even apply other factors to derive my subjective forecast. Whereas an estimate is generally an isolated objective assessment.

Both have huge degrees of variation; accuracy; precision; and risk factors, but in some instances it may very well be that my estimate and my forecast are the same for all practical purposes.

For example, If I was asked to estimate a journey, I may say that it is between 15 and 45 minutes and on average it is 30 minutes. If I am asked to forecast when I will arrive, that requires knowing not only the estimate but the time the journey starts and if there are other factors such as a need to stop to get fuel, or food. I may also add a certain weighting to the journey based on the time of day. Thus my forecast, whilst based on an estimate may not be the same as the estimate. Other times though I may simply take the estimate and it will become my forecast.

In practice though the terms are used interchangeably and likely it really makes very little difference, except when it comes to expectations. If I ask “How long will this take” I am asking for an estimate, if I ask “When will this be done” I am asking for a forecast, both are fine so long as everyone knows what is meant by the question.

Forecasting is misunderstood

Forecasting is guesswork, it may be scientific guesswork and it may be based on past experience and metrics and clever projection tools, but it is a guess. You will be wrong far more often than you are right. The more professional and clever and precise the forecast looks the more confidence you may instill in that guess. But it is still a guess, and when your audience gains undue confidence in your guess they have a tendency to believe it as fact not forecast. It might be that a guess is all that is needed but dressing a guess up and delivering it with confidence may create a perception of commitment.

A forecast is commitment (in the eyes of the one asking)

In normal circumstances by giving a forecast there is an implied commitment, even if you give caveat after caveat, and at any point where you even hint at a commitment to delivery to a fixed scope, fixed cost AND fixed date you are setting yourself up for disappointment. And sadly that is what a forecast is seen as by most people.

By all means work to a budget or a schedule or even a scope (although that is very likely to vary) but ideally ONLY one and never all three.

Understanding why a forecast is needed.

There are many reasons why people ask for forecasts, and the why is the most crucial aspect of the process. Understanding the why is the first step to providing the right information, and hopefully changing the conversation. Forecasting for stories and features is both more reliable and more accurate but the project/product level forecasting is where there is the most confusion and least understood purpose.

If possible, try to change the question of “When?” in to an understanding of “Why?”

Generally speaking we gather information to help us make decisions. If the information will have no bearing on our decisions then the gathering of that information is wasteful. Forecasting all too often falls into this category. We take time and effort to produce a forecast and yet the forecast has no bearing on any decision, that effort was wasted, or more often the level of detail in the forecast was unnecessary for the purpose for which it was used.

Making predictions of unknown work based on incomplete information and a variety of assumptions leads to poor decisions, especially when the questions being asked are not directly reflective of the decisions being made.

In my experience the Why? generally falls into three broad categories:

How long will this take?
How much will this cost? and
simple curiosity/routine.

But most people don’t ask why, they spend time creating a forecast and present it without knowing how it will be used. Some people asking for a forecast/estimate may not even know why they are asking, they just always get a forecast. But let us delve a little deeper.

How long will this take?

Why do you need to know? Some typical answers are:

I need to plan
I have dependencies
I need to prioritize
I have a deadline

How much will this cost?

Why do you need to know? Some typical answers are:

I want to know if I will get a good Return on Investment
I need to budget
I need to prioritize
I have limited available funds

Curiosity/routine

We always ask for a forecast, I need to put something in my report
I want reassurance that you know what you are doing
I want to know if my project is on track

What you will notice about these questions is that when you ask why, the first request for a forecast suddenly doesn’t make sense anymore, they are not really interested in the forecast itself, but in some other factor that they can infer from the forecast. If the intent is clear then the question can be tailored to get the required information in a better way.

e.g.

I need a forecast – Why?… I need to know when I can allocate staff to the next product/project.

In this case would a simple high level guess be sufficient? I feel confident that staff won’t be available for the next 3-6 months, in 3 months let’s review, I’ll have a better idea then…

I could put a lot of effort into a detailed forecast but an instinctive response may give all the information needed, saving us a lot of trouble.

I need a forecast – Why?… I want to ship this to maximize the Christmas shopping period – or I want to time the launch for a trade show etc.

This isn’t a request for a forecast, this is a request for an assurance that there will be something suitable available for a particular event/date. I can give you an assurance and confidence level without a detailed forecast, I may even change the priority of some features to ensure that those needs are factored into the product earlier. Or can de-scope some features to meet a certain date.

I need a forecast – Why?… I have limited funds available, and I want to know when I can start getting a return on this investment.

This isn’t a request for a forecast, it is a request to plan the product delivery so that revenue can be realized sooner and for the least investment. It may be possible to organise delivery so that future development is funded from delivering a reduced functionality product early. Or that development is spread over a longer period to meet your budget.

I need a forecast – Why?… I need to budget, The way this company works I must get approval for my project expenses and staffing in advance so I need to present forecasts of costs and timelines. This answer is twofold, first – can you challenge the process? It might be better to have a fixed staffing pool and prioritize products/projects such that the most important ones are done first and then move on to the next, in which case the forecast for this is irrelevant, it is a question of prioritization. Or if the issue is ensuring staffing for the forthcoming year could I simply say whether this product will not be completed in the next budget year?

I need a forecast – Why?… I want confidence that you know what you are doing. This is not a request for a forecast it is an assessment of trust in the team. There are many more reliable ways to ascertain confidence and trust in a team than asking for a forecast.

I need a forecast – Why?… I have a dependency on an aspect of this project. This may not be a request for a forecast of this project, but more a request to prioritize a dependency higher so it is completed sooner to enable other work to start.

I need a forecast – Why?… I want to know if my project is on track. Essentially what you are saying is that I want to track actual progress of work done, against a guess made in advance that was based on incomplete information and unclear expectations. And I will declare this project to be ahead or behind based on this. I am sure those of you reading this will know that what you are measuring here is the accuracy of the original guess, not the health of the project. But we have been doing that for decades so why stop now?

finally the closest to a genuine need for a forecast

I need a forecast – Why?… I need to prioritize or I want to know if I will get a good Return on Investment.

Both of these are very similar questions, but are really requests for estimates not forecasts a rough estimate helps me gauge the cost and when I evaluate that against my determination of the value expected to be gained from the project it may help me decide whether the project is worth doing at all or if there are other projects that are more important. E.g. If it is a short project it may impact my decision on priorities

But even here it not the estimate that has value it is just information that helps me evaluate and prioritize. If I already know that this project has huge value and will be my top priority does forecasting aid with that decision?

When does it make sense to forecast?

Listing those questions above it seems like I am suggesting that a request for a forecast is always the wrong question and is never really needed. And it is true that I did struggle to come up with a good example of when it makes sense to do a detailed project level forecast that included dates or any type of scheduling expectations.

It may be necessary for a sales contract, to have a common set of expectations. Although I would very much hope that sales contract for agile projects are for time and materials and are flexible in scope and dates, if not cost too. So for the purposes of setting expectations and in negotiations I can see that there is value in an estimate, although I do still wonder if a detailed forecast adds anything here that a reasonable cost estimate doesn’t cover. If possible I would rather work to an agreed date or an agreed budget than suggest a forecast that may lead to false expectations.

But the reality is that sometimes your customer does want one and wont tell you why, or doesn’t know why. Some customers (and managers) are willing to accept that a forecast will cost time and money and the more detailed it is the more it costs, and of course being more detailed may not make it any more accurate.

More detailed investigation for your forecasting is likely to build greater confidence but may not be more accurate and you should ask the question “will more detail have a material impact on your decisions?”, and if the extra effort wouldn’t affect your decision then it is just waste.

I would caution that if there is no other alternative and a forecast is made, that it is revised regularly and transparently, the sooner the forecast is seen as variable the more useful it is. There is so much assumption tied to a forecast that it becomes a ball and chain if there is not an expectation of it changing and so it must be refreshed regularly to prevent the early assumptions being seen as certainty, or they will lead to disappointment later.

Short term forecasting

The real value in forecasts is when the forecast is for a short frame of time. Over the short term we can have much more confidence in our forecasts, especially if we have been working on this project for a while and have historic information we can base our forecast on. There are fewer assumptions and less variables.

Can you forecast which features are likely to be included in the next release?
If I add a new feature now can you estimate a lead time for this?
Can you give me an estimate of how much this feature would cost?

By limiting the scope of the forecast to an areas where we have more confidence in our expectations the forecasts become more meaningful and whilst there is still a danger they are seen as commitments the risk is mitigated by the shorter time frame.

Alternatives to forecasting

Many of the questions above could have been resolved with much less effort than a detailed project level forecast. In most cases we could achieve sufficient accuracy for decision making with a high level estimate. E.g. A product like that is similar to ‘x’ therefore I’d estimate 4-8 months for a small team. Or as a rough estimate 12-18 months for two teams and calculate costs accordingly.

These estimates are certainly broad but if you have confidence in your teams and you believe they will use Agile principles to get value early and are able to communicate progress and be transparent with issues then I see no issue with broad estimates. It is sufficient to allocate staff and resources, to prioritize, to schedule and to determine Return on investment decisions.

For the other questions you may achieve far better answers through the use of Product/feature burndown charts, user story mapping, or even simple high level Road maps. These tools provide useful information which can be used for managing expectations, identifying dependencies and visualising the progress of a product. And crucially – aid in setting priorities and keeping the progress transparent.

I think you are missing the point…

A brief story….

A team found that many of their impediments and problems with delivery were caused by a lack of access to IT operations or lack of responsiveness, they requested a member of the IT operations team be co-located with (and become part of) the delivery team. This was approved and trialled as an experiment.

From the perspective of the development team this was a major success, delivery times improved, delays were reduced and the major constraint to delivery was aleiviated. In short software projects were delivered sooner and with less complications, the overall cost of delivery went down dramatically.

However, the management team decided to remove the IT ops team member and retrurn them to their old team on the basis that they were not being fully utilized. The notion of a human resource not being fully utilized was too much for them to bear.

[Please note when I say utilization – that is on IT Ops tasks – they were using ‘slack time’ to support the team in other ways]

When I challenged this decision, I was told that they were unable to increase headcount for IT operations staff unless it could be demonstrated that existing resources were more than 100% utilized on IT Ops tasks over the course of a normal week. So anyone not 100% utilized was considered a problem for them and became a burden on other team members.

This same IT operations group has a huge backlog of work and lead times were extremely long, they were a major bottleneck for almost every aspect of the business operations. Sadly the management team did not see any correlation between the absence of slack time and the long lead times.

Too often the dependency on IT Operations and their perceived hindrance to business operations, leads to unnecessary conflicts and frustrations. Usually felt by the staff on the IT Ops teams. But it is because IT Operations are so important that they get this reputation.

How can you justify extra resource if you are not fully utilizing the ones you have?

I have no doubt this is a common story, “how can you justify extra resource if you are not fully utilizing the ones you have?” In principle this would appear to be a perfectly reasonable and logical statement, and I can understand why many management teams fall into this trap. But it is the wrong question.

What troubled me most was that we had demonstrated that the impact and delays suffered by the delivery team in just this one example were actually costing the organization so much that they could easily pay for 5 or more additional IT operations staff with the benefits gained. And more so that this situation could be repeated all over the business. The response from the IT Ops manager was that I was missing the point, and they couldn’t justify underutilizing staff. Their measure for success was maximizing utilization of staff, and not supporting business goals, like say profitability.

I see this mainly as a failure of the business, for not making clear to management where they contribute and how they impact on the larger business objectives. That combined with a manager that either doesn’t see or doesn’t challenge this omission leads to perpetuation of dysfunctional behaviour.

Managing resources effectively not efficiently

I apologize for comparing people to resources, but in this context it is applicable.

Could you imagine if that same principle was applied to other resources? Your PC would be taken away if you didn’t use it 40 hours a week, you would be forced sell your car because my guess is that you only use it 10% of the time. You would wear the same clothes every day. Cooking would be a nightmare if you could only have food where all ingredients took exactly the same amount of prep time and cooking time. I’m getting silly now but it is the same principle, by totally ignoring the reason why we have a resource and focusing entirely on maximizing it’s utilization we behave in really rather perverse and contrary ways.

In any other context the notion that anything less than 100% utilization prohibits buying a second item (regardless of how beneficial) is plain nuts.

Please do not misunderstand, I am not encouraging waste by this, I am encouraging understanding of how resources are being used. When you are meeting the business goals and your flow is optimized, that is the time to look at whether reducing waste is possible but to do so in a way that ensures that any reduction in waste is not hurting business goals. It may turn out that just like your car, the optimum utilization is less than 100% and availability and responsiveness are valued higher than utilisation.

Summary

In this situation, I am very proud to be “missing the point” and I wish far more managers would take the time to understand how their role impacts the larger business and perhaps they too will miss the point and start asking important questions. IT Operations in particular are the lifeblood of so many businesses and their actions can make or break a business, understanding all the areas of the business that IT operations supports and how the cost of delay is often significantly higher than the individual cost to operations of that task or that ‘resource’. The ability for IT operations to prioritize and react promptly to the needs of a business is a far better measurement than measuring utilization of staff.

I’d highly recommend that anyone in IT Operations reads the book The Phoenix Project by Gene Kim, and ensure that anyone that benefits from the service they provide also read it.

Understanding how you contribute to the goals of the business is so important, and if after that you truly believe that your role should be maximizing the utilization of resources and not supporting the business achieve it’s goals, then I would suggest it may be you that is missing the point.

Why do we use WiP limits?

I participated in a great discussion this week on the use of WiP limits. For those that don’t know, WiP stands for Work in Progress, and is a measure of how many activities you have started but not completed.

It is important to note that this is not a measure of how many tasks you are currently actively working on. If I started a task but have become blocked, so I start another then my Work in Progress is 2 – even though I am only actively working on one. WiP is a measure of how many started but incomplete tasks I have.

For example: call-waiting or being put on hold during a phone call: You are talking away and get another call so you switch to the second call without hanging up, and when you get another call you take that too, how many people can you have on hold at once? You are only working on one call at a time, although you must remember the content and context of all those other calls. For the people still on hold while you have all these conversations I expect they are wishing you had a WiP limit. Your WiP-Work In Progress is the sum of all active(incomplete) phone calls.

Limiting WiP

There are many reasons for limiting Work in Progress not least because I hate being put on hold. I will go in to some later and it is likely that if you follow any type of Agile framework you already limit WiP although you may not be consciously aware of it.

If you have a backlog, then you are limiting WiP. You are consciously choosing not to start new work immediately but are prioritising it and organising it to work on later. It is likely that you are informally limiting your WiP according to what you feel you or your team can manage at a given time. It is important to realise that this is a WiP limit even though it may not be conscious or scientific.

If you follow Scrum: WiP is consciously and explicitly limited at Sprint Planning, often by estimating effort, or counting stories or calculating story points. The limit is generally set based on Velocity – our average achieved over previous sprints. If on average we finish 10 stories a Sprint, or we average 35 points, then that average is generally taken as the starting point for our WiP limit, we may choose to push for a few more, or if we are expecting to be slower due to vacation time, or known issues we may choose to commit to less. But in Scrum we don’t usually refer to it as a WiP limit. But that is exactly what it is.

KanBan is actually very similar, if we on average are completing 10 stories a week, then it is likely that we will consciously or subconsciously limit ourselves to only take on 10 new stories within a week, although in the case of KanBan this is not done in a Big Bang explicit decision but gradually over the measured period and is more a consequence than a plan (A Pull rather than a Push). We pull stories when we are ready and we pull at the pace we are working at, that just happens to be 10.

Slightly off-topic, but one of the crucial differences between Scrum and KanBan is that Scrum is a ‘Push’ model and KanBan is a ‘Pull’ model. In Scrum we Push an amount of work to the board and spend the Sprint working to Remove(complete) it. With KanBan we are aiming for a more steady amount of Work in Progress and will pull new work as current work is completed, which creates a smoother flow, but conversely lacks a unified time based goal or target. But it is important to understand that both frameworks limit WiP and both focus on ‘completed’ work being the primary objective.

Why do we limit WiP?

I’d like to think it was obvious by now and I think the basic principle is, but there are nuances that may complicate things which may be where the cause for debate comes from.

At it’s simplistic level if I can only complete an average of 10 stories a week/sprint then starting an average of 20 a week without changing anything else is nuts, all that will happen is that on average 10 or more of those stories each week will remain incomplete and stay ‘in progress’ by the start of week 4 we will still have completed 30, but will have 50 now in progress. Chances are by now we will actually go slower because we will be distracted and falling over ourselves trying to juggle more than we can possibly achieve.

Think of all those people on hold, growing frustrated at being ignored, and you trying to remember all those conversations, the phone rings again, can you cope with another caller on hold?

We limit WiP because we understand that by working only on what we can achieve, we can focus and be more effective.

Limiting WiP and WiP limits

“Ah but you have talked about Limiting WiP but you haven’t mentioned WiP limits” I hear you say…

And this is where I think the conversation really stems from and where we get into nuances. KanBan began in manufacturing where work was passed from one workstation to another and there was a measurable flow through the system. Limiting the production on one workstation so that it maintained pace with the entire system limits waste. Having one hyper-performing efficient widget maker that can produce widgets 4 times faster than they can possibly be used is wasted effort. And the same applies to a software process, we use WiP limits to regulate one part of the flow to maintain pace with the flow as a whole, the goal is to visualise bottlenecks and by visualising bottlenecks we can take steps to increase the flow of the whole system.

Other forms of WiP limit

But WiP limits can take many forms and are applicable to almost all aspects of life. The most common example used is a highway, when a road gets busy it slows down, when it gets very busy it grinds to a halt. Limiting access actually speeds things up for everyone on the road, and ultimately more cars can get through far quicker.

But there are other less obvious WiP limits. A long-distance runner wants to increase her times, she starts fast but she runs out of energy and is slowing down near the end of the distance. By setting herself a slower time per mile early on (Pace is a form of WiP limit) she has a consistent speed and reserves energy so is able to run the full distance faster by slowing down during the early part of the race.

Are you dedicated to a single product/project? That is a very low WiP limit of 1 project.

Do you use a calendar and try not to book two meetings at the same time, that is a very tight 1 item at a time WiP limit.

What about a budget? Do you manage your finances as an individual or company. A budget is a WiP limit for your money, by limiting money spent on beer you have more available for clothes. By regulating your spending you can save for a holiday. By remaining in credit at the bank you do not have to pay interest and so on.

Example of how to choose a WiP limit in a workflow

In the case of the widget maker, if the system can only use 10 widgets a day then we may set his WiP limit to 10, when he gets to 10 he is free to go and help someone else. Suddenly rather than unused widgets we have an extra pair of hands to do other work. But hold on! these widgets are easy to make and only take a short while to do, we could probably limit WiP to a lower limit and jump on the workstation as an when needed to produce more. So how do we set the WiP limit? My advice is not very scientific at this point. Start by continue doing what you currently do and just watch, if your workstation/activity columns are sub divided in to doing and done, take a look on a regular basis and see where the biggest stacks are. If one column regularly has more cards on the done column than doing, it ‘may’ be a sign that you should reduce your WiP limit for that activity. Try it and see if it improves your flow.

The largest queues of done work are usually the area that needs limiting, but be aware of ‘bursts’ of activity, for example there may be a workstation that is only available for one day a week, in which case the queue for that workstation may need to be longer – or maybe you should be asking for more regular access to the workstation. But when imposing limits do it gradually and monitor to see if it improves the flow.

Anything in a done column is normally waste, if it is done and not in production that effort cost you money and is just sat there. Sometimes that makes sense but the car industry learned that complete cars sat in storage amounted to $millions of wasted investment, that components stockpiled in advance was essentially big piles of cash that could be used more effectively, we can learn the same.

What I am saying is that a WiP limit being reached is a conversation trigger, a long queue is a conversation trigger, and so on. With so many things in Agile we create early warning systems to create conversations and make decisions when it is necessary. We make small changes to the process and watch, we see what changes and if it is better we carry on, but look for another new small improvement.

Summary

WiP limits are not unique to KanBan, and like it or not you are already limiting your work and activities in all aspects of your daily life, you just perhaps didn’t realize it. In the context of a KanBan board we do not arbitrarily impose a WiP limit because we can or because it is fun. We impose a limit when we feel it adds value, normally where we see a queue of completed work growing faster than it is being consumed. We limit various stages of the work because our goal is the output of the system (the whole team) and not the perceived utilization of individuals. One team member being 100% occupied all day producing something that will sit untouched for a week is not efficient.

We are not trying to manage a system where our goal is to keep busy a group of individuals, our goal is to create a cooperative and coordinated team working towards a common purpose in the most effective way possible. A WiP limit is just one of many tools that can be used to enhance the prioritisation of work and focus the team on the true goal.

Is Agile a Quantum Leap for a Project Manager?

I have been asked why you can’t have Project Managers in Scrum, and there are a number of discussions on boards that have entertained me, where there is a question about whether there is such a thing as an Agile Project Manager?

The problem for me is that Project Managers are measured on the wrong things, their objectives conflict with Agile goals they are almost always put in a position where they are pressured to conform to a dysfunctional notion. I see them as a sort of dysfunctional Sam Beckett from Quantum Leap “…striving to put right what once went wrong and hoping each time that..” well you get the picture. There is a notion that the plan in their mind is more important than the reality in front of them.

But in Agile we value “Responding to change over following a plan”

At it’s heart, the job of a Project Manager is to plan a project – usually in fine detail, and then to manage the project to ensure it doesn’t deviate from the plan, and when it does deviate, which of course all but the simplest projects do, the PMs job is to take corrective action to get a project back on plan. It is about dates, and deadlines and the all-important plan. (cue lightning, harsh shadows and ominous Gantt charts).

All we need is a strangely dressed hologram saying “Ziggy predicts that if we skip all testing of the software there is a 93% chance that we will meet our deadline and the software wont devolve in to a pile of un-maintainable junk for 9 months and 3 days.”

The project Manager then gets to ‘Leap’ to a new project without having to stick around to see the mess left behind.

This is in so many ways the very opposite of an Agile philosophy, I see nothing wrong in having a plan, understanding the vision and the goals is crucial. Even some milestones are often beneficial, but the more detail there is in the plan the more waste there is. Everyone including the Project Managers know perfectly well that the plan is flawed, just like every other plan they have ever produced, the difference is that in Agile we accept that the domain is too complex for a fixed and immovable plan. In agile we adjust as new information becomes available, but the traditional Project Manager wrestles with reality and desperately fights to shoehorn a complex reality into a flawed plan. We all know this, the statistics on ‘failed’ project plans are staggering and yet some still persist in this notion of creating fixed plans and punishing teams for deviating from them.

Agile in contrast focuses on getting the customer what they want, and they do this by getting something to them as early as possible and as often as is practical and then responding and re-evaluating, changing the plan to accommodate the customers evolving requirements. Maximizing value early for the customers benefit.

So can there be an Agile Project Manager? I don’t know, but if the role involves valuing following a plan over responding to change then it becomes an oxymoron. And with a Product Owner and a Scrum Master I fail to see what useful purpose they can serve in a Scrum framework.

A good PO these days is hard to find…

1: Differentiating what is needed from what is wanted

2: Understanding what is required to create a successful application

3: Communicating effectively with both technical and business stakeholders

Summary

Measuring Capitalism

TL:DR Summary:

What has this to do with Agile?

Governance

Let’s change the way we write stories

Tell me how you measure me and I’ll tell you how I will behave.

Eli Goldratt

Hypothesis rather than Story

The best architectures, requirements, and designs emerge from self-organizing teams.

Already happening?

Measurements for Goals as the basis for Story Mapping

Make it measurable

Parkinson’s Law

Not just a European thing.

Henry Ford… again.

Back to the present.

So what about overtime?

Bad news for knowledge workers

Worse news for Lawyers

What now?

Summary

Decision Makers

The primary goal is to Maximize Value Early

The 4 Steps

1 – Prioritization (Selecting Which Project/Product to Build Next?)

2 – Direction (Product Ownership – Building the right Product)

Deployment and ongoing updates should not be an after-thought.

Feedback from people actually using your software is the most valuable feedback you can get.

3 – Quality (Build the Product Right)

Quality means doing it right when no one is looking.

Henry Ford

4 – Efficiency and Improvement (Building it Better and Faster)

If you always do what you always did, you’ll always get what you always got

Henry Ford

What about efficiency of developer effort?

Accountability of others

Holding yourself accountable

Accountability techniques

A regular stand-up meeting

Make your work visible

Always review previous actions

Was the purpose achieved?

Summary

Related reading

The difference between a forecast and an estimate.

Forecasting is misunderstood

Understanding why a forecast is needed.

How long will this take?

How much will this cost?

Curiosity/routine

When does it make sense to forecast?

Short term forecasting

Alternatives to forecasting

A brief story….

How can you justify extra resource if you are not fully utilizing the ones you have?

Managing resources effectively not efficiently

Summary

Limiting WiP

Why do we limit WiP?

Limiting WiP and WiP limits

Other forms of WiP limit

Example of how to choose a WiP limit in a workflow

Summary

The best architectures, requirements, and designs
emerge from self-organizing teams.

1 – Prioritization
(Selecting Which Project/Product to Build Next?)

2 – Direction
(Product Ownership – Building the right Product)

3 – Quality
(Build the Product Right)

4 – Efficiency and Improvement
(Building it Better and Faster)