Estimating at a project level.

One of the most difficult aspects of the transition to Agile is the confusion over how estimation is done.

Estimation is difficult, the experts suggest that even with a full picture of what is required and with clear detailed and fixed requirements, the best estimators cannot realistically estimate better than within a 25% margin of error. It’s easily possible to do worse than this. But It isn’t possible to be consistently more accurate; it’s only possible to occasionally get lucky.

But in agile we start without clear requirements, we don’t have a fixed scope and chances are the requirements we do have are at a high level and there are many unknowns. I could talk about the cone of uncertainty but I’m not convinced most businesses will accept that level of uncertainty even if it is based on sound reasoning. In my experience they would rather base decisions on a specific guess than an accurate ranged estimate especially a wide range. Sounds daft when I say it like that but I bet you have experienced it.

Nevertheless it is still often necessary for commercial reasons to have a solid estimate before starting a project (Agile or otherwise), in many situations projects need to provide a good ROI or are limited by a budget.  In some situations the ability to estimate reliably could be the difference between the success and failure of a business. These estimates can be crucial.

So how do projects provide reliable and useful estimates?

First of all it is worth noting that estimates are largely misunderstood in general, they are misused and can often be damaging to the project. But still estimates are asked for and used to make important decisions.

In a survey from a few years ago*, a range of IT companies were asked about estimation strategies, the results were both worrying and yet reassuring that difficulties were universal.

* http://www.ambysoft.com/surveys/stateOfITUnion200907.html

Around 44% of the project teams in the survey described themselves as ‘Agile’ so this is a balanced pool of projects and should give an idea of estimation across the board.

When asked to give estimates to the business for project delivery around 65% of teams were asked by the business to provide estimates within the 25% margin of error range that experts in the field say is ‘impossible’. 11% were allowed no margin of error at all they had to specify a single date or a specific cost for the project,  conversely 21% were not asked to give any estimates at all. The rest allowed a margin of up to 50% on their estimates.

So how did that pan out for those companies?

Well 40% never even tracked whether those initial estimates were correct, it is difficult to draw any conclusions from this, but 40% came within that magic 25% of their estimates, which frankly is an incredible statistic, when I first read this I started questioning the validity of the survey. 40% of software project estimates were more accurate than the ‘experts’ say is possible to achieve consistently, 40% is more than just getting lucky it is frankly unbelievable.   At this point I was about to dismiss the survey as nonsense, but I read on…

How is it possible?

In order to achieve the 25% margin of error the projects did the following:

  • 18% admitted they had padded their original estimate
  • 63% de-scoped towards the end of the project to deliver on the estimated schedule.
  • 34% asked for extra funds to complete the projects on the original estimated schedule
  • 72% extended the schedule to deliver the promised scope (effectively revising the estimate and success was then measured on the revised estimate not the original)

It is impossible to tell from this how many of the projects matched the original estimates, but clearly it wasn’t very many, it is not a stretch to conclude that the vast majority of respondents de-scoped and/or extended the original estimates, including those that had already padded the original estimates.

Moving goalposts is the key

My reading of this survey is that very few if any delivered what was estimated in the originally estimated time-frame/budget. It makes very bleak reading and regardless of whether the project was or wasn’t Agile the estimates did not deliver what the business asked them to.

If we take the stated purpose as being simply to plan and budget and assume the estimates were not padded or interpreted then they hold very little value based on  the lack of accuracy.

In my opinion if any of the businesses that demanded such specific estimates went on to actually base business decisions on the accuracy of those estimates, then they were just setting themselves up for disappointment and future problems.

There is no way from this survey to conclude what the accuracy of the original estimates actually was other than to say that even with padding, de-scoping and extending schedules they were still unable to meet the original expectations and were overwhelmingly wrong and seemingly nearly always underestimated the true time/cost. This reads like a recipe for disappointed customers and shrinking profit margins.

That is a very long winded way of saying that (according to this survey at least) no one in the industry, Agile or otherwise is producing reliable estimates for software projects, we consistently get it wrong, and more worryingly fudge the figures so we never learn from our mistakes.  So any suggestion that estimating Agile projects is more difficult is not based in fact, estimating for software projects is difficult full stop.

Do estimates have value?

Now that is a different question, if I was running a business and I received a project estimate of 6 months, I would be foolish to consistently believe it will be delivered to the defined scope in that time-frame. But that doesn’t make the estimate useless.  If one project estimates 6 months and another estimates 3 months. I can conclude that the first is likely to take longer than the second, especially if the same person or group has estimated both.  Both estimates are likely wrong but chances are that on average and over time they will be wrong by a consistent margin, which makes them predictable.

If I check historic records I might be able to see that projects estimated at 6m generally take 8-12 months, or better yet I could ask the estimators to compare the current proposed project and identify a previously completed project that is closest in size and scope and use the actual figures from a sensible comparator.  Empirical evidence is so valuable I’m surprised more emphasis is not put into keeping track of past estimates and actual delivery costs and schedules.

Estimates are not commitments

Essentially we need to accept estimates as simply estimates not as a plan or a commitment.  Any PM that transposes an estimate of a software project straight into a plan is nuts, and it happens so often that in my experience developers turn white and have panic attacks when asked for an estimate, painful experience says they will be misused and ultimately the one that gave the estimate gets blamed.  If the business could be trusted to accept that estimates are not an exact science and factor in realistic contingency based on empirical evidence then developers would be less afraid to give estimates.

So how should we do it?

I have two suggestions, the first is to use an extension of the Planning Poker process.  Take a group of people that are experienced with software delivery and relatively knowledgeable about the scope and complexity of what is being asked. E.g. Product Owners, Business analysts, Project managers, representatives from development and testing.  Ask them to give estimates of a variety of projects relative to each other.  I’d use Fibonacci numbers or T-shirt estimates, to keep it at an abstract level.  If possible I’d try to include a benchmark project (or more than one) where the actual time/cost is known.

Blue-11Blue-6If we accept that the best we are going to get is a granular; relative; ball-park estimate of a project then this should give you that and more. In fact for budgeting purposes a reliable granular estimate is of far more value than an unreliable specific figure, and far more valuable than the estimates in the survey. Over time it is likely that the estimation team will learn and improve, they will get better with every completed project. I’d have far more confidence saying a project is either a Medium or Large T-shirt.  The T-shirt sizes could map to high level budgets.

My second suggestion which could be used in conjunction or independently of the first is to set a budget and ask the team to provide the best product they can within that time/cost. A good Scrum team will be able to prioritise stories and features to ensure you get the best value for money. If that budget is based on the poker estimates above it is more likely that the budget chosen is realistic and you will get the product you want.  You will also very quickly be able to tell if the project will be unable to meet the goal and can cut your losses early, rather than having to pour more money into a money-pit project that is over-running but too far down the line to cancel.

Estimation is a difficult skill to master but a group is far better than an individual.

Vision without action is a daydream. Action without vision is a nightmare

Waterfall vs Agile?

Sounds a bit daft to even be having the debate, surely by now there is no one left that still thinks waterfall is the better choice for delivery of complex software projects, unfortunately that is very far from the truth, there seem to be many that still prefer waterfall and recently I even heard it talked of in terms of the good old days.  So I delved a little deeper.

In this instance the source was a statement by someone who’s opinion was given weight. The comment was:

I could deliver ‘project x’ using waterfall if you gave me dedicated resources and we could lock ourselves away for 6-months.  

In waterfall the costs of planning are high, the costs of analysis is high, and the cost of change is very high. But by it’s nature waterfall has a clear scope(vision).   The result is that everyone is clear on the priority, the project is protected, no change and no disruption, the scope doesn’t change. Because everyone understands the cost of interruption.

In Agile the upfront cost of planning is low, the upfront cost of analysis is low and the cost of change is relatively low. The result in some cases is that ‘unnecessary’ direction change can be frequent, the project is unprotected because it can adjust, disruption can be frequent, and so the scope regularly changes for sometimes trivial reasons or low priority distractions.

In short we are not comparing apples with apples.  We are comparing focused vision-led waterfall, with an unfocused ad-hoc agile.

In the example above the team had a fairly vague goal to deliver ‘x’ but also support ‘project y’ and deliver small increments of ‘project z’, ‘project w’ and maybe even an update to ‘project v’ too. And that plan was likely to change next week when something else came along. Because of this ‘project x’ was taking far longer than anticipated. so this notion that delivery using waterfall would be quicker has arisen.

But, the problem for me is that it was not the ‘waterfall’ part of the plan that would ensure success, it was the dedicated resources and clear vision that were the crucial aspects.  Give me a clear vision and those same dedicated resources and I can be pretty sure that I would deliver what the customer wants, sooner, cheaper and more reliably using Scrum than someone else could using waterfall.

Agile is the victim of it’s own agility, it is like the free gift no one values.  Because there is less pain in changing direction in agile projects, and because the team can adjust and adapt, that doesn’t make it free and it doesn’t mean it isn’t disruptive.

Agile enables you to have flexibility in planning, it allows deferring decisions, it allows planning only small amounts, and allows changes in direction and content.  But just because you can do these things doesn’t mean you always should.

A single vision with dedicated resources is a powerful combination, Agile is a powerful tool, but as they say with great power comes great responsibility.

Agile enables you to vary the scope of your vision, to respond to change, it is not a licence to operate without a vision.

As the Japanese proverb goes:

Vision without action is a daydream.  Action without vision is a nightmare.

Who is interviewing who?

How I’d like to do it.

My advice would be to concentrate first and foremost on creating a place where people would want to work, then marketing your organisation to the candidate, they should really want to work for your company, that way you start with a better candidate pool, immediately giving you a better chance of finding the best employees.

Then – even if it feels impersonal, a series of consistent structured tests (IQ and Psychometric) and then formal structured interview questions where the results and responses are noted and then referred onto an independent panel for a decision on hiring. Forget the gut-feel, forget the technical questions, forget hostile interview techniques, they really don’t work. Throughout all this ensure the candidates are treated well and made to feel important.

Sounds like hard work, it sounds like it undermines the hiring manager, it sounds incredibly time-consuming and bureaucratic, perhaps even expensive. But if your goal is to consistently hire good quality candidates it is going to be hard work and you will have to accept that there are some things that cannot be effectively and consistently assessed in an interview situation.

My final thought is that if you have got your hiring right, it is very likely that the most capable, most experienced and most reliable person for a role is the one currently in it. Don’t lose them.  Look after good employees they are your company’s most valuable asset.  Treat them well and do whatever it takes to keep them.

recruitment tips

And next time you are in an interview situation, ask yourself who is interviewing who?

Typical interview styles

The interview itself.

What I have observed is that interviewing is generally composed of one or more of a very small number of techniques:

  1.  Free-form, gut-feel unstructured or semi-structured chat and questions between candidate and hiring manager.

  2.  More formal pre-defined questions structured and consistently asked to all candidates.

  3.  Some variety of standardised test, essentially IQ based.

  4.  Questions posed by a technical expert with a desire to highlight his superiority rather than assess your capability.

  5.  A technical test or series of technical questions intended to be pragmatic and fair.

  6.  Aggressive panel questioning

  7.  Psychometric testing: verbal/numeric/diagrammatic reasoning or personality tests.

  8.  Presentation by interviewer on why the candidate should work for you.

There are studies and statistics that rank the effectiveness of the different techniques for selecting good candidates, I won’t pour them out here – I am not an expert and I’d just embarrass myself. But in my experience 1 is the most common, often combined with 4 or 5. But logic states that 4 is a waste of time and puts off candidates, and studies show that 1 and 5 are actually very, very poor indicators of ability and capability for the role and do not result in long-term success.

So what works? 

2, 3 and 7 are by far a better method of assessing candidates, followed by an independent panel-decision based on the documented evidence taken from the interviews – the interviewer should not be allowed to make the decision independently as they display bias (subconscious or otherwise).

But 2, 3 and 7 are hard, it requires yet more work on what is already a tough and time-consuming process, so very rarely gets done.

The reciprocal nature of an interview (8) is so often overlooked, if you want the best people and you want to get the best out of them, then not only do you need to decide if they are right for you, you need to convince them that your business/team is right for them. You should be putting as much effort in to impressing candidates as they put into impressing you. You are competing for them in a buoyant market.

recruitment2

Doing it right.

As an interviewer, my best experience has been for a company that did a combination of 2, 3, 7 and 5. We put a lot of effort into interviewing, but the majority of people still failed the combination of tests, especially the technical test. The test felt ‘easy’ to us and some candidates did well, but the results didn’t match the apparent skills of candidates, in hindsight I think it confused the process. The problem is that technical tests are not effective at assessing ability during an interview, there are simply so many other factors at play.

Relying on structured questions, IQ tests and psychometric tests may feel clinical and impersonal, but very likely the best way to find the right candidate. But ego plays a role and when hiring for a team, we are only human and so many hiring managers want to rely on their own instinct, even if evidence demonstrates this is unreliable, so it is hardly a surprise that many hiring managers favour their instinct over a structured process.

What makes a good interview?

My experience of recruitment is largely anecdotal, I am not a professional interviewer or HR expert. My limited experience of interviewing has been as a candidate, a hiring manager, or sometimes even as a subject expert. I’d estimate that over my career my experience is approximately 8:1 in favour of being an interviewer over being an interviewee, but I have been on both sides of the table often enough to know how it feels from either perspective. 

Over the last 5 or so years the amount of effort I have put into recruitment has been considerable, especially considering it is not really meant to be a significant part of my role. I’d estimate that I have hired an average of one person ever 2 months or so.   I even like to think I am pretty good at it,  although as you will see I am probably mistaken in that opinion.

What is striking though is how much time this takes, very anecdotally I’d estimate that depending on the quality of the CVs that I receive, on average only 5% will make it through the screening and interviewing to getting an offer(probably less). But the 95% that get rejected still consume a lot of time, and of the 5% selected we don’t always get it right, and I’m pretty sure quite a few good candidates get missed. The process is far from ideal.

recruitment

But what makes for a good interview?

As a candidate I have had interviews lasting less than 15 minutes (where I was offered) and interviews spanning 3 days, for a graduate recruitment programme, and just about everything in between. I have had good experiences and bad.

Some, in fact most interviewers seem to forget that the candidate is also interviewing them, and that the candidate is deciding if they want to work with you in the same way you are assessing them – it is not a one way decision by any stretch. Such is the lack of understanding and sheer arrogance of some that I have been asked extremely obscure technical questions that provide no clear value, I have been challenged(insulted) to gauge my response, I have had panel of interviewers quick-firing questions and had interviewers who were clearly reading my CV for the first time during the interview.  Why would any good candidate have any desire to work for you after an interview like that?

I haven’t kept accurate records, and I appreciate this is anecdotal but I’d estimate I have turned down close to three job offers for every one I have accepted over my career. Now I am well aware that I am fortunate to have in-demand skills at a time when the market is booming, I have also had a lot of rejection.  Similarly, as a hiring manager I’ve had too many candidates turn down offers, or more frustrating get a better offer before we could complete the hiring process.

The hiring process is so expensive and so important I am surprised at how often it is treated in a cavalier manner.   Why oh why would you be so arrogant to treat candidates with anything less than the respect you would expect them to show you?

Live to work, not work to live.

It is often said that happiness is about finding balance in your life, sometimes this means making time for the different aspects of your life, work, hobbies, family and so on, but what if balance is achieved by taking pleasure in all these endeavours? Not simply making time for each but making sure each is fulfilling in its own way.

Too many people seem to suggest that they dislike work, or that it is a chore to be endured for the pay cheque. I find this an odd perspective. Why would you spend 40 hours a week, every week, doing something you endure? Chances are you spend more time at work than anything else. 

I am very lucky, I have a job I love, although I also love my bed and find getting out of bed on a morning difficult sometimes. But once I am up I am itching to get to work, I have ideas and plans and normally can’t wait to get started, at the end of the day there is usually more I want to achieve and it is only self-discipline that makes me come home at the end of the day, I set myself a time box on my working day to avoid taking on too much or allowing work to take over, I try hard to only rarely exceed it, this is a personal choice and I’m aware is not common but works for me and is something I encourage in teams I coach. 

I have covered this before but being clear on what you can achieve in a working week is vital. To quote Dickens.

“Annual income twenty pounds, annual expenditure nineteen pounds nineteen shillings and six pence, result happiness. Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.” Charles Dickens.

My philosophy is similar – if your workload can be achieved in the time available the result is happiness, if your workload exceeds the time available the result is stress and unhappiness.

If you work extra hours to get the extra work done, the result is that next week the work will grow more and then more, until at some point you will collapse or be forced to set a limit, so set the limit now where you are able to do so objectively.
When it comes to job satisfaction it is fair to say that my job is not saving the world, I am not doing something overtly exciting or grandly creative, the teams I work with are about streamlining business processes to increase automation, reduce errors and reduce staff to save the government money, as a 7 year old I was hardly saying “when I grow up I want to save the government money”. But nevertheless I love my job, I work with some really nice people and we are working hard to do a good job. I take pleasure from achieving goals, and especially seeing teams and individuals develop, I take particular pride when my advice or coaching bears fruit.

Life is shaped by experience and I have found that no matter what the work there can be pleasure in it. In the past, I have been a debt collector, I owned and ran a small retail bookshop, But I have spent most of my career in software development, some interesting projects some much less so, but it has been only rare occasions that I haven’t enjoyed work. My first post-university job was with Bell Labs working on GSM, the established guys got to work on the roll-out of GPRS, but the new guys got the ‘dull job’ we had to fix bugs on an old C++ Unix system, but it wasn’t dull at all, we had a largely absent boss so we mostly organised ourselves, every bug was a puzzle, a challenge and a learning experience (I hope that doesn’t sound too pretentious because it really was enjoyable) we had a small team that were great fun, we would lunch together most days, we had a pretty poor softball team that played in a league and lost most of the time, and we often socialised together, because work isn’t just about the work it is about the people. 

My worst experience was also at Bell Labs, the team had an ambitious boss, he wanted a juicy high profile project and so he turned down anything that didn’t fit this requirement and so the team went through an idle period where we quite literally had nothing to do, it was sole destroying, we would come to work for our required hours but there was nothing to do, we read, and looking for training, we talked and we played silly office games, we formed a league for Ball-in-the-box a game where we threw a mini American football the length of an office to bounce into a photocopy paper box, after many hours playing we became quite adept. But what we were not allowed to do was work on something, we had to be ready for the next plum project.  Thankfully a senior manager saw what was going on, and transferred the team to UMTS (3G) this was a lifesaver the morale was getting pretty low.  I find it odd that my worst experience was where I was paid to do nothing but chat have fun and play games, for me at least I get pleasure from being busy and doing something I see as productive and valuable. 

Perhaps the most influential aspect in anyone’s life is their parents, I was a “vicar’s kid” or “son of a preacher man” etc etc, my father was a minister of religion and I was the 3rd of 4 children, the only boy, as a result I have developed a pretty strong rebellious streak, but what surprises me looking back is how much my father’s work has influenced me, as the leader of a church he was an archetypal servant leader, a church is a voluntary organisation, the minister is employed only with the support of the church and he has very little power, his job is to lead and to serve. It is extremely challenging and often unrewarding. 

So now like him I am the servant leader, only in business, I see myself as having two roles one to serve the team and one to lead them to achieve, and what I have learned is that it is important to create an environment where your team can enjoy work, this is a paradox for many as the assumption seems to be that if you are having fun you are not being productive, my experience is the opposite – not being productive is no fun at all.  

Make work an enjoyable place to be and people will work harder, make them feel invested in the organisation and they will work harder still, create a team spirit and collective sense of purpose and they will amaze you.

Trust is often lacking and this is a real sticking point for many managers and businesses, but my opinion and experience is that the normal situation is that people have an inherent desire to work hard and achieve and give the opportunity they will, what often prevents this are people or processes that restrict or control, managing should be about setting goals and guidelines, creating a framework to achieve not a culture of control.

My number one piece of advice for a ‘task’ manager is to give a team a clear goal and stand back, you will be amazed. 

Your only three jobs as a leader are to remind the team of the goal, to remove any obstacle in their way and most importantly  to Trust them. 
As an aside for business leaders I’d also suggest if you want to build and retain an effective happy and productive workforce, to indulge them (sensibly) make them feel valued, give them the best tools – generally whatever they ask for, the cost is likely a fraction of their salary and tiny compared to recruiting and training a replacement if they are lured away, give perks – for example issue everyone an iPad and a pluralsite subscription, this costs very little(and is tax deductible) in comparison to an average salary (far less than 1%) but I’d bet your team would feel valued and would broadcast what a great employer you are, and chances are they might even do some training too. Provide coffee and snacks, encourage social activities, make time for the team, if people feel it is a great place to work recruitment becomes easier and they WILL work harder. And lastly pay more, especially to those that have proven themselves, being generous early saves a fortune later, if you are not paying your best people well above average it is inviting them to leave.