Archive for January, 2012

Story points are useless!

January 31, 2012 3 comments

Catchy title, isn’t it? Now when I got your attention, let me explain how they are useless.

Estimating in points (relative measurement) is much, much better than trying to use absolute measurement such as working hours or ideal engineering days. While the benefits are widely quoted, one risk needs special attention.

The danger lies in human tendency to simplify things. We try to turn complex phenomena into simple numbers. Although many parts in Scrum look simple (point-based estimations, task board, burn-down charts), they are only representations of complex issues.

Story points are hiding complexity and if this hidden information is lost, then decisions will go wrong.

I personally experienced this in a web-service development project. The project was relatively short (less than 6 months from start to finish) and it was running late from early stages, mainly due to unexpected difficulties with development and production environments. When the last 2 Sprints started, it was obvious that team could not get anything in production.

In order to get at least something out of the project, the management decided to grant more funding to get minimum product out. Before deciding about extra funding, the management wanted to know the estimated duration (=cost) of remaining work.

Product Owner and Scrum Master looked at the history data and found out that team velocity had been about 10 points per 2-week Sprint and the remaining work was estimated to 10 points. They cleverly padded the estimate 1.5x to 15 and asked funding for two Sprints (four weeks).

Then the unpleasant truth came out. The work remaining was more than 10 points, the team re-estimated it to be “about 20 points” . At the same time, the team moved to an unknown domain (legacy code) and they re-estimated their velocity to be “maximum 8 points per Sprint”. Suddenly the remaining work was not two Sprints — it was closer to three.

The problem exploded: Product Owner (responsible for investment) felt betrayed, since she had been fighting to get money for 2 extra Sprints and now team came back to her saying “Sorry, we ain’t gonna make it in two“. Scrum Master was upset, because he felt the team was lying to him. And the team was confused, since they did not betray or lie anyone. It took quite a while to handle the mess, but eventually team delivered — after very painful re-scoping and budget negotiations.

What went wrong? Lack of communication and using partial information to make significant decision. Team estimated the remaining work very early. Their estimate (“10 points”) was never updated, since current tasks occupied them and last “remaining work” seemed distant in time. Team probably knew that work was more than 10, but that information was never passed to decision-makers.

At the same time, management was assuming stable velocity (from burn-down chart it was between 9-13 points per Sprint). This was an illusion, since team was struggling with their project and somehow everyone wanted to show that velocity is “eleven plus/minus two“; maybe because stable-looking velocity caused least harm, interrupts and discussions.

Last but not least, the team was going through team forming process. Under pressure they had conflicts and in such environment it is not easy to shout out loud any risks or express conflicting opinion. Risk of delay, risk of increased work was something that team did not feel safe to discuss, or at least tell outside the team.

Lessons learned:
– Product Owner and development team must discuss openly about work, estimates and risks. Do not fall in the trap that “I protect developers from unpleasant budget discussions” or “Let’s not bother the PO about risks related to some future backlog items
Do not anchor discussion to history data. Imagine what happens in discussion if PO starts “Great work, seems you’ve done 10 or more points every Sprint and remaining work is about 10 points
Pay attention to coming work more than history. Talk about risks, confidence levels, options and alternatives
Make discussion and decisions explicit. If you want to know how long something takes, don’t ask story point estimation.
Do not hide behind story points. People outside the team will not understand it! Even if you know the risks related to estimation, other people still have tendency to simplify things and imagine that complex issues can be represented accurately with few numbers


Post Script: I wrote most of this story already some time ago. Since then, Vasco Duarte wrote an excellent blog post about Story Points and estimation and that sparked me to finish this text. Vasco talks about estimation techniques (SP vs. # of items) and I agree with him. However, no matter what approach you use, the conclusion is clear: Do not use the numbers in estimates without fully understanding risks behind the numbers.

Categories: Agile and Lean