Home > Agile and Lean > Story points are useless!

Story points are useless!

Catchy title, isn’t it? Now when I got your attention, let me explain how they are useless.

Estimating in points (relative measurement) is much, much better than trying to use absolute measurement such as working hours or ideal engineering days. While the benefits are widely quoted, one risk needs special attention.

The danger lies in human tendency to simplify things. We try to turn complex phenomena into simple numbers. Although many parts in Scrum look simple (point-based estimations, task board, burn-down charts), they are only representations of complex issues.

Story points are hiding complexity and if this hidden information is lost, then decisions will go wrong.

I personally experienced this in a web-service development project. The project was relatively short (less than 6 months from start to finish) and it was running late from early stages, mainly due to unexpected difficulties with development and production environments. When the last 2 Sprints started, it was obvious that team could not get anything in production.

In order to get at least something out of the project, the management decided to grant more funding to get minimum product out. Before deciding about extra funding, the management wanted to know the estimated duration (=cost) of remaining work.

Product Owner and Scrum Master looked at the history data and found out that team velocity had been about 10 points per 2-week Sprint and the remaining work was estimated to 10 points. They cleverly padded the estimate 1.5x to 15 and asked funding for two Sprints (four weeks).

Then the unpleasant truth came out. The work remaining was more than 10 points, the team re-estimated it to be “about 20 points” . At the same time, the team moved to an unknown domain (legacy code) and they re-estimated their velocity to be “maximum 8 points per Sprint”. Suddenly the remaining work was not two Sprints — it was closer to three.

The problem exploded: Product Owner (responsible for investment) felt betrayed, since she had been fighting to get money for 2 extra Sprints and now team came back to her saying “Sorry, we ain’t gonna make it in two“. Scrum Master was upset, because he felt the team was lying to him. And the team was confused, since they did not betray or lie anyone. It took quite a while to handle the mess, but eventually team delivered — after very painful re-scoping and budget negotiations.

What went wrong? Lack of communication and using partial information to make significant decision. Team estimated the remaining work very early. Their estimate (“10 points”) was never updated, since current tasks occupied them and last “remaining work” seemed distant in time. Team probably knew that work was more than 10, but that information was never passed to decision-makers.

At the same time, management was assuming stable velocity (from burn-down chart it was between 9-13 points per Sprint). This was an illusion, since team was struggling with their project and somehow everyone wanted to show that velocity is “eleven plus/minus two“; maybe because stable-looking velocity caused least harm, interrupts and discussions.

Last but not least, the team was going through team forming process. Under pressure they had conflicts and in such environment it is not easy to shout out loud any risks or express conflicting opinion. Risk of delay, risk of increased work was something that team did not feel safe to discuss, or at least tell outside the team.

Lessons learned:
– Product Owner and development team must discuss openly about work, estimates and risks. Do not fall in the trap that “I protect developers from unpleasant budget discussions” or “Let’s not bother the PO about risks related to some future backlog items
Do not anchor discussion to history data. Imagine what happens in discussion if PO starts “Great work, seems you’ve done 10 or more points every Sprint and remaining work is about 10 points
Pay attention to coming work more than history. Talk about risks, confidence levels, options and alternatives
Make discussion and decisions explicit. If you want to know how long something takes, don’t ask story point estimation.
Do not hide behind story points. People outside the team will not understand it! Even if you know the risks related to estimation, other people still have tendency to simplify things and imagine that complex issues can be represented accurately with few numbers

—-

Post Script: I wrote most of this story already some time ago. Since then, Vasco Duarte wrote an excellent blog post about Story Points and estimation and that sparked me to finish this text. Vasco talks about estimation techniques (SP vs. # of items) and I agree with him. However, no matter what approach you use, the conclusion is clear: Do not use the numbers in estimates without fully understanding risks behind the numbers.

Categories: Agile and Lean
  1. January 31, 2012 at 14:41

    Very good post! However your team made the mistake of using statistical data for a single event. Statistics don’t apply to single events.
    When planning 1-3 sprints they must look at the details (task level). Average velocity data only applies to *long** term planning (3+ sprints IME).

  2. February 2, 2012 at 20:04

    Vasco has an excellent point. You can’t apply a statistical average to a planned event and make the leap of faith that the event will conform to the average. Remember, you need to be within 2 or 3 standard deviations of the average to be 95% sure of being right. Even at that, there is still a 5% chance (1 time in 20) that you’ll be wrong. These rules apply regardless of your measurement technique.

    • February 2, 2012 at 23:23

      Hi Vasco and and Vin, and thanks for your comments! Using statistical information to plan single events is actually in the core of this problem, thanks for pointing it out.

      My observations about bad communication & using partial information really boil down to the fact that trend / statistical data should not be used to determine outcome on single event.

      Here, of course, the stochastic behavior was hidden since velocity _looked_ stable, almost constant. And “single event” was hidden to expression “how many more Sprints are needed”. Despite this camouflage, it was about stochastic behavior vs. single event.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: