Thursday, March 17, 2011

Humans can't estimate tasks

As I said in my last blog entry I’ve been looking at some of the academic research on task time estimation. Long long ago, well 1979, two researchers, Kahneman and Tversky described “The Planning Fallacy.”

The Planning Fallacy is now well established in academic literature and there is even a Planning Fallacy wikipedia page. All the other literature I looked at takes this fallacy as a starting point. What the fallacy says is two-fold:
  • Humans systematically underestimate how long it will take to do a task
  • Humans are over confident in their own estimates
Breaking out of the fallacy is hard. Simply saying “estimate better” or “remember how long previous tasks took” won’t work. You might just succeed in increasing the estimate to something longer than it takes to do but you are still no more accurate.

Curiously the literature does show that although human’s can’t estimate how long a task will take, the estimate they produce do correlate with actual time spent on a task. In other words: any time estimate is likely to be too small but relative to other estimates the estimate is good.

Second, it seems the planning fallacy holds retrospectively. If you are asked to record how long you spend on a task you are quite likely to underestimate it. There seems no reason to believe retrospective estimation is significantly more accurate than future estimation.

Something else that comes out of the research is: psychologists and others who study this stuff still don’t completely understand what the brain is up to, some of the studies contradict each other and there are plenty of subtle differences which influence estimates.

Third, although we don’t like to admit it deadlines play a big role in both estimating and doing. If a deadline is mentioned before an estimate is given people tend to estimate within the deadline. People are also quite good at meeting deadlines (assuming no outside blocks that is), partly this is because estimating then doing is a lot about time management. i.e. managing your time to fit within the estimate.

While deadlines seem like a good way of bring work to a conclusion it doesn’t seem a particularly good idea to base deadlines on estimates. Consider two scenarios:

  • Simple estimate becomes a deadline: If we ask people to estimate how long a piece of work will take they will probably underestimate. So if this estimate is then used as a deadline the deadline may well be missed.
  • Pessimistic estimate becomes deadline: If people are encouraged, coerced, or scared into giving pessimistic estimates the estimate will be too long. If this estimate is then used as a deadline work is likely to be completed inside the time but there will be "slack" time. The actual time spent on the task may be the same either way but the total elapsed (end-to-end) time will be longer.
I’ve written my findings down in full, albeit somewhat roughly, together with a full list of the articles I examined in depth. “Estimation and Retrospective Estimation” can be downloaded from my website.

I would love to spend more time digging into this subject but I can’t. Anyone else want to?

6 comments:

  1. Very interesting indeed.

    What about Planning Poker? I've been told that it should solve this problem or at least result in better estimates.
    http://www.mountaingoatsoftware.com/topics/planning-poker

    ReplyDelete
  2. Thanks Anonymous,
    What I haven't included in this blog entry - although there is a little of it in the longer essay - is: how do we compensate for this?

    That is another blog entry I have yet to write.
    Planning poker is part of the answer, I think, also the XP style planning game.
    See my "Two ways to fill an iteration" from last year
    http://allankelly.blogspot.com/2010/06/two-ways-to-fill-iterations.html

    ReplyDelete
  3. Simple statistics.

    Step 1) Estimate
    Step 2) Estimate how accurate your estimate is

    I use the -50%/+50% rule. I.e. if I estimate two weeks, I'll communicate "I will finish between 1-4 weeks". That is, I consider my estimate to be nearly 100% precise within one standard deviation. It is extremely rarely that I'll not be within such a frame of my estimate.

    ReplyDelete
  4. Have you looked at Joel Spolsky's Evidence Based Scheduling, which, for a large part, is about estimates? It can be found at http://www.joelonsoftware.com/items/2007/10/26.html

    ReplyDelete
  5. Hi Alan,

    as time goes on, im beginning to believe that Fred Taylor was on to something with his 21lb shovel. The estimates of a given project can be very telling after the fact. Looking at the average release plan estimate, and then the actual. The planning fallacy does seem to hold true, we are overly optimistic; however, what i've found is the stories all end up being roughly 21lbs.

    Stuart - @hsiboy

    ReplyDelete
  6. I've stolen Benjamin Mitchell's (blog.benjaminm.net) quib about delivering 20Kg of software. I guess it depends if you are metric or imperial.

    ReplyDelete

Note: only a member of this blog may post a comment.