PERT analysis

The 1950s were a time of great innovation and advance, some developments like the Hula Hoop and the Hovercraft may well resulted individual effort in inspiration but some – like the space race was motivated by the need to win the Cold War. When the Soviets launched Sputnik in 1958, the Americans became convinced that their project management system was deeply flawed and launched a number of initiatives aimed at tacking this deficiency. One of these – instigated by the US Navy – was the Project Evaluation and Review Technique or PERT.

After looking at the outcomes of hundreds of projects, the use of PERT analysis concluded that – as Helmuth von Moltke the Elder said 100 years previously, “No battle plan survives the first contact with the enemy”  Another way of saying that single outcome predictions are pointless, Molktke was himself drawing on work by Napoleon 50 years earlier still when he wrote that “Plans are usually useless but time spent planning is invaluable” a reference to the benefits of scenario planning for the variety of different possible outcomes which may befall and decision, action or endeavour.

Specifically, PERT analysis concluded that – while projects almost never hit their original targets (sometimes coming in over budget, rarely under) a more effective way of estimating the time and money needed was to aggregate together various optimistic, pessimistic and realistic estimates giving each one a weighting according to its likelihood (or probability). While theoretically every organisation would have different probabilities, the ones created by the US Navy fit most purposes and are as follows.

Since people’s estimates were usually optimistic because people naturally overestimate how much of any system they can control and put the interventions of fate down as freak occurrences which – now the lessons have been learned will not happen again… the weighted estimate was to be four times the most pessimistic plus the realistic, plus two times the optimistic, divided by seven or:

4P  +   R  +  2O

7

Crucially the point of this analysis was NOT to tame uncertainty in the sense that the result would magically predict the future, but rather the figure created – along with the other figures for other projects would correctly predict the average time and budget needed. The figure produced is essentially just an expectation calculation. Instead of two possible outcomes (win or lose) in this case there are three and they are weighted according to their probabilities in exactly the same way as the win or lose figures were.

In this sense though – since the actual result was going to be roughly one of the three figures, PERT analysis represents an industrial game of Deal or No Deal with seven boxes containing the following amounts:

Optimistic
Optimistic
Realistic
Pessimistic
Pessimistic
Pessimistic
Pessimistic

At some point one of the boxes will be opened and that will be the result but until that time all we have is the multi outcome expectation. Until then, there is no “one” future but many possibilities, each with its own attendant probability. This is one of the reasons for James Surowiecki’s observation in The Wisdom of Crowds that expert forecasts are usually less accurate than the combined guesses of a crowd of people of people who are diverse, independent and decentralized. The aggregation of such different opinions acts as a kind of Montecarlo analysis that gives all possible futures consideration in the calculation and produces a kind of “expectation” or average of all possible futures. (See Chapter Two)

Indeed when people throughout history have made single outcome predictions and prognostications, they are setting themselves up to fail: “Who the hell wants to hear actors talk?” asked Jack Warner. Everything that can be invented has been invented” declared Charles H Duell, Commissioner of patents in 1899. “640KB ought to be enough for anybody.” Said Bill Gates in 1981

And we don’t laugh at any of these individuals: they were the best guesses that we had from experts in their field. If someone had told you in August 2001 that the world’s airlines would lose more money in the next 18 months than they had collectively made in profit since their inception you’d have thought they were crazy and yet that is exactly what went on to happen. If someone had told you in spring 2008 that Britain’s banking system would be all but nationalized by Christmas…?

The only meaningful way of talking about the future is to use the language of probability. By doing so we not only avoid the pitfalls of “pinning our colours to the mast” we also manage to tame uncertainty as well as we can – just ask any project manager!

Posted 01:48pm by Caspar and filed in Innovation, Uncertainty

Judgement in a Changing World

According to Jonathan Freedland in Saturday’s Guardian Al Gore had it but Chris Huhne, David Cameron and Sir Philip Hampton all, sadly, lack it.  Indeed, Freedland continues, many others in positions of power over the years all want for this vital skill: Alan Greenspan, Peter Mandelsohn, Paddy Ashdown, Michael Gove. He’s talking, of course, of judgement and he uses as his evidence the fact that all of these people have at some point pinned their colours to a prophecy or prognosis which turned out to be untrue.

By this definition, of course, we can add pretty much most of the world’s politicians, financiers and, for that matter, columnists of the fourth estate, which is probably why all three of these professions are so wildly unpopular at the moment as people cry out in vain for some semblance of order in an increasingly unpredictable world.

The reality, however, is that if the yardstick by which we measure judgement is accurate prediction then we’re all going to fall short of the mark at one time or another. In my first submission to this publication six months ago I argued that the butterfly effect rendered any meaningful prediction of our future practically impossible because doing so required virtually complete knowledge of every tiny actor within incomprehensibly vast and complex systems.

Indeed, as the pace of change increases and the level of feedback between the different actors on our planet grows exponentially – barring monumental scientific progress in this area – effective prediction of the future is going to become less and less useful because our lives are simply more and more susceptible to the tiniest changes in anything and everything around us. Everyday, I am buffeted by the lives of the hundreds of people I come into contact with on a daily basis via the plethora of different media, all of us shuffling deadlines, changing plans and reallocating resources in response to the shifting sands of what we call reality.

It wasn’t always thus. A few thousand years ago, I would have had a role in my family unit and obviously some level of interaction with the hundred or so people in my community but broadly speaking a tsunami in Japan would not have changed the value of my pension pot and traffic jams on the other side of London would have had no effect on my ability to make my meeting on time and thus my daughter’s birthday party in the evening.

There are paintings on the walls of Lascaux in France which are separated by literally thousands of years which are – to all intents and purposes – stylistically the same. And yet a cursory study of the definition of “art” now reveals at least 35 different and distinct means of expression for the twentieth century alone!

The most successful music artist of the 50s and 60s, Elvis Presley probably had about three main “looks” culminating of course in the iconic Vegas jumpsuit of the 70s. In the 80s, Madonna became famous for restyling herself with every album – something I remember being a massive deal at the time. Now, the most successful artist on the planet, Lady Gaga, looks almost unrecognisable every time she leaves the front door, sometimes dressed in meat and other times as a man!

In business, behemoths of industry like Chrysler and Kodak founded on continuity go to the wall while new, agile, companies like Groupon and Google appear to create vast sums of capital value in periods of time too short to make any sense fifty years ago. Moreover, they appear to change their offering as the world changes around them in the most remarkable manner and in ways which appear to rip up the rulebooks. YouTube actually started off for a few weeks as a video dating site before finding a killer app as a place for people to upload homemade movies and subsequently morphing into a centralised repository for television and music video. Meanwhile I’ve lost track of the different ways that Twitter and Facebook choose to represent my information. I’m obviously getting old…

Society is changing, that is obvious. What is less obvious – until we take a time out and stare at it – is the extent to which change itself is changing and, therefore, that the way in which society deals with change needs to change as well!

This is not a new idea per se. Alvin Toffler predicted exactly this 40 years ago in his seminal book “Future Shock” which foretold of the social disorientation that would happen in the future as the pace of change accelerated to a point where there is no status quo. It is remarkably prescient for any book written about today’s society from the perspective of 1970 – particularly for one telling us how much everything’s going to change – coining terms like “information overload” and describing very accurately how we’ll all feel with things like Google News and Times for iPad at our fingertips. But where does all this leave the subject of judgement and how can anyone appear to have it in a world that is changing so rapidly and with accelerating speed?

In 1984 Philip Tetlock was a newly tenured academic in the National Research Council who had an idea that was derided by some as a kind of madness. He embarked on a study with 284 experts in their fields – spanning numerous disciplines – and collected a staggering 27,450 judgements from them all over the course of twenty years. Inspired as it was by the cold war, the study focussed on political events on the global stage before being written up as the book “Expert Political Judgement: How Good Is It? How Can We Know?” in 2005. Its conclusions were startling.

Overall, the experts fared less well than chance and only slightly better than basic computer algorithms but it was the nuances beneath this headline that made the most fascinating reading because obviously some experts did better than others. But probably not in the way that you’d think. Good judgement did not correlate with academic discipline, age or experience. It wasn’t concentrated in any particular area of the political spectrum. And it wasn’t a question of gender or any other trait of background or upbringing.

The single most significant factor that determined an expert’s powers of judgement was the degree of confidence to which they expressed their prediction. And the correlation was inverse, that is to say, the less confident they were – on average – the more accurate. In other words, the more convinced and convincing an expert’s prediction appeared to be the less accurate it actually was. And in an age where politicians, journalists and financiers get to the top by being convincing, that’s a problem.  It may mean that we have to re-write the rules on what makes good judgement and slowly but surely turn to new and better judges in the future.