What does ‘random’ look like?
Posted: November 24, 2011 Filed under: Miscellaneous | Tags: cognitive illusion, Ed Purcell, gambler's fallacy, London Blitz, Poisson, statistics, Steven Jay Gould, Steven Pinker, The Better Angles of our Nature, Thomas Pynchon, Waitomo 8 Comments
Which plot is random?
The exhibit shows two seemingly similar patterns – but one of them is random and one isn’t. Can you tell which is which? More in a moment.
I’ve taken these plots from Steven Pinker’s new book, The better angels of our nature: why violence has declined. It’s in a Chapter titled The statistics of deadly quarrels where he discusses the statistical patterning of wars and takes a small detour into “a paradox of utility”, specifically our tendency to see randomness as regularity with little clustering.
This cognitive illusion has relevance to all disciplines but is of particular interest to anyone interested in spatial issues, as a couple of these examples show.
Professor Pinker, who’s a psychologist at Harvard, cites the example of the London blitz, when Londoners noticed a few sections of the city were hit by German V-2 rockets many times, while other parts were not hit at all:
They were convinced that the rockets were targeting particular kinds of neighborhoods. But when statisticians divided a map of London into small squares and counted the bomb strikes, they found that the strikes followed the distribution of a Poisson process—the bombs, in other words, were falling at random. The episode is depicted in Thomas Pynchon’s 1973 novel Gravity’s Rainbow, in which statistician Roger Mexico has correctly predicted the distribution of bomb strikes, though not their exact locations. Mexico has to deny that he is a psychic and fend off desperate demands for advice on where to hide
Another example is The Gambler’s Fallacy – the belief that after a long run of (say) heads, the next toss will be tails:
Tversky and Kahneman showed that people think that genuine sequences of coin flips (like TTHHTHTTTT) are fixed, because they have more long runs of heads or of tails than their intuitions allow, and they think that sequences that were jiggered to avoid long runs (like HTHTTHTHHT) are fair
The exhibit above shows a simulated plot of the stars on the left. On the right it shows the pattern made by glow worms on the ceiling of the famous Waitomo caves, New Zealand. The stars show constellation-like forms but the virtual planetarium produced by the glow worms is relatively uniform.
That’s because glow worms are gluttonous and inclined to eat anything that comes within snatching distance, so they keep their distance from each other and end up relatively evenly spaced i.e. non-randomly. Says Pinker:
The one on the left, with the clumps, strands, voids, and filaments (and perhaps, depending on your obsessions, animals, nudes, or Virgin Marys) is the array that was plotted at random, like stars. The one on the right, which seems to be haphazard, is the array whose positions were nudged apart, like glowworms
Thus random events will occur in clusters, because “it would take a non-random process to space them out. The human mind has great difficulty appreciating this law of probability”.
By the way, this is an interesting book. As summarised by Peter Singer in his review for the New York Times, “the central thesis of “Better Angels” is that our era is less violent, less cruel and more peaceful than any previous period of human existence:
The decline in violence holds for violence in the family, in neighborhoods, between tribes and between states. People living now are less likely to meet a violent death, or to suffer from violence or cruelty at the hands of others, than people living in any previous century.
Even if you don’t buy his big thesis, this is a fascinating journey through the history of civilisation, philosophy, psychology, and much more.
To finish, here’s a puzzle Professor Pinker poses at the start of the chapter (he reckons almost no one gets it right. The answer is in the comments):
Suppose you live in a place that has a constant chance of being struck by lightning at any time throughout the year. Suppose that the strikes are random: every day the chance of a strike is the same, and the rate works out to one strike a month. Your house is hit by lightning today, Monday. What is the most likely day for the next bolt to strike your house?
I’ve looked at interesting statistical “stories” before –e.g. see Can selection bias shoot down an argument.
The Melbourne Urbanist 
Interesting article. Presumably your chance of a lightening strike is the same as before since the lightening would have no memory, or is it more likely to be tomorrow due to clustering?
Does he discuss the changes in the expectations of violence in the book?
You might also be interested in Daniel Kahneman’s Thinking, Fast and Slow. I haven’t read it but from interviews of him I have heard it sounds well worth a read – or at least listen to the LSE podcast of him talking about it. Unfortunately I already have a large stack of books waiting to be read.
That’s pretty good. From Pinker:
There’s a coincidence. I got Kahneman’s new book on Tuesday (the marvel of e-books!) and will have a look at it on the weekend (I use a Sony Reader).
Of course, in most scientific applications you’ll have the opposite information: a sequence of events with varying durations between them and (potentially) some change in frequency compared to previous observations. What the scientist would really like to know is whether there is in fact a change in the expected probability, or whether the trend is merely random noise.
Or to put it another way, would Pinker recommend dropping Ricky Ponting?
Slightly OT – the same applies to extremes of Climate Change. For extremes, we have a sequence of events, varying times between them, we don’t know if they occur more often or less often, we don’t know if they are worse or not and we don’t know if they reflect real change in climate or are just part of a totally random sequence. (Also depends on our definition of ‘extreme’!)
But when we get a ‘heat wave’ or a long ‘drought’ and the news says that it is the worst since “X” (a date sometime in the past) we wonder.
What we do know is that disasters are worse because (a) the population in subject areas has increased – so more people are affected, (b) inflation means the nominal cost has increased – although the real cost per person may be constant, and (c) neglect of proper precautions may make what was a survivable disaster when the infrastructure was built unsurvivable now, when the infrastructure has been allowed to deteriorate (think New Orleans and Katrina).
Apologies if this is too off topic, but disasters are – one hopes – random events. Aren’t they?
Hmm, I thought the answer was “today” – i.e. straight away, in the very next quantum unit of time…
BTW, don’t suppose you can do a giveaway for this book!
wizofaus, the highest probability occurs in the next specific unit of time, but because the units are days, that becomes Tuesday. Assuming the lightning strike was randomly distributed within a day, the probability of it occurring again that day (Monday) is 0.015. If the units of time were months then there’d be an average 63% of chance of getting out of the current month without a strike (0.97^15), and 25% chance of the following month as well (0.97^15). The probability of going an entire year (365 days) without lightning would be 0.0014%, you’d need to be a week into December before you’d expect your next lightning strike the following year.
My guess:
Most likely day for the *next* lightning strike is tomorrow. Days after that have the same chance of a lighting strike, but a higher chance of one in the interim.
I want it known I thought we were putting our own answers in the comments, and hadn’t read the answer first!