I was listening to Tim Harford’s More or Less podcast last week in the wake of all of the excitement over the possible detection of the Higgs boson. More or Less is a show all about statistical abuses in the news and in life, and I strongly recommend it. In this episode, they had Robert Matthews, a physicist from Aston Universities talking about the nature of statistical significance. He did a good job, but his discussion was oddly lacking any reference to the Reverend Thomas Bayes.
This post will be a little bit technical (but nothing on the level of what we’ve seen in some of our relativity discussions), so you may want to get a bit of scratch paper out. It deals with how we actually discover new things, and in particular, how we assess the uncertainty of things that we can’t possibly know.
I should also add that while I will throw around a few example numbers, and that these resemble the real numbers put out by the ATLAS and CMS experiments at the LHC and the Tevatron at Fermilab this is really meant to be an overview.
Take the plot at the top of the page. That’s a measurement of the signal and noise from the ATLAS collaboration. They were, as you know, looking for the Higgs Boson. The idea of the plot is that if there is no Higgs than the signal (the solid line) should appear within the green curve approximately 68% of the time (so-called ), and within the yellow curve approximately 95% of the time ().
This “68% of the time” or “95% of the time” needs some explanation. An experiment is going to be noisy. Sometimes the noise is going to produce an uptick, and sometimes a downtick. If there were no Higgs, and if you ran the same experiment thousands of times, you’d expect to find yourself within the yellow 95% of the time, above the yellow 2.5% of the time, and below the yellow 2.5% of the time.
In other words, a result is suggestive, but even naively, it’s not overwhelming. I wouldn’t bet my life on something where 1 time in 40, even no signal at all could produce as significant a result.
There are other complications that I’m going to ignore here, since they don’t change the basic picture. One is that in searching for the Higgs, you get to look at lots of different possible masses. If you could look at 40 different masses, and all of them are independent, 1 of them is likely to be a result. This is the “look elsewhere” result that is often cited in these results.
For our purposes, we want to only take into account the data around 125GeV, and ask what we can say about whether a Higgs at that mass is likely or not. Taking all of this into account, the ATLAS result is at . This means:
If there is no Higgs, the odds of finding a signal this far above the expected background level is about 1%
This does not mean that the probability that the Higgs is real is 99%. That was Matthews’s point. To figure out how likely the Higgs is, we need to say a bit about Bayesian inference.
Suppose I told you that I had a magic genie that grants wishes. I could prove it. I wish for a coin that comes up heads every time. To test my conjecture, I flip a coin 5 times in a row and get heads every time. That’s only a 1 in 32 probability (about 3%)!
Would you assume that I really did have a genie?
- My argument (following the same logic that we applied to the Higgs) might be that the probability of the genie is 97%.
- The argument against is that while 5 heads in a row (especially if I called it ahead of time) is unlikely, the odds of genies are even more unlikely.
All of this can be formalized in a relation known as Bayes Theorem, which says that if I have two events, A & B, their probabilities can be related with the expression:
Don’t be scared off by the notation. It’s more straightforward than you might think. and simply mean the probability of A or B being true before we do any experiments at all.
In this case:
- = There really is a genie with the properties I described.
- = I flip 5 heads in a row.
is “The probability of A given that B has already occurred.” In our case,
- = The probability that we flip 5 heads if there really is a genie. = 100%. (That’s the rule for a genie.)
Computing the other terms are a bit more complicated. For one thing, we don’t really know ahead of time what the a priori probability of having a genie is (). This is really a measure of our pre-existing belief in genies.
At one extreme, I might say that there is literally zero chance of genies, or I might pick an incredibly small number, perhaps 1 in a million. At the other extreme, I might say that I’ve seen and interacted with genies before, so the chance is 100%. We really don’t know.
The good news, though, is that once we pick this number, we can compute everything else. The a priori probability is simply:
Put into words, this is simply the probability of flipping 5 heads on a fair coin (with no genie) added to the probability that there’s a genie.
From this, we can compute the “posterior probability” of there being a genie given that I’ve just flipped 5 heads, by plugging into Bayes theorem:
Naturally, if we already knew that there was a genie, our certainty was 100% before and after we did the experiment.
On the other extreme, if we really don’t believe in genies, this experiment isn’t going to help very much. If we thought that particular type of genies were a one in a million chance (a rather generous assumption) then even after seeing this “proof” we would only raise that belief from 0.0001% to 0.0032%.
And the Higgs
The calculation is very similar for the Higgs. We’re assuming that everything else has been ruled out and the “look elsewhere” effect has already been taken into account. We’re just asking about a Higgs at or near 125 GeV.
In this case, our two events might be:
- = There is no Higgs.
- = The ATLAS team finds a measurement (or higher).
We already know one of these terms, the probability of getting such a large amount of noise if there really is no Higgs
The number we don’t know, of course, is , the probability that there’s no Higgs. Theory strongly supports the existence of a Higgs particle in approximately the range we’re talking about, so I might be inclined to put the a priori probability of no Higgs as quite low, perhaps 10%. But let’s argue from the other direction. Suppose that you would have given the Higgs only a 5% chance of being real, that means:
which can then be plugged in to compute
where I’ve assumed (not entirely accurately) that if there really were a Higgs, then we’d see a result 100% of the time.
Just to make this clear. The idea is that we’ll only get a signal this large in the 5% probability that there is a Higgs plus the 95% probability of no Higgs times the 1% chance of getting the level of noise that we see. That adds up to 6%.
Plugging in the numbers, we’d get:
Or in other words, even if you believed with 95% confidence ahead of time that there shouldn’t be a Higgs, after conducting this experiment, your posterior probability is more like 16%. The likelihood of a Higgs is therefore 84%.
More generally, we can look at the posterior probabilities for any prior we like:
Note that the y-axis has a log scaling, and that it represents the probability of no Higgs (after seeing the data from ATLAS).
Some things to remember
- I picked my prior probability of the Higgs rather arbitrarily. If you think it 99.99% probable that there’s no Higgs (or any probability you like) then it’s going to be that much harder to convince you.
- ATLAS isn’t the only experiment in town. CMS detected something like a result, and the Tevatron had something like a result. Combined, these are at the level (about 0.1% probability by pure chance). Using our 5% prior likelihood of the Higgs really being in this range, we get about a 98% posterior probability. This is why I’m so confident that Higgs will ultimately be officially detected.
- The particle physics community usually relies on a result to “detect” a particle — less than 1 part in a million for . Even if you were 99.999% ahead of time that there was no Higgs (and really, how did you come to such a precise number?), a detection would still have a 97% probability of being real.
Apologies for the fairly technical post, and to the statistics community, apologies for playing fast and loose with some terminology.
In either case, I hope this helps to explain why there are some people who aren’t quite ready to believe the Higgs is real (yet) and why others (like me) are.