John Kruschke’s career as an experimental psychologist took an unexpected turn when he began to question traditional statistical methods

If you think statistics is dull and dry, and the math outright intimidating – well, you may have a point. It can be intimidating. As for dull and dry? Think again.

Between scientists and their findings, questions and answers, data and the trends within it, is the science of statistics. Statistics, Kruschke explains, helps you detect the signal in that “heap of noisy numbers. You need statistics to find out what kind of trend you can describe in this heap of numbers and how big it is relative to the noise.”

Does a new drug have a benefit? Will caffeine boost your IQ? Does chocolate prevent heart disease? Red wine?

The science of statistics, as Professor John Kruschke describes it, provides the most critical plot twists in the detective story at the core of science itself.

Statistical Conversions

Kruschke’s close encounter with statistics began almost 30 years ago when he started teaching the subject to first-year graduate students. He began the stint, like most statistics professors, teaching the traditional “frequentist” approach. The more he taught this approach, however, the more he began to question it. As he notes in a short bio, he reached a point where he “could no longer teach corrections for multiple comparisons with a clear conscience. The perils of p values provoked him to find a better way.”

Ultimately, his career took what he calls “a complete digression,” when he adopted an alternative statistical approach – Bayesian statistics – that was just then gaining currency. But it was less a digression perhaps, than an all-out conversion experience, one that was also taking place more broadly among statisticians and scientists of many kinds, who were turning to Bayesian statistics as an alternative to more traditional, and potentially problematic, statistical methods.

Kruschke himself has since become a leading proponent of Bayesian statistics, which he has enthusiastically shared and continues to share with his many students, readers, and audiences around the world. He has produced numerous articles on the topic and two editions of a major textbook, Doing Bayesian Data Analysis. He has taught scores of workshops for groups with varied professional and academic leanings, from the Federal Aviation Association, the Food and Drug Administration, to medical economists in Norway, health scientists in Scotland, and many universities across the United States and Europe. Most recently, he was an editor for a special volume of essays devoted to Bayesian statistics in the Psychonomic Bulletin and Review, for which he also wrote two essays intended for an audience of Bayesian “newcomers.” All of these efforts have helped bridge the gap between statisticians, who develop statistical techniques, and others who use statistical techniques in their work.

Named for its eighteenth-century originator Reverend Thomas Bayes, Bayesian statistics had remained in the shadows of the traditional, institutionally entrenched “frequentist” method. Then in the 1990’s and 2000’s computational advances, a collapsing wall of philosophical resistance with respect to features such as making use of prior possibilities – and researchers like Kruschke – brought it further into the daylight.

Does a new drug have a benefit? Will caffeine boost your IQ? Does chocolate prevent heart disease? Red wine?

“The Perils of P Values”

Kruschke’s critique of traditional statistics starts with the tendency for people to rely heavily on p (or probability) values when determining whether trends in their data are meaningful. P values quantify the likelihood that a trend occurs just by chance. The higher the p value, the more likely it is that the pattern of results resulted from normal variability in the data, or the probability that the pattern of results does not reflect a “true” effect.

He gives an example: Say you want to know whether a new smart drug improves people’s IQ. You give the smart drug to one group and a placebo to another. You then give both groups an IQ test and find that the group taking the smart drug had an average score that was 10 points higher than the placebo group. You now want to know whether that difference is an effect of the drug or if it is within the normal range of variability between two groups.

At this point traditional statistics seeks to find the p value by asking, “What is the probability I would have gotten a difference this big by chance alone if really there was no difference between the smart drug and the placebo?” To determine this number, you have to consider all the random influences on the sample size and all the ways you intend to look at the data. “All those influences affect the p value because the p value is based on what might have happened if you collected and tested your data the same way, but the null hypothesis was true,” Kruschke explains. “What frequentist statistics gives us is the probability of getting imaginary data if the world were a hypothetical way, but what we intuitively want is the opposite -- the probability of there being a real difference given the real data we actually observed.”

The “backward, counter-intuitive logic,” of frequentist statistics, says Kruschke, “becomes the bane of every first-year statistics student. Yet, once people understand what the p value is really about and how fickle that number is, they feel so unburdened. It’s like a revelation. The veil has been removed from their eyes.”

The backward, counter-intuitive logic of frequentist statistics becomes the bane of every first-year statistics student.

John Kruschke

The Bayesian Edge

By contrast, the Bayesian method takes you through a different, “more intuitive,” kind of reasoning, one Kruschke sees best exemplified by the fictional detective Sherlock Holmes. For any given crime, Holmes typically begins with a set of suspects, some of whom initially appear more suspicious than others. But as he goes about his sleuthing, collecting new information, this information alters his view about who the most likely culprit might be. This process of “reallocating credibility across possibilities,” translated into mathematical formulas, is the first foundational principle of Bayesian statistics.

In place of the mental gymnastics required to create the idealized world of traditional statistics, the Bayesian approach prompts you to work with the data you already have, to consider ‘What effect can I actually believe, given what I actually have?’”

Having to establish degrees of suspicion at the outset, the “prior probabilities,” was a philosophical point of contention for traditional statisticians, who thought it was too subjective. But this concern, Kruschke explains, has since been outweighed by the practical applicability and usefulness of the method.

While traditional methods often lead to simple black and white findings – whether an effect is simply significant or not, based on the p value alone – and sparse information about the data, Bayesian methods “give you a more nuanced idea of what you can believe about the trends in your data and how uncertain you are about them.”

In this way the Bayesian view is more in line with a certain sense of what the world is like: it is infinitely variable and nuanced. As Holmes would say, “Nothing is more deceptive than an obvious fact.” Or as Kruschke puts it, “We are all already Bayesian.” The logic behind Bayesian statistics is straightforward and intuitive to us, aligned with the way we already think and reason.

The conversion to Bayesian statistics has begun to take root more widely among scientists, social scientists and many others - the U.S. Coast Guard, for example - who now use Bayesian statistics to guide their operations. Eventually, Kruschke believes, Bayesian methods will gain a greater footing in the institutional structures, academic curricula, journal policies and granting agencies, where the traditional methods are deeply entrenched.

Yet each year in his classroom he witnesses a piece of that conversion in motion – the excitement of a new generation taking the Bayesian methods into their labs, using them in their dissertations and in the world at large where they will, in turn, pass them on to others.