# “The” multiple comparisons problem

Andrew Gelman posted recently with the title “Bayesian inference completely solves the multiple comparisons problem”. Bayesians have been making a claim that sounds like this for many years, so it would be easy to misunderstand and think he was making a much weaker claim than he actually is.

There are at least two multiple comparisons problems, andI’d like to suggest some terminology:

• The first-person multiple comparisons problem: I have data relevant to a collection of parameters $$\{\theta_i\}_{i=1}^N$$ and I want to make sure I arrive at sensible beliefs or take sensible decisions even if $$N$$ is quite large

• The second-person multiple comparisons problem: You want me to publish my findings in such a way that you arrive at sensible beliefs or take sensible decisions even if $$N$$ is quite large

The first-person problem is, fairly uncontroversially, solved automatically by Bayesian inference.  Frequentists aren’t bad at it either.

The second-person problem isn’t automatically solved by Bayesian inference, I’ve written about this (and as a commenter points out, Xiao-Li Meng has written related things). Andrew Gelman has said as much, in his ‘Garden of Forking Paths’ metaphor for researcher degrees-of-freedom.

The new claim is for a particular (important) case of the problem: Prof Gelman shows that a fairly strong but reasonable prior will ensure that badly-underpowered studies almost never draw false positive conclusions at 95% posterior probability. In that sense, if everyone switched to Bayesian inference there would be a lot fewer bogus newspaper stories.  There would be a higher rate of false negatives, but in the fields he cares most about there’s a huge asymmetry between underclaiming and overclaiming, so it doesn’t matter.