Peter Green used this line (from I Corinthians) for his Royal Statistical Society Presidential Address in 2003, which anyone interested in the future of statistics should read. I’ve been planning to steal it ever since then, and the time seems right.
Roger, Jeff, and Rafa at Simply Statistics are holding an unconference on the future of statistics, some time before dawn tomorrow morning New Zealand time. I probably won’t be attending, but if you’re in a more compatible time zone it promises to be interesting. It’s also sparked some Twitter chatter on the future of statistics. As you’d expect given the promoters, the chatter has focused on the importance of computation and applications and argued that theory is overrated. To some extent I agree, but I’m writing this in defense of methodological pluralism.
I think it’s unquestionably true that the academic statistical community overvalues mathematical formalism. It can be easier to publish sterile generalisations or pointless complications of mathematical statistics than useful simulation studies or high-quality applications. Much of the community has not really caught up with the fact that computation is thousands of times cheaper than it was three decades ago, and this has real implications for the best ways to solve problems. My colleague Alastair Scott (Chicago, 1965) tells the story of joining a discussion with other members of his generation about the most important advances in statistics over their careers. He suggested computing, which had not been brought up by anyone else and was received with some surprise.
On the other hand, some of the discussion reminds me of non-statisticians finding that, say, Andrew Gelman or Don Rubin is more knowledgeable and sensible than whoever taught them statistics in QMETH 101 and taking this as strong evidence for Bayesian statistics over frequentist statistics. It’s certainly true that, say, Hadley Wickham or Roger Peng’s research is of more benefit to humankind than the median piece of asymptotic statistics. But that’s mostly because they are really smart and hardworking. If everyone learned lots of statistical computing and graphics and less theory it wouldn’t turn everyone into Roger or Hadley. It would mean that useless papers on Edgeworth expansions of the overgeneralized beta distribution were replaced by useless simulations or ungeneralisable data analyses or pointless graphs. Beating Sturgeon’s Law just isn’t that easy – as any journal editor can tell you.
The point of mathematical statistics is that it tells you how to simplify certain problems that are too hard to think about heuristically. That’s only a minority of scientific problems, but it’s an important minority. A huge amount of cognitive effort has gone into developing mathematical tools for thinking about inference, and these tools are valuable today. I’ve given some examples in past posts, and lots of people could give you others.
So, what would I recommend? Diversity. The heretics are right that we shouldn’t have all PhD programs teaching two semesters from TSH and TPE; but some programs should. Some of them should focus on computation and algorithms. Some should teach more modern theory from van der Vaart instead. Some, like Santa Cruz and Duke, should focus on decision theory and Bayesian methods. Perhaps some should just concentrate on particular areas of application.
Statistics needs Savage Bayesians and moderate borrowing-strength Bayesians; we need applied statisticians who know the difference between DNA and RNA and probabilists who know the difference between \(l_2\) and \(L_2(P)\); we need Big Data and randomised experiements; and we need mathematical statisticians who understand why asymptotic approximations are useful.
I spent a lot of time and effort on statistical computing before it became popular, against the advice of my seniors. I understand the attraction in elevating Chambers and Friedman and casting down Cramer and Kolmogorov. I can see the poetic justice in mathematical statistics becoming a peripheral subject in graduate programs. But I think it would be a terrible waste.