Estimator vs estimate

In sandwich variance estimators, the middle of the sandwich¹ is the variance of the estimating function. If we have independent observations and estimate by solving $$\sum _i{U}_i\left(\theta \right)=0$$ we want $var\left[{\sum }_i{U}_i\right(\theta \left)\right]$, which we estimate by ${\sum }_i{U}_i\left(\hat{\theta }\right)U_i(\hat{\theta }{)}^T$. There are two issues here you might miss.

First, some people (younger me, for example) worry that $\sum U_i\left(\hat{\theta }\right)$ is always zero, by construction, so that surely $var[\sum U_i(\hat{\theta }\left)\right]$ will also be zero. This is a nice simple mistake of confusing the random variable $\hat{\theta }$ with its value in a particular data set. In our data, we might have $\hat{\theta }=42.69$, so that $$\sum _i{U}_i\left(42.69\right)=0.$$ In other data sets sampled from the same distribution, $\hat{\theta }$ will have some other value and ${\sum }_i{U}_i\left(42.69\right)$ should be small, but it won’t be zero. When we write $$var\left[\sum _i{U}_i\right(\hat{\theta }\left)\right]$$ the notation $\hat{\theta }$ doesn’t mean the estimator as a random variable, it means the value that the estimator took in this sample. We want $${var}_{\theta =42.69}\left[\sum _i{U}_i\right(42.69\left)\right].$$

In other data sets sampled from the same distribution we won’t have $\hat{\theta }=42.69$, so $var[\sum U_i(42.69\left)\right]$ is not zero and is a reasonable estimator of $var[\sum U_i({\theta }_0\left)\right]$ if 42.69 is close to ${\theta }_0$, which we know happens with high probability.

And, in addition, (assuming independence) $${var}_{\theta =42.69}\left[\sum _i{U}_i\right(42.69\left)\right].$$ is well estimated by $$\sum _i{U}_i\left(42.69\right)U_i(42.69{)}^T.$$ which we can evaluate without assuming a particular parametric model to compute the variances. We can prove this without a great deal of subtlety: just use Chebyshev’s inequality on the iid sum.

Things get more interesting when we have dependence. We need to include some crossproduct terms $U_i{U}_j^T$ in the variance calculation, representing non-zero covariances of $U_i$ and $U_j$. It’s still true² that $$E\left[\sum _{i,j}{U}_i\right({\theta }_0\left)U_j\right({\theta }_0\left)\right]=var\left[\sum _i{U}_i\right({\theta }_0\left)\right],$$ but it’s no longer true that

$$\sum _{i,j}{U}_i\left(\hat{\theta }\right)U_j\left(\hat{\theta }\right)\approx var\left[\sum _i{U}_i\right({\theta }_0\left)\right].$$ The left-hand side of this is identically zero, just like it wasn’t for the iid case. The left-hand side evaluates to $$\left(\sum _i{U}_i\right(\hat{\theta }\left)\right){\left(\sum _i{U}_i\right(\hat{\theta }\left)\right)}^T={00}^T.$$ That’s actually a bit surprising. The bias^{3 4} from using $\hat{\theta }$ instead of ${\theta }_0$ has grown from asymptotically negligible to the whole estimator. ⁵

The difference between ${\theta }_0$ and $\hat{\theta }$ is $O_p\left(n^{-1/2}\right)$, so the difference between $U_i^2\left({\theta }_0\right)$ and $U_i^2\left(\hat{\theta }\right)$ is $O_p\left(n^{-1}\right)$, giving a bias of size $O_p\left(1\right)$ in a variance matrix that’s of order $n$. So in the iid case the bias is asymptotically negligible. In the completely correlated case we have $n^2$ copies of the bias, so it’s only $O_p\left(n\right)$. That doesn’t prove there’s a problem, since $O_p$ terms only give upper bounds, but we already knew there was a problem and it makes sense.

This counting argument suggests that we need $o\left(n^2\right)$ terms in the sum to get a sandwich estimator that works. Or, given that we’d like a little margin to get a rate of convergence, maybe $O\left(n^{2-\delta }\right)$ terms. That’s potentially a problem – $n\times n$ is not $O\left(n^{2-\delta }\right)$⁶ – but in some settings we know that some of the $cov[U_i,U_j]$ terms are exactly zero and we can leave them out.

In longitudinal data with $m$ observations on each of $M$ units, we have $n=mM$ data points but all pairs $(i,j)$ of observations on two distinct units will have zero covariance. We can replace $U_i\left(\hat{\theta }\right)U_j\left(\hat{\theta }\right)$ for those pairs by just 0. The number of pairs we have left is $M{m}^2$, so if $M\to \infty$ we have $M{m}^2/n^2\to 0$ and if $M\to \infty$ and $m$ is bounded by any power of $M$ we have $M{m}^2=O\left(n^{2-\delta }\right)$. In crossed clustering with $m$ and $M$ distinct groups of two types, again we need $m,M\to \infty$ and $m$ bounded above and below by some power of $M$. In both these cases the proofs are again just brute-forcing counting applications of Chebyshev’s inequality and a suitable Taylor series expansion

For time series and spatial data it’s a bit more tricky since we don’t ever have exact independence of $U_i$ and $U_j$. Here we need to drop $(i,j)$ terms where $cov[U_i,U_j]$ is small enough, and we need to chose the threshold for small enough to get stricter with increasing $n$ at a rate that lets us use Chebyshev’s inequality on the non-zero terms we keep but still bound the bias from dropping terms.^{7 8}

So, in conclusion, sometimes seems like it should matter that we’re using $\hat{\theta }$ instead of ${\theta }_0$ in the sandwich estimator but it really doesn’t, and sometimes it doesn’t seem like it should matter but it really does.