I have mentioned Stein’s method in passing, a few times on this blog. Today I want to talk about Stein’s method in a bit of detail.
1. What Is Stein’s Method?
Stein’s method, due to Charles Stein, is actually quite old, going back to 1972. But there has been a great deal of interest in the method lately. As in many things, Stein was ahead of his time.
Stein’s method is actually a collection of methods for showing that the distribution of some random variable is close to Normal. What makes Stein’s method so powerful is that it can be used in a variety of situations (including cases with dependence) and it tells you how far is from Normal.
I will follow Chen, Goldstein and Shao (2011) quite closely.
2. The Idea
Let be iid with mean 0 and variance 1. Let
and let . We want to bound
where is the cdf of a standard Normal. For simplicity, we will assume here
but this is not needed. The key idea is this:
for all smooth if and only if . This suggests the following idea: if we can show that is close to 0, then should be almost Normal.
More precisely, let and let be any function such that . The Stein function associated with is a function satisfying
It then follows that
and showing that is close to normal amounts to showing that is small.
Is there such an ? In fact, letting , the Stein function is
More precisely, is the unique solution to the above equation subject to the side condition .
Let’s be more specific. Choose any and let . Let denote the Stein function for ; thus satisfies
The unique bounded solution to this equation is
The function has the following properties:
where . Also,
Let . From the equation it follows that
We have reduced the problem of bounding to the problem of bounding .
3. Zero Bias Coupling
As I said, Stein’s method is really a collection of methods. One of the easiest to explain is “zero-bias coupling.”
Let . Recall that are iid, mean 0, variance 1. Let
For each define the leave-one-out quantity
which plays a crucial role. Note that is independent of . We also make use of the following simple fact: for any and , since the density of the Normal is bounded above by .
Recall that if and only if
for all absolutely continuous functions (for which the expectations exists). Inspired by this, Goldstein and Reinert (1997) introduced the following definition. Let be any mean 0 random variable with variance . Say that has the -zero bias distribution if
for all absolutely continuous functions for which . Zero-biasing is a transform that maps one random variable into another random variable . (More precisely, it maps the distribution of into the distribution of .) The Normal is the fixed point of this map. The following result shows that exists and is unique.
Theorem 1 Let be any mean 0 random variable with variance . There exists a unique distribution corresponding to a random variable such that
for all absolutely continuous functions for which . The distribution of has density
Proof: It may be verified that and integrates to 1. Let us verify that (2) holds. For simplicity, assume that . Given an absolutely continuous there is a such that . Then
Here is a way to construct explicitly when dealing with a sum.
Lemma 2 Let be independent, mean 0 random variables and let . Let . Let be independent and zero-bias. Define
where . Then has the -zero bias distribution. In particular, suppose that have mean 0 common variance and let . Let be a random integer from 1 to . Then has the -zero bias distribution.
Now we can prove a Central Limit Theorem using zero-biasing.
Theorem 3 Suppose that . Let . Then
Proof: Let be independent random variables where is zero-bias for . Let be chosen randomly from and let
By a symmetric argument, we deduce that
Let . From the properties of the Stein function given earlier we have
Combining these inequalities we have
This is just the tip of the iceberg. If you want to know more about Stein’s method, see the references below. I hope I have given you a brief hint of what it is all about.
Chen, Goldstein and Shao. (2011). Normal Approximation by Stein’s Method. Springer.
Nourdin and Peccati (2012). Normal Approximations With Malliavin Calculus. Cambridge.
Ross, N. (2011). Fundamentals of Stein’s method . Probability Surveys, 210-293. link