Some Facts about Bootstrapping

This post documents some facts about bootstrapping that I encountered during my PhD. In particular, this post highlights some asymptotic results relating to

• The mean absolute deviation of normal variables

• The quantile of absolute deviations from the mean

• The distribution of standard deviations

Consider the setting where we sample $T$ observations and create $B$ samples of size $T$ by sampling with replacement. In other words, we have:

• The original sample $S := \{X_i\}_{i \in [T]}$

• $\bar{\mu}$ is the sample mean statistic computed over $S$

• The samples from $S$ with replacement: $S_b := \{X^{(b)}_i\}_{i=1}^T, \ b \in [B] := \{1,2,...,B\}$

• $\bar{\mu}^{(b)}$ is the statistic computed over the $T$ observations associated with batch $b\in[B]$

Part 1: Normality

Mean Absolute Deviation and Standard Deviation

Geary, R. C. proved the following: If $X \sim N(\mu, \sigma^2)$ then $\mathbb{E}[|X - \mu|] = \sqrt{\frac{2}{\pi}} \sigma$

Throughout this post $N(\cdot, \cdot)$ takes the mean as its first argument and the variance (or covariance matrix, in the multivariate case) as its second.

Geary, The Ratio of the Mean Deviation to the Standard Deviation as a Test of Normality., Biometrika, 27(3), 310–32, 1935.

Central limit theorem

The central limit theorem states that if $\{X_i \}_{i \in [T]}$ are $T$ samples drawn independently and identically distributed from a distribution with mean $\mu$ and variance $\sigma^2$, then $\sqrt{T} (\frac{1}{T}\sum_{t \in [T]}{X_t} - \mu)/{\sigma} \overset{d}{\rightarrow} Z$, where $Z$ follows a standard normal distribution. The sample estimate of the mean for batch $b \in [B]$ is given by $\bar{\mu}^{(b)} := \frac{1}{T} \sum_{t \in [T]} X_t^{(b)}$, therefore, for large $T$:

where $\overset{\cdot}{\sim}$ reads "is approximately distributed as" — the exact statement is the $\sqrt{T}$-scaled one, $\sqrt{T}(\bar{\mu}^{(b)} - \mu) \overset{d}{\rightarrow} N(0, \sigma^2)$. In our setting, the distribution is the empirical distribution defined by the sample $S$, implying that $\mu$ and $\sigma^2$ are given by

and

Therefore $\bar{\mu}^{(b)} = \frac{1}{T} \sum_{t \in [T]} X^{(b)}_t$ can be thought of as a sample from $N(\mu, \frac{\sigma^2}{T})$, i.e. with standard error $\sigma/\sqrt{T}$.

Part 2: Quantile of Absolute Deviations

In the work by Ho, M., Sun, Z., and Xin, J. , the optimal portfolio solves:

where $\beta \in \mathbb{R}_+^N$ and $\alpha_i \in \mathbb{R}_+ \ \forall i \in [N]$ denote the penalty parameters. Ho, M., Sun, Z., and Xin, J. show that $\alpha_i$ and $\beta_i$ correspond to the sizes of box-uncertainty sets used in the robust counterpart of the mean-variance optimization problem. Specifically, $\alpha_i$ and $\beta_i$ correspond to uncertainty in the diagonal of the estimated covariance matrix $\Sigma_{ii}$ and expected return $\mu_i$ respectively for asset $i = 1,2,..., N$. Ho, M., Sun, Z., and Xin, J. use a bootstrapping approach to select the values of $\beta_i$ and $\alpha_i$; they proceed as follows: Let $\{\boldsymbol{r}^{(t)} \in \mathbb{R}^N\}_{t \in [T]}$ denote $T$ observations of the random variable $\boldsymbol{r}$ used to estimate $\boldsymbol{\Sigma}$ and $\boldsymbol{\mu}$. The estimation errors are computed by re-sampling $T$ observations with replacement from the original set of observations $B$ times. Each re-sampling yields estimates of the mean and covariance $\bar{\boldsymbol{\mu}}^{(b)}, \bar{\boldsymbol{\Sigma}}^{(b)} \ b \in [B]$. Let $p_1$ and $p_2$ denote the investor's aversion to estimation risk of the mean and variance, respectively. The values for $\beta_i$ and $\alpha_i$ are defined as the quantiles of the bootstrapped deviations

and

Quantile of absolute deviation for standard normal

One can express the quantile of absolute deviation of a normal variable in a closed-form equation. The results can be derived as follows: letting $Z \sim N(0,1)$ then the quantile of the absolute deviation satisfies $P(|Z| \leq x) = p$ which is the same as $2\Phi(x) - 1 = p$ which implies

This is useful because we see that the sample means $\bar{\boldsymbol{\mu}}^{(b)}$ converge in distribution to a normal distribution. Ho, Sun & Xin, Weighted elastic net penalized mean-variance portfolio design and computation., SIAM Journal on Financial Mathematics, 6(1), 1220–1244, 2015.

Part 3: Distribution of standard deviations can be obtained via the delta method

The links below were useful in understanding the delta method and its application to the distribution of standard deviations.

• Question about standard deviation and central limit theorem

• Why does the second-order delta method approximation to the variance of Bernoulli r.v. result in a negative chi-square?

• Elements of Large-Sample Theory

• Standard Deviation Distribution

The case of determining an approximation for the distribution of $\boldsymbol{\Sigma}$ is more complex. For simplicity, we consider the case where $\boldsymbol{\Sigma}$ is a scalar denoted by $\sigma^2$. An approach referred to as the delta method can be used to obtain the asymptotic distribution of the estimator $\hat{\sigma}^2 := \frac{1}{T}\sum_{i \in [T]} X_i^2 - (\frac{1}{T} \sum_{i \in [T]} X_i)^2$.

Let $\mathbf{V}_i := (X_i, X_i^2)^{\intercal}$ be the per-observation vector and let $\mathbf{W} := \frac{1}{T}\sum_{i \in [T]} \mathbf{V}_i$, so that $W^{(1)} = \frac{1}{T} \sum_{t \in [T]} X_t$ and $W^{(2)} = \frac{1}{T} \sum_{i \in [T]} X_i^2$. Write $\boldsymbol{\mu}_{\mathbf{W}} := \mathbb{E}[\mathbf{V}]$, and let $\Xi$ be the covariance of a single observation,

The multivariate central limit theorem then gives $\sqrt{T}(\mathbf{W} - \boldsymbol{\mu}_{\mathbf{W}} ) \overset{d}{\rightarrow} N(\mathbf{0}, \Xi)$. Note that $\Xi$ must be the per-observation covariance, not $\mathbb{V}[\mathbf{W}]$ — the two differ by a factor of $T$.

In general, for any differentiable $g(\mathbf{W})$ one can write the following first-order Taylor expansion $g({\mathbf{W}}) \approx {g(\boldsymbol{\mu}_{\mathbf{W}}) + } \nabla g(\boldsymbol{\mu}_{\mathbf{W}})({\mathbf{W}} - \boldsymbol{\mu}_{\mathbf{W}})$. One can approximate the asymptotic variance of $g(\mathbf{W})$ by $\nabla g(\boldsymbol{\mu}_{\mathbf{W}})^{\intercal} {\mathbf{\Xi}} \nabla g(\boldsymbol{\mu}_{\mathbf{W}})$ — so that $\mathbb{V}[g({\mathbf{W}})] \approx \frac{1}{T} \nabla g(\boldsymbol{\mu}_{\mathbf{W}})^{\intercal} {\mathbf{\Xi}} \nabla g(\boldsymbol{\mu}_{\mathbf{W}})$ — and as such $\sqrt{T}(g({\mathbf{W}}) - g({\boldsymbol{\mu}_{\mathbf{W}}}) )\overset{d}{\rightarrow} N(0,\nabla g(\boldsymbol{\mu}_{\mathbf{W}})^{\intercal} {\mathbf{\Xi}} \nabla g(\boldsymbol{\mu}_{\mathbf{W}}))$.

One can set $g(W_1,W_2) = W^{(2)} - (W^{(1)})^2$ and use the first-order delta method as described above to obtain asymptotic convergence condition. The first-order delta method does not always work. Convergence to normality does not hold for Bernoulli variables with $p = 0.5$ because the variance of $g$ ends up being zero.

To circumvent this, the second-order Taylor approximation must be used. In the case of Bernoulli random variables $g$ can be expressed as a function of a single random parameter:

since $\frac{1}{T} \sum_{t \in [T]} X_t = \frac{1}{T} \sum_{t \in [T]} X_t^2$. In this case,

If the mean is $p=1/2$, then taking the second-order taylor expansion around the mean implies $\hat{p}(1-\hat{p}) = 1/4 + 0*(\hat{p}-0.5) + 1/2 (-2) (\hat{p} - 1/2)^2$. It is then true that $\hat{p}(1-\hat{p})$ is asymptotically negative $\chi^2$ and is not normal because $\hat{p} = \frac{1}{T} \sum_{t \in [T]} X_t$ is asymptotically normal by the central limit theorem and $\hat{p}(1-\hat{p})$ is a constant minus an asymptotically normal variable squared (as shown in the above Taylor series. In the case that the first-order delta method is applicable and the first-order Taylor series introduces uncertainty in $g$, it follows:

where $\text{CM}_4(X)$ denotes the 4th-order central moment of $X$; equivalently, for large $T$, $\hat{\sigma}^2 \overset{\cdot}{\sim} N {\big(}\sigma^2, \frac{\text{CM}_4(X) - \sigma^4}{T}{\big)}$.

From the variance to the standard deviation

Part 3 is really after the distribution of $\hat{\sigma}$, not $\hat{\sigma}^2$. One more application of the delta method, with $h(v) = \sqrt{v}$ and $h'(\sigma^2) = \frac{1}{2\sigma}$, gives

This is the result the bootstrapped quantiles of Part 2 are implicitly relying on.

Google Colab Notebook

This notebook demonstrates the facts highlighted above.