Fourth recitation

Hanzhe Li

Contents

Reading guide

This chapter focuses on continuous distributions, multivariate normal vectors, change-of-variables formulas, GOE, and several integral estimates.
For multivariate problems, the main objects are densities, linear transformations, covariance matrices, and orthogonal invariance.
Jensen's inequality, tail integral formulas, and normal tail bounds will be used later for laws of large numbers and concentration inequalities.

Tip. In normal distribution problems, zero covariance implies independence only when the variables come from a linear Gaussian structure.

Exercise 3.1

Note

For continuous densities and convolution problems, handle normalization, substitutions, Jacobians, and independence separately. Check the support before doing each integral.

Problem

Find the normalizing constants for the following densities:

$f(x)=\frac{1}{C}\frac{1}{\sqrt{x(1-x)}}$ , $0<x<1$ ;
$f(x)=\frac{1}{C}e^{-x-e^{-x}}$ , $x\in\mathbb{R}$ .

Proof

f(x)=\frac{1}{C}\frac{1}{\sqrt{x(1-x)}},\quad 0<x<1.

From $\int_{0}^{1}f(x)\,dx=1$ ,

\frac{1}{C}\int_{0}^{1}\frac{dx}{\sqrt{x(1-x)}}=1.

Compute the integral. Let $x=\sin^2 t$ . Then $dx=2\sin t\cos t\,dt$ , and

\sqrt{x(1-x)}=\sin t\cos t,\quad \int_{0}^{1}\frac{dx}{\sqrt{x(1-x)}}=\int_{0}^{\pi/2}2\,dt=\pi.

Thus

\frac{\pi}{C}=1 \quad\Rightarrow\quad C=\pi.

Since $\int e^{-x-e^{-x}}\,dx=e^{-e^{-x}}+C$ , direct evaluation gives $C=1$ .

Problem

Find the normalizing constant for

\int_{\mathbb{R}^2} |x_1 - x_2| e^{-\frac{1}{2}(x_1^2 + x_2^2)} \, dx_1 dx_2.

Proof

Let $u=x_1-x_2$ and $v=x_1+x_2$ . Then

|J|=\frac12.

Therefore

\text{the integral} = \int_{\mathbb{R}^2} |u| e^{-\frac{1}{4}(u^2 + v^2)} \cdot \frac{1}{2} \, du dv.

Hence

= 2 \int_{0}^{+\infty} u e^{-\frac{1}{4}u^2} \, du \cdot \int_{0}^{+\infty} e^{-\frac{1}{4}v^2} \, dv = 4\sqrt{\pi}.

Thus $C=4\sqrt{\pi}$ .

Problem

Let $X,Y$ be independent exponential random variables with parameter $1$ . Find the density of $U=X+Y$ .

Proof

Directly,

f_U(u)=\int_0^u e^{-x}e^{x-u}\,dx=u e^{-u}\mathbb{1}_{u\geq 0}.

Problem

Let $X,Y,Z$ be independent, and suppose each is uniformly distributed on $[0,1]$ .

(1) Find the density of $-\log X$ .

(2) Prove that $W=(XY)^Z$ is also uniformly distributed on $[0,1]$ .

Proof

(1) We have

\mathbb{P}(-\log X \leq x)=\mathbb{P}(X\geq e^{-x})=1-e^{-x},

so the density is $e^{-x}\mathbb{1}_{x\geq 0}$ .

(2) Since $W=(XY)^Z$ ,

-\log W=Z(-\log X-\log Y).

It is enough to prove that $Z(-\log X-\log Y)$ has the exponential distribution with parameter $1$ . Let

T=-\log X-\log Y.

By the previous problem,

f_T(t) = te^{-t} \mathbb{1}_{t \geq 0}.

Also,

f_{T,Z}(t, z) = te^{-t} \mathbb{1}_{t \geq 0} \mathbb{1}_{0 \leq z \leq 1}.

It remains to prove that $R=TZ$ has the exponential distribution with parameter $1$ .

Use the change of variables $r=tz$ , $s=t$ . Then

J = \begin{pmatrix} \frac{\partial z}{\partial r} & \frac{\partial z}{\partial s} \\ \frac{\partial t}{\partial r} & \frac{\partial t}{\partial s} \end{pmatrix} = \begin{pmatrix} \frac{1}{s} & -\frac{r}{s^2} \\ 0 & 1 \end{pmatrix}.

Thus

|\det(J)| = \frac{1}{s}.

By the density transformation formula,

f_{R,S}(r,s)=e^{-s}\mathbb{1}_{0\leq r\leq s}.

Therefore

f_R(r)=\int_{\mathbb{R}}e^{-s}\mathbb{1}_{0\leq r\leq s}\,ds =\mathbb{1}_{r\geq 0}\int_r^{+\infty}e^{-s}\,ds =e^{-r}\mathbb{1}_{r\geq 0}.

Exercise 3.2

Note

There are three common ways to compute moments: direct integration, recursion or integration by parts, and generating functions. It is useful to compare the moments of the normal law and the semicircle law.

Problem

Let $X$ have the standard normal distribution, and let $Y$ have the standard Wigner semicircle law. Find all their moments.

Proof

Both random variables are symmetric, so we only need the moments of order $2k$ .

\begin{aligned} \mathbb{E}[X^{2k}] &= \int_{-\infty}^{+\infty} x^{2k} \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}x^2} \, dx \\ &= \frac{2^{k+\frac{1}{2}}}{\sqrt{2\pi}} \int_{0}^{+\infty} u^{k-\frac{1}{2}} e^{-u} \, du \quad (\text{let } u = \frac{x^2}{2}) \\ &= \frac{2^{k+\frac{1}{2}}}{\sqrt{2\pi}} \, \Gamma\!\left(k + \frac{1}{2}\right) \\ &= \frac{2^{k+\frac{1}{2}}}{\sqrt{2\pi}} \cdot \left( \frac{\sqrt{\pi} \, (2k-1)!!}{2^k} \right) \quad (\text{using } \Gamma(n+1)=n\Gamma(n) \text{ and } \Gamma(\tfrac12)=\sqrt{\pi}) \\ &= (2k-1)!!. \end{aligned}

For the semicircle law,

\begin{aligned} \mathbb{E}[Y^{2k}] &= 2 \int_{0}^{2} \frac{1}{2\pi} y^{2k} \sqrt{4 - y^2} \, dy \\ &\overset{\text{let } y = 2\sin\theta}{=} \frac{2^{2k+2}}{\pi} \int_{0}^{\frac{\pi}{2}} (\sin\theta)^{2k} \cos^2\theta \, d\theta \\ &= \frac{2^{2k+2}}{\pi} \int_{0}^{\frac{\pi}{2}} \left[ (\sin\theta)^{2k} - (\sin\theta)^{2k+2} \right] \, d\theta. \end{aligned}

From real analysis,

\begin{aligned} \int_{0}^{\frac{\pi}{2}} \sin^n x \, dx = \begin{cases} \dfrac{n-1}{n} \dfrac{n-3}{n-2} \cdots \dfrac{1}{2} \cdot \dfrac{\pi}{2}, & n \text{ positive even}, \\ \dfrac{n-1}{n} \dfrac{n-2}{n-3} \cdots \dfrac{1}{3} \cdot 1, & n \text{ positive odd}. \end{cases} \end{aligned}

Substitution gives the final answer

\mathbb{E}[Y^{2k}]=\frac{1}{k+1}\binom{2k}{k}.

Problem

The joint density of $(X,Y)$ is

f(x, y) = C^{-1} x (y - x) e^{-y}, \quad 0 \leq x \leq y < \infty.

Find the constant $C$ , the conditional density $f_{X|Y}$ , and the conditional expectation $\mathbb{E}[Y|X]$ .

Proof

\begin{aligned} &\int_{0}^{+\infty} \int_{0}^{y} Cx (y - x)e^{-y} \, dx \, dy \\ &= \int_{0}^{+\infty} Ce^{-y} \, dy \int_{0}^{y} x(y - x) \, dx \\ &= \int_{0}^{+\infty} C \cdot \frac{1}{6} y^3 e^{-y} \, dy \\ &= C \cdot \frac{1}{6} \cdot 3! \\ &= C. \end{aligned}

Thus $C=1$ .

\begin{aligned} f_{X|Y}(x) &= \frac{f(x, y)}{f_Y(y)} \\ &= \frac{x(y - x)e^{-y} \mathbb{1}_{0 \leq x \leq y}}{\frac{1}{6}y^3 e^{-y}} \\ &= \frac{6x(y - x)}{y^3} \mathbb{1}_{0 \leq x \leq y}. \end{aligned}

Similarly,

\begin{aligned} f_{Y|X}(y) &= \frac{f(x, y)}{f_X(x)} \\ &= \frac{x(y - x)e^{-y} \mathbb{1}_{0 \leq x \leq y}}{x e^{-x}} \\ &= (y - x)e^{-(y-x)} \mathbb{1}_{y \geq x}. \end{aligned}

Therefore

\begin{aligned} \mathbb{E}[Y \mid X] &= \int_{x}^{\infty} y (y - x) e^{-(y-x)} \, dy \\ &= X + 2. \end{aligned}

Problem

Let $g$ be continuously differentiable on $\mathbb{R}$ , and assume that both $g$ and $g'$ are bounded. Prove that

X \sim N(\mu, \sigma^2) \implies \mathbb{E}[(X - \mu)g(X)] = \sigma^2 \mathbb{E}[g'(X)].

Proof

It is enough to consider $Z\sim N(0,1)$ .

\begin{aligned} & \mathbb{E} f^{\prime}(Z)=\frac{1}{\sqrt{2 \pi}} \int_{\mathbb{R}} e^{-t^2 / 2} f^{\prime}(t) d t \\ & \quad=\frac{1}{\sqrt{2 \pi}} \int_0^{\infty} f^{\prime}(t) \int_t^{\infty} w e^{-w^2 / 2} d w d t-\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^0 f^{\prime}(t) \int_{-\infty}^t w e^{-w^2 / 2} d w d t \\ & \quad \overset{Fubini}{=} \frac{1}{\sqrt{2 \pi}} \int_0^{\infty} w e^{-w^2 / 2}\left[\int_0^w f^{\prime}(t) d t\right] d w-\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^0 w e^{-w^2 / 2}\left[\int_w^0 f^{\prime}(t) d t\right] d w \\ & \quad = \frac{1}{\sqrt{2 \pi}} \int_0^{\infty} w e^{-w^2 / 2}f(w) d w+\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^0 w e^{-w^2 / 2}f(w) d w \\ & \quad=\mathbb{E}[Z f(Z)] . \end{aligned}

Conversely, suppose $Z$ satisfies

\mathbb{E}(Zg(Z))=\mathbb{E}(g^{\prime}(Z)).

Consider the Stein equation

xf(x)-f^{\prime}(x)=h(x)-\mathbb{E}(h(G)),

where $G\sim N(0,1)$ . For a fixed $h$ , this is an ordinary differential equation in $f$ and has a unique solution. Take $h(x)=\mathbb{1}_{x\leq z}$ . Substituting $x=Z$ and taking expectations gives

0=\mathbb{E}(Zf(Z))-\mathbb{E}(f^{\prime}(Z))=\mathbb{P}(Z\leq z)-\mathbb{P}(G\leq z).

Thus $Z\overset{\mathrm{law}}{=}G$ .

Problem

Let $\{X_r : 1 \leq r \leq n\}$ be i.i.d. random variables with finite variance. Set

\overline{X} = \frac{1}{n} \sum_{k=1}^n X_k.

Find $\operatorname{Cov}(\overline{X}, X_k - \overline{X})$ .

Proof

\begin{aligned} &\operatorname{Cov}\left(\bar{X}, X_k-\bar{X}\right)\\ =&\operatorname{Cov}\left(\bar{X}, X_k\right) -\operatorname{Cov}\left(\bar{X}, \bar{X}\right)\\ =&\operatorname{Cov}\left(\frac{X_k}{n}, X_k\right) -\operatorname{Var}(\bar{X})\\ =&\frac{\operatorname{Var}(X_k)}{n}-\frac{1}{n^2}\operatorname{Var}\!\left(\sum_{k=1}^n X_k\right)\\ =&\frac{\operatorname{Var}(X_k)}{n}-\frac{1}{n}\operatorname{Var}(X_k)\\ =&0. \end{aligned}

Problem

Let $X$ be a nonnegative random variable. Prove that for every $r>0$ ,

\mathbb{E}[X^r] = \int_0^\infty r t^{r-1}\mathbb{P}(X \geq t)\,dt.

Proof

\begin{aligned} \mathbb{E}[X^r] &= \mathbb{E}\!\left[\int_0^X r t^{r-1}\,dt\right] \\ &= \mathbb{E}\!\left[\int_0^{+\infty} r t^{r-1}\mathbb{1}_{t \leq X}\,dt\right] \\ &= \int_0^{\infty} r t^{r-1}\mathbb{E}[\mathbb{1}_{t \leq X}]\,dt \qquad \text{(Fubini)} \\ &= \int_0^{\infty} r t^{r-1}\mathbb{P}(X \geq t)\,dt. \end{aligned}

Problem

For i.i.d. random variables $X$ and $Y$ , prove:

(1) $U=X+Y$ and $V=X-Y$ are uncorrelated, but need not be independent.

(2) If $X,Y\sim N(0,1)$ , then $U$ and $V$ are independent.

Proof

(1)

\mathbb{E}[(X+Y)(X-Y)]=\mathbb{E} X^2-\mathbb{E} Y^2=0.

Also,

\mathbb{E}[X+Y]\mathbb{E}[X-Y]=\mathbb{E}[X]^2-\mathbb{E}[Y]^2=0.

Thus $U$ and $V$ are uncorrelated. If $X,Y$ are independent and both have the Bernoulli distribution $B(1,\frac12)$ , then $X+Y$ and $X-Y$ are not independent.

(2)

\begin{aligned} & f_{X , Y}(x , y)=\frac{1}{2 \pi} e^{-\frac{1}{2} x^2-\frac{1}{2} y^2} \\ & \text { let } u=x+y \quad v=x-y \\ & J=\left(\begin{array}{cc} \frac{\partial x}{\partial u} & \frac{\partial x}{\partial v} \\[5pt] \frac{\partial y}{\partial u} & \frac{\partial y}{\partial v} \end{array}\right)=\left(\begin{array}{cc} \frac{1}{2} & \frac{1}{2} \\[5pt] \frac{1}{2} & -\frac{1}{2} \end{array}\right) \\ & |\operatorname{det} J|=\frac{1}{2} \\ & \begin{aligned} f_{U ,V}(u , v) & =\frac{1}{4 \pi} e^{-\frac{1}{4} u^2-\frac{1}{4} v^2} \\ & =\frac{1}{\sqrt{2 \pi} \cdot \sqrt{2}} e^{-\frac{1}{2}\left(\frac{u}{\sqrt{2}}\right)^2}\cdot \frac{1}{\sqrt{2 \pi} \cdot \sqrt{2}} e^{-\frac{1}{2}\left(\frac{v}{\sqrt{2}}\right)^2}. \end{aligned} \end{aligned}

Therefore $U$ and $V$ are independent.

Exercise 3.3

Note

Linear transformations of multivariate normal vectors are most cleanly described by covariance matrices. After a linear transformation, first compute the mean and covariance.

Problem

Let $(X,Y)$ have a bivariate standard normal distribution. Find:

(1) the joint density and marginal densities of $X+Y$ and $X-Y$ ;

(2) $\mathbb{E}[X-Y\mid X+Y]$ and $\operatorname{Var}(X-Y\mid X+Y)$ .

Proof

Set $U=X+Y$ and $V=X-Y$ , and suppose

(X,Y)\sim N(0,\Sigma).

For a bivariate standard normal vector, write

\Sigma=\left(\begin{array}{cc} 1 & \rho \\ \rho & 1 \end{array}\right).

Let

D=\left(\begin{array}{cc} 1 & 1\\ 1& -1 \end{array}\right).

Then

(U,V)=(X,Y)D.

By Theorem 3.3.3 in the notes,

(U,V)\sim N(0,D^T\Sigma D):=N(0,\Sigma').

A direct calculation gives

\Sigma'=\left(\begin{array}{cc} 2+2\rho & 0 \\ 0& 2-2\rho \end{array}\right).

Thus $U,V$ are independent, $U\sim N(0,2+2\rho)$ , and $V\sim N(0,2-2\rho)$ .

(2) Since $U,V$ are independent,

\mathbb{E}[V\mid U]=\mathbb{E}[V]=0,\qquad \operatorname{Var}(V\mid U)=\operatorname{Var}(V)=2-2\rho.

Problem

Let $X=(X_1,X_2,\cdots,X_n)$ have the multivariate normal distribution $N(0,\Sigma)$ , where $\Sigma=(\sigma_{ij})_{i,j=1}^n$ is positive definite. Prove that

U = \sum_{k=1}^n a_k X_k \text{ and } V = \sum_{k=1}^n b_k X_k \text{ are independent if and only if } \sum_{j,k=1}^n a_j b_k \sigma_{jk} = 0,

where $a_1,\cdots,a_n,b_1,\cdots,b_n$ are real numbers. Also find $\mathbb{E}[U\mid V]$ when $b_1,\cdots,b_n$ are not all zero.

Proof

(1) Linear combinations of the components of a multivariate normal vector are again jointly normal. Hence $(U,V)$ is bivariate normal. Clearly,

\mathbb{E}[U]=\mathbb{E}[V]=0.

Moreover,

\begin{aligned} &U,V\text{ independent}\\ \Leftrightarrow &\operatorname{Cov}(U,V)=0\\ \Leftrightarrow &\mathbb{E}[UV]=0\\ \Leftrightarrow &\sum_{j,k=1}^n a_jb_k\mathbb{E}[X_jX_k] =0\\ \Leftrightarrow &\sum_{j,k=1}^n a_jb_k\sigma_{jk} =0. \end{aligned}

(2) For normal variables, uncorrelatedness is equivalent to independence. Thus

\operatorname{Cov}\!\left(U-\frac{\operatorname{Cov}(U, V)}{\operatorname{Var}(V)}V , V\right)=0.

Hence $U-\frac{\operatorname{Cov}(U,V)}{\operatorname{Var}(V)}V$ is independent of $V$ . Therefore

\mathbb{E}\!\left[U-\frac{\operatorname{Cov}(U,V)}{\operatorname{Var}(V)}V\mid V\right] =\mathbb{E}\!\left[U-\frac{\operatorname{Cov}(U,V)}{\operatorname{Var}(V)}V\right]=0.

By linearity of conditional expectation,

\mathbb{E}[U \mid V] = \frac{\operatorname{Cov}(U, V)}{\operatorname{Var}(V)} \, V.

Substituting the covariances gives

\mathbb{E}[U \mid V] = \frac{\sum_{j,k} a_j b_k \sigma_{jk}}{\sum_{j,k} b_j b_k \sigma_{jk}} V.

Problem

Let $X_1,X_2,\cdots,X_n$ be i.i.d. $N(\mu,\sigma^2)$ random variables, and let $\overline{X}=\frac{1}{n}\sum_{i=1}^n X_i$ . Find $\rho(X_1,\overline{X})$ .

Proof

By assumption, $X_1,X_2,\ldots,X_n$ are i.i.d., with $\mathbb{E}(X_i)=\mu$ and $\operatorname{Var}(X_i)=\sigma^2$ . Also,

\overline{X} = \frac{1}{n} \sum_{j=1}^n X_j.

First compute the covariance:

\operatorname{Cov}(X_1, \overline{X}) = \operatorname{Cov}\left(X_1, \frac{1}{n} \sum_{j=1}^n X_j\right) = \frac{1}{n} \sum_{j=1}^n \operatorname{Cov}(X_1, X_j).

By independence, $\operatorname{Cov}(X_1,X_j)=0$ when $j\ne 1$ , while $\operatorname{Cov}(X_1,X_1)=\operatorname{Var}(X_1)=\sigma^2$ . Hence

\operatorname{Cov}(X_1, \overline{X}) = \frac{\sigma^2}{n}.

Also,

\operatorname{Var}(\overline{X}) = \frac{1}{n^2} \sum_{j=1}^n \operatorname{Var}(X_j) = \frac{\sigma^2}{n},

and

\operatorname{Var}(X_1)=\sigma^2.

Thus

\rho(X_1, \overline{X}) = \frac{\operatorname{Cov}(X_1, \overline{X})}{\sqrt{\operatorname{Var}(X_1) \operatorname{Var}(\overline{X})}} = \frac{\frac{\sigma^2}{n}}{\sqrt{\sigma^2 \cdot \frac{\sigma^2}{n}}} = \frac{1}{\sqrt{n}}.

Problem

Let $e$ be a fixed unit vector in $\mathbb{R}^n$ with $n\geq 2$ . Let $X\sim N(0,I_n)$ , where $I_n$ is the identity matrix. Let $Z$ be the square of the length of the projection of $e$ onto the line spanned by $X$ . Find the density of $Z$ .

Proof

The squared projection length is

Z = \frac{(e^\top X)^2}{\|X\|^2}.

Since $X\sim N(0,I_n)$ , by an orthogonal transformation we may assume that $e$ is the first coordinate vector $(1,0,\ldots,0)^\top$ . This is allowed because $X$ is spherically symmetric. Let $Y=(Y_1,Y_2,\ldots,Y_n)^\top$ be the transformed vector. Then $Y\sim N(0,I_n)$ , and

e^\top X = Y_1, \quad \|X\|^2 = \sum_{i=1}^n Y_i^2.

Thus

Z = \frac{Y_1^2}{Y_1^2 + Y_2^2 + \cdots + Y_n^2}.

Let $U=Y_1^2\sim \chi^2_1$ and $V=Y_2^2+\cdots+Y_n^2\sim \chi^2_{n-1}$ . Then $U$ and $V$ are independent, and

Z = \frac{U}{U+V}.

If $U\sim\chi^2_a$ and $V\sim\chi^2_b$ are independent, then

\frac{U}{U+V} \sim \operatorname{Beta}\left(\frac{a}{2}, \frac{b}{2}\right).

Here $a=1$ and $b=n-1$ , so

Z \sim \operatorname{Beta}\left(\frac{1}{2}, \frac{n-1}{2}\right).

Its density is

f_Z(z) = \frac{\Gamma\left(\frac{n}{2}\right)}{\Gamma\left(\frac{1}{2}\right) \Gamma\left(\frac{n-1}{2}\right)} z^{\frac{1}{2} - 1} (1-z)^{\frac{n-1}{2} - 1}, \quad 0 < z < 1.

Equivalently,

f_Z(z) = \frac{\Gamma\left(\frac{n}{2}\right)}{\sqrt{\pi} \, \Gamma\left(\frac{n-1}{2}\right)} z^{-1/2} (1-z)^{\frac{n-3}{2}}, \quad 0 < z < 1,

and it is $0$ elsewhere.

Exercise 3.4

Note

Complex Gaussian and GOE problems often use invariance. Find the symmetry first; it is usually cleaner than computing a density directly.

Problem

Let $Z\sim N_C(0,1)$ . Prove that

\mathbb{E}[Z^k \bar{Z}^l] = \begin{cases} k! & , \quad k = l, \\ 0 & , \quad k \neq l. \end{cases}

Proof

\begin{aligned} &\text { Using } f(z)=\frac{1}{\pi} \mathrm{e}^{-|z|^{2}}, \text { we have }\\ & \mathbb{E}\left[Z^{k} \bar{Z}^{l}\right]=\int_{\mathbb{C}} \frac{1}{\pi} \mathrm{e}^{-|z|^{2}} z^{k} \bar{z}^{l} \mathrm{~d} z \\ &\stackrel{z=re^{i\theta}}{=} \int_{0}^{2 \pi} \int_{0}^{+\infty} \frac{1}{\pi} \mathrm{e}^{-r^{2}}\left(r \mathrm{e}^{\mathrm{i} \theta}\right)^{k}\left(r \mathrm{e}^{-\mathrm{i} \theta}\right)^{l} r \mathrm{~d} r \mathrm{~d} \theta \\ & =\int_{0}^{2 \pi} \int_{0}^{+\infty} \frac{1}{\pi} r^{k+l+1} \mathrm{e}^{i(k-l) \theta} \mathrm{e}^{-r^{2}} \mathrm{~d} r \mathrm{~d} \theta \\ & =\left\{\begin{array}{ll} 2 \int_{0}^{+\infty} r^{2 k+1} \mathrm{e}^{-r^{2}} \mathrm{~d} r=\Gamma(k+1)=k!\quad\quad\quad & ,k=l, \\ 0 & ,k \neq l . \end{array}\right. \end{aligned}

Problem

Let $Z_1,Z_2\sim N_C(0,1)$ be independent.

$1$ Find the density of $Z_1/Z_2$ .

$2$ Use Exercise 3.4.1 to compute $\mathbb{E}[|Z_1-Z_2|^{2n}]$ for positive integers $n$ .

Proof

(1) Since $Z_1,Z_2\overset{\text{i.i.d.}}{\sim}N_\mathbb C(0,1)$ ,

f_{Z_1, Z_2}(z_1, z_2) = \frac{1}{\pi^2} e^{-|z_1|^2 - |z_2|^2}, \quad z_1, z_2 \in \mathbb{C}.

Let

W = \frac{Z_1}{Z_2}, \quad V = Z_2.

Then

Z_1 = WV, \quad Z_2 = V.

The Jacobian determinant of this transformation is $|v|^2$ . Hence

f_{W,V}(w, v) = f_{Z_1, Z_2}(wv, v) \cdot |v|^2 = \frac{|v|^2}{\pi^2} e^{-|v|^2(1 + |w|^2)}.

Integrating out $v$ gives the marginal density of $W$ :

f_W(w) = \int_{\mathbb{C}} f_{W,V}(w, v) \, dv.

Let $r=|v|$ . Then $dv=r\,dr\,d\theta$ , and the integral does not depend on $\theta$ :

f_W(w) = \frac{1}{\pi^2} \int_0^{\infty} \int_0^{2\pi} r^2 e^{-r^2(1+|w|^2)} \cdot r \, d\theta \, dr = \frac{2\pi}{\pi^2} \int_0^{\infty} r^3 e^{-r^2(1+|w|^2)} \, dr.

Let $a=1+|w|^2$ and $t=r^2$ . Then $r^3dr=\frac{t}{2}dt$ , so

\int_0^{\infty} r^3 e^{-a r^2} dr = \frac{1}{2} \int_0^{\infty} t e^{-a t} dt = \frac{1}{2a^2}.

Thus

f_W(w) = \frac{2\pi}{\pi^2} \cdot \frac{1}{2(1+|w|^2)^2} = \frac{1}{\pi (1+|w|^2)^2}.

$2$ Since $Z_1,Z_2\sim N_\mathbb C(0,1)$ are independent, $Z_1-Z_2\sim N_\mathbb C(0,2)$ . Let

Y=\frac{Z_1-Z_2}{\sqrt{2}}\sim N_C(0,1).

Then

|Z_1 - Z_2|^{2n} = 2^n |Y|^{2n} = 2^n(Y\bar Y)^n.

By Exercise 3.4.1 with $k=n,l=n$ ,

\mathbb{E}[(Y\bar Y)^n]=\mathbb{E}[Y^n\bar Y^n]=n!.

Therefore

\mathbb{E}[|Z_1-Z_2|^{2n}]=2^n n!.

Problem

(Moments of GOE) Let $H$ have the GOE $_n$ distribution, and set $a_k=\mathbb{E}[\operatorname{tr}(H^k)]$ . Compute the first six moments $a_k$ , $k=1,\dots,6$ .

Proof

By symmetry, only the even moments need to be considered. In random matrix notation, $\operatorname{tr}=\frac1n\operatorname{Tr}$ , where $\operatorname{Tr}$ is the usual trace and $n$ is the matrix dimension. By Wick's formula,

\begin{aligned} \mathbb{E}\operatorname{Tr}(H^{2k}) &=\sum_{i_1,\cdots , i_{2k}} \mathbb{E}( h_{i_1i_2}h_{i_2i_3}\cdots h_{i_{2k}i_1}) \\ &=\sum_{i_1,\cdots , i_{2k}} \sum_{\pi \in \mathcal{P}(2k)}\prod_{(p,q)\in \pi }\mathbb{E}(h_{i_pi_{p+1}}h_{i_qi_{q+1}})\\ &=\sum_{i_1,\cdots , i_{2k}} \sum_{\pi \in \mathcal{P}(2k)}\prod_{(p,q)\in \pi }\mathbb{E}(\delta_{i_pi_q}\delta_{i_{p+1}i_{q+1}}+\delta_{i_pi_{q+1}}\delta_{i_{p+1}i_q}). \end{aligned}

For $k=1$ ,

a_2 = \dfrac{1}{n}\sum_{i_1,i_2} \mathbb{E}[h_{i_1 i_2} h_{i_2 i_1}] = \dfrac{1}{n}\sum_{i_1,i_2} \mathbb{E}[h_{i_1 i_2}^2]=\dfrac{1}{n}(2n+(n^2-n))=n+1.

For $k=2$ ,

a_4 = \dfrac{1}{n}\sum_{i_1,i_2,i_3,i_4} \mathbb{E}[h_{i_1i_2} h_{i_2i_3} h_{i_3i_4} h_{i_4i_1}].

Wick's formula gives three pairings:

$(1,2)(3,4)$ : the term is nonzero only when $i_1=i_3$ . If $i_1=i_2=i_3=i_4$ , the contribution is $4$ ; if $i_1=i_2\neq i_4$ or $i_1=i_4\neq i_2$ , the contribution is $2$ ; if $i_1\neq i_2$ and $i_1\neq i_4$ , the contribution is $1$ . The total contribution is

4n+2n(n-1)+2n(n-1)+n(n-1)^2.

$(1,3)(2,4)$ : this is the crossing pairing. It is nonzero only when $i_1=i_3,i_2=i_4$ , or when $i_2=i_3,i_1=i_4$ . The total contribution is $4n+n(n-1)=n^2+3n$ .
$(1,4)(2,3)$ : this is the same as the first type, and contributes $n^3+2n^2+n$ .

After summing,

a_4=2n^2+5n+5.

For $k=3$ , cyclic symmetry gives five cases, represented by the pairings

(1,2)(3,4)(5,6),\quad (1,2)(3,6)(4,5),\quad (1,2)(3,5)(4,6),

(1,3)(2,5)(4,6),\quad (1,4)(2,5)(3,6).

These cases contain $2,3,6,3,1$ pairings respectively. For the first representative, $(1,2)(3,4)(5,6)$ :

\mathbb{E} h_{i_1i_2}h_{i_2i_3}\neq 0

if and only if $i_1=i_2,i_2=i_3$ , or $i_1=i_3$ ;

\mathbb{E} h_{i_3i_4}h_{i_4i_5}\neq 0

if and only if $i_3=i_4,i_4=i_5$ , or $i_3=i_5$ ;

\mathbb{E} h_{i_5i_6}h_{i_6i_1}\neq 0

if and only if $i_5=i_6,i_6=i_1$ , or $i_1=i_5$ .

There are eight cases, each contributing $1$ . For instance, choosing the second relation in all three places gives $i_1=i_3=i_5$ and free indices $i_2,i_4,i_6$ , so there are $n^4$ possibilities. Continuing in the same way gives

n^4+3n^2+3n^2+n.

The other four representative pairings contribute

n^4+3n^3+3n^2+n,\quad n^3+4n^2+3n,\quad 3n^2+5n,\quad n^3+4n^2+3n.

The total is

5n^4+22n^3+52n^2+41n.

M. Ledoux, in "A recursion formula for the moments of the Gaussian orthogonal ensemble", gives a five-term recursion for the exact GOE moments. Write

\mathbb{E}(\operatorname{Tr}(H^{2p}))=b_p^N=\sum_{k\geq 1}\eta_k(p)N^k.

Then

\begin{aligned} (p + 1)\eta_k(p) &= (8p - 2)\eta_{k-1}(p - 1) - (4p - 1)\eta_k(p - 1) \\ &\quad + p(2p - 3)(10p - 9)\eta_k(p - 2) - 8(2p - 3)\eta_{k-2}(p - 2) \\ &\quad + 8(2p - 3)\eta_{k-1}(p - 2) - 10(2p - 3)(2p - 4)(2p - 5)\eta_{k-1}(p - 3) \\ &\quad + 5(2p - 3)(2p - 4)(2p - 5)\eta_k(p - 3) \\ &\quad - 2(2p - 3)(2p - 4)(2p - 5)(2p - 6)(2p - 7)\eta_k(p - 4). \end{aligned}

Problem

Suppose $\{x_{ij}\}_{i,j=1}^n$ are i.i.d. $N(0,1)$ random variables, and let

X_n=(x_{ij})_{i,j=1}^n.

Construct the symmetric matrix

H=\frac{1}{\sqrt{2}}(X_n+X_n^t).

Prove that $H$ has the GOE distribution.

Proof

Since $H=\frac{1}{\sqrt{2}}(X_n+X_n^t)$ , its entries are

H_{ij}=\frac{1}{\sqrt{2}}(x_{ij}+x_{ji}).

Because $\{x_{ij}\}$ are i.i.d. $N(0,1)$ , we can compute the distribution of each entry of $H$ .

For diagonal entries,

H_{ii}=\frac{1}{\sqrt{2}}(x_{ii}+x_{ii})=\sqrt{2}\,x_{ii}.

Since $x_{ii}\sim N(0,1)$ ,

H_{ii}\sim N(0,2).

For off-diagonal entries with $i<j$ ,

H_{ij}=\frac{1}{\sqrt{2}}(x_{ij}+x_{ji}).

Since $x_{ij}$ and $x_{ji}$ are independent and both have distribution $N(0,1)$ , their sum has distribution $N(0,2)$ . Multiplying by $1/\sqrt2$ gives

H_{ij}\sim N(0,1).

There are $n$ diagonal entries and $\frac{n(n-1)}2$ off-diagonal entries, for a total of $\frac{n(n+1)}2$ independent entries.

The joint density is the product of the densities of these independent entries:

\begin{aligned} f(H) &= \prod_{i=1}^n \frac{1}{\sqrt{2\pi \cdot 2}} \exp\left(-\frac{H_{ii}^2}{4}\right) \times \prod_{i<j} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{H_{ij}^2}{2}\right) \\ &= 2^{-\frac{n}{2}} (2\pi)^{-\frac{n(n+1)}{4}} \exp\left(-\frac{1}{4}\sum_{i=1}^n H_{ii}^2 - \frac{1}{2}\sum_{i<j} H_{ij}^2\right). \end{aligned}

Since

\frac{1}{4}\sum_{i=1}^n H_{ii}^2+\frac{1}{2}\sum_{i<j}H_{ij}^2 =\frac14\operatorname{tr}(H^2),

we get

f(H)=2^{-\frac{n}{2}}(2\pi)^{-\frac{n(n+1)}4}e^{-\frac14\operatorname{tr}(H^2)}.

Problem

(Orthogonal invariance of GOE) Let $H$ have the GOE distribution. Prove that for every orthogonal matrix $Q$ , the matrix $QHQ^{-1}$ also has the GOE distribution.

Proof

Let $H\sim\mathrm{GOE}_n$ . Its joint density is

f(H)=C_n e^{-\frac14\operatorname{tr}(H^2)},

where

C_n=2^{-\frac n2}(2\pi)^{-\frac{n(n+1)}4}.

Set

\widetilde H=QHQ^{-1}=QHQ^t,

because $Q$ is orthogonal.

First, $\widetilde H$ is still symmetric:

\widetilde H^t=(QHQ^t)^t=QH^tQ^t=QHQ^t=\widetilde H.

The trace is invariant:

\operatorname{tr}(\widetilde H^2) =\operatorname{tr}(QHQ^tQHQ^t) =\operatorname{tr}(QH^2Q^t) =\operatorname{tr}(H^2Q^tQ) =\operatorname{tr}(H^2).

Thus the exponential part of the density is unchanged:

\exp\left(-\frac14\operatorname{tr}(\widetilde H^2)\right) =\exp\left(-\frac14\operatorname{tr}(H^2)\right).

For the Jacobian, $\widetilde H=QHQ^t$ is a linear transformation, so there exists a matrix $A$ such that

\operatorname{Vec}(\widetilde H)=A\operatorname{Vec}(H),

where $\operatorname{Vec}$ stacks the entries of a matrix into an $n^2$ -dimensional vector. A linear map is orthogonal exactly when it preserves the Euclidean norm. Here

\|\operatorname{Vec}(\widetilde H)\|_2^2=\operatorname{tr}(\widetilde H^2)=\operatorname{tr}(H^2)=\|\operatorname{Vec}(H)\|_2^2.

Thus $A$ is orthogonal, and the absolute value of the Jacobian determinant is $1$ .

The density after the change of variables is therefore

f(\widetilde H)=f(H)|\det(J)|^{-1}=f(H)=C_n e^{-\frac14\operatorname{tr}(\widetilde H^2)}.

Thus $\widetilde H=QHQ^{-1}$ also has the GOE distribution.

Exercise 4.1

Note

This section prepares tail integrals, Jensen's inequality, and moment bounds. Later we will turn them into probability bounds.

Problem

For a nonnegative random variable $X$ , prove that

\sum_{n=1}^\infty \mathbb{P}(X \geq n) \leq \mathbb{E}[X] \leq \sum_{n=1}^\infty \mathbb{P}(X \geq n) + 1.

Proof

For a nonnegative random variable $X$ ,

X=\sum_{i=0}^{\infty}X\mathbb 1_{i\leq X<i+1}.

Also,

i\mathbb{1}_{i\leq X<i+1}\leq X\mathbb{1}_{i\leq X<i+1}\leq (i+1)\mathbb{1}_{i\leq X<i+1}.

Hence

\begin{aligned} \mathbb{E}[X] &=\sum_{i=0}^{\infty}\mathbb{E}[X\mathbb{1}_{i\leq X<i+1}]\\ &\leq \sum_{i=0}^{\infty}(i+1)\mathbb{E}[\mathbb{1}_{i\leq X<i+1}]\\ &\leq 1+\sum_{i=0}^{\infty}i\mathbb{E}[\mathbb{1}_{i\leq X<i+1}]\\ &\leq 1+\sum_{i=1}^{\infty}\sum_{j=1}^{i}\mathbb{E}[\mathbb{1}_{i\leq X<i+1}]\\ &\quad \text{switch the order of summation}\\ &\leq 1+\sum_{j=1}^{\infty}\sum_{i=j}^{\infty}\mathbb{E}[\mathbb{1}_{i\leq X<i+1}]\\ &\leq 1+\sum_{j=1}^{\infty}\mathbb{P}(X\geq j). \end{aligned}

The other side is similar.

Problem

(Jensen's inequality) A function $u:\mathbb{R}\to\mathbb{R}$ is called convex if for every $a\in\mathbb{R}$ there exists $\lambda=\lambda_a$ such that

u(x)\geq u(a)+\lambda_a(x-a),\quad \forall x\in\mathbb{R}.

A convex function $u$ is called strictly convex if $\lambda_a$ is strictly increasing in $a$ .

Prove that if $u$ is convex and $X$ has an expectation, then

\mathbb{E}[u(X)]\geq u(\mathbb{E}[X]).

Prove that if $u$ is strictly convex and $\mathbb{E}[u(X)]=u(\mathbb{E}[X])$ , then $X$ is a constant with probability $1$ .

Proof

(1) Take $a=\mathbb{E}[X]$ . By convexity,

\mathbb{E}[u(X)] \geq \mathbb{E}[u(a)+\lambda_a(X-a)] =u(a)+\lambda_a(\mathbb{E}[X]-a) =u(a) =u(\mathbb{E}[X]).

(2) Equality can hold only when $X=\mathbb{E}[X]$ a.e. Hence $X$ is a constant with probability $1$ .

Problem

Let $X$ be a nonnegative random variable. Prove that for every $r>0$ ,

\mathbb{E}[X^r] = \int_0^\infty r x^{r-1} \mathbb{P}(X > x) \, dx.

Proof

\begin{aligned} \mathbb{E} [X^{r}] &=\mathbb{E} \int_{0}^{X} rt^{r-1}\,dt \\ &=\mathbb{E} \int_{0}^{\infty} r t^{r-1}\mathbb{1}_{X>t}\,dt \\ &\quad \text{use Fubini's theorem to switch the order of integration} \\ &=\int_{0}^{\infty} r t^{r-1}\mathbb{E}[\mathbb{1}_{X>t}]\,dt \\ &=\int_{0}^{\infty} r t^{r-1}\mathbb{P}(X>t)\,dt. \end{aligned}

Problem

Fix $r>0$ .

If $\mathbb{E}[|X|^r]<\infty$ , prove that

\lim_{x\to+\infty}x^r\mathbb{P}(|X|\geq x)=0.

\lim_{x\to+\infty}x^r\mathbb{P}(|X|\geq x)=0,

prove that $\mathbb{E}[|X|^s]<\infty$ for every $s\in(0,r)$ . Does $\mathbb{E}[|X|^r]<\infty$ necessarily hold? Give a reason or a counterexample.

Proof

(1) We have

x^r\mathbb{P}(|X|\geq x) =x^r\int_{\mathbb R}\mathbb{1}_{|X|\geq x}\,d\mathbb{P} \leq \int_{\mathbb R}|X|^r\mathbb{1}_{|X|\geq x}\,d\mathbb{P}.

Also, $|X|^r\mathbb{1}_{|X|\geq x}\leq |X|^r$ , and $|X|^r$ is integrable. By the dominated convergence theorem,

\lim_{n\to\infty}\int_{\mathbb R}|X|^r\mathbb{1}_{|X|\geq n}\,d\mathbb{P} =\int_{\mathbb R}\lim_{n\to\infty}|X|^r\mathbb{1}_{|X|\geq n}\,d\mathbb{P} =0.

This proves the claim.

(2) For every $\varepsilon>0$ , choose $M$ such that for all $x>M$ ,

x^r\mathbb{P}(|X|\geq x)<\varepsilon.

By the tail integral formula,

\begin{aligned} \mathbb{E}[|X|^s] &=\int_{0}^{\infty} s t^{s-1}\mathbb{P}(|X|>t)\,dt\\ &=\int_{0}^{M} s t^{s-1}\mathbb{P}(|X|>t)\,dt +\int_{M}^{\infty} s t^{s-1}\mathbb{P}(|X|>t)\,dt\\ &\leq C_1+\int_{M}^{\infty} s t^{s-1}\frac{\varepsilon}{t^r}\,dt\\ &= C_1+C_2\varepsilon. \end{aligned}

Thus $\mathbb{E}[|X|^s]<\infty$ .

However, $\mathbb{E}[|X|^r]<\infty$ need not hold. For example, take

\mathbb{P}(|X|\geq x)\sim \frac{1}{x^r\log x}.

The tail integral formula shows that $\mathbb{E}[|X|^r]$ diverges.

Exercise 3.5

Note

The normal tail probability has order $e^{-x^2/2}/x$ . Integration by parts is the main tool here.

Problem

For $X\sim N(0,1)$ , prove the standard normal tail estimate

\dfrac{x}{x^2+1}\dfrac{1}{\sqrt{2\pi }}e^{-x^2/2}\leq \mathbb{P}(X\geq x)\leq \dfrac{1}{x}\dfrac{1}{\sqrt{2\pi }}e^{-x^2/2}.

Proof

The upper bound follows from

\mathbb{E}(\mathbb{1}_{X\geq x})\leq \mathbb{E}\left(\frac{X}{x}\mathbb{1}_{X\geq x}\right).

Thus

\mathbb{P}\{X>x\} \leq \frac{1}{\sqrt{2\pi}}\int_x^\infty \frac{u}{x}e^{-u^2/2}\,du =\frac{1}{x}\frac{1}{\sqrt{2\pi}}e^{-x^2/2}.

For the lower bound, define

f(x)=xe^{-x^2/2}-(x^2+1)\int_x^\infty e^{-u^2/2}\,du.

We have $f(0)<0$ , $\lim_{x\to\infty}f(x)=0$ , and

f'(x)=(1-x^2+x^2+1)e^{-x^2/2} -2x\int_x^\infty e^{-u^2/2}\,du =-2x\left(\int_x^\infty e^{-u^2/2}\,du-\frac{e^{-x^2/2}}{x}\right).

For $x>0$ , the upper bound already proved gives $f'(x)>0$ . Hence $f(x)\leq 0$ , which is the desired lower bound.

Define functions $H_n$ , $n\geq 0$ , by $H_0=1$ and $(-1)^nH_n\phi=\phi^{(n)}$ . Prove that $H_n(x)$ is a degree $n$ polynomial with leading term $x^n$ , and that

\int_{-\infty}^{+\infty}H_m(x)H_n(x)\phi(x)\,dx = \begin{cases} m!, & m=n,\\ 0, & m\neq n. \end{cases}

Also prove

\sum_{n=0}^{\infty}H_n(x)\frac{t^n}{n!}=e^{xt-\frac12t^2}.

Solution

By definition,

(-1)^nH_n(x)\phi(x)=\phi^{(n)}(x),

where

\phi(x)=\frac{1}{\sqrt{2\pi}}e^{-x^2/2}.

We compute $\phi'=-x\phi$ and $\phi''=-\phi+x^2\phi'$ . By induction,

H_n(x)=-H_{n-1}'(x)+xH_{n-1}(x).

It follows that $H_n$ is monic of degree $n$ .

For orthogonality, assume $m\geq n$ . Then

\int H_mH_n\phi\,dx=(-1)^n\int H_m\phi^{(n)}\,dx.

Integrating by parts $n$ times, with boundary terms equal to $0$ , gives

(-1)^n\int H_m\phi^{(n)}\,dx=\int H_m^{(n)}\phi\,dx.

If $m=n$ , then $H_m^{(n)}=n!$ ; if $m>n$ , orthogonality follows by the same integration-by-parts argument. Since $\int\phi=1$ , the stated relation follows.

Finally, by Taylor expansion,

\phi(x-t)=\sum_{n=0}^{\infty}\frac{(-t)^n}{n!}\phi^{(n)}(x).

But

\phi(x-t)=\frac{1}{\sqrt{2\pi}}e^{-(x-t)^2/2}=\phi(x)e^{xt-t^2/2}.

Substituting $\phi^{(n)}(x)=(-1)^nH_n(x)\phi(x)$ and cancelling $\phi(x)$ gives the generating function.

For positive integers $m,n$ , compute the correlation coefficient $\rho(H_m(X),H_n(Y))$ .

Solution

From the generating function in (1),

e^{Xt-\frac{t^2}{2}}=\sum_{i=0}^{\infty}H_i(X)\frac{t^i}{i!}, \quad e^{Ys-\frac{s^2}{2}}=\sum_{j=0}^{\infty}H_j(Y)\frac{s^j}{j!}.

First,

\mathbb{E}\left(e^{Xt-\frac{t^2}{2}}\right) =e^{-\frac{t^2}{2}}\mathbb{E}(e^{tX}) =1.

Comparing coefficients gives $\mathbb{E}(H_0(X))=1$ and $\mathbb{E}(H_n(X))=0$ for $n\geq 1$ .

Next consider the joint generating function:

\mathbb{E}\left(e^{Xt-\frac{t^2}{2}}e^{Ys-\frac{s^2}{2}}\right) =e^{-\frac{t^2+s^2}{2}}\mathbb{E}(e^{tX+sY}).

Since $(X,Y)$ is standard bivariate normal, $tX+sY\sim N(0,t^2+s^2+2\rho ts)$ , so

\mathbb{E}(e^{tX+sY})=e^{\frac12(t^2+s^2+2\rho ts)}.

Hence

\mathbb{E}\left(e^{Xt-\frac{t^2}{2}}e^{Ys-\frac{s^2}{2}}\right) =e^{\rho ts}.

On the other hand, expanding the generating functions gives

\mathbb{E}\left(\sum_{i=0}^{\infty}H_i(X)\frac{t^i}{i!} \sum_{j=0}^{\infty}H_j(Y)\frac{s^j}{j!}\right) =\sum_{i,j=0}^{\infty}\mathbb{E}[H_i(X)H_j(Y)]\frac{t^is^j}{i!j!}.

Since

e^{\rho ts}=\sum_{n=0}^{\infty}\frac{(\rho ts)^n}{n!} =\sum_{n=0}^{\infty}\rho^n\frac{t^ns^n}{n!},

comparing the coefficients of $t^is^j$ gives

\mathbb{E}[H_i(X)H_j(Y)] = \begin{cases} \rho^n n!, & i=j=n,\\ 0, & i\neq j. \end{cases}

Together with $\mathbb{E}(H_n(X))=\mathbb{E}(H_n(Y))=0$ for $n\geq 1$ and

\operatorname{Var}(H_n(X))=\mathbb{E}[H_n^2(X)]=n!,

we get

\rho(H_m(X),H_n(Y)) = \begin{cases} \rho^n, & m=n\geq 1,\\ 0, & m\neq n. \end{cases}

Let $P(x)$ and $Q(y)$ be nonconstant polynomials. Prove that

|\rho(P(X),Q(Y))|\leq |\rho(X,Y)|.

Solution

Expand $P$ and $Q$ in Hermite polynomials:

P(x)=\sum_{i=1}^{k}a_iH_i(x), \quad Q(y)=\sum_{j=1}^{l}b_jH_j(y).

The constant terms do not affect covariance. By (2),

\operatorname{Cov}(P(X),Q(Y)) =\sum_{i=1}^{\min(k,l)}a_ib_i i!\rho^i.

Also,

\operatorname{Var}(P(X))=\sum_{i=1}^{k}a_i^2i!, \quad \operatorname{Var}(Q(Y))=\sum_{j=1}^{l}b_j^2j!.

It remains to prove

\left|\sum_i a_ib_i i!\rho^i\right| \leq |\rho|\sqrt{\sum_i a_i^2i!}\sqrt{\sum_i b_i^2i!}.

By Cauchy-Schwarz and $|\rho|^i\leq |\rho|$ for $i\geq 1$ and $|\rho|<1$ ,

\begin{aligned} \left|\sum_{i=1} a_ib_i i!\rho^i\right| &\leq \sum_{i=1}|a_i||b_i|i!|\rho|^i\\ &\leq |\rho|\sum_{i=1}(|a_i|\sqrt{i!})(|b_i|\sqrt{i!})\\ &\leq |\rho|\sqrt{\sum_{i=1}a_i^2i!}\sqrt{\sum_{i=1}b_i^2i!}. \end{aligned}

Hence $|\rho(P,Q)|\leq |\rho|$ , where $|\rho|=|\rho(X,Y)|$ .

End-of-chapter check

The original problems and solutions in this chapter come from the corresponding TeX source files.
You can first read only the problem boxes, write down the main identities, and then open the proof or solution.
If a conclusion uses independence, countable additivity, a change-of-variables formula, or a moment condition, it is worth marking that point explicitly.