归一化确定均值和方差且最大化熵的概率分布为 正太分布（借助mathematica）

PRML 机器学习

---

《PRML》习题2.14，把多元概率分布改为了一维的，就是说只证一维的

概率分布$p(x)$的熵为

$$H[x] = -\int p(x)\ln p(x)\mathrm{d}x$$
现在对这个概率分布$p(x)$有三个约束条件：
$$\begin{align*} \int p(x) \mathrm{d}x = 1 \qquad & // 概率分布特性 \\ \int p(x)x \mathrm{d}x = \mu \qquad & // 均值已知 \\ \int p(x)(x-\mu)^2 \mathrm{d}x = \sigma^2 \qquad & //方差已知 \end{align*}$$
求证，使熵最大时的概率分布$p(x)$为正太分布

要求概率分布$p(x)$的具体形式，很容易想到用变分法来解，把熵看作是函数$p$的一个泛函$H[p]$，带约束的极值，引入拉格朗日乘数法

$$\begin{align*} F(H,a,b,c) & = -\int p(x)\ln p(x) \mathrm{d}x + a\left( \int p(x) \mathrm{d}x -1 \right) \\ &+ b\left( \int p(x) x\mathrm{d}x -\mu \right) + c\left( \int p(x)(x-\mu)^2\mathrm{d}x-\sigma^2 \right) \\ \frac{\delta F}{\delta p} & = -(1+\ln p(x)) + a + bx + c(x-\mu)^2 = 0 \\ \frac{\partial F}{\partial a} & = \int p(x) \mathrm{d}x -1 = 0 \\ \frac{\partial F}{\partial b} & = \int p(x) x\mathrm{d}x -\mu = 0 \\ \frac{\partial F}{\partial c} & = \int p(x)(x-\mu)^2\mathrm{d}x-\sigma^2 = 0 \end{align*}$$

整理得到

$$\begin{align*} \ln p(x) & = c(x-\mu)^2 + bx +a - 1 \\ p(x) & = e^{c(x-\mu)^2 + bx +a - 1} \end{align*}$$

然后带入上面的约束

$$\begin{align*} \int p(x)\mathrm{d}x & = \int e^{c(x-\mu)^2 + bx +a - 1} \mathrm{d}x \\ & = \frac{e^{-1+a-\frac{b^2}{4c}+b \mu}\sqrt{\pi}}{\sqrt{-c}} \quad // 借助mathematica \\ & = 1 \end{align*}$$

$$\begin{align*} \int p(x)x\mathrm{d}x & = \int e^{c(x-\mu)^2 + bx +a - 1}x \mathrm{d}x \\ & = \frac{e^{-1 +a -\frac{b^2}{4c}+b \mu}\sqrt{\pi}(b-2c\mu)}{2(-c)^{\frac{3}{2}}} \quad // 借助mathematica \\ & = \mu \end{align*}$$

$$\begin{align*} \int p(x)x\mathrm{d}(x-\mu)^2 & = \int e^{c(x-\mu)^2 + bx +a - 1}(x-\mu)^2 \mathrm{d}x \\ & = \frac{(b^2-2c)e^{-1+a-\frac{b^2}{4a}+b \mu}\sqrt{\pi}}{4(-c)^{\frac{5}{2}}} \quad // 借助mathematica \\ & = \sigma^2 \end{align*}$$

最后再借助$mathematica$求解方程组

得到

$$\begin{align*} c & = -\frac{1}{2\sigma^2} \\ b & = 0 \\ a & = \frac{1}{4}\ln \frac{e^4}{4\pi^2\sigma^4} \\ & = \frac{1}{4}\left( 4-\ln (4\pi^2\sigma^4)\right) \\ & = 1 - \ln (2\pi\sigma^2)^{\frac{1}{2}} \end{align*}$$

带入上面的方程得

$$\begin{align*} p(x) & = e^{c(x-\mu)^2 + bx +a - 1} \\ & = e^{-\frac{(x-\mu^2)}{2\sigma^2} + 1-\ln(2\pi\sigma^2)^{\frac{1}{2}}-1} \\ & = e^{-\frac{(x-\mu^2)}{2\sigma^2}}\cdot e^{\ln(2\pi\sigma^2)^{-\frac{1}{2}}} \\ & = (2\pi\sigma^2)^{-\frac{1}{2}} \cdot e^{-\frac{(x-\mu^2)}{2\sigma^2}} \\ & = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu^2)}{2\sigma^2}} \end{align*}$$

泛函求导

泛函求导下次再补充
可以参照：泛函求导

Others

$Mathematica$代码
HN.nb10.2kB