knowledge distilling

deeplearning

---

Distilling the Knowledge in a Neural Network

Distillation

$$q_i=\frac{exp(\frac{z_i}{T})}{\sum_jexp(\frac{z_i}{T})}$$
T:温度系数，用来让softmax更加soft

Matching logits is a special case of distillation

target probabilities:$p_i=\frac{exp(\frac{v_i}{T})}{\sum_jexp(\frac{v_j}{T})}$,

$$loss=\sum_i p_ilog(q_i)=\sum_i p_ilog(\frac{exp(\frac{z_i}{T})}{\sum_jexp(\frac{z_i}{T})})$$
$$\frac{\partial loss}{\partial z_i}=\frac{1}{T}(q_i-p_i)=\frac{1}{T}(\frac{exp(\frac{z_i}{T})}{\sum_jexp(\frac{z_i}{T})}-\frac{exp(\frac{v_i}{T})}{\sum_jexp(\frac{v_j}{T})})$$
当T很大时：
$$\frac{\partial loss}{\partial z_i}=\frac{1}{T}(\frac{1+(\frac{z_i}{T})}{N+\sum_j(\frac{z_i}{T})}-\frac{1+(\frac{v_i}{T})}{N+\sum_j(\frac{v_j}{T})})$$
假定$z_i$是0均值的，那：
$$\frac{\partial loss}{\partial z_i}=\frac{1}{NT^2}(z_i-v_i)$$
这样就可以认为优化的loss是mse

A Gift from Knowledge Distillation:Fast Optimization, Network Minimization and Transfer Learning

本文思想是学习一种认知的过程
对$F^1\in \mathbb{R}^{h\times w\times m}$,$F^2\in \mathbb{R}^{h\times w\times n}$,

$$G_{ij}(x;W)=\sum_{s=1}^h\sum_{t=1}^w \frac{F^1_{s,t,i}(x;W)\times F^2_{s,t,j}(x;W)}{h\times w}$$
实验没法证明fast optimization

Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

Maximum Mean Discrepancy

$$L_{MMD^2}=||\frac{1}{N}\sum_{i=1}^N\phi(x^i)-\frac{1}{M}\sum_{i=1}^M\phi(y^j)||_2^2$$
论文中，作者验证了三种常用的核函数，他们分别是

线性核:$k(x,y)=x^Ty$
多项式核:$k(x,y)=(x^Ty+c)^d$, 文中采用$d=2$, $c=0$
高斯核:$k(x, y)=exp(-\frac{|x-y|^2}{2\sigma^2})$:高斯核中$\sigma$设为两网络特征的平均距离
利用kernel trick将feature map映射到高维空间，是因为他认为attention map的做法粒度还是比较大。当kernel取二阶多项式核的时候，根据gram矩阵的性质，feature map的channel之间的内积可以转化为feature map空间的相关性，即让student模型学习teacher模型的feature map通道之间的关系或者空间像素点之间的关系。

$$L_{MMD^2}=\frac{1}{{C_T}^2}\sum_{i=1}^{C_T}\sum_{i'=1}^{C_T}k(\frac{f_T^i}{||f_T^i||_2},\frac{f_T^{i'}}{||f_T^{i'}||_2})\\ +\frac{1}{{C_S}^2}\sum_{j=1}^{C_S}\sum_{j'=1}^{C_S}k(\frac{f_T^j}{||f_T^j||_2},\frac{f_T^{j'}}{||f_T^{j'}||_2})\\ -\frac{2}{C_TC_S}\sum_{i=1}^{C_T}\sum_{j=1}^{C_S}k(\frac{f_T^i}{||f_T^i||_2},\frac{f_T^{j}}{||f_T^{j}||_2})$$
效果如下图：

但是Resnet1001不应该这么高的精度，BN-Inception也没有达到它的精度