fobos

未分类

$w_{t+\frac{1}{2}} = w_t - \eta_t g_t\\ w _{t+1} = \underset{w}{\arg \min} \{\frac{1}{2} ||w-w_{t+\frac{1}{2}}||^2 + \eta_{t+\frac{1}{2}}r(w)\}$

第一项是标准梯度下降；第二项是微调，处理正则化，产生稀疏性。rr 是正则函数。

$\textbf{0} \in \partial \bigg\{\frac{1}{2} \big|\big|w-w_{t+\frac{1}{2}} \big|\big|^2 + \eta_{t+\frac{1}{2}} r(w) \bigg\}$

$\textbf 0 \in w_{t+1} - w_t + \eta_t g_t^f + \eta_{t+\frac{1}{2}} \partial r(w_{t+1}) \\ \textbf 0= w_{t+1} - w_t + \eta_t g_t^f + \eta_{t+\frac{1}{2}} g_{t+1}^r$

$\textbf{0} \in \partial \bigg\{\frac{1}{2} \big|\big|w-w_{t+\frac{1}{2}} \big|\big|^2 + \eta_{t+\frac{1}{2}} r(w) \bigg\}$
$a_{ij}=\frac{exp(a_{ij}')}{\sum_{(i,j) \in R_x}exp(a_{ij}')}$

$\textbf{0} \in \partial \bigg\{\frac{1}{2} \big|\big|w-w_{t+\frac{1}{2}} \big|\big|^2 + \eta_{t+\frac{1}{2}} r(w) \bigg\}$

$\textbf 0 \in w_{t+1} - w_t + \eta_t g_t^f + \eta_{t+\frac{1}{2}} \partial r(w_{t+1}) \\ \textbf 0= w_{t+1} - w_t + \eta_t g_t^f + \eta_{t+\frac{1}{2}} g_{t+1}^r$