FTLO is a optimizer for training neural networks

FTLO是一个用于训练神经网络的优化器算法

$$ v \leftarrow \beta_2 v + (\beta_2 \cdot \gamma) g $$

$$ \theta \leftarrow \theta - \eta g - \alpha v $$

Here, the coefficients $\alpha$ and $\gamma$ decay with the time step $t$ (starting from 1):

其中，系数 $\alpha$ 和 $\gamma$ 随时间步 $t$（从 $1$ 开始）进行衰减：

• Momentum coefficient $\alpha$ | 动量系数 $\alpha$：

$$ \alpha = \frac{\beta_1}{\eta \cdot (t+1)^P} $$

• Learning rate $\gamma$ for $v$ update | $v$ 更新学习率 $\gamma$：

$$ \gamma = \frac{\eta}{(t+1)^Q} $$

$g \text{ is the gradient, and } t \text{ is the current time step.}$

$g \text{ 为梯度，} t \text{ 为当前时间步。}$

We identified a parameter combination that outperforms Adam through comparative experiments with fixed random seeds on the MNIST task

我们在 MNIST 任务上，通过固定随机种子对比实验，找到了超越 Adam 的参数组合

The FTLO performs well in tasks far from the optimal solution or those that are high-dimensional and complex (such as MNIST). However, when the initial point is very close to the optimum (for example, starting at $(0, 0)$ for the Rosenbrock function), its momentum term may cause severe oscillations.

FTLO在远离最优解或高维、复杂任务（如 MNIST）中表现出色。然而，当初始点非常接近最优解（例如 Rosenbrock 函数的 $(0, 0)$ 起点）时，其动量项可能导致剧烈振荡。

Adjusting the decay parameters according to the initial conditions is key to using FTLO:

根据初始条件调整衰减参数，是使用 FTLO 的关键：

Since the momentum term $\alpha v$ and the $v$ update learning rate $\gamma$ in the FTLO may lead to aggressive update steps during the early stages of training, we strongly recommend applying gradient clipping before optimizer.step() to ensure numerical stability across various tasks (especially those prone to gradient explosion).

由于FTLO的动量项 $\alpha v$ 和 $v$ 更新学习率 $\gamma$ 在训练初期可能导致激进的更新步骤，为了确保在各种任务（尤其具有梯度爆炸风险的任务）上的数值稳定性，我们强烈建议在 optimizer.step() 之前应用梯度裁剪

In our MNIST experiments, we used CLIP_NORM = 1.0

在 MNIST 实验中，我们使用了 CLIP_NORM = 1.0