SwiftTD: A Fast and Robust Algorithm for Temporal Difference Learning
Khurram Javed, Arsalan Sharifnassab, Richard S. Sutton
SwiftTD is a fast and robust algorithm for temporal difference learning. It can learn from large step-sizes effectively, without diverging. They key to its success is (1) its ability to increase the step-size of important signals, while reducing it for less important ones, and (2) a bound on the rate of learning that prevents updates that are too large, and can cause numerical instability and divergence. This demo highlights the capabilites of SwiftTD for learning to predict sum of discounted future rewards. The user can select an Atari game, and visualize the learning process. The user can also change the hyperparameters of the algorithm, and see the effect on learning. Finally, the user can visualize the step-sizes that the algorithm learns.
Agent state construction
The canvas above shows the gameplay frame and the meta-learned step-sizes. The step-sizes tell us which pixels are getting credit for learning. If you let the learner run for a while, you will see that it assigns credit to pixels that are important for predicting the onset of reward.
Metrics
We visualize the reward signal, the predictions, and the rate of learning.
Predictions
This graph shows the predictions of the system and the reward. The left y-axis (-3 to +3) is for predictions, and the right y-axis is for the reward. After learning, the agent's prediction should anticipate the onset of reward.
Rate of learning
The rate of learning is the effective speed at which the system learns. A value of 0.5 means the learner halves a prediction's error after an update; a value above 1 means it overshoots the target. Ideally we don't want to overshoot.