Step update rule
From Piki
The Step update rule is the most basic form of a first order gradient descent rule.
| Step | |
| Family | Gradient |
| Direction | Backpass |
| Deployable | Yes |
| Supervised | Yes |
Contents |
Usage
The Step Rule is used with error backpropagation based systems. It uses propagated error information together with gradient information to progress towards an error minimum. If the update rule is in batch mode (i.e. receives multiple samples at once) it also uses a technique called momentum. It means that previous update is added to the current, scaled with a momentum factor. This means that the update can move faster along flat surfaces, resulting in a faster update.
Algorithm
Weight update:
where 
is the learning rate and 
is the momentum factor.
Settings
The settings can be modified using the settings browser.
| Step settings |
|---|
|
|
General advice
- Large step size equals faster, but more imprecise learning. Larger step sizes may be desirable early on in the learning process while it may be beneficial to reduce them later for fine tuning.
- In standard neural network topologies where you have a straight line of connected weight layers and function layers the effect of the step will be altered in each layer due to the non-linear compression that the functions perform. In practical terms it means that you can have larger step sizes early in the chain and smaller ones later on to balance it out.
See also
- Update rule - General article on update rules.
- List of update rules - List of all update rules.