Implementing SGD in Minima

[bhavnicksm]

Introduction

We need to write our own SGD implementation from Pytorch. This issue is to keep all the resources for writing optimizers in one place in PyTorch.

We want to create a BaseOptimiser, which would have general commands in the future and can be inherited for every optimizer. For e.g. instead of momentum, we wish to enable Nesterov acceleration, enabling nesterov=True.

Resources

• Blog on writing custom optimisers by Daniel Mcneela
• Source code of Torch.Optim.SGD
• Source code of Torch.Optim.Adam
• Blog on difference of PyTorch Implementation and Normal SGD
• Ways to visualize a optimizer
• LabLab's implementation of the Adam and other Optimisers