Ben's Mini Deep Learning Framework

Description

I'm building this mini DL framework to get a better understanding of how things work under the hood. This is part of my ongoing efforts to keep my brain sticky (or neuroplastic). Your brain is the most important neural network, after all.

setup.py

This adds a .pth file to the site-packages directory containing the project's root dir. This allows for absolute imports anywhere without having to install the package or use sys.path.append(). Using this right now because I know it works, but I plan to try pip install --editable . in the future.

Completed

• Add a Dense layer
• Add auto differentiation
• Add basic loss functions
• Fix autodiff for scalars and shape mismatches
• fix backward_pow func (need to handle edge cases)
• Test backprop on dense layer
• Add optimizers
• Add checks for numerical instability
• Finish replicating/reverse engineering Karpathy's micrograd
• make a nn module that mirrors micrograd but with my logic
• update my backward method to use Top sort?
• make a test suite that mirrors micrograd/write sanity checks for Tensor class
• figure out why certain operation chains are failing and others aren't
• Update nn to use new tensor class
• Add activation functions with tests
• Test log
• Test exp
• Test clip
• fix tensor backward add method
• clean out old methods in Tensor once fully implemented and tested
• Test the MLP on a simple synthetic dataset (train/val), get it converging, add necessary methods
• update no_grad context manager via thread local approach
• fix the reshape method, and test
• write a test for mean
• make a sequential model class like pytorch
• fix the reshape problem in pytorch that makes a 1D bias vector into 2D (this is bad)
  • make a simple test case fot this with a 1D bias vector

TODO TASKS

• Just found this amazing book, Understanding Deep Learning with detailed explanations and example implementations. Putting this here so I can use it as a guide moving forward.

• softmax backprop stuff

• get MNISTv4 working with my lib

  • finish adding subscriptable indexing to tensor class (getitem backward)
  • fix clip test
  • write a test for dot product
  • write a test for sum with axis and keepdims
  • add keepdims to sum, max
  • write a test for max with axis and keepdims
  • write softmax and log softmax forward test
  • fix BinaryCELoss test
  • write log softmax func and test
  • write negative log likelihood func and test
  • write CCELoss
  • get CatCELoss test passing

• add keepdims to rest of reduction ops

• add a simple dataloader, with option to load all in memory at once

• write tests for the sequential model class

• figure out what basic layers I need to implement

• Clean up unused code in lib

• Add data handling

• Add evaluation

• Add dynamic lr optimizer

• Add regularization

• Add examples and docs

• write a test for the new no_grad context manager

TODO IDEAS

• watch this video by Karpathy on backprop and learn from/copy his abstractions
• replicate some of these models/demos with this lib
• reverse engineer some of these operator abstractions to see how they work
• Look into this jacobian matrix optimization

Backlog