Zero Order implementation does not converge in CIFAR-10 dataset.

[amritansh6]

Hi I have implemented Mezo in cifar-10 dataset but the model does not seems to be converging even after large epochs, by taking appropriate parameters it seems to be going down very slowly. And if you increase learning rate it crashes after a certain point giving Nan Values.

[lebrice]

Hey there, I'm not an author of the paper or involved in this repo at all (I'm just some researcher who read the paper and finds it interesting), but why would you expect Mezo to learn CIFAR10? From the paper, the algorithm seems only applicable for the fine-tuning of LLMs.

Also, if you're interested, I made a small, neat, and well-tested implementation of MeZO (as far as I understand it) here: https://github.com/lebrice/MeZO/tree/main
This hasn't been reviewed by the authors yet, so take it with a grain of salt, but it should make it very easy to play around with it.

Hope that helps! :)