Skip to content

Latest commit

 

History

History
65 lines (42 loc) · 4.24 KB

README.md

File metadata and controls

65 lines (42 loc) · 4.24 KB

[ICML 2024 Spotlight] Sparse and Structured Hopfield Networks

Official implementation of the paper Sparse and Structured Hopfield Networks accepted at International Conference on Machine Learning, 2024.

Saul Santos, Vlad Niculae, Daniel McNamee and André Martins

Abstract: Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.


If you use this code in your work, please cite our paper.


Resources

All material is made available under the MIT license. You can use, redistribute, and adapt the material for non-commercial purposes, as long as you give appropriate credit by citing our paper and indicating any changes that you've made.

Synthetic, MNIST and Multiple Instance Learning Experiments

Python requirements and installation

This code was tested on Python 3.10.10. To install, follow these steps:

  1. In a virtual environment, first install Cython: pip install cython
  2. Clone the Eigen repository to the main folder: git clone git@gitlab.com:libeigen/eigen.git
  3. Clone the LP-SparseMAP fork repository to main folder, and follow the installation instructions found there
  4. Install the requirements: pip install -r requirements.txt
  5. Run the corresponding scripts

Reproducibility

MNIST MIL

Run the script MNIST_bags.py with the desired parameters (nomenclature can be found in the beginning of the script)

Benchmarks MIL

Download and upzip the dataset

$ wget http://www.cs.columbia.edu/~andrews/mil/data/MIL-Data-2002-Musk-Corel-Trec9-MATLAB.tgz 

Run the script MIL_Data_2002.py with the desired parameters (nomenclature can be found in the beginning of the script)

Countours and Basins of Attraction

Run the scripts countours.py and basins.py

Metastable State Counting

Run the script MNIST_metastable.py

Spectra Experiments

Python requirements and installation

Follow the instructions of the branch in hopfield-spectra

Acknowledgment

The experiments in this work benefit from the following open-source codes: