Self-Supervised Learning #221
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change introduces frontend bundle for custom projector that is very bare in its functionality. Python counterpart will follow shortly. While the change is sizable, most of it is configuration and fixture that contain very bare logic. Co-authored-by: Owen Vallis <owensvallis@gmail.com>
* Update contrastive model to sum the two example level losses before computing the mean loss * Add support for converting cosing similarity to angular similarity * Update typing issues.
* refactore example * Fixed SiamSiam to SimSiam -- experiment with decoder * implemented saving * Set the SiamSiamLoss back to SimSiamLoss. Was reverted during the previous merge. * Add Batch type alias to cover all basic types accepted by tf.keras.Model.fit() Add typing to Augmenter Update unsupervised notebook. * Updates to unsupervised notebook. Co-authored-by: Owen Vallis <ovallis@google.com>
* Kaggle working * Updates and Markdown for Kaggle notebook. * Pull updates from Kaggle notebook into kaggle_train script and add argparse for data path. * Clean up Kaggle examples in the examples directory. Update links in the examples README. Co-authored-by: Elie Bursztein <github@elie.net>
Add Encoder and Projector to get_encoder, and build predictor as second MLP layer. Test on the frozen global max pooling layer only. Add cosine decay for SGD
PY3.10 is breaking pip install packages.
No pretrain eval was over writing the pretrianed encoder.
This is closer to the original paper. Updates to the unsupervised notebook.
Update Notebook to use the cifair10 resnet used in the Kerias simsiam example. Also update the notebook params to confirm we get similar outputs as the keras example. Add support for selecting the loss type in the simsiam loss. Fix angular distance. Was previously returning angular similarity.
…attribute directly.
Architectures previously took a callable stack of augmentation layers that would be added after the input of the model. This could cause issues with saving and training on TPU. Users are now expected to add augmentation to either the data samplers / datasets or manually add it to the model.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
abeltheo
pushed a commit
to abeltheo/similarity
that referenced
this pull request
Mar 23, 2023
* SSL initial working implementation * Self supervised initial version * version bump * Add dev suffix to version number. * Fix nightly pypi token var in workflow. * [nightly] Increase version to 0.15.0.dev1 * [nightly] Increase version to 0.15.0.dev2 * vis: Introduce frontend projector (tensorflow#176) This change introduces frontend bundle for custom projector that is very bare in its functionality. Python counterpart will follow shortly. While the change is sizable, most of it is configuration and fixture that contain very bare logic. Co-authored-by: Owen Vallis <owensvallis@gmail.com> * [nightly] Increase version to 0.15.0.dev3 * * Add SimClr loss * Update contrastive model to sum the two example level losses before computing the mean loss * Add support for converting cosing similarity to angular similarity * Update typing issues. * Initial contrastive loss (tensorflow#180) * refactore example * Fixed SiamSiam to SimSiam -- experiment with decoder * implemented saving * Set the SiamSiamLoss back to SimSiamLoss. Was reverted during the previous merge. * Add Batch type alias to cover all basic types accepted by tf.keras.Model.fit() Add typing to Augmenter Update unsupervised notebook. * Updates to unsupervised notebook. Co-authored-by: Owen Vallis <ovallis@google.com> * Kaggle (tensorflow#182) * Kaggle working * Updates and Markdown for Kaggle notebook. * Pull updates from Kaggle notebook into kaggle_train script and add argparse for data path. * Clean up Kaggle examples in the examples directory. Update links in the examples README. Co-authored-by: Elie Bursztein <github@elie.net> * Updates to make simclr loss work as expected. * Update unsupervised hello world to work with simsiam. Add Encoder and Projector to get_encoder, and build predictor as second MLP layer. Test on the frozen global max pooling layer only. Add cosine decay for SGD * More updates to the unsupervised notebook. * Updates the nightly workflow to purge the pip cache before installing tf-sim * Fix workflow versions to python 3.9 PY3.10 is breaking pip install packages. * [nightly] Increase version to 0.15.0.dev4 * Fix bug in unsupervised notebook. No pretrain eval was over writing the pretrianed encoder. * [nightly] Increase version to 0.15.0.dev5 * Reduce mean over each simsiam loss seperately before summing. This is closer to the original paper. Updates to the unsupervised notebook. * Make SimSiam a subclass of Loss. Update Notebook to use the cifair10 resnet used in the Kerias simsiam example. Also update the notebook params to confirm we get similar outputs as the keras example. Add support for selecting the loss type in the simsiam loss. Fix angular distance. Was previously returning angular similarity. * Move loss_type cond from SimSiam.call() to the __init__. * Fix mypy typing for simsiam loss. * Unsupervised notebook updates. * [nightly] Increase version to 0.15.0.dev6 * [nightly] Increase version to 0.15.0.dev7 * [nightly] Increase version to 0.15.0.dev8 * [nightly] Increase version to 0.15.0.dev9 * [nightly] Increase version to 0.15.0.dev10 * [nightly] Increase version to 0.15.0.dev11 * [nightly] Increase version to 0.15.0.dev12 * [nightly] Increase version to 0.15.0.dev13 * [nightly] Increase version to 0.15.0.dev14 * [nightly] Increase version to 0.15.0.dev15 * [nightly] Increase version to 0.15.0.dev16 * [nightly] Increase version to 0.15.0.dev17 * [nightly] Increase version to 0.15.0.dev18 * [nightly] Increase version to 0.15.0.dev19 * [nightly] Increase version to 0.15.0.dev20 * Set the direction of the False Positive Count metric to 'min' as we would like to minimize the FP rate. * [nightly] Increase version to 0.15.0.dev21 * [nightly] Increase version to 0.15.0.dev22 * Unpacking the Lookup objects could fail if the Lookup sets were of different lengths. * Add support for loading variable length Lookup sets. * Imput label and distance values to make all Lookup sets the same length * * The label is set to 0x7FFFFFFF as this is unlikely to be a class label. * * The distance is set to math.inf * print a warning to the user to notify them that we are imputing the values and the number of short Lookup sets. * Add tests for all utils.py functions. * [nightly] Increase version to 0.15.0.dev23 * [nightly] Increase version to 0.15.0.dev24 * [nightly] Increase version to 0.15.0.dev25 * [nightly] Increase version to 0.15.0.dev26 * [nightly] Increase version to 0.15.0.dev27 * Initial version of the Barlow contrastive loss. * Updates to the simclr and simsiam contrastive losses. * Refactor contrastive model to subclass similarity model and add support for indexing. * Small updates to the example notebooks. * [nightly] Increase version to 0.15.0.dev28 * Fix divide by zero in the barlow column wise normalization and only call barlow loss once for each pair of views. * [nightly] Increase version to 0.15.0.dev29 * [nightly] Increase version to 0.15.0.dev30 * [nightly] Increase version to 0.15.0.dev31 * [nightly] Increase version to 0.15.0.dev32 * [nightly] Increase version to 0.15.0.dev33 * [nightly] Increase version to 0.15.0.dev34 * [nightly] Increase version to 0.15.0.dev35 * [nightly] Increase version to 0.15.0.dev36 * [nightly] Increase version to 0.15.0.dev37 * Add tests and typeing for effnet architectures. * Add tests for confusion matrix and rename module to avoid clobering the module name when importing. * add tets for neighbor vix. * fix typing warnings in new modules. * Update unsupervised draft to include barlow twins tests. * Add draft of tf record kaggle notebook. * [nightly] Increase version to 0.15.0.dev38 * [nightly] Increase version to 0.15.0.dev39 * [nightly] Increase version to 0.15.0.dev40 * [nightly] Increase version to 0.15.0.dev41 * [nightly] Increase version to 0.15.0.dev42 * Fix var mismatch in sampler after merge from master. * Add WarmUpCosine learning rate schedule. (tensorflow#197) * Add WarmUpCosine learning rate schedule. This is required for the Barlow Twins loss. * Make WarmUpCosine lr schedule serializable. * [nightly] Increase version to 0.15.0.dev43 * [nightly] Increase version to 0.15.0.dev44 * [nightly] Increase version to 0.15.0.dev45 * Add GeneralizeMeanPooling2D layer (tensorflow#198) * Add GeneralizeMeanPooling2D layer GeM adds support for global mean pooling using the generalized mean. This enables the pooling to increase or decrease the contrast between the feature map activation's. Add tests for GeM and MetricEmbedding layers. * Fix mypy errors. * Add 1D version for GeM pooling. * [nightly] Increase version to 0.15.0.dev46 * [nightly] Increase version to 0.15.0.dev47 * Refactor GeM to reduce code duplication and add test coverage for 1D. * Rename lr schedule module to schedules to make import cleaner. - Fix mypy error by defining return type of GeM output_shape. * [nightly] Increase version to 0.15.0.dev48 * [nightly] Increase version to 0.15.0.dev49 * [nightly] Increase version to 0.15.0.dev50 * Replace GlobalMeanPooling2d wiht GeneralizedMeanPooling2D and add None to augmenter typing. * Add support for scaling vix by max pixel value. * [Bug] Cast steps to dtype in WarmUpCosine. Rename test module to schedules. * [nightly] Increase version to 0.15.0.dev51 * Update changelog style. * Add license to schedules module. * Refer to Layers using the full module path rather than importing the class directly. * Rename effnet p param to gem_p to make it clear that we are setting the power on the gem layer. * Major updates to the contrastive model. - Add support for indexing. - Add test_step for tracking validation loss. - Add forward pass method to make it easier to pass data to the training model. - Update the predict method so that it now passes through the backbone and projector and returns the embedding output layer. - Various other fixes. * Major updates to the unsupervised hello world notebook. - Add support for using effnet as the backbone. - Clean up the projector and predictor models. - Provide support for switching between the various sefl-supervised algos. - Add example of passing validation data and match classification metrics. - Update eval code to ensure we take the correct output from the projector. * [nightly] Increase version to 0.15.0.dev52 * Fix mypy errors in ContrastiveModel. - distance can be Union[Distance, str] - embedding_output was an Optional[int], but should just default to 0 and be a simple int. * Update example notebooks. * [nightly] Increase version to 0.15.0.dev53 * [nightly] Increase version to 0.15.0.dev54 * [nightly] Increase version to 0.15.0.dev55 * Update supervised hello world. * [nightly] Increase version to 0.15.0.dev56 * [nightly] Increase version to 0.15.0.dev57 * [nightly] Increase version to 0.15.0.dev58 * [nightly] Increase version to 0.15.0.dev59 * [nightly] Increase version to 0.15.0.dev60 * Added soft_nearest_neighbor_loss. Closes tensorflow#103 (tensorflow#203) * [nightly] Increase version to 0.15.0.dev61 * Add ActivationStdLoggingLayer. Used for tracking the mean std of a layer's activations during training. * Add area_range arg to simclr crop_and_resize. This enables us to call simclr resize and crop with a custom upper and lower bound. * Basic formatting and cleanup of doc strings and args. * Add resent architectures. Add ResNet50Sim and ResNet18Sim architectures. Add support for standard keras.application args in efficitent. * Updates to contrastive model. - Explicitly add sub models losses to combined loss. - Track contrastive_loss, regularization_loss, and combined_loss separately - Add metrics property to correctly reset metrics state. * Updates to unsupervised notebook * Remove contrastive metrics module. We now track the layer std using the metric logging layer. * [nightly] Increase version to 0.15.0.dev62 * Support inner model losses. The backbone, projector, and predictor models may have layer losses like kernel_regularization. We now check for these and add additional loss trackers if required. This enables us to separately monitor the contrastive and regularization losses, if they exist. * Fix mypy errors in simsiam loss. * Various fixes to ResNet18 Now uses SimSiam kernel intialization. This seems to be critical for proper training on the cifar10 dataset. * Initial working version of SimSiam on cifar10 * Added SNR distance. Closes tensorflow#64 (tensorflow#205) Added Signal-to-Noise Ratio distance metric as defined in [Signal-to-Noise Ratio: A Robust Distance Metric for Deep Metric Learning](https://arxiv.org/abs/1904.02616) * Small updates for SimSiam example. Switch to using LeCunn.Uniform for the kernel_initalizers. Stride should be 1 for the first Conv2d layer in the ResNet18 backbone. * [nightly] Increase version to 0.15.0.dev63 * [nightly] Increase version to 0.15.0.dev64 * Updates for contrastive model saving. * [nightly] Increase version to 0.15.0.dev65 * [nightly] Increase version to 0.15.0.dev66 * [nightly] Increase version to 0.15.0.dev67 * [nightly] Increase version to 0.15.0.dev68 * [nightly] Increase version to 0.15.0.dev69 * [nightly] Increase version to 0.15.0.dev70 * [nightly] Increase version to 0.15.0.dev71 * Update losses to use Loss reduction. Losses previously computed the mean loss over the examples within the call() method. This may create issues when using multi GPU training. The call() method now returns the per example loss, and the final loss is computed using the losses.Loss reduction method. We also updated the from_config() method to include the parent class's reduction and name args. * Resnet18 returns as a SimilarityModel. We may want Resnet18 as a regular model, but keeping the output type as SimilarityModel to avoid mixed output types. * Fix various mypy and linter errors. * Add support for contrastive_model save and load. * Update unsupervised notebook with save and load. * Update the save and load. Add updated example and docs for save and load in the supervised hello world. * Updates to visualization notebook. * [nightly] Increase version to 0.15.0.dev72 * Unsupervised notebook update. * [nightly] Increase version to 0.15.0.dev73 * [nightly] Increase version to 0.15.0.dev74 * [nightly] Increase version to 0.15.0.dev75 * [nightly] Increase version to 0.15.0.dev76 * Notes on the unsupervised notebook draft. * [nightly] Increase version to 0.15.0.dev77 * [nightly] Increase version to 0.15.0.dev78 * [nightly] Increase version to 0.15.0.dev79 * Remove get_backbone() method and just have users access the backbone attribute directly. * Add new diagrams and updated copy to teh unsupervised notebook. * [nightly] Increase version to 0.15.0.dev80 * [nightly] Increase version to 0.15.0.dev81 * First finished draft of unsupervised_hello_world notebook * Updates to the README file. Add self-supervised info. * [nightly] Increase version to 0.15.0.dev82 * [nightly] Increase version to 0.15.0.dev83 * Update README.md * Remove augmentation arg from architectures. Architectures previously took a callable stack of augmentation layers that would be added after the input of the model. This could cause issues with saving and training on TPU. Users are now expected to add augmentation to either the data samplers / datasets or manually add it to the model. * Clean up example dir. * Fix flake8 errors in architectures. * Update API docs. * Bump version to 0.15.0 Co-authored-by: Elie Bursztein <github@elie.net> Co-authored-by: Github Actions Bot <> Co-authored-by: Stephan Lee <stephanwlee@gmail.com> Co-authored-by: Abhishar Sinha <24841841+abhisharsinha@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial support for self-supervised learning along with new notebook examples and numerous improvements and optimizations.
This is a minor version bump to 0.15 and includes breaking changes. See the change log and Readme for more details.