Merging 0.18 #358
Merged
Merging 0.18 #358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* refactor distances call signature and add appropriate tests * refactor metrics for new distance call signature * make similarity losses compatible with asymmetric and non-square distance matrices * adapt and add test * remove newline
* initiate XBM loss * add todo * add XBM tests * WIP: XBM serialization * XBM serialization * class docstring * remove todo * improve docstring * remove comment
* Use list(range()) instead of comprehension as it is more pythonic. * Create barlow.py * Bump three in /tensorflow_similarity/visualization/projector_v2 (#228) Bumps [three](https://github.com/mrdoob/three.js) from 0.132.2 to 0.137.0. - [Release notes](https://github.com/mrdoob/three.js/releases) - [Commits](https://github.com/mrdoob/three.js/commits) --- updated-dependencies: - dependency-name: three dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Restructure class to be like Augmenter * Minor fixing of dead links (#230) * Fixed dead links * augmenter main to master * Spelling changes Auto Augment * MixupAndCutmix main to master * RandAugment main to master * RandomErasing main to master * Update SimCLRAugmenter.md * Update ClassificationMatch.md * Update ClassificationMetric.md * Update Evaluator.md * Update MemoryEvaluator.md * Update SimilarityModel.md * Update BinaryAccuracy.md * Update F1Score.md * Update FalsePositiveRate.md * Update NegativePredictiveValue.md * Update Precision.md * Update Recall.md Co-authored-by: Owen Vallis <owensvallis@gmail.com> * Fix minor typos (#226) Co-authored-by: Owen Vallis <owensvallis@gmail.com> * Update barlow.py * Update barlow.py * Update setup.py * Update barlow.py * Update barlow.py * Update barlow.py * Update barlow.py * Update barlow.py * revisions * Update __init__.py * Update __init__.py * Update color_jitter.py * Update barlow.py * Update barlow.py * Update barlow.py * Update setup.py Co-authored-by: Owen S Vallis <ovallis@google.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Owen Vallis <owensvallis@gmail.com> Co-authored-by: Genrry Hernandez <genrryhernandez@gmail.com>
* Create barlow.py * Restructure class to be like Augmenter * Update barlow.py * Update barlow.py * Update setup.py * Update barlow.py * Update barlow.py * Update barlow.py * Update barlow.py * Update barlow.py * revisions * Update __init__.py * Update __init__.py * Update color_jitter.py * Update barlow.py * Update barlow.py * Update barlow.py * Update setup.py * fixed some bugs * Remove seed instance variable Co-authored-by: Owen Vallis <owensvallis@gmail.com>
Merged 0.17.0 into master and bumping the dev versions to prepare for the next release.
* fix formatting * small fixes * adding reset to Search * adding reset to stores * add reset to indexer * add more tests --------- Co-authored-by: Ali Zand <zand@google.com>
* Update similarity_model.py Update verbose printing to display the count of indexed items. Verbose output was missing an f-string prefix and also returned the entire shape. Now we just return the number of examples. * 0.17 patches (#325) * fixes #323 Default indexer distance is now cosine in Sim Model. Calling create_index method now defaults to cosine distance. Additionally, auto distance defaults to cosine if no distance is passed to compile. * fixes #322 remove all calls to tf.convert_to_tensor in SimModel. * Update gitignore to exclude models and datasets from the example notebooks. * Update multi-modal notebook to remove the call to compile. * Patch bump * Remove check for tf.shape in index. Input can also be tuple or dict, so we should use len() here. * Update github workflow tests to use TF >= 2.8 * Tensor slice sampler (#329) * Create tfdata_sampler.py Initial version of new tf.data.Dataset sampler. * Refactor and clean up the tf data sampler. * Add initial tests for tfdata_sampler * Reformat TFDataSampler test file. * Fix proto dep issue in github workflow tests. py 3.10 breaks with protobuf > 3.20.x * Setting env var didn't work. Trying again with pinning the protobuf version to 3.20.1 * Check TF version before creating the tf dataset counter. * Format file * Remove as_numpy_iterator when creating the list of grouped datasets. * Also move class_list filter to before the group_by function * Apply the total_examples_per_class as a take() function on each grouped dataset * Remove as much casting as possible from the dataset. Certain functions expect an int64 though and require casting. * Refactor to move the filter by class list out of the window_group_by function. * Add class list filter test. * Move augment_fn and load_fn to before the repeat and batch functions. This change means the aug and load functions apply per example now. This will make it easier to apply random augmentations per example and is more consistent with how we implemented it in the existing memory sampler. This change also improves the tests for all parts of the module. * Add support for handling tuple and dict values for y. This change adds support for passing a callable to parse the correct class id element for batch sampling. By default y is assumed to be a 1D tensor with the class ids and the function is lambda y:y. Otherwise we accept an int or str and construct a parser to get the class id tensor. * Update email for github actions bot to fix CLA errors in PR * Fix import order and remove typing imports * Fix import check in search init. * Small updates to tfdata_sampler doc string
* Refactor similiarity unit tests to Tensorflow TestCase, reduce usage of Numpy API * Refactor similarity unittests to reduce usage of numpy and increase overall coverage * Merge branch 'development' of https://github.com/tensorflow/similarity into development * reformat tf search initialization file * Update indexer test files from recent push * Refactor similiarity unit tests to Tensorflow TestCase, reduce usage of Numpy API * Refactor similarity unittests to reduce usage of numpy and increase overall coverage * Merge branch 'development' of https://github.com/tensorflow/similarity into development * reformat tf search initialization file * Update indexer test files from recent push * Cleaned up and reformatted fiels * Sort test_file_samplers file
* Cleanup imports to legacy tensorflow.python.keras for tensorflow_similarity Remove call to conv_utils and explicitly define normalize_data_format for channels check. * reformat layers.py
* ensure no -1 is returned from FAISS * use string keys for redis store * update indexing colab * ensure faiss returned -1s are filtered out, given faiss returns -1s wherever it does not find the neighbors * fix a bug with distances * fix formatting --------- Co-authored-by: Ali Zand <zand@google.com>
Merging master back into dev for consistency.
* Checking changes for 0.18 release. * Refactor setup to make as many deps optional as possible [Note: might come back to this if it's awkward to run the hello world] * Clean up types and try to increase consistency between the different Search and Store classes * Implement lazy loading for all Search and Store modules to avoid having to load the deps. * Fix and update tests * Migrate to using pathlib where possible * Other small fixes and updates. * Add all deps needed for testing in the git workflow and try using only typing for search and store utils. * Migrate basic flow test to tf.TestCase. * Update to use BatchNorm with synchronized and finish eval of Notebooks. * Bump TF version check for BatchNorm Sync to 2.12 * Reorder the Search class methods to better align with Search base class * Update Faiss to return cosine distance values for distance type cosine * Add support for using inner_product with Faiss * Small refactor to Search methods * Flatten the example directory. * Refactor distance, store, search, and losses deserialization and serialization to better support custom 'str' for loading. * Fix distance imports in the models. * Add new distance module. * Flaky test on git workflows. Try updating to match local venv. * Set typing imports to only import during type checking. This cleans up our imports and prevents us from loading unused modules. * Update all notebooks * Remove all but the basic model and sampler modules from main package imports * Update the dataset and hyper_param scripts for benchmarking * Clean up the Redis tests and mocks to check all methods called. * Fix mock import for redis test. * Wrap tf.data.Dataset in CPU context to avoid GPU OOM errors. * Add ModuleNotFoundError with pip install info for missing extra_require deps when loading the relevent modules.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
owenvallis
approved these changes
Sep 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merging all the contribution of 0.18 into Main