CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project DOES NOT adhere to Semantic Versioning at the moment.

Unreleased

Added

• Support for building C bindings through the FAISS_ENABLE_C_API CMake option.
• Serializing the indexes with the python pickle module
• Support for the NNDescent k-NN graph building method
• Support for the NSG graph indexing method
• Residual quantizers: support as codec and unoptimized search

Changed

• The order of xb an xq was different between faiss.knn and faiss.knn_gpu. Also the metric argument was called distance_type.
• The typed vectors (LongVector, LongLongVector, etc.) of the SWIG interface have been deprecated. They have been replaced with Int32Vector, Int64Vector, etc.

Fixed

• Fixed a bug causing kNN search functions for IndexBinaryHash and IndexBinaryMultiHash to return results in a random order.
• Copy constructor of AlignedTable had a bug leading to crashes when cloning IVFPQ indices.

1.7.0 - 2021-01-27

1.6.5 - 2020-11-22

1.6.4 - 2020-10-12

Added

• Arbitrary dimensions per sub-quantizer now allowed for GpuIndexIVFPQ.
• Brute-force kNN on GPU (bfKnn) now accepts int32 indices.
• Nightly conda builds now available (for CPU).
• Faiss is now supported on Windows.

1.6.3 - 2020-03-24

Added

• Support alternative distances on GPU for GpuIndexFlat, including L1, Linf and Lp metrics.
• Support METRIC_INNER_PRODUCT for GpuIndexIVFPQ.
• Support float16 coarse quantizer for GpuIndexIVFFlat and GpuIndexIVFPQ. GPU Tensor Core operations (mixed-precision arithmetic) are enabled on supported hardware when operating with float16 data.
• Support k-means clustering with encoded vectors. This makes it possible to train on larger datasets without decompressing them in RAM, and is especially useful for binary datasets (see https://github.com/facebookresearch/faiss/blob/master/tests/test_build_blocks.py#L92).
• Support weighted k-means. Weights can be associated to each training point (see https://github.com/facebookresearch/faiss/blob/master/tests/test_build_blocks.py).
• Serialize callback in python, to write to pipes or sockets (see https://github.com/facebookresearch/faiss/wiki/Index-IO,-cloning-and-hyper-parameter-tuning).
• Reconstruct arbitrary ids from IndexIVF + efficient remove of a small number of ids. This avoids 2 inefficiencies: O(ntotal) removal of vectors and IndexIDMap2 on top of indexIVF. Documentation here: https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes.
• Support inner product as a metric in IndexHNSW (see https://github.com/facebookresearch/faiss/blob/master/tests/test_index.py#L490).
• Support PQ of sizes other than 8 bit in IndexIVFPQ.
• Demo on how to perform searches sequentially on an IVF index. This is useful for an OnDisk index with a very large batch of queries. In that case, it is worthwhile to scan the index sequentially (see https://github.com/facebookresearch/faiss/blob/master/tests/test_ivflib.py#L62).
• Range search support for most binary indexes.
• Support for hashing-based binary indexes (see https://github.com/facebookresearch/faiss/wiki/Binary-indexes).

Changed

• Replaced obj table in Clustering object: now it is a ClusteringIterationStats structure that contains additional statistics.

Removed

• Removed support for useFloat16Accumulator for accumulators on GPU (all accumulations are now done in float32, regardless of whether float16 or float32 input data is used).

Fixed

• Some python3 fixes in benchmarks.
• Fixed GpuCloner (some fields were not copied, default to no precomputed tables with IndexIVFPQ).
• Fixed support for new pytorch versions.
• Serialization bug with alternative distances.
• Removed test on multiple-of-4 dimensions when switching between blas and AVX implementations.

1.6.2 - 2020-03-10

1.6.1 - 2019-12-04

1.6.0 - 2019-09-24

Added

• Faiss as a codec: We introduce a new API within Faiss to encode fixed-size vectors into fixed-size codes. The encoding is lossy and the tradeoff between compression and reconstruction accuracy can be adjusted.
• ScalarQuantizer support for GPU, see gpu/GpuIndexIVFScalarQuantizer.h. This is particularly useful as GPU memory is often less abundant than CPU.
• Added easy-to-use serialization functions for indexes to byte arrays in Python (faiss.serialize_index, faiss.deserialize_index).
• The Python KMeans object can be used to use the GPU directly, just add gpu=True to the constuctor see gpu/test/test_gpu_index.py test TestGPUKmeans.

Changed

• Change in the code layout: many C++ sources are now in subdirectories impl/ and utils/.

1.5.3 - 2019-06-24

Added

• Basic support for 6 new metrics in CPU IndexFlat and IndexHNSW (facebookresearch#848).
• Support for IndexIDMap/IndexIDMap2 with binary indexes (facebookresearch#780).

Changed

• Throw python exception for OOM (facebookresearch#758).
• Make DistanceComputer available for all random access indexes.
• Gradually moving from long to uint64_t for portability.

Fixed

• Slow scanning of inverted lists (facebookresearch#836).

1.5.2 - 2019-05-28

Added

• Support for searching several inverted lists in parallel (parallel_mode != 0).
• Better support for PQ codes where nbit != 8 or 16.
• IVFSpectralHash implementation: spectral hash codes inside an IVF.
• 6-bit per component scalar quantizer (4 and 8 bit were already supported).
• Combinations of inverted lists: HStackInvertedLists and VStackInvertedLists.
• Configurable number of threads for OnDiskInvertedLists prefetching (including 0=no prefetch).
• More test and demo code compatible with Python 3 (print with parentheses).

Changed

• License was changed from BSD+Patents to MIT.
• Exceptions raised in sub-indexes of IndexShards and IndexReplicas are now propagated.
• Refactored benchmark code: data loading is now in a single file.

1.5.1 - 2019-04-05

Added

• MatrixStats object, which reports useful statistics about a dataset.
• Option to round coordinates during k-means optimization.
• An alternative option for search in HNSW.
• Support for range search in IVFScalarQuantizer.
• Support for direct uint_8 codec in ScalarQuantizer.
• Better support for PQ code assignment with external index.
• Support for IMI2x16 (4B virtual centroids).
• Support for k = 2048 search on GPU (instead of 1024).
• Support for renaming an ondisk invertedlists.
• Support for nterrupting computations with interrupt signal (ctrl-C) in python.
• Simplified build system (with --with-cuda/--with-cuda-arch options).

Changed

• Moved stats() and imbalance_factor() from IndexIVF to InvertedLists object.
• Renamed IndexProxy to IndexReplicas.
• Most CUDA mem alloc failures now throw exceptions instead of terminating on an assertion.
• Updated example Dockerfile.
• Conda packages now depend on the cudatoolkit packages, which fixes some interferences with pytorch. Consequentially, faiss-gpu should now be installed by conda install -c pytorch faiss-gpu cudatoolkit=10.0.

1.5.0 - 2018-12-19

Added

• New GpuIndexBinaryFlat index.
• New IndexBinaryHNSW index.

1.4.0 - 2018-08-30

Added

• Automatic tracking of C++ references in Python.
• Support for non-intel platforms, some functions optimized for ARM.
• Support for overriding nprobe for concurrent searches.
• Support for floating-point quantizers in binary indices.

Fixed

• No more segfaults due to Python's GC.
• GpuIndexIVFFlat issues for float32 with 64 / 128 dims.
• Sharding of flat indexes on GPU with index_cpu_to_gpu_multiple.

1.3.0 - 2018-07-10

Added

• Support for binary indexes (IndexBinaryFlat, IndexBinaryIVF).
• Support fp16 encoding in scalar quantizer.
• Support for deduplication in IndexIVFFlat.
• Support for index serialization.

Fixed

• MMAP bug for normal indices.
• Propagation of io_flags in read func.
• k-selection for CUDA 9.
• Race condition in OnDiskInvertedLists.

1.2.1 - 2018-02-28

Added

• Support for on-disk storage of IndexIVF data.
• C bindings.
• Extended tutorial to GPU indices.