Facebook sync (Jun 2019)

[beauby]

Changelog:

Bugfixes:

• slow scanning of inverted lists (Why does it take more time to search the entire xq than to search sequentially for a subset of all the shards of the xq? #836).

Features:

• add basic support for 6 new metrics in CPU IndexFlat and IndexHNSW (Support as many distances as possible from scipy.spatial.distance #848);
• add support for IndexIDMap/IndexIDMap2 with binary indexes ([Feature Request] IdMap/IdMap2 for IndexBinary #780).

Misc:

• throw python exception for OOM ( std::bad_alloc #758);
• make DistanceComputer available for all random access indexes;
• gradually moving from long to int64_t for portability.

[mdouze]

Add in changelog

• support for 6 new distance types in CPU index Flat and HNSW

• refactoring to make DistanceComputer available for all random access indexes

[mdouze]

Good to go IMO.

[Enet4]

I could not compile this on my machine:

utils_simd.cpp: In function ‘float faiss::fvec_Linf(const float*, const float*, size_t)’:
utils_simd.cpp:484:12: error: ‘fvec_Linf_ref’ was not declared in this scope
     return fvec_Linf_ref (x, y, d);
            ^~~~~~~~~~~~~
utils_simd.cpp:484:12: note: suggested alternative: ‘fvec_L1_ref’
     return fvec_Linf_ref (x, y, d);
            ^~~~~~~~~~~~~
            fvec_L1_ref

A reference implementation is seen in utils_simd.cpp for fvec_L1_ref, but fvec_Linf_ref appears to be missing. I think something like this would do the trick:

float fvec_Linf_ref (const float * x,
                   const float * y,
                   size_t d)
{
    float res = 0;
    for (size_t i = 0; i < d; i++) {
        const float tmp = fabs(x[i] - y[i]);
        if (tmp > res) {
            res = tmp;
        }
    }
    return res;
}

As for int64_t not being found in some Travis jobs, it's likely missing a few #include <cstdint> in some places.

[mdouze]

@Enet4 thanks for flagging. We forgot to test without the -mavx2 flag, which exposes this bug.

[beauby]

Only test left failing on OSX is a matter of system python not finding Homebrew-installed scipy.