Is there a way to rebuild index in IVF-index after update ivf centers?

[zhanghan9797]

Hi,
First, i init a ivf index like this:

quantizer = faiss.IndexFlatIP(emb_size)
index = faiss.IndexIVFFlat(quantizer, emb_size, ivf_centers_num, faiss.METRIC_INNER_PRODUCT)

Then, I update IndexIVFFlat's centers like this:

coarse_quantizer = faiss.downcast_index(index.quantizer)  
faiss.copy_array_to_vector(center_vecs.ravel(),coarse_quantizer.xb)

After that,i want to add corpus to this index and search, however this error occur:

terminate called after throwing an instance of 'faiss::FaissException'
  what():  Error in faiss::IndexIVF::search_preassigned(faiss::Index::idx_t, const float*, faiss::Index::idx_t, const idx_t*, const float*, float*, faiss::Index::idx_t*, bool, const faiss::IVFSearchParameters*, faiss::IndexIVFStats*) const::<lambda(faiss::Index::idx_t, float, float*, faiss::Index::idx_t*)> at /project/faiss/faiss/IndexIVF.cpp:469: Error: 'key < (idx_t)nlist' failed: Invalid key=140461314990592 nlist=10000

How can I create a new ivf index according my ivf_centers ? Thanks in advance.

[mdouze]

So this is a very low level way of accessing the centroids.
The safest way of doing this is to do

index.quantizer.reset()
index.quantizer.add(center_vecs) 

also note that "re-training" an index is not recommended, so this operation should happen only when the index is still empty.

[zhanghan9797]

Why does the query latency become very large after updating the centers?

start = time.perf_counter()
_, dev_I = index.search(query_embeddings, 1000)
end = time.perf_counter() - start
print('{:.6f}s for the total'.format(end))
print('{:.6f}s for the every'.format(end / np.array(query_embeddings).shape[0]))