How to query HNSW index with float[]

[nemo83]

Hello,

I've recently come across HNSW and found out REDIS has added support to it. So instead of building an api that manages an HNSW index, I am thinking of just deploying a nice REDIS cluster and use their HNSW capabilities.

My vectors are float[] and the loading of the index, apparenlty, should be working fine as I'm using the jsonSet. So when I create the index, the data are loaded from the json, correctly deserialised into float32 and the index is built. Or at least that's what I think is happening.

Now my problem is that I don't seem to be able to query the index.

This is my code:

float[] tensor = ...

String queryString = "*=>[KNN 10 @vector $tensor]";

Query q = new Query(queryString)
                .addParam("tensor", tensor)
                .dialect(2);

SearchResult searchResult = jedisClient.ftSearch("word-index", q)
int results = totalResults
System.out.println(results); // gives me 0

In the tests I found:
https://github.com/redis/jedis/blob/master/src/test/java/redis/clients/jedis/modules/search/SearchTest.java#L440

That seem to be working for strings, bytes and floats (but not floatp[]?!)

Any suggestions would be greatly appreciated.

[sazzad16]

This is what I got:

it's a byte array, you can translate your float[] to byte[]

This may not be a Jedis specific issue. You may find better answer asking in RediSearch.

[nemo83]

I tried, but how does the library/redis engine knows that that byte[] shuold be deserialised in a float[] rather than a String?

Something is missing.

[sazzad16]

My understanding, it would be the user who converts float[] to byte[].

[nemo83]

Yeah I managed to make some progress.

Still not the full exercise. Will update as soon as I have something working for future reference

Thanks for the help! <3

[nemo83]

Some additional info

I'm now creating an index as follows:

 Map<String, Object> attr = new HashMap<>();
        attr.put("TYPE", "FLOAT32");
        attr.put("DIM", 1024);
        attr.put("DISTANCE_METRIC", "COSINE");
        attr.put("INITIAL_CAP", 100);
        attr.put("M", 16);
        attr.put("EF_CONSTRUCTION", 200);
        attr.put("EF_RUNTIME", 200);

        Schema sc = new Schema().addHNSWVectorField("v", attr);
        assertEquals("OK", client.ftCreate(index, IndexOptions.defaultOptions(), sc));

loading items in an hashset as follows
client.hset(hash.getBytes(), "v".getBytes(), getVectorBytes(tensor.vector()));

with getVectorBytes defined as

    static byte[] getVectorBytes(float[] vec) throws IOException {
        ByteArrayOutputStream bas = new ByteArrayOutputStream();
        DataOutputStream dos = new DataOutputStream(bas);
        for (float f : vec) {
            dos.writeFloat(f);
        }
        dos.flush();
        dos.close();
        bas.flush();
        bas.close();
        byte[] bytes = bas.toByteArray();
        return bytes;
    }

Loading and indexing seems to work fine, but the query:

        Query query = new Query("*=>[KNN 10 @v $vec]")
                .addParam("vec", getVectorBytes(tensor.vector()))
                .setSortBy("__v_score", true)
                .returnFields("__v_score")
                .dialect(2);
        List<Document> documents = client.ftSearch(index, query).getDocuments();
        documents.forEach(doc -> {
            System.out.println(doc.getId());
            System.out.println(doc.get("__v_score"));
        });

returns NaN as score value, and the result is not in the correct order (if compared to vanilla hnsw java lib)

[lfernando2091]

Same problem, getVectorBytes this function for some reason creates a bad byte[], I don't even know how to fix that, but I tested against Python to do the same thing end the output are different.

[philippn]

It seems to be related to little-endian vs big-endian encoding; maybe this link here helps: https://forum.redis.com/t/ftsearch-on-vector-embeddings-returns-vector-score-of-nan/2271/2

[lfernando2091]

My solution:

https://github.com/lfernando2091/openai-webflux-kotlin/blob/2810e4740be02c9e87b7020c979dcdb34742ce4e/src/main/kotlin/io/lfernando/openai/utils/Embedding.kt#L21-L31