Deserialization issue with value size

[georgemcz]

Under some circumstances, we receive an exception during multi-search. I found that it is an issue with the extreme number of dropped tokens during the search - it fails when Hit.TextMatchInfo.NumTokensDropped shall contain deserialized json value:

"num_tokens_dropped":18446744073709551615 from text_match_info

which clearly does not fit into the int type. It seems that the typesense itself allows there some larger range.

Mitigation - bump the type to long, optionally also for other num_ types, this fix seems to work.

public record TextMatchInfo  
{ 
    //...

    [JsonPropertyName("num_tokens_dropped")]  
    public long NumTokensDropped { get; set; }     // changed from int to long  
    
    //...
}

[runeanielsen]

I think this is fine, you're welcome to make a PR with the change.

[runeanielsen]

@georgemcz this has now been released in version 7.20.0.

[adonistseriotis]

@runeanielsen Hey, this error has been occurring to me as well. I figured that long is not enough. It needs ulong.

[georgemcz]

Could be that you are dropping more tokens 😀
Shall I bump it, or you will?

[adonistseriotis]

Could be that you are dropping more tokens 😀
Shall I bump it, or you will?

Can I configure it somehow from the cloud dashboard?
I have applied the change locally for my use case to work. I can also pull request the change as well.
Nevertheless 'ulong' should capture all use cases.

[runeanielsen]

I am fine with switching it to ulong, feel free to make a PR.

[georgemcz]

Done. Just forced to do it over web gui, hopefully all is ok

[runeanielsen]

This has now been released in version 7.21.0.