Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throw error if ES version is not supported #5068

Closed
Tracked by #2810
TuanaCelik opened this issue Jun 2, 2023 · 4 comments · Fixed by #5320
Closed
Tracked by #2810

Throw error if ES version is not supported #5068

TuanaCelik opened this issue Jun 2, 2023 · 4 comments · Fixed by #5320
Assignees
Labels
Contributions wanted! Looking for external contributions P2 Medium priority, add to the next sprint if no P1 available topic:document_store topic:elasticsearch type:documentation Improvements on the docs type:enhancement

Comments

@TuanaCelik
Copy link
Member

There was a breaking change with Elasticsearch 7.6 as mentioned in this issue: #5066 which means Haystack doesn't support the vector similarity function.
I suggest we start:

@jscheel
Copy link

jscheel commented Jun 2, 2023

I think we can easily support this by storing the ES cluster version internally in the document store, then tweaking the painless script args in _get_vector_similarity_query based on that. Specifically, we would just need to look at the version number to determine if the script should look like this:

"source": f"{similarity_fn_name}(params.query_vector,'{self.embedding_field}') + 1000"

or like this:

 "source": f"{similarity_fn_name}(params.query_vector,doc['{self.embedding_field}']) + 1000"

There doesn't seem to be support for getting the cluster version number in painless, so it would be maintained in the store itself. I can put together a PR if you are open to it.

@ZanSara ZanSara added the Contributions wanted! Looking for external contributions label Jun 7, 2023
@ZanSara
Copy link
Contributor

ZanSara commented Jun 7, 2023

@jscheel please feel free to open a PR!

@ZanSara ZanSara added the P2 Medium priority, add to the next sprint if no P1 available label Jun 7, 2023
@jscheel
Copy link

jscheel commented Jun 7, 2023 via email

@jscheel
Copy link

jscheel commented Jul 13, 2023

@bogdankostic awesome! Thanks for taking care of this! Sorry I haven't been able to look at it. Just got back into the swing of things after my trip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Contributions wanted! Looking for external contributions P2 Medium priority, add to the next sprint if no P1 available topic:document_store topic:elasticsearch type:documentation Improvements on the docs type:enhancement
Projects
None yet
4 participants