Add support for Elasticsearch 8

[masci]

Is your feature request related to a problem? Please describe.
Haystack should be compatible with more recent Elasticsearch releases, possibly also leveraging new features

Describe the solution you'd like
The kNN search API was changed with version 8 and most likely will speed up dense retrieval. Other features could also be interesting for Haystack, e.g. pytorch model support

• Checking compatibility with old elasticsearch versions and probably have two ElasticsearchDocumentStores (on version 8, one version 7)
• Try running the latest version 7.x version
• Fix error: cannot import name 'RequestsHttpConnection' from 'elasticsearch' feat: prepare ES8 support by fixing 'cannot import RequestsHttpConnection' #2991
• Check migration guides and identify blockers/necessary changes https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html
• Implement/support new features added in release 8.x (kNN search API etc.)

Tasks

Beta Give feedback

1. refactor!: extract elasticsearch #4668
   breaking change topic:CI topic:build/distribution topic:dependencies topic:document_store topic:pinecone topic:retriever topic:tests type:documentation +9
2. ci: run Elasticsearch 8.6 in compatibility mode #3853
   ignore-for-release-notes topic:windows +2
3. Pin elasticsearch to version 7.17.4 for testing #2993
   topic:document_store topic:elasticsearch +2
4. Make ElasticsearchDocumentStore compatible with ES8 #5027
   P1 topic:elasticsearch +2
   
5. Throw error if ES version is not supported #5068
   Contributions wanted! P2 topic:document_store topic:elasticsearch type:documentation type:enhancement +6
   

[UmarEm]

Hey @masci @ZanSara @julian-risch Can we get the timeline/plan for Elasticsearch 8.x support. You had earlier mentioned, that the support for Elasticsearch 8.x is scheduled for this quater
It would be great, if we could know what is the progress, and when can we expect the update.

Thank you

[nickchomey]

Above it says 8.3.3 is pinned for testing, but ES 8.4 moved the ANN mechanism into the _search endpoint, which probably would require various changes on your end. I'm just sharing this in case you missed the recent announcement/update.

And any guidance (however tentative) that you might be able to provide about when this will be supported by Haystack would be very appreciated. Thanks for the great work!

https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html#integrate_ann_into_search_endpoint

[julian-risch]

I noticed that Haystack's dependency BEIR pins the version of the elasticsearch client to elasticsearch==7.9.1. However, BEIR is just an optional dependency (not mandatory) and is only installed for the installation options all, all-gpu, and beir.

[anakin87]

I noticed that Haystack's dependency BEIR pins the version of the elasticsearch client to elasticsearch==7.9.1. However, BEIR is just an optional dependency (not mandatory) and is only installed for the installation options all, all-gpu, and beir.

Similar problem (beir pins faiss-cpu): #3218

[julian-risch]

Slightly related: A user needed just a few lines of code to use Haystack with Elastic on GCP #4547

[julian-risch]

ES8 is now supported and the last related issue has been merged: #5320

[btofan]

Does the current solution support the approximate KNN feature that elasticsearch offers? According to the elasticsearch documentation (cf. https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html#approximate-knn), a specific kind of indexing and querying is required for approximate KNN search. I couldn't find the solution in the current commits. (This was also one of the main intentions of this issue and the related issues #2833, #2368.)