Add vector search with embedding generation workload

[vpehkone]

Signed-off-by: Vesa Pehkonen vesa.pehkonen@intel.com

Description

Add vector search with embedding generation workload

Issues Resolved

#198

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[akashsha1]

@VijayanB / @wbeckler - PR for benchmarking vectorsearch with embedding i.e. neural search. Please help add necessary reviewers for this benchmark.

Feedback is appreciated.

[wbeckler]

What is the licensing of this content?

[navneet1v]

@vpehkone thanks for raising the PR. Can we move this benchmarks to folder named: semantic_search rather than vectorSearch_embedding

[vpehkone]

@navneet1v Addressed all comments: changed workload name to semantic_search, moved common code to OSB, and changed dataset to trec-covid. Can you please review?

[IanHoang]

Since we have two semantic search workloads that are looking to be added, let's rename the workload to be more specific (such as treccovid_semantic_search).

[VijayanB]

Since we have two semantic search workloads that are looking to be added, let's rename the workload to be more specific (such as treccovid_semantic_search).

@IanHoang If difference is just usage of datasets, do we need two different workloads? Why not merge and let individual procedures use their own corpus?

[vpehkone]

Since we have two semantic search workloads that are looking to be added, let's rename the workload to be more specific (such as treccovid_semantic_search).

@IanHoang If difference is just usage of datasets, do we need two different workloads? Why not merge and let individual procedures use their own corpus?

@VijayanB These workload are very different. Trec-covid semantic search generates embeddings and does vector search. Noaa semantic search does range, aggregate, term, etc... searches. It does not make sense to merge these.

[martin-gaievski]

Since we have two semantic search workloads that are looking to be added, let's rename the workload to be more specific (such as treccovid_semantic_search).

@IanHoang If difference is just usage of datasets, do we need two different workloads? Why not merge and let individual procedures use their own corpus?

@VijayanB These workload are very different. Trec-covid semantic search generates embeddings and does vector search. Noaa semantic search does range, aggregate, term, etc... searches. It does not make sense to merge these.

@vpehkone @VijayanB what would you say about rebasing that workload on a single dataset? We're not using this workload and trec-covid for checking correctness. We can use some field of text type to generate embeddings. Here is the example of doc from noaa:

{
  "TMIN": 8.3,
  "SNOW": "0",
  "WSF5": 11.2,
  "SNWD": "0",
  "PRCP": 0,
  "station": {
    "wmo_id": "72398",
    "state": "MARYLAND",
    "name": "SALISBURY WICOMICO RGNL AP",
    "location": {
      "lon": -75.5103,
      "lat": 38.3406
    },
    "elevation": 14.6,
    "country_code": "US",
    "id": "USW00093720",
    "country": "United",
    "state_code": "MD"
  },
  "WDF2": "20",
  "TMAX": 18.3,
  "AWND": 3.1,
  "TRANGE": {
    "lte": 18.3,
    "gte": 8.3
  },
  "date": "2015-04-15T00:00:00",
  "TAVG": 12.9,
  "WSF2": 8.1,
  "WDF5": "20"
}

we can use field station.name instead of text to generate embeddings.
Two workloads then can be merged into one as rest of operations are independent, maybe some operations can be reused.
I'm not sure the other way around is possible as trec-covid mapping is simple and is lacking integer and keyword fields that are needed for hybrid query.

[navneet1v]

@martin-gaievski what is the reason for merging the noaa dataset with Semantic Search? I think its better to keep semantic search as a separate use case and workload. Having a uber name as semantic search is really good. We can have more dataset in semantic search later. But for now having a simple Semantic Search dataset with trec-covid as dataset is pretty neat I would say.

[VijayanB]

@IanHoang Can this be added directly to opensearch-benchmark?

[IanHoang]

@VijayanB Sorry for the lates response. Yes, this can be technically contributed directly into the OpenSearch Benchmark repository . If this is going to be reused by other workloads potentially, we should include it there. @vpehkone Please feel free to make this quick change in OSB repository and we can review it quickly and get this shipped.

[vpehkone]

@IanHoang I don't know how it could be possible to add a registration of parameter source function/class to OSB. OSB cannot know where to look at it or the name of the source function/class? Please let me know if you have an idea how it works, and I will try to implement it.

[IanHoang]

@vpehkone We plan to add some documentation to the official OSB documentation regarding this. In the interest of time, we can have this merged in

[IanHoang]

@vpehkone does the data corpora need to be added to a cloud repository (similar to this workload.json) If so, we might need to add a files.txt, similar to other workloads.

@IanHoang Yes, I added files.txt. Can you let me know how add the data corpora to the cloud repository? Then I can update the corpus url to workload.json.

We'll need to download the corpora manually and add it to our cloud repository. I can coordinate this with you offline in the slack community.

[IanHoang]

@vpehkone Please add this to the workload.json as the base-url: https://dbyiw3u3rf9yr.cloudfront.net/corpora/treccovid

[IanHoang]

@vpehkone This can now be updated to https://dbyiw3u3rf9yr.cloudfront.net/corpora/treccovid. Please confirm if document count is also correct. Is 129192 from the uncompressed version of documents.json.bz2?

$ wc -l documents.json.bz2
253809 documents.json.bz2

[vpehkone]

Updated the new URL for documents and queries. Yes, the document count 129192 is right and it is for uncompressed documents.

[navneet1v]

excited to see this workload getting merged. :) thanks @vpehkone making this happen and @IanHoang for reviewing the code.