Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search

Introduction

We introduce the “Shopping Queries Data Set”, a large dataset of difficult search queries, released with the aim of fostering research in the area of semantic matching of queries and products. For each query, the dataset provides a list of up to 40 potentially relevant results, together with ESCI relevance judgements (Exact, Substitute, Complement, Irrelevant) indicating the relevance of the product to the query. Each query-product pair is accompanied by additional information. The dataset is multilingual, as it contains queries in English, Japanese, and Spanish.

The primary objective of releasing this dataset is to create a benchmark for building new ranking strategies and simultaneously identifying interesting categories of results (i.e., substitutes) that can be used to improve the customer experience when searching for products. The three different tasks that are studied in the literature (see https://amazonkddcup.github.io/) using this Shopping Queries Dataset are:

Task 1 - Query-Product Ranking: Given a user specified query and a list of matched products, the goal of this task is to rank the products so that the relevant products are ranked above the non-relevant ones.

Task 2 - Multi-class Product Classification: Given a query and a result list of products retrieved for this query, the goal of this task is to classify each product as being an Exact, Substitute, Complement, or Irrelevant match for the query.

Task 3 - Product Substitute Identification: This task will measure the ability of the systems to identify the substitute products in the list of results for a given query.

Dataset

We provide two different versions of the data set. One for task 1 which is reduced version in terms of number of examples and ones for tasks 2 and 3 which is a larger.

The training data set contain a list of query-result pairs with annotated E/S/C/I labels. The data is multilingual and it includes queries from English, Japanese, and Spanish languages. The examples in the data set have the following fields: example_id, query, query_id, product_id, product_locale, esci_label, small_version, large_version, split, product_title, product_description, product_bullet_point, product_brand, product_color and source

The Shopping Queries Data Set is a large-scale manually annotated data set composed of challenging customer queries.

There are 2 versions of the dataset. The reduced version of the data set contains 48,300 unique queries and 1,118,011 rows corresponding each to a <query, item> judgement. The larger version of the data set contains 130,652 unique queries and 2,621,738 judgements. The reduced version of the data accounts for queries that are deemed to be “easy”, and hence filtered out. The data is stratified by queries in two splits train, and test.

A summary of our Shopping Queries Data Set is given in the two tables below showing the statistics of the reduced and larger version, respectively. These tables include the number of unique queries, the number of judgements, and the average number of judgements per query (i.e., average depth) across the three different languages.

	Total	Total	Total	Train	Train	Train	Test	Test	Test
Language	# Queries	# Judgements	Avg. Depth	# Queries	# Judgements	Avg. Depth	# Queries	# Judgements	Avg. Depth
English (US)	29,844	601,354	20.15	20,888	419,653	20.09	8,956	181,701	20.29
Spanish (ES)	8,049	218,774	27.18	5,632	152,891	27.15	2,417	65,883	27.26
Japanese (JP)	10,407	297,883	28.62	7,284	209,094	28.71	3,123	88,789	28.43
Overall	48,300	1,118,011	23.15	33,804	781,638	23.12	14,496	336,373	23.20

Table 1: Summary of the Shopping queries data set for task 1 (reduced version) - the number of unique queries, the number of judgements, and the average number of judgements per query.

	Total	Total	Total	Train	Train	Train	Test	Test	Test
Language	# Queries	# Judgements	Avg. Depth	# Queries	# Judgements	Avg. Depth	# Queries	# Judgements	Avg. Depth
English (US)	97,345	1,818,825	18.68	74,888	1,393,063	18.60	22,458	425,762	18.96
Spanish (ES)	15,180	356,410	23.48	11,336	263,063	23.21	3,844	93,347	24.28
Japanese (JP)	18,127	446,053	24.61	13,460	327,146	24.31	4,667	118,907	25.48
Overall	130,652	2,621,288	20.06	99,684	1,983,272	19.90	30,969	638,016	20.60

Table 2: Summary of the Shopping queries data set for tasks 2 and 3 (larger version) - the number of unique queries, the number of judgements, and the average number of judgements per query.

Usage

The dataset has the following files:

• shopping_queries_dataset_examples.parquet contains the following columns : example_id, query, query_id, product_id, product_locale, esci_label, small_version, large_version, split
• shopping_queries_dataset_products.parquet contains the following columns : product_id, product_title, product_description, product_bullet_point, product_brand, product_color, product_locale
• shopping_queries_dataset_sources.csv contains the following columns : query_id, source

Load examples, products and sources

import pandas as pd
df_examples = pd.read_parquet('shopping_queries_dataset_examples.parquet')
df_products = pd.read_parquet('shopping_queries_dataset_products.parquet')
df_sources = pd.read_csv("shopping_queries_dataset_sources.csv")

Merge examples with products

df_examples_products = pd.merge(
    df_examples,
    df_products,
    how='left',
    left_on=['product_locale','product_id'],
    right_on=['product_locale', 'product_id']
)

Filter and prepare for Task 1

df_task_1 = df_examples_products[df_examples_products["small_version"] == 1]
df_task_1_train = df_task_1[df_task_1["split"] == "train"]
df_task_1_test = df_task_1[df_task_1["split"] == "test"]

Filter and prepare data for Task 2

df_task_2 = df_examples_products[df_examples_products["large_version"] == 1]
df_task_2_train = df_task_2[df_task_2["split"] == "train"]
df_task_2_test = df_task_2[df_task_2["split"] == "test"]

Filter and prepare data for Task 3

df_task_3 = df_examples_products[df_examples_products["large_version"] == 1]
df_task_3["subtitute_label"] = df_task_3["esci_label"].apply(lambda esci_label: 1 if esci_label == "S" else 0 )
del df_task_3["esci_label"]
df_task_3_train = df_task_3[df_task_3["split"] == "train"]
df_task_3_test = df_task_3[df_task_3["split"] == "test"]

Merge queries with sources (optional)

df_examples_products_source = pd.merge(
    df_examples_products,
    df_sources,
    how='left',
    left_on=['query_id'],
    right_on=['query_id']
)

Baselines

In order to ensure the feasibility of the proposed tasks, we will provide the results obtained by standard baseline models run on this data sets. For example, for the first task (ranking), we have run a BERT model. For the remaining two tasks (classification) we will provide the results of the multilingual BERT-based models as the initial baseline.

Requirements

We launched the baselines experiments creating an environment with Python 3.6 and installing the packages dependencies shown below:

numpy==1.19.2
pandas==1.1.5
transformers==4.16.2
scikit-learn==0.24.1
sentence-transformers==2.1.0

For installing the dependencies we can launch the following command:

pip install -r requirements.txt

Reproduce published results

For a task K, we provide the same scripts, one for training the model (and preprocessing the data for tasks 2 and 3): launch-experiments-taskK.sh; and a second script for getting the predictions for the public test set using the model trained on the previous step: launch-predictions-taskK.sh.

Task 1 - Query Product Ranking

For task 1, we fine-tuned 3 models one for each product_locale.

For us locacale we fine-tuned MS MARCO Cross-Encoders. For es and jp locales multilingual MPNet. We used the query and title of the product as input for these models.

To get the nDCG score of the ranking models is needed the terrier source code (download version 5.5 here)

cd ranking/
./launch-experiments-task1.sh
./launch-predictions-task1.sh $TERRIER_PATH

Task 2 - Multiclass Product Classification

For task 2, we trained a Multilayer perceptron (MLP) classifier whose input is the concatenation of the representations provided by BERT multilingual base for the query and title of the product.

cd classification_identification/
./launch-experiments-task2.sh
./launch-predictions-task2.sh

Task 3 - Product Substitute Identification

For task 3, we followed the same approach as in task 2.

cd classification_identification/
./launch-experiments-task3.sh
./launch-predictions-task3.sh

Results

The following table shows the baseline results obtained through the different public tests of the three tasks.

Task	Metrics	Scores
1	nDCG	0.83
2	Macro F1, Micro F1	0.23, 0.62
3	Macro F1, Micro F1	0.44, 0.76

Security

See CONTRIBUTING for more information.

Cite

Please cite our paper if you use this dataset for your own research:

@article{reddy2022shopping,
title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
author={Chandan K. Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu Xing and Karthik Subbian},
year={2022},
eprint={2206.06588},
archivePrefix={arXiv}
}

License

This project is licensed under the Apache-2.0 License.