The first edition of the Situated and Interactive Multimodal Conversations (SIMMC) Track at the Dialog State Tracking Challenge (DSTC) 9 came to a successful end! The challenge saw a total of 11 model entries from 5 teams across the world, setting a new state-of-the-art in all three subtasks:
- Subtask 1 (Assistant API Prediction)
- Action accuracy increased by 3 points (79.3% to 82.5%)
- Action Attribute accuracy increased by ~10 points (63.7% to 73.9%)
- Subtask 2 (Assistant Response Generation)
- BLEU score increased by 0.067 points (0.061 to 0.128)
- Recall@1 increased by 45 points (7.2% to 52.6%)
- Subtask 3 (Dialog State Tracking)
- Slot F-1 increased by 16.7 points (62.4% to 79.1%)
- Intent F-1 increased by 16 points (62.1% to 78.1%)
Congratulations to the winners and runners-up, and a big thanks to all the participants!
We launched the SIMMC challenge in June, 2020. Please checkout the main page, paper, and the Facebook AI blog for more details.
Please checkout the challenge page, paper, and the Facebook AI blog for further details about the challenge, dataset, tasks, and the evaluation metrics.
Results Summary
| Team | Affiliation | Github |
|---|---|---|
| 1 | National Taiwan University, Taiwan | Link |
| 2 | Sogang University, South Korea | Link |
| 3 | Sogang University, South Korea | Link |
| 4 | Institute for Infocomm Research, A-STAR, Singapore | Link |
| 5 | LINKS Foundation and Politecnico di Torino, Italy | Link |
Detailed Results
All the detailed results are here.