Stanford Question Answering Dataset (SQuAD) is a new reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. With 100,000+ question-answer pairs on 500+ articles, SQuAD is significantly larger than previous reading comprehension datasets.
Since the release of our dataset (and paper), the community has made rapid progress! Here are the ExactMatch (EM) and F1 scores of the best models evaluated on the test and development sets of v1.1. Will your model outperform humans on the QA task?
| Model | Test EM | Test F1 | Dev EM | Dev F1 |
|---|---|---|---|---|
| Human Performance (Stanford University) (Rajpurkar et al. '16) | 82.3 | 91.2 | 81.4 | 91.0 |
| r-net (Microsoft Research Asia) | TBD | TBD | 65.8 | 75.0 |
| Co-attention (Allen Institute for Artificial Intelligence & University of Washington) | 61.8 | 72.5 | 62.2 | 72.6 |
| No Chunker and Neural Chunk Ranker with Fine Attention (IBM) | TBD | TBD | 61.8 | 70.7 |
| Match-LSTM with Ans-Ptr (Boundary) (Singapore Management University) (Wang & Jiang '16) | 60.5 | 70.7 | 59.4 | 70.0 |
| Trained Chunker and Neural Chunk Ranker (IBM) | TBD | TBD | 58.7 | 68.4 |
| Match-LSTM with Ans-Ptr (Sequence) (Singapore Management University) (Wang & Jiang '16) | 54.5 | 67.7 | 54.8 | 68.0 |
| Attention and Chunking Single Model (IBM) | TBD | TBD | 48.0 | 64.5 |
| Logistic Regression Baseline (Stanford University) (Rajpurkar et al. '16) | 40.4 | 51.0 | 39.8 | 51.0 |
We've built a few resources to help you get started with the dataset.
To get a feel of the dataset, you can explore it visually.
Explore SQuADDownload a copy of the dataset (distributed under the CC BY-SA 4.0 license):
To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v1.1.py <path_to_dev-v1.1> <path_to_predictions>.
Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:
Submission TutorialBecause SQuAD is an ongoing effort, we expect the dataset to evolve.
To keep up to date with major changes to the dataset, please subscribe:
Ask us questions at our google group or at pranavsr@stanford.edu.