Stanford Question Answering Dataset (SQuAD) is a new reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. With 100,000+ question-answer pairs on 500+ articles, SQuAD is significantly larger than previous reading comprehension datasets.
Explore SQuAD and model predictionsRead the paper (Rajpurkar et al. '16)We've built a few resources to help you get started with the dataset.
Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):
To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v1.1.py <path_to_dev-v1.1> <path_to_predictions>.
Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:
Submission TutorialBecause SQuAD is an ongoing effort, we expect the dataset to evolve.
To keep up to date with major changes to the dataset, please subscribe:
Ask us questions at our google group or at pranavsr@stanford.edu.
Since the release of our dataset, the community has made rapid progress! Here are the ExactMatch (EM) and F1 scores of the best models evaluated on the test and development sets of v1.1. Will your model outperform humans on the QA task?
| Rank | Model | EM | F1 |
|---|---|---|---|
| 1 Aug 2017 | DCN+ (ensemble) Salesforce Research | 78.706 | 85.619 |
| 2 Jul 2017 | Interactive AoA Reader (ensemble) Joint Laboratory of HIT and iFLYTEK Research | 77.845 | 85.297 |
| 3 Jun 2017 | r-net (ensemble) Microsoft Research Asia http://aka.ms/rnet | 77.688 | 84.666 |
| 4 Jul 2017 | r-net (single model) Microsoft Research Asia http://aka.ms/rnet | 75.705 | 83.496 |
| 4 Jul 2017 | smarnet (ensemble) Eigen Technology & Zhejiang University | 75.989 | 83.475 |
| 5 Jul 2017 | DCN+ (single model) Salesforce Research | 74.866 | 82.806 |
| 5 May 2017 | MEMEN (ensemble) Eigen Technology & Zhejiang University https://arxiv.org/abs/1707.09098 | 75.370 | 82.658 |
| 6 Mar 2017 | ReasoNet (ensemble) MSR Redmond https://arxiv.org/abs/1609.05284 | 75.034 | 82.552 |
| 7 Jul 2017 | Mnemonic Reader (ensemble) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 74.268 | 82.371 |
| 8 Jul 2017 | Interactive AoA Reader (single model) Joint Laboratory of HIT and iFLYTEK Research | 73.639 | 81.931 |
| 8 Jul 2017 | SEDT (ensemble model) CMU https://arxiv.org/abs/1703.00572 | 74.090 | 81.761 |
| 9 Jul 2017 | SSAE (ensemble) Tsinghua University | 74.080 | 81.665 |
| 10 Apr 2017 | SEDT+BiDAF (ensemble) CMU https://arxiv.org/abs/1703.00572 | 73.723 | 81.530 |
| 10 Feb 2017 | BiDAF (ensemble) Allen Institute for AI & University of Washington https://arxiv.org/abs/1611.01603 | 73.744 | 81.525 |
| 11 May 2017 | jNet (ensemble) USTC & National Research Council Canada & York University https://arxiv.org/abs/1703.04617 | 73.010 | 81.517 |
| 11 Jan 2017 | Multi-Perspective Matching (ensemble) IBM Research https://arxiv.org/abs/1612.04211 | 73.765 | 81.257 |
| 12 Apr 2017 | T-gating (ensemble) Peking University | 72.758 | 81.001 |
| 13 Nov 2016 | Dynamic Coattention Networks (ensemble) Salesforce Research https://arxiv.org/abs/1611.01604 | 71.625 | 80.383 |
| 14 Jul 2017 | smarnet (single model) Eigen Technology & Zhejiang University | 71.415 | 80.160 |
| 15 Jul 2017 | Mnemonic Reader (single model) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 70.995 | 80.146 |
| 15 Apr 2017 | QFASE NUS | 71.898 | 79.989 |
| 16 Mar 2017 | jNet (single model) USTC & National Research Council Canada & York University https://arxiv.org/abs/1703.04617 | 70.607 | 79.821 |
| 16 Apr 2017 | Ruminating Reader (single model) New York University https://arxiv.org/abs/1704.07415 | 70.639 | 79.456 |
| 17 Mar 2017 | ReasoNet (single model) MSR Redmond https://arxiv.org/abs/1609.05284 | 70.555 | 79.364 |
| 17 Mar 2017 | Document Reader (single model) Facebook AI Research https://arxiv.org/abs/1704.00051 | 70.733 | 79.353 |
| 17 Dec 2016 | FastQAExt German Research Center for Artificial Intelligence https://arxiv.org/abs/1703.04816 | 70.849 | 78.857 |
| 18 Apr 2017 | Multi-Perspective Matching (single model) IBM Research https://arxiv.org/abs/1612.04211 | 70.387 | 78.784 |
| 18 May 2017 | RaSoR (single model) Google NY, Tel-Aviv University https://arxiv.org/abs/1611.01436 | 70.849 | 78.741 |
| 19 Apr 2017 | SEDT+BiDAF (single model) CMU https://arxiv.org/abs/1703.00572 | 68.478 | 77.971 |
| 20 Jun 2017 | PQMN (single model) KAIST & AIBrain & Crosscert | 68.331 | 77.783 |
| 21 Apr 2017 | T-gating (single model) Peking University | 68.132 | 77.569 |
| 21 Jul 2017 | SEDT (single model) CMU https://arxiv.org/abs/1703.00572 | 68.163 | 77.527 |
| 22 Nov 2016 | BiDAF (single model) Allen Institute for AI & University of Washington https://arxiv.org/abs/1611.01603 | 67.974 | 77.323 |
| 22 Dec 2016 | FastQA German Research Center for Artificial Intelligence https://arxiv.org/abs/1703.04816 | 68.436 | 77.070 |
| 23 Oct 2016 | Match-LSTM with Ans-Ptr (Boundary) (ensemble) Singapore Management University https://arxiv.org/abs/1608.07905 | 67.901 | 77.022 |
| 24 Feb 2017 | Iterative Co-attention Network Fudan University | 67.502 | 76.786 |
| 25 Nov 2016 | Dynamic Coattention Networks (single model) Salesforce Research https://arxiv.org/abs/1611.01604 | 66.233 | 75.896 |
| 26 Oct 2016 | Match-LSTM with Bi-Ans-Ptr (Boundary) Singapore Management University https://arxiv.org/abs/1608.07905 | 64.744 | 73.743 |
| 27 Feb 2017 | Attentive CNN context with LSTM NLPR, CASIA | 63.306 | 73.463 |
| 28 Nov 2016 | Fine-Grained Gating Carnegie Mellon University https://arxiv.org/abs/1611.01724 | 62.446 | 73.327 |
| 28 Sep 2016 | Dynamic Chunk Reader IBM https://arxiv.org/abs/1610.09996 | 62.499 | 70.956 |
| 29 Aug 2016 | Match-LSTM with Ans-Ptr (Boundary) Singapore Management University https://arxiv.org/abs/1608.07905 | 60.474 | 70.695 |
| 30 Aug 2016 | Match-LSTM with Ans-Ptr (Sentence) Singapore Management University https://arxiv.org/abs/1608.07905 | 54.505 | 67.748 |
| Human Performance Stanford University (Rajpurkar et al. '16) | 82.304 | 91.221 |