Stanford Question Answering Dataset (SQuAD) is a new reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. With 100,000+ question-answer pairs on 500+ articles, SQuAD is significantly larger than previous reading comprehension datasets.
Explore SQuAD and model predictionsRead the paper (Rajpurkar et al. '16)We've built a few resources to help you get started with the dataset.
Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):
To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v1.1.py <path_to_dev-v1.1> <path_to_predictions>.
Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:
Submission TutorialBecause SQuAD is an ongoing effort, we expect the dataset to evolve.
To keep up to date with major changes to the dataset, please subscribe:
Ask us questions at our google group or at pranavsr@stanford.edu.
Since the release of our dataset, the community has made rapid progress! Here are the ExactMatch (EM) and F1 scores of the best models evaluated on the test and development sets of v1.1. Will your model outperform humans on the QA task?
| Rank | Model | EM | F1 |
|---|---|---|---|
| 1 Sep 20, 2017 | AIR-FusionNet (ensemble) Microsoft Business AI Solutions Team | 78.842 | 85.936 |
| 2 Aug 16, 2017 | DCN+ (ensemble) Salesforce Research | 78.706 | 85.619 |
| 3 Jul 25, 2017 | Interactive AoA Reader (ensemble) Joint Laboratory of HIT and iFLYTEK Research | 77.845 | 85.297 |
| 3 Sep 01, 2017 | r-net (ensemble) Microsoft Research Asia http://aka.ms/rnet | 78.244 | 85.206 |
| 4 Aug 21, 2017 | Reinforced Mnemonic Reader (ensemble) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 77.678 | 84.888 |
| 5 Sep 08, 2017 | AIR-FusionNet (single model) Microsoft Business AI Solutions team | 75.968 | 83.900 |
| 6 Jul 17, 2017 | r-net (single model) Microsoft Research Asia http://aka.ms/rnet | 75.705 | 83.496 |
| 6 Jul 14, 2017 | smarnet (ensemble) Eigen Technology & Zhejiang University | 75.989 | 83.475 |
| 7 Aug 18, 2017 | Reg-RaSoR (single model) Google NY, Tel-Aviv University | 75.789 | 83.261 |
| 8 Jul 10, 2017 | DCN+ (single model) Salesforce Research | 74.866 | 82.806 |
| 8 Sep 19, 2017 | SLQA (ensemble model) Alibaba iDST NLP | 75.212 | 82.681 |
| 8 May 21, 2017 | MEMEN (ensemble) Eigen Technology & Zhejiang University https://arxiv.org/abs/1707.09098 | 75.370 | 82.658 |
| 9 Mar 09, 2017 | ReasoNet (ensemble) MSR Redmond https://arxiv.org/abs/1609.05284 | 75.034 | 82.552 |
| 10 Jul 14, 2017 | Mnemonic Reader (ensemble) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 74.268 | 82.371 |
| 11 Jul 25, 2017 | Interactive AoA Reader (single model) Joint Laboratory of HIT and iFLYTEK Research | 73.639 | 81.931 |
| 12 Aug 21, 2017 | Reinforced Mnemonic Reader (single model) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 73.188 | 81.816 |
| 12 Jul 29, 2017 | SEDT (ensemble model) CMU https://arxiv.org/abs/1703.00572 | 74.090 | 81.761 |
| 13 Jul 06, 2017 | SSAE (ensemble) Tsinghua University | 74.080 | 81.665 |
| 14 Apr 22, 2017 | SEDT+BiDAF (ensemble) CMU https://arxiv.org/abs/1703.00572 | 73.723 | 81.530 |
| 14 Feb 22, 2017 | BiDAF (ensemble) Allen Institute for AI & University of Washington https://arxiv.org/abs/1611.01603 | 73.744 | 81.525 |
| 15 May 01, 2017 | jNet (ensemble) USTC & National Research Council Canada & York University https://arxiv.org/abs/1703.04617 | 73.010 | 81.517 |
| 15 Jan 24, 2017 | Multi-Perspective Matching (ensemble) IBM Research https://arxiv.org/abs/1612.04211 | 73.765 | 81.257 |
| 16 Sep 20, 2017 | BiDAF + Self Attention (single model) Allen Institute for Artificial Intelligence | 72.139 | 81.048 |
| 16 Apr 12, 2017 | T-gating (ensemble) Peking University | 72.758 | 81.001 |
| 17 Sep 08, 2017 | SLQA (single model) Alibaba iDST NLP | 72.023 | 80.648 |
| 18 Nov 01, 2016 | Dynamic Coattention Networks (ensemble) Salesforce Research https://arxiv.org/abs/1611.01604 | 71.625 | 80.383 |
| 19 Jul 14, 2017 | smarnet (single model) Eigen Technology & Zhejiang University | 71.415 | 80.160 |
| 20 Jul 14, 2017 | Mnemonic Reader (single model) NUDT and Fudan University https://arxiv.org/abs/1705.02798 | 70.995 | 80.146 |
| 20 Apr 13, 2017 | QFASE NUS | 71.898 | 79.989 |
| 21 Mar 24, 2017 | jNet (single model) USTC & National Research Council Canada & York University https://arxiv.org/abs/1703.04617 | 70.607 | 79.821 |
| 21 Apr 02, 2017 | Ruminating Reader (single model) New York University https://arxiv.org/abs/1704.07415 | 70.639 | 79.456 |
| 22 Mar 08, 2017 | ReasoNet (single model) MSR Redmond https://arxiv.org/abs/1609.05284 | 70.555 | 79.364 |
| 22 Mar 14, 2017 | Document Reader (single model) Facebook AI Research https://arxiv.org/abs/1704.00051 | 70.733 | 79.353 |
| 22 Dec 28, 2016 | FastQAExt German Research Center for Artificial Intelligence https://arxiv.org/abs/1703.04816 | 70.849 | 78.857 |
| 23 Apr 14, 2017 | Multi-Perspective Matching (single model) IBM Research https://arxiv.org/abs/1612.04211 | 70.387 | 78.784 |
| 23 May 13, 2017 | RaSoR (single model) Google NY, Tel-Aviv University https://arxiv.org/abs/1611.01436 | 70.849 | 78.741 |
| 24 Aug 30, 2017 | SimpleBaseline (single model) Technical University of Vienna | 69.600 | 78.236 |
| 25 Apr 12, 2017 | SEDT+BiDAF (single model) CMU https://arxiv.org/abs/1703.00572 | 68.478 | 77.971 |
| 26 Jun 25, 2017 | PQMN (single model) KAIST & AIBrain & Crosscert | 68.331 | 77.783 |
| 27 Apr 12, 2017 | T-gating (single model) Peking University | 68.132 | 77.569 |
| 27 Jul 29, 2017 | SEDT (single model) CMU https://arxiv.org/abs/1703.00572 | 68.163 | 77.527 |
| 28 Nov 28, 2016 | BiDAF (single model) Allen Institute for AI & University of Washington https://arxiv.org/abs/1611.01603 | 67.974 | 77.323 |
| 29 Sep 19, 2017 | AllenNLP BiDAF (single model) Allen Institute for AI http://allennlp.org/ | 67.618 | 77.151 |
| 29 Dec 28, 2016 | FastQA German Research Center for Artificial Intelligence https://arxiv.org/abs/1703.04816 | 68.436 | 77.070 |
| 30 Oct 26, 2016 | Match-LSTM with Ans-Ptr (Boundary) (ensemble) Singapore Management University https://arxiv.org/abs/1608.07905 | 67.901 | 77.022 |
| 31 Feb 05, 2017 | Iterative Co-attention Network Fudan University | 67.502 | 76.786 |
| 32 Nov 01, 2016 | Dynamic Coattention Networks (single model) Salesforce Research https://arxiv.org/abs/1611.01604 | 66.233 | 75.896 |
| 33 Oct 26, 2016 | Match-LSTM with Bi-Ans-Ptr (Boundary) Singapore Management University https://arxiv.org/abs/1608.07905 | 64.744 | 73.743 |
| 34 Feb 19, 2017 | Attentive CNN context with LSTM NLPR, CASIA | 63.306 | 73.463 |
| 35 Nov 02, 2016 | Fine-Grained Gating Carnegie Mellon University https://arxiv.org/abs/1611.01724 | 62.446 | 73.327 |
| 35 Sep 28, 2016 | Dynamic Chunk Reader IBM https://arxiv.org/abs/1610.09996 | 62.499 | 70.956 |
| 36 Aug 27, 2016 | Match-LSTM with Ans-Ptr (Boundary) Singapore Management University https://arxiv.org/abs/1608.07905 | 60.474 | 70.695 |
| 37 Aug 27, 2016 | Match-LSTM with Ans-Ptr (Sentence) Singapore Management University https://arxiv.org/abs/1608.07905 | 54.505 | 67.748 |
| Human Performance Stanford University (Rajpurkar et al. '16) | 82.304 | 91.221 |