SQuAD

The Stanford Question Answering Dataset

What is SQuAD?

Stanford Question Answering Dataset (SQuAD) is a new reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. With 100,000+ question-answer pairs on 500+ articles, SQuAD is significantly larger than previous reading comprehension datasets.

Explore SQuAD and model predictionsRead the paper (Rajpurkar et al. '16)

Getting Started

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v1.1.py <path_to_dev-v1.1> <path_to_predictions>.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Because SQuAD is an ongoing effort, we expect the dataset to evolve.

To keep up to date with major changes to the dataset, please subscribe:

Have Questions?

Ask us questions at our google group or at pranavsr@stanford.edu.

Star

Leaderboard

Since the release of our dataset, the community has made rapid progress! Here are the ExactMatch (EM) and F1 scores of the best models evaluated on the test set of v1.1. Will your model outperform humans on the QA task?

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar et al. '16)		82.30491.221

1

Jan 22, 2018		Hybrid AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.48289.281

1

Mar 06, 2018		QANet (ensemble)

Google Brain & CMU

82.74489.045

1

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (ensemble model)

Microsoft Research Asia & NUDT

82.84988.764

2

Jan 05, 2018		SLQA+ (ensemble)

Alibaba iDST NLP

82.44088.607

3

Feb 02, 2018		Reinforced Mnemonic Reader (ensemble model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		82.28388.533

3

Feb 27, 2018		QANet (single model)

Google Brain & CMU

82.20988.608

3

Jan 03, 2018		r-net+ (ensemble)

Microsoft Research Asia

82.65088.493

4

Dec 22, 2017		AttentionReader+ (ensemble)

Tencent DPDAC NLP

81.79088.163

4

Dec 17, 2017		r-net (ensemble)

Microsoft Research Asia

http://aka.ms/rnet		82.13688.126

5

Feb 27, 2018		QANet (single model)

Google Brain & CMU

80.92987.773

5

Nov 17, 2017		BiDAF + Self Attention + ELMo (ensemble)

Allen Institute for Artificial Intelligence

81.00387.432

5

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.91987.492

6

Mar 06, 2018		MARS (single model)

YUANFUDAO research NLP

80.66787.465

7

Feb 12, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.48987.454

8

Jan 22, 2018		Hybrid AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

80.02787.288

8

Jan 13, 2018		SLQA+

single model

80.43687.021

9

Jan 12, 2018		EAZI+ (ensemble)

Yiwise NLP Group

80.42686.912

9

Jan 04, 2018		{EAZI} (ensemble)

Yiwise NLP Group

80.43686.912

10

Feb 23, 2018		MAMCN+ (single model)

Samsung Research

79.69286.727

10

Mar 20, 2018		DNET (ensemble)

QA geeks

80.16486.721

11

Mar 15, 2018		Unnamed submission by fensefa

80.02786.612

12

Jan 03, 2018		r-net+ (single model)

Microsoft Research Asia

79.90186.536

12

Feb 12, 2018		BiDAF + Self Attention + ELMo + A2D (single model)

Microsoft Research Asia & NUDT

79.99686.711

13

Dec 28, 2017		SLQA+ (single model)

Alibaba iDST NLP

79.19986.590

13

Dec 05, 2017		SAN (ensemble model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		79.60886.496

13

Jan 29, 2018		Reinforced Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		79.54586.654

14

Oct 17, 2017		Interactive AoA Reader+ (ensemble)

Joint Laboratory of HIT and iFLYTEK

79.08386.450

15

Feb 01, 2018		Unnamed submission by null

78.99986.151

16

Oct 24, 2017		FusionNet (ensemble)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1711.07341		78.97886.016

17

Oct 22, 2017		DCN+ (ensemble)

Salesforce Research

https://arxiv.org/abs/1711.00106		78.85285.996

18

Nov 03, 2017		BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence

78.58085.833

19

Jan 02, 2018		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		78.43385.517

19

Nov 30, 2017		SLQA(ensemble)

Alibaba iDST NLP

78.32885.682

19

Mar 19, 2018		aviqa (ensemble)

aviqa team

78.49685.469

20

Jan 03, 2018		MEMEN (single model)

Zhejiang University

https://arxiv.org/abs/1707.09098		78.23485.344

20

Jan 29, 2018		test

single

78.08785.348

21

Jul 25, 2017		Interactive AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

77.84585.297

22

Jan 10, 2018		Unnamed submission by null

77.43685.130

23

Dec 06, 2017		AttentionReader+ (single)

Tencent DPDAC NLP

77.34284.925

23

Mar 20, 2018		DNET (single model)

QA geeks

77.64684.905

24

Jan 23, 2018		MARS (single model)

YUANFUDAO research NLP

76.85984.739

24

Mar 13, 2018		Unnamed submission by fensefa

77.48984.735

25

Nov 06, 2017		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		76.99684.630

25

Dec 21, 2017		Jenga (ensemble)

Facebook AI Research

77.23784.466

25

Dec 19, 2017		FRC (single model)

in review

76.24084.599

25

Nov 01, 2017		SAN (single model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		76.82884.396

26

Oct 13, 2017		r-net (single model)

Microsoft Research Asia

http://aka.ms/rnet		76.46184.265

26

Dec 13, 2017		RaSoR + TR + LM (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		77.58384.163

27

Oct 22, 2017		Conductor-net (ensemble)

CMU

76.14683.991

28

Sep 08, 2017		FusionNet (single model)

Microsoft Business AI Solutions team

https://arxiv.org/abs/1711.07341		75.96883.900

29

Oct 22, 2017		Interactive AoA Reader+ (single model)

Joint Laboratory of HIT and iFLYTEK

75.82183.843

29

Jul 14, 2017		smarnet (ensemble)

Eigen Technology & Zhejiang University

75.98983.475

30

Mar 15, 2018		AVIQA-v2 (single model)

aviqa team

75.92683.305

31

Aug 18, 2017		RaSoR + TR (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		75.78983.261

32

Oct 23, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		75.08783.081

32

Nov 01, 2017		Mixed model (ensemble)

Sean

75.26582.769

33

Nov 17, 2017		two-attention-self-attention (ensemble)

guotong1988

75.22382.716

33

May 21, 2017		MEMEN (ensemble)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1707.09098		75.37082.658

34

Mar 09, 2017		ReasoNet (ensemble)

MSR Redmond

https://arxiv.org/abs/1609.05284		75.03482.552

34

Jan 02, 2018		Conductor-net (single model)

CMU

https://arxiv.org/abs/1710.10504		74.40582.742

34

Feb 06, 2018		Jenga (single model)

Facebook AI Research

74.37382.845

34

Oct 31, 2017		SLQA (single model)

Alibaba iDST NLP

74.48982.815

34

Jul 10, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		74.86682.806

35

Feb 13, 2018		SSR-BiDAF

ensemble model

74.54182.477

36

Jul 14, 2017		Mnemonic Reader (ensemble)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		74.26882.371

36

Oct 27, 2017		Unnamed submission by null

74.48982.312

36

Dec 23, 2017		S^3-Net (ensemble)

Kangwon National University in South Korea

74.12182.342

37

Nov 06, 2017		Conductor-net (single)

CMU

https://arxiv.org/abs/1710.10504		73.24081.933

37

Jul 29, 2017		SEDT (ensemble model)

CMU

https://arxiv.org/abs/1703.00572		74.09081.761

38

Dec 14, 2017		Jenga (single model)

Facebook AI Research

73.30381.754

38

Jul 06, 2017		SSAE (ensemble)

Tsinghua University

74.08081.665

39

Apr 22, 2017		SEDT+BiDAF (ensemble)

CMU

https://arxiv.org/abs/1703.00572		73.72381.530

39

Feb 22, 2017		BiDAF (ensemble)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		73.74481.525

40

May 01, 2017		jNet (ensemble)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		73.01081.517

40

Jan 24, 2017		Multi-Perspective Matching (ensemble)

IBM Research

https://arxiv.org/abs/1612.04211		73.76581.257

40

Jul 25, 2017		Interactive AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

73.63981.931

41

Sep 20, 2017		BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1710.10723		72.13981.048

42

Dec 15, 2017		S^3-Net (single model)

Kangwon National University in South Korea

71.90881.023

42

Oct 22, 2017		Conductor-net (single)

CMU

72.59081.415

42

Nov 16, 2017		two-attention-self-attention (single model)

guotong1988

72.60081.011

42

Apr 12, 2017		T-gating (ensemble)

Peking University

72.75881.001

43

Mar 03, 2018		AVIQA (single model)

aviqa team

72.48580.550

44

Nov 01, 2016		Dynamic Coattention Networks (ensemble)

Salesforce Research

https://arxiv.org/abs/1611.01604		71.62580.383

44

Apr 13, 2017		QFASE

NUS

71.89879.989

44

Nov 06, 2017		attention+self-attention (single model)

guotong1988

71.69880.462

45

Jul 14, 2017		smarnet (single model)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1710.02772		71.41580.160

46

Jul 14, 2017		Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		70.99580.146

46

Oct 27, 2017		M-NET (single)

UFL

71.01679.835

47

Apr 02, 2017		Ruminating Reader (single model)

New York University

https://arxiv.org/abs/1704.07415		70.63979.456

48

Mar 08, 2017		ReasoNet (single model)

MSR Redmond

https://arxiv.org/abs/1609.05284		70.55579.364

48

Mar 14, 2017		Document Reader (single model)

Facebook AI Research

https://arxiv.org/abs/1704.00051		70.73379.353

48

Dec 28, 2016		FastQAExt

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		70.84978.857

48

Mar 24, 2017		jNet (single model)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		70.60779.821

49

Apr 14, 2017		Multi-Perspective Matching (single model)

IBM Research

https://arxiv.org/abs/1612.04211		70.38778.784

49

May 13, 2017		RaSoR (single model)

Google NY, Tel-Aviv University

https://arxiv.org/abs/1611.01436		70.84978.741

50

Feb 05, 2018		SSR-BiDAF

single model

69.44378.358

50

Aug 30, 2017		SimpleBaseline (single model)

Technical University of Vienna

69.60078.236

51

Apr 12, 2017		SEDT+BiDAF (single model)

CMU

https://arxiv.org/abs/1703.00572		68.47877.971

52

Jun 25, 2017		PQMN (single model)

KAIST & AIBrain & Crosscert

68.33177.783

53

Apr 12, 2017		T-gating (single model)

Peking University

68.13277.569

53

Jul 29, 2017		SEDT (single model)

CMU

https://arxiv.org/abs/1703.00572		68.16377.527

54

Nov 28, 2016		BiDAF (single model)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		67.97477.323

54

Feb 22, 2018		Unnamed submission by null

68.47877.220

54

Jan 22, 2018		FABIR (Single Model)

in review

67.74477.605

54

Feb 22, 2018		Unnamed submission by null

68.42577.077

54

Dec 28, 2016		FastQA

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		68.43677.070

55

Oct 26, 2016		Match-LSTM with Ans-Ptr (Boundary) (ensemble)

Singapore Management University

https://arxiv.org/abs/1608.07905		67.90177.022

55

Sep 19, 2017		AllenNLP BiDAF (single model)

Allen Institute for AI

http://allennlp.org/		67.61877.151

56

Feb 05, 2017		Iterative Co-attention Network

Fudan University

67.50276.786

57

Nov 01, 2016		Dynamic Coattention Networks (single model)

Salesforce Research

https://arxiv.org/abs/1611.01604		66.23375.896

57

Jan 03, 2018		newtest

single model

66.52775.787

58

Feb 24, 2018		Unnamed submission by null

65.99275.469

59

Jan 03, 2018		baseline

single model

64.79674.272

60

Dec 09, 2017		Unnamed submission by ravioncodalab

64.43973.921

60

Oct 26, 2016		Match-LSTM with Bi-Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		64.74473.743

61

Feb 19, 2017		Attentive CNN context with LSTM

NLPR, CASIA

63.30673.463

62

Nov 02, 2016		Fine-Grained Gating

Carnegie Mellon University

https://arxiv.org/abs/1611.01724		62.44673.327

62

Sep 21, 2017		OTF dict+spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		64.08373.056

63

Sep 21, 2017		OTF spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.89772.016

64

Sep 21, 2017		OTF spelling+lemma (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.60471.968

65

Sep 28, 2016		Dynamic Chunk Reader

IBM

https://arxiv.org/abs/1610.09996		62.49970.956

66

Aug 27, 2016		Match-LSTM with Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		60.47470.695

67

Jan 05, 2018		PivRet (single model)

anonymous

58.76469.276

68

Aug 27, 2016		Match-LSTM with Ans-Ptr (Sentence)

Singapore Management University

https://arxiv.org/abs/1608.07905		54.50567.748