SQuAD2.0

The Stanford Question Answering Dataset

What is SQuAD?

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

New SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 new, unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD2.0 is a challenging natural language understanding task for existing models, and we release SQuAD2.0 to the community as the successor to SQuAD1.1. We are optimistic that this new dataset will encourage the development of reading comprehension systems that know what they don't know.

Explore SQuAD2.0 and model predictionsSQuAD2.0 paper (Rajpurkar & Jia et al. '18)

SQuAD 1.1, the previous version of the SQuAD dataset, contains 100,000+ question-answer pairs on 500+ articles.

Explore SQuAD1.1 and model predictionsSQuAD1.0 paper (Rajpurkar et al. '16)

Getting Started

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v2.0.py <path_to_dev-v2.0> <path_to_predictions>.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Because SQuAD is an ongoing effort, we expect the dataset to evolve.

To keep up to date with major changes to the dataset, please subscribe:

Have Questions?

Ask us questions at our google group or at pranavsr@stanford.edu and robinjia@stanford.edu.

Star

Leaderboard

SQuAD2.0 tests the ability of a system to not only answer reading comprehension questions, but also abstain when presented with a question that cannot be answered based on the provided paragraph. How will your system compare to humans on this task?

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar & Jia et al. '18)		86.83189.452

1

Mar 20, 2019		BERT + DAE + AoA (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

87.14789.474

2

Mar 15, 2019		BERT + ConvLSTM + MTL + Verifier (ensemble)

Layer 6 AI

86.73089.286

3

Mar 05, 2019		BERT + N-Gram Masking + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert		86.67389.147

4

May 21, 2019		XLNet (single model)

XLNet Team

86.34689.133

5

Apr 13, 2019		SemBERT(ensemble)

Shanghai Jiao Tong University

86.16688.886

5

May 14, 2019		SG-Net (ensemble)

Anonymous

86.21188.848

6

Mar 16, 2019		BERT + DAE + AoA (single model)

Joint Laboratory of HIT and iFLYTEK Research

85.88488.621

7

May 14, 2019		SG-Net (single model)

Anonymous

85.22987.926

8

Apr 11, 2019		SemBERT (single model)

Shanghai Jiao Tong University

84.80087.864

8

Mar 05, 2019		BERT + N-Gram Masking + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert		85.15087.715

9

Apr 16, 2019		Insight-baseline-BERT (single model)

PAII Insight Team

84.83487.644

9

Jan 15, 2019		BERT + MMFT + ADA (ensemble)

Microsoft Research Asia

85.08287.615

9

Mar 13, 2019		BERT + ConvLSTM + MTL + Verifier (single model)

Layer 6 AI

84.92488.204

9

Jun 04, 2019		BertCNN (single model)

Anonymous

85.24087.901

10

Jan 10, 2019		BERT + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert		84.29286.967

11

Dec 21, 2018		PAML+BERT (ensemble model)

PINGAN GammaLab

83.45786.122

11

Mar 20, 2019		Bert-raw (ensemble)

None

83.60486.036

11

Dec 13, 2018		BERT finetune baseline (ensemble)

Anonymous

83.53686.096

12

Dec 16, 2018		Lunet + Verifier + BERT (ensemble)

Layer 6 AI NLP Team

83.46986.043

13

May 14, 2019		ATB (single model)

Anonymous

82.88286.002

13

Jan 14, 2019		BERT + MMFT + ADA (single model)

Microsoft Research Asia

83.04085.892

14

Jan 10, 2019		BERT + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert		82.97285.810

14

Feb 26, 2019		BERT with Something (ensemble)

Anonymous

83.05185.737

14

Dec 15, 2018		Lunet + Verifier + BERT (single model)

Layer 6 AI NLP Team

82.99586.035

15

Feb 15, 2019		BERT + NeurQuRI (ensemble)

2SAH

82.80385.703

15

Mar 11, 2019		Bert-raw (ensemble)

None

83.11985.510

15

Feb 16, 2019		Bert-raw (ensemble)

None

83.17585.635

16

Feb 27, 2019		BERT + NeurQuRI (ensemble)

2SAH

82.71385.584

16

May 13, 2019		BERT-Base + QA Pre-training (single model)

Anonymous

82.72485.491

16

Dec 16, 2018		PAML+BERT (single model)

PINGAN GammaLab

82.57785.603

17

Mar 03, 2019		Unnamed submission by null

82.43185.178

17

Nov 16, 2018		AoA + DA + BERT (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.37485.310

18

Feb 28, 2019		BERT_s (single model)

Anonymous

81.97984.846

18

Dec 12, 2018		BERT finetune baseline (single model)

Anonymous

82.12684.820

18

Dec 10, 2018		Candi-Net+BERT (ensemble)

42Maru NLP Team

82.12684.624

18

Jun 06, 2019		Unnamed submission by zw10

81.73184.862

19

Feb 28, 2019		BERT-large+UBFT (single model)

anonymous

81.57384.535

20

Feb 15, 2019		BERT + NeurQuRI (single model)

2SAH

81.25784.342

20

Feb 25, 2019		BERT with Something (single model)

Anonymous

81.11084.386

20

Nov 16, 2018		AoA + DA + BERT (single model)

Joint Laboratory of HIT and iFLYTEK Research

81.17884.251

21

Mar 20, 2019		Bert-raw (single)

None

80.69383.922

21

Mar 07, 2019		BERT + UnAnsQ (single model)

Anonymous

80.74983.851

22

Apr 07, 2019		BERT + AL (single model)

Anonymous

80.71583.827

23

Dec 19, 2018		Candi-Net+BERT (single model)

42Maru NLP Team

80.65983.562

24

Jan 09, 2019		Unnamed submission by null

80.51283.539

24

Jan 22, 2019		BERT + NeurQuRI (single model)

2SAH

80.59183.391

25

May 28, 2019		Bert

Single Model

https://senseforth.ai		80.42283.118

25

Apr 19, 2019		Unnamed submission by null

80.35483.329

26

Feb 16, 2019		Bert-raw (single model)

None

80.34383.243

26

Jan 09, 2019		Unnamed submission by null

80.34383.221

27

Feb 19, 2019		BERT + UDA (single model)

Anonymous

80.00583.208

27

Dec 03, 2018		PwP+BERT (single model)

AITRICS

80.11783.189

28

Apr 10, 2019		bert (single model)

vinda msqjmxx

79.97183.184

28

Apr 04, 2019		BISAN-CC (single model)

Seoul National University & Hyundai Motors

80.20883.149

28

Mar 11, 2019		Bert-raw (single)

None

80.41183.457

29

Nov 08, 2018		BERT (single model)

Google AI Language

80.00583.061

30

Feb 12, 2019		BERT + Sparse-Transformer

single model

79.94883.023

30

Feb 28, 2019		ST_bl

single model

80.14082.962

30

Dec 05, 2018		Candi-Net+BERT (single model)

42Maru NLP Team

80.38882.908

30

Dec 06, 2018		NEXYS_BASE (single model)

NEXYS, DGIST R7

79.77982.912

30

Mar 07, 2019		BERT uncased (single model)

Anonymous

79.74583.020

31

Feb 01, 2019		{bert-finetuning} (single model)

ksai

79.63282.852

32

Mar 14, 2019		{Anonymous} (single model)

Anonymous

78.87682.524

32

Nov 09, 2018		L6Net + BERT (single model)

Layer 6 AI

79.18182.259

33

Mar 14, 2019		BISAN (single model)

Seoul National University & Hyundai Motors

78.48181.531

33

Apr 24, 2019		BERT + WIAN (ensemble)

Infosys Limited

78.65081.497

34

Jan 09, 2019		Unnamed submission by null

78.30181.350

35

Dec 14, 2018		BERT+AC(single model)

Hithink RoyalFlush

78.05281.174

36

Nov 06, 2018		SLQA+BERT (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158		77.00380.209

37

Jan 05, 2019		synss (single model )

bert_finetune

76.05579.329

38

Dec 18, 2018		ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai		74.74678.227

38

Nov 05, 2018		MIR-MRC(F-Net) (single model)

Kangwon National University, Natural Language Processing Lab. & ForceWin, KP Lab.

74.79177.988

39

May 23, 2019		{BERTcw} (single model)

private

74.38577.308

40

Sep 13, 2018		nlnet (single model)

Microsoft Research Asia

74.27277.052

41

Dec 22, 2018		Unnamed submission by null

73.23476.790

41

Dec 29, 2018		MMIPN

Single

73.50576.424

42

Apr 20, 2019		BERT-Base (single model)

Dining Philosophers

73.09976.236

43

Oct 12, 2018		YARCS (ensemble)

IBM Research AI

72.67075.507

44

Nov 14, 2018		BERT+Answer Verifier (single model)

Pingan Tech Olatop Lab

71.66675.457

44

Nov 10, 2018		Unnamed submission by null

72.58075.075

45

Sep 17, 2018		Unet (ensemble)

Fudan University & Liulishuo Lab

https://arxiv.org/abs/1810.06638		71.41774.869

46

Jan 19, 2019		{BERT-base} (single-model)

Anonymous

70.76374.449

46

Aug 28, 2018		SLQA+ (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158		71.46274.434

46

Apr 24, 2019		BERT-Base (single)

GreenflyAI

https://greenfly.ai		71.69974.430

46

Aug 15, 2018		Reinforced Mnemonic Reader + Answer Verifier (single model)

NUDT

https://arxiv.org/abs/1808.05759		71.76774.295

47

Sep 14, 2018		SAN (ensemble model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556		71.31673.704

48

Nov 10, 2018		Unnamed submission by null

70.71873.403

49

Sep 26, 2018		Multi-Level Attention Fusion(MLAF) (single model)

Chonbuk National University, Cognitive Computing Lab.

69.47672.857

49

Aug 21, 2018		FusionNet++ (ensemble)

Microsoft Business Applications Group AI Research

https://arxiv.org/abs/1711.07341		70.30072.484

49

Sep 14, 2018		Unet (single model)

Fudan University & Liulishuo Lab

69.26272.642

50

Dec 20, 2018		DocQA + NeurQuRI (single model)

2SAH

68.76671.662

51

Sep 13, 2018		BiDAF++ with pair2vec (single model)

UW and FAIR

68.02171.583

51

Aug 21, 2018		SAN (single model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556		68.65371.439

51

Nov 10, 2018		Unnamed submission by null

68.65371.124

52

Jul 13, 2018		VS^3-NET (single model)

Kangwon National University in South Korea

67.89770.884

52

Jun 24, 2018		KACTEIL-MRC(GFN-Net) (single model)

Kangwon National University, Natural Language Processing Lab.

68.21370.878

53

Jan 01, 2019		EBB-Net (single model)

Enliple AI

66.61070.303

54

Jun 25, 2018		KakaoNet2 (single model)

Kakao NLP Team

65.71969.381

55

Jul 11, 2018		abcNet (single model)

Fudan University & Liulishuo AI Lab

65.25669.206

55

Sep 13, 2018		BiDAF++ (single model)

UW and FAIR

65.65168.866

56

Jun 27, 2018		BSAE AddText (single model)

reciTAL.ai

63.33867.422

57

Aug 14, 2018		eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com		63.32766.633

57

May 30, 2018		BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

63.37266.251

58

Nov 27, 2018		Tree-LSTM + BiDAF + ELMo (single model)

Carnegie Mellon University

57.70762.341

58

May 30, 2018		BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

59.33262.305

59

May 30, 2018		BiDAF-No-Answer (single model)

University of Washington [modified by Stanford]

59.17462.093

60

May 20, 2019		BNA + SoftDrop (single model)

Shanghai University of Finance and Economics and Shanghai Key Laboratory of Data Science, Fudan University

58.50862.045

61

May 20, 2019		BNA + HardDrop (single model)

Shanghai University of Finance and Economics and Shanghai Key Laboratory of Data Science, Fudan University

56.54559.546

SQuAD1.1 Leaderboard

Here are the ExactMatch (EM) and F1 scores evaluated on the test set of SQuAD v1.1.

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar et al. '16)		82.30491.221

1

May 21, 2019		XLNet (single model)

XLNet Team

89.89895.080

2

Oct 05, 2018		BERT (ensemble)

Google AI Language

https://arxiv.org/abs/1810.04805		87.43393.160

3

May 14, 2019		ATB (single model)

Anonymous

86.94092.641

4

Feb 14, 2019		KT-NET (single model)

Baidu NLP

85.94492.425

5

Feb 28, 2019		ST_bl

single model

85.43091.976

5

Sep 26, 2018		nlnet (ensemble)

Microsoft Research Asia

85.95491.677

6

Feb 16, 2019		BERT+Sparse-Transformer

single model

85.12591.623

6

Sep 09, 2018		nlnet (ensemble)

Microsoft Research Asia

85.35691.202

6

Mar 14, 2019		BISAN (single model)

Seoul National University & Hyundai Motors

85.31491.756

6

Oct 05, 2018		BERT (single model)

Google AI Language

https://arxiv.org/abs/1810.04805		85.08391.835

7

Feb 21, 2019		WD1 (single model)

Anonymous

83.80490.429

7

Jul 08, 2018		r-net (ensemble)

Microsoft Research Asia

84.00390.147

7

Apr 21, 2019		Common-sense Governed BERT-123 (single model)

Jerry AGI Ragtag

83.93090.613

7

Jul 11, 2018		QANet (ensemble)

Google Brain & CMU

84.45490.490

7

Feb 19, 2019		WD (single model)

Anonymous

84.40290.561

8

Jun 20, 2018		MARS (ensemble)

YUANFUDAO research NLP

83.98289.796

9

Mar 19, 2018		QANet (ensemble)

Google Brain & CMU

83.87789.737

9

Sep 09, 2018		nlnet (single model)

Microsoft Research Asia

83.46890.133

10

Sep 01, 2018		MARS (single model)

YUANFUDAO research NLP

83.18589.547

11

Jun 21, 2018		MARS (single model)

YUANFUDAO research NLP

83.12289.224

11

Jun 20, 2018		QANet (single)

Google Brain & CMU

82.47189.306

11

Jan 22, 2018		Hybrid AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.48289.281

11

Mar 06, 2018		QANet (ensemble)

Google Brain & CMU

82.74489.045

12

Dec 23, 2018		MMIPN

Single

81.58088.948

12

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (ensemble model)

Microsoft Research Asia & NUDT

82.84988.764

13

Feb 27, 2018		QANet (single model)

Google Brain & CMU

82.20988.608

13

Jan 05, 2018		SLQA+ (ensemble)

Alibaba iDST NLP

82.44088.607

14

Feb 02, 2018		Reinforced Mnemonic Reader (ensemble model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		82.28388.533

14

Jan 03, 2018		r-net+ (ensemble)

Microsoft Research Asia

82.65088.493

14

May 09, 2018		MARS (single model)

YUANFUDAO research NLP

82.58788.880

15

Dec 22, 2017		AttentionReader+ (ensemble)

Tencent DPDAC NLP

81.79088.163

16

May 09, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

81.53888.130

16

Dec 17, 2017		r-net (ensemble)

Microsoft Research Asia

http://aka.ms/rnet		82.13688.126

17

May 09, 2018		Reinforced Mnemonic Reader + A2D + DA (single model)

Microsoft Research Asia & NUDT

81.40188.122

17

Apr 03, 2018		KACTEIL-MRC(GF-Net+) (ensemble)

Kangwon National University, Natural Language Processing Lab.

81.49687.557

17

Dec 17, 2018		ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai		81.30788.909

17

Apr 23, 2018		r-net (single model)

Microsoft Research Asia

81.39188.170

18

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.91987.492

19

Feb 12, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.48987.454

19

Feb 27, 2018		QANet (single model)

Google Brain & CMU

80.92987.773

19

Nov 17, 2017		BiDAF + Self Attention + ELMo (ensemble)

Allen Institute for Artificial Intelligence

81.00387.432

20

Apr 12, 2018		AVIQA+ (ensemble)

aviqa team

80.61587.311

21

Jan 22, 2018		Hybrid AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

80.02787.288

21

Jan 12, 2018		EAZI+ (ensemble)

Yiwise NLP Group

80.42686.912

21

Jan 04, 2018		{EAZI} (ensemble)

Yiwise NLP Group

80.43686.912

21

Jan 13, 2018		SLQA+

single model

80.43687.021

22

Mar 20, 2018		DNET (ensemble)

QA geeks

80.16486.721

23

Feb 12, 2018		BiDAF + Self Attention + ELMo + A2D (single model)

Microsoft Research Asia & NUDT

79.99686.711

23

Feb 23, 2018		MAMCN+ (single model)

Samsung Research

79.69286.727

24

Jan 29, 2018		Reinforced Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		79.54586.654

24

Jan 03, 2018		r-net+ (single model)

Microsoft Research Asia

79.90186.536

25

Dec 05, 2017		SAN (ensemble model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		79.60886.496

25

Apr 10, 2018		Unnamed submission by null

80.02786.612

26

Dec 28, 2017		SLQA+ (single model)

Alibaba iDST NLP

79.19986.590

27

Oct 17, 2017		Interactive AoA Reader+ (ensemble)

Joint Laboratory of HIT and iFLYTEK

79.08386.450

27

Nov 05, 2018		MIR-MRC(F-Net) (single model)

ForceWin, KP Lab.

79.08386.288

28

Feb 01, 2018		Unnamed submission by null

78.99986.151

29

Oct 24, 2017		FusionNet (ensemble)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1711.07341		78.97886.016

29

Jun 01, 2018		MDReader

single model

79.03186.006

30

Oct 22, 2017		DCN+ (ensemble)

Salesforce Research

https://arxiv.org/abs/1711.00106		78.85285.996

31

Mar 29, 2018		KACTEIL-MRC(GF-Net+) (single model)

Kangwon National University, Natural Language Processing Lab.

78.66485.780

31

Nov 03, 2017		BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence

78.58085.833

32

May 09, 2018		KakaoNet (single model)

Kakao NLP Team

78.40185.724

33

Nov 30, 2017		SLQA(ensemble)

Alibaba iDST NLP

78.32885.682

33

Mar 19, 2018		aviqa (ensemble)

aviqa team

78.49685.469

33

Jan 02, 2018		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		78.43385.517

33

Sep 18, 2018		BiDAF++ with pair2vec (single model)

UW and FAIR

78.22385.535

33

Jan 03, 2018		MEMEN (single model)

Zhejiang University

https://arxiv.org/abs/1707.09098		78.23485.344

33

Jun 01, 2018		MDReader0

single model

78.17185.543

34

Jan 29, 2018		test

single

78.08785.348

35

Jul 25, 2017		Interactive AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

77.84585.297

36

Dec 13, 2017		RaSoR + TR + LM (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		77.58384.163

36

Jan 10, 2018		Unnamed submission by null

77.43685.130

36

Sep 18, 2018		BiDAF++ (single model)

UW and FAIR

77.57384.858

37

Apr 10, 2018		Unnamed submission by null

77.48984.735

37

Dec 06, 2017		AttentionReader+ (single)

Tencent DPDAC NLP

77.34284.925

37

Mar 20, 2018		DNET (single model)

QA geeks

77.64684.905

38

Dec 21, 2017		Jenga (ensemble)

Facebook AI Research

77.23784.466

38

Dec 19, 2017		FRC (single model)

in review

76.24084.599

38

May 14, 2018		VS^3-NET (single model)

Kangwon National University in South Korea

76.77584.491

38

Jan 23, 2018		MARS (single model)

YUANFUDAO research NLP

76.85984.739

39

Nov 01, 2017		SAN (single model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		76.82884.396

40

Oct 13, 2017		r-net (single model)

Microsoft Research Asia

http://aka.ms/rnet		76.46184.265

40

Sep 26, 2018		{gqa} (single model)

FAIR

77.09083.931

40

Oct 22, 2017		Conductor-net (ensemble)

CMU

76.14683.991

40

Nov 06, 2017		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		76.99684.630

41

Sep 08, 2017		FusionNet (single model)

Microsoft Business AI Solutions team

https://arxiv.org/abs/1711.07341		75.96883.900

42

Oct 22, 2017		Interactive AoA Reader+ (single model)

Joint Laboratory of HIT and iFLYTEK

75.82183.843

42

Jul 14, 2017		smarnet (ensemble)

Eigen Technology & Zhejiang University

75.98983.475

43

Mar 15, 2018		AVIQA-v2 (single model)

aviqa team

75.92683.305

43

Oct 18, 2018		KAR (single model)

York University

https://arxiv.org/abs/1809.03449		76.12583.538

44

Aug 18, 2017		RaSoR + TR (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		75.78983.261

45

Oct 23, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		75.08783.081

45

Oct 05, 2018		Unnamed submission by null

74.95083.294

45

Nov 01, 2017		Mixed model (ensemble)

Sean

75.26582.769

46

Nov 17, 2017		two-attention-self-attention (ensemble)

guotong1988

75.22382.716

46

Oct 31, 2017		SLQA (single model)

Alibaba iDST NLP

74.48982.815

46

Mar 09, 2017		ReasoNet (ensemble)

MSR Redmond

https://arxiv.org/abs/1609.05284		75.03482.552

46

Jul 10, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		74.86682.806

46

May 21, 2017		MEMEN (ensemble)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1707.09098		75.37082.658

47

Aug 14, 2018		eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com		74.60482.501

48

Feb 13, 2018		SSR-BiDAF

ensemble model

74.54182.477

49

Oct 27, 2017		Unnamed submission by null

74.48982.312

49

Jul 14, 2017		Mnemonic Reader (ensemble)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		74.26882.371

50

Dec 23, 2017		S^3-Net (ensemble)

Kangwon National University in South Korea

74.12182.342

50

Jan 02, 2018		Conductor-net (single model)

CMU

https://arxiv.org/abs/1710.10504		74.40582.742

50

Feb 06, 2018		Jenga (single model)

Facebook AI Research

74.37382.845

51

Jan 24, 2017		Multi-Perspective Matching (ensemble)

IBM Research

https://arxiv.org/abs/1612.04211		73.76581.257

51

Nov 06, 2017		Conductor-net (single)

CMU

https://arxiv.org/abs/1710.10504		73.24081.933

51

Dec 14, 2017		Jenga (single model)

Facebook AI Research

73.30381.754

51

Jul 06, 2017		SSAE (ensemble)

Tsinghua University

74.08081.665

52

Apr 22, 2017		SEDT+BiDAF (ensemble)

CMU

https://arxiv.org/abs/1703.00572		73.72381.530

52

Feb 22, 2017		BiDAF (ensemble)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		73.74481.525

52

Jul 29, 2017		SEDT (ensemble model)

CMU

https://arxiv.org/abs/1703.00572		74.09081.761

52

Jul 25, 2017		Interactive AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

73.63981.931

53

May 01, 2017		jNet (ensemble)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		73.01081.517

54

Oct 22, 2017		Conductor-net (single)

CMU

72.59081.415

55

Sep 20, 2017		BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1710.10723		72.13981.048

56

Dec 15, 2017		S^3-Net (single model)

Kangwon National University in South Korea

71.90881.023

56

Nov 16, 2017		two-attention-self-attention (single model)

guotong1988

72.60081.011

56

Apr 12, 2017		T-gating (ensemble)

Peking University

72.75881.001

56

Apr 17, 2018		Unnamed submission by null

72.83180.622

56

Apr 17, 2018		Unnamed submission by null

72.83180.622

57

Mar 03, 2018		AVIQA (single model)

aviqa team

72.48580.550

58

Nov 06, 2017		attention+self-attention (single model)

guotong1988

71.69880.462

59

Nov 01, 2016		Dynamic Coattention Networks (ensemble)

Salesforce Research

https://arxiv.org/abs/1611.01604		71.62580.383

60

Jul 14, 2017		smarnet (single model)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1710.02772		71.41580.160

61

Jul 14, 2017		Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		70.99580.146

61

Apr 13, 2017		QFASE

NUS

71.89879.989

62

Apr 22, 2018		MAMCN (single model)

Samsung Research

70.98579.939

62

Oct 27, 2017		M-NET (single)

UFL

71.01679.835

62

May 23, 2018		AttReader (single)

College of Computer & Information Science, SouthWest University, Chongqing, China

71.37379.725

63

May 13, 2017		RaSoR (single model)

Google NY, Tel-Aviv University

https://arxiv.org/abs/1611.01436		70.84978.741

63

Apr 02, 2017		Ruminating Reader (single model)

New York University

https://arxiv.org/abs/1704.07415		70.63979.456

63

Mar 14, 2017		Document Reader (single model)

Facebook AI Research

https://arxiv.org/abs/1704.00051		70.73379.353

63

Dec 28, 2016		FastQAExt

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		70.84978.857

63

Mar 24, 2017		jNet (single model)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		70.60779.821

64

Mar 08, 2017		ReasoNet (single model)

MSR Redmond

https://arxiv.org/abs/1609.05284		70.55579.364

65

Apr 14, 2017		Multi-Perspective Matching (single model)

IBM Research

https://arxiv.org/abs/1612.04211		70.38778.784

66

Feb 05, 2018		SSR-BiDAF

single model

69.44378.358

66

Aug 30, 2017		SimpleBaseline (single model)

Technical University of Vienna

69.60078.236

67

Apr 12, 2017		SEDT+BiDAF (single model)

CMU

https://arxiv.org/abs/1703.00572		68.47877.971

67

Feb 22, 2018		Unnamed submission by null

68.47877.220

67

Jan 22, 2018		FABIR

Single Model

https://arxiv.org/abs/1810.09580		67.74477.605

67

Apr 12, 2017		T-gating (single model)

Peking University

68.13277.569

67

Jul 29, 2017		SEDT (single model)

CMU

https://arxiv.org/abs/1703.00572		68.16377.527

68

Nov 28, 2016		BiDAF (single model)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		67.97477.323

68

Dec 28, 2016		FastQA

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		68.43677.070

68

Sep 19, 2017		AllenNLP BiDAF (single model)

Allen Institute for AI

http://allennlp.org/		67.61877.151

68

Feb 22, 2018		Unnamed submission by null

68.42577.077

68

Jun 25, 2017		PQMN (single model)

KAIST & AIBrain & Crosscert

68.33177.783

69

Oct 26, 2016		Match-LSTM with Ans-Ptr (Boundary) (ensemble)

Singapore Management University

https://arxiv.org/abs/1608.07905		67.90177.022

70

Feb 05, 2017		Iterative Co-attention Network

Fudan University

67.50276.786

71

Nov 01, 2016		Dynamic Coattention Networks (single model)

Salesforce Research

https://arxiv.org/abs/1611.01604		66.23375.896

71

Jan 03, 2018		newtest

single model

66.52775.787

72

Feb 24, 2018		Unnamed submission by null

65.99275.469

73

Jan 10, 2018		Unnamed submission by null

64.79674.272

74

Dec 09, 2017		Unnamed submission by ravioncodalab

64.43973.921

74

Oct 26, 2016		Match-LSTM with Bi-Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		64.74473.743

75

Feb 19, 2017		Attentive CNN context with LSTM

NLPR, CASIA

63.30673.463

76

Nov 02, 2016		Fine-Grained Gating

Carnegie Mellon University

https://arxiv.org/abs/1611.01724		62.44673.327

76

Sep 21, 2017		OTF dict+spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		64.08373.056

77

Sep 21, 2017		OTF spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.89772.016

78

Sep 21, 2017		OTF spelling+lemma (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.60471.968

79

Sep 28, 2016		Dynamic Chunk Reader

IBM

https://arxiv.org/abs/1610.09996		62.49970.956

80

Aug 27, 2016		Match-LSTM with Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		60.47470.695

81

Sep 18, 2018		Unnamed submission by null

59.05869.436

82

Jan 10, 2018		Unnamed submission by null

58.76469.276

83

Aug 27, 2016		Match-LSTM with Ans-Ptr (Sentence)

Singapore Management University

https://arxiv.org/abs/1608.07905		54.50567.748

84

Nov 14, 2018		Unnamed submission by jinhyuklee

52.54462.780

85

Oct 26, 2018		Unnamed submission by minjoon

52.53362.757