SQuAD2.0

The Stanford Question Answering Dataset

What is SQuAD?

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.


New SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 new, unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD2.0 is a challenging natural language understanding task for existing models, and we release SQuAD2.0 to the community as the successor to SQuAD1.1. We are optimistic that this new dataset will encourage the development of reading comprehension systems that know what they don't know.

Explore SQuAD2.0 and model predictionsSQuAD2.0 paper (Rajpurkar & Jia et al. '18)

SQuAD 1.1, the previous version of the SQuAD dataset, contains 100,000+ question-answer pairs on 500+ articles.

Explore SQuAD1.1 and model predictionsSQuAD1.0 paper (Rajpurkar et al. '16)

Getting Started

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v2.0.py <path_to_dev-v2.0> <path_to_predictions>.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Because SQuAD is an ongoing effort, we expect the dataset to evolve.

To keep up to date with major changes to the dataset, please subscribe:

Have Questions?

Ask us questions at our google group or at pranavsr@stanford.edu and robinjia@stanford.edu.

Star

Leaderboard

SQuAD2.0 tests the ability of a system to not only answer reading comprehension questions, but also abstain when presented with a question that cannot be answered based on the provided paragraph. How will your system compare to humans on this task?

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar & Jia et al. '18)
86.83189.452

1

Mar 20, 2019
BERT + DAE + AoA (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

87.14789.474

2

Mar 15, 2019
BERT + ConvLSTM + MTL + Verifier (ensemble)

Layer 6 AI

86.73089.286

3

Mar 05, 2019
BERT + N-Gram Masking + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert
86.67389.147

4

May 21, 2019
XLNet (single model)

Google Brain & CMU

86.34689.133

5

Apr 13, 2019
SemBERT(ensemble)

Shanghai Jiao Tong University

86.16688.886

5

May 14, 2019
SG-Net (ensemble)

Anonymous

86.21188.848

6

Mar 16, 2019
BERT + DAE + AoA (single model)

Joint Laboratory of HIT and iFLYTEK Research

85.88488.621

7

Jun 22, 2019
BNDVnet (ensemble model)

PAOS

85.85088.449

8

Mar 13, 2019
BERT + ConvLSTM + MTL + Verifier (single model)

Layer 6 AI

84.92488.204

8

May 14, 2019
SG-Net (single model)

Anonymous

85.22987.926

8

Jun 10, 2019
Unnamed submission by null

85.24087.901

9

Mar 05, 2019
BERT + N-Gram Masking + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert
85.15087.715

10

Jan 15, 2019
BERT + MMFT + ADA (ensemble)

Microsoft Research Asia

85.08287.615

10

Jun 19, 2019
BNDVnet (single model)

PAOS

85.00387.833

11

Jun 18, 2019
BertSpan (ensemble)

Anonymous

84.82387.489

11

Apr 11, 2019
SemBERT (single model)

Shanghai Jiao Tong University

84.80087.864

11

Apr 16, 2019
Insight-baseline-BERT (single model)

PAII Insight Team

84.83487.644

12

Jan 10, 2019
BERT + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert
84.29286.967

13

Dec 16, 2018
Lunet + Verifier + BERT (ensemble)

Layer 6 AI NLP Team

83.46986.043

13

Mar 20, 2019
Bert-raw (ensemble)

None

83.60486.036

13

Dec 21, 2018
PAML+BERT (ensemble model)

PINGAN GammaLab

83.45786.122

13

Dec 13, 2018
BERT finetune baseline (ensemble)

Anonymous

83.53686.096

14

Dec 15, 2018
Lunet + Verifier + BERT (single model)

Layer 6 AI NLP Team

82.99586.035

15

May 14, 2019
ATB (single model)

Anonymous

82.88286.002

15

Jan 14, 2019
BERT + MMFT + ADA (single model)

Microsoft Research Asia

83.04085.892

15

Jun 20, 2019
SENSEFORTH + BERT

single

https://senseforth.ai
83.14285.873

16

Jan 10, 2019
BERT + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert
82.97285.810

16

Feb 26, 2019
BERT with Something (ensemble)

Anonymous

83.05185.737

17

Feb 15, 2019
BERT + NeurQuRI (ensemble)

2SAH

82.80385.703

17

Mar 11, 2019
Bert-raw (ensemble)

None

83.11985.510

17

Feb 16, 2019
Bert-raw (ensemble)

None

83.17585.635

18

Feb 27, 2019
BERT + NeurQuRI (ensemble)

2SAH

82.71385.584

18

May 13, 2019
BERT-Base + QA Pre-training (single model)

Anonymous

82.72485.491

18

Dec 16, 2018
PAML+BERT (single model)

PINGAN GammaLab

82.57785.603

19

Mar 03, 2019
Unnamed submission by null

82.43185.178

19

Nov 16, 2018
AoA + DA + BERT (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.37485.310

20

Feb 28, 2019
BERT_s (single model)

Anonymous

81.97984.846

20

Dec 12, 2018
BERT finetune baseline (single model)

Anonymous

82.12684.820

20

Dec 10, 2018
Candi-Net+BERT (ensemble)

42Maru NLP Team

82.12684.624

20

Jun 06, 2019
Unnamed submission by zw10

81.73184.862

21

Feb 28, 2019
BERT-large+UBFT (single model)

anonymous

81.57384.535

22

Feb 15, 2019
BERT + NeurQuRI (single model)

2SAH

81.25784.342

22

Feb 25, 2019
BERT with Something (single model)

Anonymous

81.11084.386

22

Nov 16, 2018
AoA + DA + BERT (single model)

Joint Laboratory of HIT and iFLYTEK Research

81.17884.251

23

Mar 20, 2019
Bert-raw (single)

None

80.69383.922

23

Mar 07, 2019
BERT + UnAnsQ (single model)

Anonymous

80.74983.851

24

Apr 07, 2019
BERT + AL (single model)

Anonymous

80.71583.827

25

Dec 19, 2018
Candi-Net+BERT (single model)

42Maru NLP Team

80.65983.562

26

Jan 09, 2019
Unnamed submission by null

80.51283.539

27

Mar 11, 2019
Bert-raw (single)

None

80.41183.457

27

Jan 22, 2019
BERT + NeurQuRI (single model)

2SAH

80.59183.391

28

Feb 16, 2019
Bert-raw (single model)

None

80.34383.243

28

Jan 09, 2019
Unnamed submission by null

80.34383.221

29

Feb 19, 2019
BERT + UDA (single model)

Anonymous

80.00583.208

29

Dec 03, 2018
PwP+BERT (single model)

AITRICS

80.11783.189

30

Apr 10, 2019
bert (single model)

vinda msqjmxx

79.97183.184

30

Apr 04, 2019
BISAN-CC (single model)

Seoul National University & Hyundai Motors

80.20883.149

30

May 28, 2019
Bert

Single Model

https://senseforth.ai
80.42283.118

31

Nov 08, 2018
BERT (single model)

Google AI Language

80.00583.061

31

Dec 05, 2018
Candi-Net+BERT (single model)

42Maru NLP Team

80.38882.908

31

Feb 28, 2019
ST_bl

single model

80.14082.962

31

Feb 12, 2019
BERT + Sparse-Transformer

single model

79.94883.023

32

Mar 07, 2019
BERT uncased (single model)

Anonymous

79.74583.020

32

Dec 06, 2018
NEXYS_BASE (single model)

NEXYS, DGIST R7

79.77982.912

33

Feb 01, 2019
{bert-finetuning} (single model)

ksai

79.63282.852

34

Mar 14, 2019
{Anonymous} (single model)

Anonymous

78.87682.524

34

Nov 09, 2018
L6Net + BERT (single model)

Layer 6 AI

79.18182.259

35

Mar 14, 2019
BISAN (single model)

Seoul National University & Hyundai Motors

78.48181.531

35

Apr 24, 2019
BERT + WIAN (ensemble)

Infosys Limited

78.65081.497

36

Jan 09, 2019
Unnamed submission by null

78.30181.350

37

Dec 14, 2018
BERT+AC(single model)

Hithink RoyalFlush

78.05281.174

38

Nov 06, 2018
SLQA+BERT (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158
77.00380.209

39

Jan 05, 2019
synss (single model )

bert_finetune

76.05579.329

40

Dec 18, 2018
ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai
74.74678.227

40

Nov 05, 2018
MIR-MRC(F-Net) (single model)

Kangwon National University, Natural Language Processing Lab. & ForceWin, KP Lab.

74.79177.988

41

May 23, 2019
{BERTcw} (single model)

private

74.38577.308

42

Sep 13, 2018
nlnet (single model)

Microsoft Research Asia

74.27277.052

43

Dec 22, 2018
Unnamed submission by null

73.23476.790

43

Dec 29, 2018
MMIPN

Single

73.50576.424

44

Apr 20, 2019
BERT-Base (single model)

Dining Philosophers

73.09976.236

45

Oct 12, 2018
YARCS (ensemble)

IBM Research AI

72.67075.507

46

Nov 10, 2018
Unnamed submission by null

72.58075.075

47

Apr 24, 2019
BERT-Base (single)

GreenflyAI

https://greenfly.ai
71.69974.430

47

Sep 17, 2018
Unet (ensemble)

Fudan University & Liulishuo Lab

https://arxiv.org/abs/1810.06638
71.41774.869

48

Jan 19, 2019
{BERT-base} (single-model)

Anonymous

70.76374.449

48

Aug 28, 2018
SLQA+ (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158
71.46274.434

48

Nov 14, 2018
BERT+Answer Verifier (single model)

Pingan Tech Olatop Lab

71.66675.457

48

Aug 15, 2018
Reinforced Mnemonic Reader + Answer Verifier (single model)

NUDT

https://arxiv.org/abs/1808.05759
71.76774.295

49

Sep 14, 2018
SAN (ensemble model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556
71.31673.704

50

Nov 10, 2018
Unnamed submission by null

70.71873.403

51

Sep 14, 2018
Unet (single model)

Fudan University & Liulishuo Lab

69.26272.642

51

Aug 21, 2018
FusionNet++ (ensemble)

Microsoft Business Applications Group AI Research

https://arxiv.org/abs/1711.07341
70.30072.484

51

Sep 26, 2018
Multi-Level Attention Fusion(MLAF) (single model)

Chonbuk National University, Cognitive Computing Lab.

69.47672.857

52

Dec 20, 2018
DocQA + NeurQuRI (single model)

2SAH

68.76671.662

53

Aug 21, 2018
SAN (single model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556
68.65371.439

53

Nov 10, 2018
Unnamed submission by null

68.65371.124

54

Jul 13, 2018
VS^3-NET (single model)

Kangwon National University in South Korea

67.89770.884

54

Jun 24, 2018
KACTEIL-MRC(GFN-Net) (single model)

Kangwon National University, Natural Language Processing Lab.

68.21370.878

54

Sep 13, 2018
BiDAF++ with pair2vec (single model)

UW and FAIR

68.02171.583

55

Jan 01, 2019
EBB-Net (single model)

Enliple AI

66.61070.303

56

Jun 25, 2018
KakaoNet2 (single model)

Kakao NLP Team

65.71969.381

57

Sep 13, 2018
BiDAF++ (single model)

UW and FAIR

65.65168.866

57

Jul 11, 2018
abcNet (single model)

Fudan University & Liulishuo AI Lab

65.25669.206

58

Jun 27, 2018
BSAE AddText (single model)

reciTAL.ai

63.33867.422

59

Aug 14, 2018
eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com
63.32766.633

59

May 30, 2018
BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

63.37266.251

60

Nov 27, 2018
Tree-LSTM + BiDAF + ELMo (single model)

Carnegie Mellon University

57.70762.341

60

May 30, 2018
BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

59.33262.305

61

May 30, 2018
BiDAF-No-Answer (single model)

University of Washington [modified by Stanford]

59.17462.093

62

May 20, 2019
Unnamed submission by Simon

58.50862.045

63

May 20, 2019
Unnamed submission by Simon

56.54559.546

SQuAD1.1 Leaderboard

Here are the ExactMatch (EM) and F1 scores evaluated on the test set of SQuAD v1.1.

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar et al. '16)
82.30491.221

1

May 21, 2019
XLNet (single model)

Google Brain & CMU

89.89895.080

2

Oct 05, 2018
BERT (ensemble)

Google AI Language

https://arxiv.org/abs/1810.04805
87.43393.160

3

May 14, 2019
ATB (single model)

Anonymous

86.94092.641

4

Feb 14, 2019
KT-NET (single model)

Baidu NLP

85.94492.425

5

Oct 05, 2018
BERT (single model)

Google AI Language

https://arxiv.org/abs/1810.04805
85.08391.835

5

Mar 14, 2019
BISAN (single model)

Seoul National University & Hyundai Motors

85.31491.756

5

Sep 26, 2018
nlnet (ensemble)

Microsoft Research Asia

85.95491.677

6

Feb 16, 2019
BERT+Sparse-Transformer

single model

85.12591.623

6

Sep 09, 2018
nlnet (ensemble)

Microsoft Research Asia

85.35691.202

6

Jun 03, 2019
DPN (single model)

Anonymous

84.97892.019

6

Feb 28, 2019
ST_bl

single model

85.43091.976

7

Feb 19, 2019
WD (single model)

Anonymous

84.40290.561

7

Jul 11, 2018
QANet (ensemble)

Google Brain & CMU

84.45490.490

7

May 08, 2019
Common-sense Governed BERT-123 (single model)

MST/EOI

82.94391.074

7

Apr 21, 2019
Common-sense Governed BERT-123 (single model)

Jerry AGI Ragtag

83.93090.613

7

Jun 20, 2018
MARS (ensemble)

YUANFUDAO research NLP

83.98289.796

8

Mar 19, 2018
QANet (ensemble)

Google Brain & CMU

83.87789.737

8

Feb 21, 2019
WD1 (single model)

Anonymous

83.80490.429

8

Jul 08, 2018
r-net (ensemble)

Microsoft Research Asia

84.00390.147

9

Sep 09, 2018
nlnet (single model)

Microsoft Research Asia

83.46890.133

10

Sep 01, 2018
MARS (single model)

YUANFUDAO research NLP

83.18589.547

11

Jan 22, 2018
Hybrid AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.48289.281

11

Jun 21, 2018
MARS (single model)

YUANFUDAO research NLP

83.12289.224

12

Mar 06, 2018
QANet (ensemble)

Google Brain & CMU

82.74489.045

12

Jun 20, 2018
QANet (single)

Google Brain & CMU

82.47189.306

12

May 09, 2018
MARS (single model)

YUANFUDAO research NLP

82.58788.880

12

Feb 19, 2018
Reinforced Mnemonic Reader + A2D (ensemble model)

Microsoft Research Asia & NUDT

82.84988.764

12

Dec 23, 2018
MMIPN

Single

81.58088.948

12

Jan 05, 2018
SLQA+ (ensemble)

Alibaba iDST NLP

82.44088.607

13

Feb 02, 2018
Reinforced Mnemonic Reader (ensemble model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798
82.28388.533

13

Jan 03, 2018
r-net+ (ensemble)

Microsoft Research Asia

82.65088.493

14

Dec 17, 2017
r-net (ensemble)

Microsoft Research Asia

http://aka.ms/rnet
82.13688.126

14

Dec 22, 2017
AttentionReader+ (ensemble)

Tencent DPDAC NLP

81.79088.163

14

Feb 27, 2018
QANet (single model)

Google Brain & CMU

82.20988.608

15

May 09, 2018
Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

81.53888.130

15

Dec 17, 2018
ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai
81.30788.909

15

Apr 23, 2018
r-net (single model)

Microsoft Research Asia

81.39188.170

15

Apr 03, 2018
KACTEIL-MRC(GF-Net+) (ensemble)

Kangwon National University, Natural Language Processing Lab.

81.49687.557

15

May 09, 2018
Reinforced Mnemonic Reader + A2D + DA (single model)

Microsoft Research Asia & NUDT

81.40188.122

16

Feb 27, 2018
QANet (single model)

Google Brain & CMU

80.92987.773

17

Feb 12, 2018
Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.48987.454

17

Nov 17, 2017
BiDAF + Self Attention + ELMo (ensemble)

Allen Institute for Artificial Intelligence

81.00387.432

17

Feb 19, 2018
Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.91987.492

18

Apr 12, 2018
AVIQA+ (ensemble)

aviqa team

80.61587.311

19

Jan 22, 2018
Hybrid AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

80.02787.288

19

Jan 12, 2018
EAZI+ (ensemble)

Yiwise NLP Group

80.42686.912

19

Jan 04, 2018
{EAZI} (ensemble)

Yiwise NLP Group

80.43686.912

20

Mar 20, 2018
DNET (ensemble)

QA geeks

80.16486.721

20

Jan 13, 2018
SLQA+

single model

80.43687.021

21

Feb 23, 2018
MAMCN+ (single model)

Samsung Research

79.69286.727

21

Feb 12, 2018
BiDAF + Self Attention + ELMo + A2D (single model)

Microsoft Research Asia & NUDT

79.99686.711

21

Apr 10, 2018
Unnamed submission by null

80.02786.612

22

Jan 03, 2018
r-net+ (single model)

Microsoft Research Asia

79.90186.536

22

Dec 28, 2017
SLQA+ (single model)

Alibaba iDST NLP

79.19986.590

22

Dec 05, 2017
SAN (ensemble model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556
79.60886.496

22

Jan 29, 2018
Reinforced Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798
79.54586.654

23

Oct 17, 2017
Interactive AoA Reader+ (ensemble)

Joint Laboratory of HIT and iFLYTEK

79.08386.450

23

Nov 05, 2018
MIR-MRC(F-Net) (single model)

ForceWin, KP Lab.

79.08386.288

24

Feb 01, 2018
Unnamed submission by null

78.99986.151

25

Oct 24, 2017
FusionNet (ensemble)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1711.07341
78.97886.016

25

Jun 01, 2018
MDReader

single model

79.03186.006

26

Oct 22, 2017
DCN+ (ensemble)

Salesforce Research

https://arxiv.org/abs/1711.00106
78.85285.996

27

Nov 03, 2017
BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence

78.58085.833

27

Mar 29, 2018
KACTEIL-MRC(GF-Net+) (single model)

Kangwon National University, Natural Language Processing Lab.

78.66485.780

28

Nov 30, 2017
SLQA(ensemble)

Alibaba iDST NLP

78.32885.682

29

Jun 01, 2018
MDReader0

single model

78.17185.543

29

Sep 18, 2018
BiDAF++ with pair2vec (single model)

UW and FAIR

78.22385.535

29

Jan 02, 2018
Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504
78.43385.517

29

Mar 19, 2018
aviqa (ensemble)

aviqa team

78.49685.469

29

May 09, 2018
KakaoNet (single model)

Kakao NLP Team

78.40185.724

30

Jan 03, 2018
MEMEN (single model)

Zhejiang University

https://arxiv.org/abs/1707.09098
78.23485.344

30

Jan 29, 2018
test

single

78.08785.348

31

Jul 25, 2017
Interactive AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

77.84585.297

32

Jan 10, 2018
Unnamed submission by null

77.43685.130

32

Mar 20, 2018
DNET (single model)

QA geeks

77.64684.905

33

Sep 18, 2018
BiDAF++ (single model)

UW and FAIR

77.57384.858

34

Apr 10, 2018
Unnamed submission by null

77.48984.735

34

Dec 06, 2017
AttentionReader+ (single)

Tencent DPDAC NLP

77.34284.925

35

Jan 23, 2018
MARS (single model)

YUANFUDAO research NLP

76.85984.739

35

Nov 06, 2017
Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504
76.99684.630

35

Dec 13, 2017
RaSoR + TR + LM (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609
77.58384.163

35

Dec 21, 2017
Jenga (ensemble)

Facebook AI Research

77.23784.466

36

Nov 01, 2017
SAN (single model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556
76.82884.396

36

May 14, 2018
VS^3-NET (single model)

Kangwon National University in South Korea

76.77584.491

37

Oct 13, 2017
r-net (single model)

Microsoft Research Asia

http://aka.ms/rnet
76.46184.265

37

Dec 19, 2017
FRC (single model)

in review

76.24084.599

38

Oct 22, 2017
Conductor-net (ensemble)

CMU

76.14683.991

38

Sep 26, 2018
{gqa} (single model)

FAIR

77.09083.931

39

Sep 08, 2017
FusionNet (single model)

Microsoft Business AI Solutions team

https://arxiv.org/abs/1711.07341
75.96883.900

40

Oct 22, 2017
Interactive AoA Reader+ (single model)

Joint Laboratory of HIT and iFLYTEK

75.82183.843

40

Jul 14, 2017
smarnet (ensemble)

Eigen Technology & Zhejiang University

75.98983.475

41

Mar 15, 2018
AVIQA-v2 (single model)

aviqa team

75.92683.305

41

Oct 18, 2018
KAR (single model)

York University

https://arxiv.org/abs/1809.03449
76.12583.538

42

Aug 18, 2017
RaSoR + TR (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609
75.78983.261

43

Oct 23, 2017
DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106
75.08783.081

43

Oct 05, 2018
Unnamed submission by null

74.95083.294

43

Nov 01, 2017
Mixed model (ensemble)

Sean

75.26582.769

44

Nov 17, 2017
two-attention-self-attention (ensemble)

guotong1988

75.22382.716

44

Oct 31, 2017
SLQA (single model)

Alibaba iDST NLP

74.48982.815

44

Mar 09, 2017
ReasoNet (ensemble)

MSR Redmond

https://arxiv.org/abs/1609.05284
75.03482.552

44

Jul 10, 2017
DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106
74.86682.806

44

May 21, 2017
MEMEN (ensemble)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1707.09098
75.37082.658

45

Aug 14, 2018
eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com
74.60482.501

46

Feb 13, 2018
SSR-BiDAF

ensemble model

74.54182.477

47

Oct 27, 2017
Unnamed submission by null

74.48982.312

47

Jul 14, 2017
Mnemonic Reader (ensemble)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798
74.26882.371

48

Dec 23, 2017
S^3-Net (ensemble)

Kangwon National University in South Korea

74.12182.342

48

Jan 02, 2018
Conductor-net (single model)

CMU

https://arxiv.org/abs/1710.10504
74.40582.742

48

Feb 06, 2018
Jenga (single model)

Facebook AI Research

74.37382.845

49

Jan 24, 2017
Multi-Perspective Matching (ensemble)

IBM Research

https://arxiv.org/abs/1612.04211
73.76581.257

49

Nov 06, 2017
Conductor-net (single)

CMU

https://arxiv.org/abs/1710.10504
73.24081.933

49

Dec 14, 2017
Jenga (single model)

Facebook AI Research

73.30381.754

49

Jul 06, 2017
SSAE (ensemble)

Tsinghua University

74.08081.665

50

Apr 22, 2017
SEDT+BiDAF (ensemble)

CMU

https://arxiv.org/abs/1703.00572
73.72381.530

50

Feb 22, 2017
BiDAF (ensemble)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603
73.74481.525

50

Jul 29, 2017
SEDT (ensemble model)

CMU

https://arxiv.org/abs/1703.00572
74.09081.761

50

Jul 25, 2017
Interactive AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

73.63981.931

51

May 01, 2017
jNet (ensemble)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617
73.01081.517

52

Oct 22, 2017
Conductor-net (single)

CMU

72.59081.415

53

Sep 20, 2017
BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1710.10723
72.13981.048

54

Dec 15, 2017
S^3-Net (single model)

Kangwon National University in South Korea

71.90881.023

54

Nov 16, 2017
two-attention-self-attention (single model)

guotong1988

72.60081.011

54

Apr 12, 2017
T-gating (ensemble)

Peking University

72.75881.001

54

Apr 17, 2018
Unnamed submission by null

72.83180.622

54

Apr 17, 2018
Unnamed submission by null

72.83180.622

55

Mar 03, 2018
AVIQA (single model)

aviqa team

72.48580.550

56

Nov 06, 2017
attention+self-attention (single model)

guotong1988

71.69880.462

57

Nov 01, 2016
Dynamic Coattention Networks (ensemble)

Salesforce Research

https://arxiv.org/abs/1611.01604
71.62580.383

58

Jul 14, 2017
smarnet (single model)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1710.02772
71.41580.160

59

Jul 14, 2017
Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798
70.99580.146

59

Apr 13, 2017
QFASE

NUS

71.89879.989

60

Apr 22, 2018
MAMCN (single model)

Samsung Research

70.98579.939

60

Oct 27, 2017
M-NET (single)

UFL

71.01679.835

60

May 23, 2018
AttReader (single)

College of Computer & Information Science, SouthWest University, Chongqing, China

71.37379.725

61

May 13, 2017
RaSoR (single model)

Google NY, Tel-Aviv University

https://arxiv.org/abs/1611.01436
70.84978.741

61

Apr 02, 2017
Ruminating Reader (single model)

New York University

https://arxiv.org/abs/1704.07415
70.63979.456

61

Mar 14, 2017
Document Reader (single model)

Facebook AI Research

https://arxiv.org/abs/1704.00051
70.73379.353

61

Dec 28, 2016
FastQAExt

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816
70.84978.857

61

Mar 24, 2017
jNet (single model)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617
70.60779.821

62

Mar 08, 2017
ReasoNet (single model)

MSR Redmond

https://arxiv.org/abs/1609.05284
70.55579.364

63

Apr 14, 2017
Multi-Perspective Matching (single model)

IBM Research

https://arxiv.org/abs/1612.04211
70.38778.784

64

Feb 05, 2018
SSR-BiDAF

single model

69.44378.358

64

Aug 30, 2017
SimpleBaseline (single model)

Technical University of Vienna

69.60078.236

65

Apr 12, 2017
SEDT+BiDAF (single model)

CMU

https://arxiv.org/abs/1703.00572
68.47877.971

65

Feb 22, 2018
Unnamed submission by null

68.47877.220

65

Jan 22, 2018
FABIR

Single Model

https://arxiv.org/abs/1810.09580
67.74477.605

65

Apr 12, 2017
T-gating (single model)

Peking University

68.13277.569

65

Jul 29, 2017
SEDT (single model)

CMU

https://arxiv.org/abs/1703.00572
68.16377.527

66

Nov 28, 2016
BiDAF (single model)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603
67.97477.323

66

Dec 28, 2016
FastQA

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816
68.43677.070

66

Sep 19, 2017
AllenNLP BiDAF (single model)

Allen Institute for AI

http://allennlp.org/
67.61877.151

66

Feb 22, 2018
Unnamed submission by null

68.42577.077

66

Jun 25, 2017
PQMN (single model)

KAIST & AIBrain & Crosscert

68.33177.783

67

Oct 26, 2016
Match-LSTM with Ans-Ptr (Boundary) (ensemble)

Singapore Management University

https://arxiv.org/abs/1608.07905
67.90177.022

68

Feb 05, 2017
Iterative Co-attention Network

Fudan University

67.50276.786

69

Nov 01, 2016
Dynamic Coattention Networks (single model)

Salesforce Research

https://arxiv.org/abs/1611.01604
66.23375.896

69

Jan 03, 2018
newtest

single model

66.52775.787

70

Feb 24, 2018
Unnamed submission by null

65.99275.469

71

Jan 10, 2018
Unnamed submission by null

64.79674.272

72

Dec 09, 2017
Unnamed submission by ravioncodalab

64.43973.921

72

Oct 26, 2016
Match-LSTM with Bi-Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905
64.74473.743

73

Feb 19, 2017
Attentive CNN context with LSTM

NLPR, CASIA

63.30673.463

74

Nov 02, 2016
Fine-Grained Gating

Carnegie Mellon University

https://arxiv.org/abs/1611.01724
62.44673.327

74

Sep 21, 2017
OTF dict+spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286
64.08373.056

75

Sep 21, 2017
OTF spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286
62.89772.016

76

Sep 21, 2017
OTF spelling+lemma (single)

University of Montreal

https://arxiv.org/abs/1706.00286
62.60471.968

77

Sep 28, 2016
Dynamic Chunk Reader

IBM

https://arxiv.org/abs/1610.09996
62.49970.956

78

Aug 27, 2016
Match-LSTM with Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905
60.47470.695

79

Sep 18, 2018
Unnamed submission by null

59.05869.436

80

Jan 10, 2018
Unnamed submission by null

58.76469.276

81

Aug 27, 2016
Match-LSTM with Ans-Ptr (Sentence)

Singapore Management University

https://arxiv.org/abs/1608.07905
54.50567.748

82

Nov 14, 2018
Unnamed submission by jinhyuklee

52.54462.780

83

Oct 26, 2018
Unnamed submission by minjoon

52.53362.757