SQuAD2.0

The Stanford Question Answering Dataset

What is SQuAD?

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.

Explore SQuAD2.0 and model predictionsSQuAD2.0 paper (Rajpurkar & Jia et al. '18)

SQuAD 1.1, the previous version of the SQuAD dataset, contains 100,000+ question-answer pairs on 500+ articles.

Explore SQuAD1.1 and model predictionsSQuAD1.0 paper (Rajpurkar et al. '16)

Getting Started

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use python evaluate-v2.0.py <path_to_dev-v2.0> <path_to_predictions>.

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Because SQuAD is an ongoing effort, we expect the dataset to evolve.

To keep up to date with major changes to the dataset, please subscribe:

Have Questions?

Ask us questions at our google group or at pranavsr@stanford.edu and robinjia@stanford.edu.

Star

Leaderboard

SQuAD2.0 tests the ability of a system to not only answer reading comprehension questions, but also abstain when presented with a question that cannot be answered based on the provided paragraph.

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar & Jia et al. '18)		86.83189.452

1

Feb 21, 2021		FPNet (ensemble)

Ant Service Intelligence Team

90.87193.183

2

Feb 24, 2021		IE-Net (ensemble)

RICOH_SRCB_DML

90.75893.044

3

Apr 06, 2020		SA-Net on Albert (ensemble)

QIANXIN

90.72493.011

4

May 05, 2020		SA-Net-V2 (ensemble)

QIANXIN

90.67992.948

4

Apr 05, 2020		Retro-Reader (ensemble)

Shanghai Jiao Tong University

http://arxiv.org/abs/2001.09694		90.57892.978

4

Feb 05, 2021		FPNet (ensemble)

YuYang

90.60092.899

5

Apr 18, 2021		TransNets + SFVerifier + SFEnsembler (ensemble)

Senseforth AI Research

https://www.senseforth.ai/		90.48792.894

5

Dec 01, 2020		EntitySpanFocusV2 (ensemble)

RICOH_SRCB_DML

90.52192.824

5

Jul 31, 2020		ATRLP+PV (ensemble)

Hithink RoyalFlush

90.44292.877

5

May 04, 2020		ELECTRA+ALBERT+EntitySpanFocus (ensemble)

SRCB_DML

90.44292.839

6

Jun 21, 2020		ELECTRA+ALBERT+EntitySpanFocus (ensemble)

SRCB_DML

90.42092.799

6

Sep 11, 2020		EntitySpanFocus+AT (ensemble)

RICOH_SRCB_DML

90.45492.748

6

Mar 12, 2020		ALBERT + DAAF + Verifier (ensemble)

PINGAN Omni-Sinitic

90.38692.777

7

Feb 05, 2021		MixEnsemble (ensemble)

Anonymous

90.19492.594

8

Jan 10, 2020		Retro-Reader on ALBERT (ensemble)

Shanghai Jiao Tong University

http://arxiv.org/abs/2001.09694		90.11592.580

9

Jan 12, 2021		Answer Dependent Classify (single model)

YITU

90.05992.517

9

Jan 30, 2021		ANet

ensemble

90.08192.457

10

Nov 06, 2019		ALBERT + DAAF + Verifier (ensemble)

PINGAN Omni-Sinitic

90.00292.425

11

Sep 18, 2019		ALBERT (ensemble model)

Google Research & TTIC

https://arxiv.org/abs/1909.11942		89.73192.215

11

Feb 25, 2020		Albert_Verifier_AA_Net (ensemble)

QIANXIN

89.74392.180

11

Jun 28, 2020		ELECTRA+ATRLP+PV (single model)

Hithink RoyalFlush

89.55192.366

11

Jan 10, 2021		Span Extract + Classify (single model)

Anonymous

89.56292.226

11

Mar 28, 2020		Retro-Reader on ELECTRA (single model)

Shanghai Jiao Tong University

http://arxiv.org/abs/2001.09694		89.56292.052

11

Mar 27, 2020		albert+KD+transfer (ensemble)

Anonymous

89.46192.134

12

Nov 18, 2020		ROaD-Electra

single model

89.44992.118

13

Feb 03, 2021		ELECTRA + E-Verifier (ensemble)

Midea NLP Team

89.34891.985

13

Jan 04, 2021		ELECTRA + ROBERTA + ALBERT (ensemble)

Midea NLP Team

89.32591.994

13

Jan 12, 2021		2task (single model)

Ted

89.32591.939

14

Apr 21, 2020		albert+KD+transfer+twopass (single)

SPPD

89.11191.877

14

Apr 18, 2020		ALBERT + MTDA + SFVerifier (ensemble model)

Senseforth AI Research

https://www.senseforth.ai/		89.23591.739

15

Apr 15, 2020		ALBERT + SFVerifier (ensemble model)

Senseforth AI Research

https://www.senseforth.ai/		89.13391.666

15

Apr 23, 2020		ELECTRA+RL+EV (single model)

Hithink RoyalFlush

89.02191.765

16

Dec 08, 2019		ALBERT+Entailment DA (ensemble)

CloudWalk

88.76191.745

16

May 02, 2020		ELECTRA+EntitySpanFocus (Single model)

SRCB_DML

88.87491.546

17

Apr 14, 2020		SA-Net on Electra (single model)

QIANXIN

88.85191.486

18

Mar 06, 2020		ELECTRA (single model)

Google Brain & Stanford

88.71691.365

19

Aug 13, 2020		ELECTRA_ATT (single model)

Shanghai Jiao Tong University

88.61491.303

20

Feb 24, 2020		ALBERT (Single model)

SRCB_DML

88.59291.286

20

Feb 20, 2020		Tuned ALBERT (ensemble model)

Group Data & Analytics Cell | Aditya Birla Group)

https://www.adityabirla.com/About/group-data-and-analytics		88.63791.230

20

Jun 24, 2020		ALBERT + IG + NE (single model)

Anonymous

88.56991.287

21

Jun 24, 2020		ALBERT + IG (single model)

Anonymous

88.52491.256

21

Jan 19, 2020		Retro-Reader on ALBERT (single model)

Shanghai Jiao Tong University

http://arxiv.org/abs/2001.09694		88.10791.419

21

Jul 22, 2019		XLNet + DAAF + Verifier (ensemble)

PINGAN Omni-Sinitic

88.59290.859

21

Mar 13, 2020		aanet_v2.0 (single model)

QIANXIN

88.43490.918

21

Dec 08, 2019		ALBERT+Entailment DA Verifier (single model)

CloudWalk

87.84791.265

21

Jan 07, 2020		ALBERT + SFVerifier (single model)

Senseforth AI Research

https://www.senseforth.ai/		88.19790.830

21

Sep 16, 2019		ALBERT (single model)

Google Research & TTIC

https://arxiv.org/abs/1909.11942		88.10790.902

21

Mar 30, 2020		MTL (single model)

HAPTIK AI RESEARCH

https://haptik.ai		88.10790.902

21

Jul 26, 2019		UPM (ensemble)

Anonymous

88.23190.713

21

Feb 10, 2020		SkERT-Large (single model)

Skelter Labs

87.99490.944

21

Aug 04, 2019		XLNet + SG-Net Verifier (ensemble)

Shanghai Jiao Tong University & CloudWalk

https://arxiv.org/abs/1908.05147		88.17490.702

21

May 21, 2020		albert+KD+transfer+twopass (single)

SPPD

87.94990.818

21

Feb 29, 2020		ALBERT+RL (single model)

Hithink RoyalFlush

87.87090.823

21

May 22, 2020		albert_xxlarge (single model)

Zheyu Ye

87.80290.872

21

Nov 15, 2019		XLNet (single model)

Google Brain & CMU

87.92690.689

22

Feb 12, 2020		Tuned ALBERT (single model)

Group Data & Analytics Cell | Aditya Birla Group)

https://www.adityabirla.com/About/group-data-and-analytics		87.84790.532

22

Feb 10, 2020		ALBERT 1.1 (single model)

Anonymous

87.70090.588

23

Apr 04, 2020		LUKE (single model)

Studio Ousia & NAIST & RIKEN AIP

https://arxiv.org/abs/2010.01057		87.42990.163

24

Aug 04, 2019		XLNet + SG-Net Verifier++ (single model)

Shanghai Jiao Tong University & CloudWalk

https://arxiv.org/abs/1908.05147		87.23890.071

25

Jul 26, 2019		UPM (single model)

Anonymous

87.19389.934

25

Nov 27, 2019		RoBERTa+Verify (ensemble)

CW

86.93390.037

25

Mar 20, 2019		BERT + DAE + AoA (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

87.14789.474

25

Jul 20, 2019		RoBERTa (single model)

Facebook AI

86.82089.795

26

Nov 12, 2019		RoBERTa+Verify (single model)

CW

86.44889.586

26

Mar 15, 2019		BERT + ConvLSTM + MTL + Verifier (ensemble)

Layer 6 AI

86.73089.286

27

Mar 05, 2019		BERT + N-Gram Masking + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert		86.67389.147

27

May 29, 2020		Enhanced Albert+Verifier (ensemble)

Microsoft STCA AIC

86.09889.634

27

Oct 16, 2019		Xlnet+Verifier

single model

86.59489.082

28

Aug 30, 2019		Xlnet+Verifier (single model)

Ping An Life Insurance Company AI Team

86.57289.063

28

May 30, 2020		Enhanced Albert+Verifier3 (ensemble)

Microsoft STCA AIC

85.82789.778

28

Dec 09, 2019		XLNET-V2-123+ (single model)

MST/EOI

http://tia.today		86.40389.148

29

May 21, 2019		XLNet (single model)

Google Brain & CMU

86.34689.133

30

May 14, 2019		SG-Net (ensemble)

Shanghai Jiao Tong University

https://arxiv.org/abs/1908.05147		86.21188.848

30

Apr 14, 2019		SemBERT (ensemble)

Shanghai Jiao Tong University

https://arxiv.org/abs/1909.02209		86.16688.886

30

Sep 29, 2019		BERTSP (single model)

NEUKG

http://www.techkg.cn/--please		85.83888.921

30

Sep 22, 2020		RoBERTa-Large (ensemble model)

SAIL

85.87288.793

30

Mar 16, 2019		BERT + DAE + AoA (single model)

Joint Laboratory of HIT and iFLYTEK Research

85.88488.621

30

Jul 22, 2019		SpanBERT (single model)

FAIR & UW

85.74888.709

31

Sep 22, 2020		RoBERTa-Large (single model)

SAIL

85.17388.425

31

May 14, 2019		SG-Net (single model)

Shanghai Jiao Tong University

https://arxiv.org/abs/1908.05147		85.22987.926

31

Mar 13, 2019		BERT + ConvLSTM + MTL + Verifier (single model)

Layer 6 AI

84.92488.204

31

Mar 05, 2019		BERT + N-Gram Masking + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert		85.15087.715

31

Jun 19, 2019		BNDVnet (single model)

PAOS

85.00387.833

31

Jan 15, 2019		BERT + MMFT + ADA (ensemble)

Microsoft Research Asia

85.08287.615

31

Apr 11, 2019		SemBERT (single model)

Shanghai Jiao Tong University

https://arxiv.org/abs/1909.02209		84.80087.864

31

Sep 13, 2019		xlnet (single model)

VerifiedXiaoPAI

84.64288.000

31

Apr 16, 2019		Insight-baseline-BERT (single model)

PAII Insight Team

84.83487.644

32

Sep 03, 2019		Hanvon_model (single model)

Hanvon_WuHan

84.72187.117

33

Jan 10, 2019		BERT + Synthetic Self-Training (ensemble)

Google AI Language

https://github.com/google-research/bert		84.29286.967

34

Nov 08, 2019		BERT + Multiple-CNN (ensemble)

Kyonggi University (ICL) & KISTI

84.20286.767

35

Jul 22, 2019		Tuned BERT-1seq Large Cased (single model)

FAIR & UW

83.75186.594

36

Mar 20, 2019		Bert-raw (ensemble)

None

83.60486.036

36

Dec 13, 2018		BERT finetune baseline (ensemble)

Anonymous

83.53686.096

36

Dec 21, 2018		PAML+BERT (ensemble model)

PINGAN GammaLab

83.45786.122

36

Dec 16, 2018		Lunet + Verifier + BERT (ensemble)

Layer 6 AI NLP Team

83.46986.043

37

Dec 15, 2018		Lunet + Verifier + BERT (single model)

Layer 6 AI NLP Team

82.99586.035

37

Jun 21, 2019		SENSEFORTH + BERT

single

https://senseforth.ai		83.14285.873

37

Jan 14, 2019		BERT + MMFT + ADA (single model)

Microsoft Research Asia

83.04085.892

37

May 14, 2019		ATB (single model)

Anonymous

82.88286.002

37

Feb 16, 2019		Bert-raw (ensemble)

None

83.17585.635

37

Feb 26, 2019		BERT with Something (ensemble)

Anonymous

83.05185.737

37

Jan 10, 2019		BERT + Synthetic Self-Training (single model)

Google AI Language

https://github.com/google-research/bert		82.97285.810

37

Jul 22, 2019		Tuned BERT Large Cased (single model)

FAIR & UW

82.80385.863

37

Mar 11, 2019		Bert-raw (ensemble)

None

83.11985.510

37

Feb 15, 2019		BERT + NeurQuRI (ensemble)

2SAH

82.80385.703

38

Feb 28, 2019		BERT + NeurQuRI (ensemble)

2SAH

82.71385.584

38

May 13, 2019		BERT-Base + QA Pre-training (single model)

Anonymous

82.72485.491

38

Dec 16, 2018		PAML+BERT (single model)

PINGAN GammaLab

82.57785.603

39

Nov 16, 2018		AoA + DA + BERT (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.37485.310

40

Dec 12, 2018		BERT finetune baseline (single model)

Anonymous

82.12684.820

40

Sep 22, 2020		BERT-Base PMI-Masking Additional Data (single model)

AI21 Labs

82.02484.854

41

Feb 28, 2019		BERT_s (single model)

Anonymous

81.97984.846

42

Feb 28, 2019		BERT-large+UBFT (single model)

anonymous

81.57384.535

43

Feb 15, 2019		BERT + NeurQuRI (single model)

2SAH

81.25784.342

43

Feb 25, 2019		BERT with Something (single model)

Anonymous

81.11084.386

43

Nov 16, 2018		AoA + DA + BERT (single model)

Joint Laboratory of HIT and iFLYTEK Research

81.17884.251

44

Mar 20, 2019		Bert-raw (single)

None

80.69383.922

44

Mar 07, 2019		BERT + UnAnsQ (single model)

Anonymous

80.74983.851

44

Sep 22, 2020		BERT-Base PMI-Masking (single model)

AI21 Labs

80.89683.604

45

Jan 22, 2019		BERT + NeurQuRI (single model)

2SAH

80.59183.391

45

Mar 12, 2019		Bert-raw (single)

None

80.41183.457

46

Sep 22, 2020		PMI-Masking Additional Data Random Baseline (single model)

AI21 Labs

80.37783.262

47

Feb 16, 2019		Bert-raw (single model)

None

80.34383.243

47

May 29, 2019		Bert

Single Model

https://senseforth.ai		80.42283.118

47

Sep 23, 2020		PMI-Masking Pure-PMI (single model)

AI21 Labs

80.24183.175

48

Apr 04, 2019		BISAN-CC (single model)

Seoul National University & Hyundai Motors

80.20883.149

48

Dec 03, 2018		PwP+BERT (single model)

AITRICS

80.11783.189

48

Jul 22, 2019		Original BERT Large Cased (single model)

FAIR & UW

79.97183.266

48

Feb 19, 2019		BERT + UDA (single model)

Anonymous

80.00583.208

49

Apr 10, 2019		bert (single model)

vinda msqjmxx

79.97183.184

49

Feb 28, 2019		ST_bl

single model

80.14082.962

49

Nov 09, 2018		BERT (single model)

Google AI Language

80.00583.061

50

Sep 23, 2020		PMI-Masking Additional Data Pure-PMI (single model)

AI21 Labs

79.99383.039

51

Feb 12, 2019		BERT + Sparse-Transformer

single model

79.94883.023

51

Sep 22, 2020		PMI-Masking Random Baseline (single model)

AI21 Labs

80.03882.796

51

Mar 07, 2019		BERT uncased (single model)

Anonymous

79.74583.020

51

Dec 06, 2018		NEXYS_BASE (single model)

NEXYS, DGIST R7

79.77982.912

52

Feb 02, 2019		{bert-finetuning} (single model)

ksai

79.63282.852

53

Feb 25, 2020		BERT-Large-Cased

single model

79.61082.692

54

Nov 09, 2018		L6Net + BERT (single model)

Layer 6 AI

79.18182.259

54

Mar 14, 2019		{Anonymous} (single model)

Anonymous

78.87682.524

55

Apr 24, 2019		BERT + WIAN (ensemble)

Infosys Limited

78.65081.497

56

Aug 03, 2020		AMBERT (single model)

ByteDance

78.59481.445

56

Mar 14, 2019		BISAN (single model)

Seoul National University & Hyundai Motors

78.48181.531

57

Dec 26, 2019		BERT-Large-Cased

single model

78.35781.500

58

Dec 14, 2018		BERT+AC (single model)

Hithink RoyalFlush

78.05281.174

59

Aug 03, 2020		BERT (single model)

ByteDance

77.31980.310

60

Nov 06, 2018		SLQA+BERT (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158		77.00380.209

61

Aug 03, 2020		AMBERT-H (single model)

ByteDance

76.71079.659

61

Aug 03, 2020		AMBERT-S (single model)

ByteDance

76.56379.776

62

Jan 05, 2019		synss (single model)

bert_finetune

76.05579.329

63

Apr 05, 2021		BERT-Base-L (single model)

Anonymous

75.45778.232

64

Dec 19, 2018		ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai		74.74678.227

64

Aug 29, 2020		BERT-Base-V (single model)

Anonymous

75.07377.805

64

Nov 05, 2018		MIR-MRC(F-Net) (single model)

Kangwon National University, Natural Language Processing Lab. & ForceWin, KP Lab.

74.79177.988

65

Aug 06, 2020		BERT-Base-DT (single model)

Anonymous

74.76977.706

66

Dec 03, 2020		BERT-Base-V2

single model

74.65677.404

66

Feb 25, 2021		BERT-Base-DP (single model)

Anonymous

74.57777.464

67

Aug 14, 2020		BERT-Base-Add (single model)

Anonymous

74.32977.396

67

May 23, 2019		{BERTcw} (single model)

private

74.38577.308

68

Sep 13, 2018		nlnet (single model)

Microsoft Research Asia

74.27277.052

69

Jan 13, 2020		batch2 (single model)

THU

73.74276.858

70

Dec 29, 2018		MMIPN

Single

73.50576.424

71

Aug 09, 2020		BERT-Base-Baseline (single model)

Anonymous

73.30276.284

72

Apr 20, 2019		BERT-Base (single model)

Dining Philosophers

73.09976.236

73

Oct 12, 2018		YARCS (ensemble)

IBM Research AI

72.67075.507

73

Apr 23, 2020		BERT-base

single model

72.07275.513

73

Apr 25, 2020		BERTBase (single model)

Anonymous

72.07275.513

74

Nov 14, 2018		BERT+Answer Verifier (single model)

Pingan Tech Olatop Lab

71.66675.457

75

Sep 17, 2018		Unet (ensemble)

Fudan University & Liulishuo Lab

https://arxiv.org/abs/1810.06638		71.41774.869

75

Apr 25, 2019		BERT-Base (single)

GreenflyAI

https://greenfly.ai		71.69974.430

75

Aug 15, 2018		Reinforced Mnemonic Reader + Answer Verifier (single model)

NUDT

https://arxiv.org/abs/1808.05759		71.76774.295

75

Aug 28, 2018		SLQA+ (single model)

Alibaba DAMO NLP

http://www.aclweb.org/anthology/P18-1158		71.46274.434

75

Apr 25, 2021		HYDRA_BERT (single model)

JAIST

71.29374.578

76

Jan 19, 2019		{BERT-base} (single-model)

Anonymous

70.76374.449

76

Sep 14, 2018		SAN (ensemble model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556		71.31673.704

77

Aug 21, 2018		FusionNet++ (ensemble)

Microsoft Business Applications Group AI Research

https://arxiv.org/abs/1711.07341		70.30072.484

77

Sep 26, 2018		Multi-Level Attention Fusion(MLAF) (single model)

Chonbuk National University, Cognitive Computing Lab.

69.47672.857

78

Sep 14, 2018		Unet (single model)

Fudan University & Liulishuo Lab

69.26272.642

79

Dec 20, 2018		DocQA + NeurQuRI (single model)

2SAH

68.76671.662

80

Aug 21, 2018		SAN (single model)

Microsoft Business Applications AI Research

https://arxiv.org/abs/1712.03556		68.65371.439

80

Sep 13, 2018		BiDAF++ with pair2vec (single model)

UW and FAIR

68.02171.583

80

Jun 25, 2018		KACTEIL-MRC(GFN-Net) (single model)

Kangwon National University, Natural Language Processing Lab.

68.21370.878

80

Jul 13, 2018		VS^3-NET (single model)

Kangwon National University in South Korea

67.89770.884

81

Jan 02, 2019		EBB-Net (single model)

Enliple AI

66.61070.303

82

Jun 25, 2018		KakaoNet2 (single model)

Kakao NLP Team

65.71969.381

83

Sep 13, 2018		BiDAF++ (single model)

UW and FAIR

65.65168.866

83

Jul 11, 2018		abcNet (single model)

Fudan University & Liulishuo AI Lab

65.25669.206

84

Jun 27, 2018		BSAE AddText (single model)

reciTAL.ai

63.33867.422

85

Aug 14, 2018		eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com		63.32766.633

85

May 30, 2018		BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

63.37266.251

86

May 30, 2018		BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence [modified by Stanford]

59.33262.305

87

May 30, 2018		BiDAF-No-Answer (single model)

University of Washington [modified by Stanford]

59.17462.093

87

Nov 27, 2018		Tree-LSTM + BiDAF + ELMo (single model)

Carnegie Mellon University

57.70762.341

SQuAD1.1 Leaderboard

Here are the ExactMatch (EM) and F1 scores evaluated on the test set of SQuAD v1.1.

RankModelEMF1
Human Performance

Stanford University

(Rajpurkar et al. '16)		82.30491.221

1

Apr 10, 2020		LUKE (single model)

Studio Ousia & NAIST & RIKEN AIP

90.20295.379

2

May 21, 2019		XLNet (single model)

Google Brain & CMU

89.89895.080

3

Dec 11, 2019		XLNET-123++ (single model)

MST/EOI

http://tia.today		89.85694.903

3

Aug 11, 2019		XLNET-123 (single model)

MST/EOI

89.64694.930

4

Sep 25, 2019		BERTSP (single model)

NEUKG

http://www.techkg.cn/		88.91294.584

4

Jul 21, 2019		SpanBERT (single model)

FAIR & UW

88.83994.635

5

Jul 03, 2019		BERT+WWM+MT (single model)

Xiaoi Research

88.65094.393

6

Jul 21, 2019		Tuned BERT-1seq Large Cased (single model)

FAIR & UW

87.46593.294

7

Oct 05, 2018		BERT (ensemble)

Google AI Language

https://arxiv.org/abs/1810.04805		87.43393.160

8

May 14, 2019		ATB (single model)

Anonymous

86.94092.641

9

Jul 21, 2019		Tuned BERT Large Cased (single model)

FAIR & UW

86.52192.617

9

Jul 04, 2019		BERT+MT (single model)

Xiaoi Research

86.45892.645

10

Feb 14, 2019		KT-NET (single model)

Baidu NLP

85.94492.425

10

Sep 27, 2018		nlnet (ensemble)

Microsoft Research Asia

85.95491.677

10

Feb 28, 2019		ST_bl

single model

85.43091.976

11

Nov 21, 2019		EL-BERT (single model)

YeonTaek Oh

85.33591.807

12

Mar 14, 2019		BISAN (single model)

Seoul National University & Hyundai Motors

85.31491.756

12

Jun 03, 2019		DPN (single model)

Anonymous

84.97892.019

12

Oct 05, 2018		BERT (single model)

Google AI Language

https://arxiv.org/abs/1810.04805		85.08391.835

12

Jul 11, 2019		BERT-uncased (single model)

Anonymous

84.92691.932

12

Feb 16, 2019		BERT+Sparse-Transformer

single model

85.12591.623

12

Sep 09, 2018		nlnet (ensemble)

Microsoft Research Asia

85.35691.202

12

Jul 21, 2019		Original BERT Large Cased (single model)

FAIR & UW

84.32891.281

12

Feb 19, 2019		WD (single model)

Anonymous

84.40290.561

12

Jul 11, 2018		QANet (ensemble)

Google Brain & CMU

84.45490.490

12

Apr 21, 2019		Common-sense Governed BERT-123 (single model)

Jerry AGI Ragtag

83.93090.613

13

Feb 21, 2019		WD1 (single model)

Anonymous

83.80490.429

13

Jul 08, 2018		r-net (ensemble)

Microsoft Research Asia

84.00390.147

13

May 08, 2019		Common-sense Governed BERT-123 (single model)

MST/EOI

82.94391.074

13

Jun 20, 2018		MARS (ensemble)

YUANFUDAO research NLP

83.98289.796

14

Mar 19, 2018		QANet (ensemble)

Google Brain & CMU

83.87789.737

14

Sep 09, 2018		nlnet (single model)

Microsoft Research Asia

83.46890.133

15

Sep 01, 2018		MARS (single model)

YUANFUDAO research NLP

83.18589.547

16

Jun 21, 2018		MARS (single model)

YUANFUDAO research NLP

83.12289.224

17

Mar 06, 2018		QANet (ensemble)

Google Brain & CMU

82.74489.045

17

Jun 20, 2018		QANet (single)

Google Brain & CMU

82.47189.306

17

Jan 22, 2018		Hybrid AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

82.48289.281

17

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (ensemble model)

Microsoft Research Asia & NUDT

82.84988.764

17

May 09, 2018		MARS (single model)

YUANFUDAO research NLP

82.58788.880

17

Jan 03, 2018		r-net+ (ensemble)

Microsoft Research Asia

82.65088.493

17

Jan 05, 2018		SLQA+ (ensemble)

Alibaba iDST NLP

82.44088.607

17

Jul 14, 2019		BERT (single model)

KTNET

82.06288.947

17

Feb 28, 2018		QANet (single model)

Google Brain & CMU

82.20988.608

17

Feb 02, 2018		Reinforced Mnemonic Reader (ensemble model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		82.28388.533

17

Dec 24, 2018		MMIPN

Single

81.58088.948

17

Dec 17, 2017		r-net (ensemble)

Microsoft Research Asia

http://aka.ms/rnet		82.13688.126

17

Dec 17, 2018		ARSG-BERT (single model)

TRINITI RESEARCH LABS, Active.ai

https://active.ai		81.30788.909

17

Dec 22, 2017		AttentionReader+ (ensemble)

Tencent DPDAC NLP

81.79088.163

18

May 09, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

81.53888.130

18

Apr 23, 2018		r-net (single model)

Microsoft Research Asia

81.39188.170

18

May 09, 2018		Reinforced Mnemonic Reader + A2D + DA (single model)

Microsoft Research Asia & NUDT

81.40188.122

18

Apr 03, 2018		KACTEIL-MRC(GF-Net+) (ensemble)

Kangwon National University, Natural Language Processing Lab.

81.49687.557

18

Feb 27, 2018		QANet (single model)

Google Brain & CMU

80.92987.773

18

Nov 17, 2017		BiDAF + Self Attention + ELMo (ensemble)

Allen Institute for Artificial Intelligence

81.00387.432

18

Feb 19, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.91987.492

18

Mar 11, 2020		batch (single model)

THU

79.85988.263

18

Feb 12, 2018		Reinforced Mnemonic Reader + A2D (single model)

Microsoft Research Asia & NUDT

80.48987.454

18

Apr 12, 2018		AVIQA+ (ensemble)

aviqa team

80.61587.311

19

Jan 13, 2018		SLQA+

single model

80.43687.021

19

Jan 04, 2018		{EAZI} (ensemble)

Yiwise NLP Group

80.43686.912

19

Jan 12, 2018		EAZI+ (ensemble)

Yiwise NLP Group

80.42686.912

19

Jan 22, 2018		Hybrid AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

80.02787.288

19

Jan 06, 2020		BERT-INDEPENDENT-DSS-FILTERED (single model)

Brno University of Technology

79.59787.374

19

Mar 20, 2018		DNET (ensemble)

QA geeks

80.16486.721

20

Feb 13, 2018		BiDAF + Self Attention + ELMo + A2D (single model)

Microsoft Research Asia & NUDT

79.99686.711

21

Jan 03, 2018		r-net+ (single model)

Microsoft Research Asia

79.90186.536

21

Feb 23, 2018		MAMCN+ (single model)

Samsung Research

79.69286.727

22

Jan 29, 2018		Reinforced Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		79.54586.654

22

Dec 05, 2017		SAN (ensemble model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		79.60886.496

22

Dec 28, 2017		SLQA+ (single model)

Alibaba iDST NLP

79.19986.590

23

Oct 18, 2017		Interactive AoA Reader+ (ensemble)

Joint Laboratory of HIT and iFLYTEK

79.08386.450

23

Nov 05, 2018		KACTEIL-MRC(GF-Net+Distillation) (single model)

Kangwon National University, Natural Language Processing Lab.

79.08386.288

24

Jun 02, 2018		MDReader

single model

79.03186.006

24

Oct 24, 2017		FusionNet (ensemble)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1711.07341		78.97886.016

25

Oct 22, 2017		DCN+ (ensemble)

Salesforce Research

https://arxiv.org/abs/1711.00106		78.85285.996

26

Mar 30, 2018		KACTEIL-MRC(GF-Net+) (single model)

Kangwon National University, Natural Language Processing Lab.

78.66485.780

26

Nov 03, 2017		BiDAF + Self Attention + ELMo (single model)

Allen Institute for Artificial Intelligence

78.58085.833

27

May 10, 2018		KakaoNet (single model)

Kakao NLP Team

78.40185.724

28

Nov 30, 2017		SLQA (ensemble)

Alibaba iDST NLP

78.32885.682

28

Mar 19, 2018		aviqa (ensemble)

aviqa team

78.49685.469

28

Jan 02, 2018		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		78.43385.517

28

Sep 18, 2018		BiDAF++ with pair2vec (single model)

UW and FAIR

78.22385.535

28

Jun 01, 2018		MDReader0

single model

78.17185.543

28

Jan 03, 2018		MEMEN (single model)

Zhejiang University

https://arxiv.org/abs/1707.09098		78.23485.344

28

Jan 29, 2018		test

single

78.08785.348

29

Jul 26, 2017		Interactive AoA Reader (ensemble)

Joint Laboratory of HIT and iFLYTEK Research

77.84585.297

30

Mar 20, 2018		DNET (single model)

QA geeks

77.64684.905

31

Sep 18, 2018		BiDAF++ (single model)

UW and FAIR

77.57384.858

31

Dec 06, 2017		AttentionReader+ (single)

Tencent DPDAC NLP

77.34284.925

31

Dec 14, 2017		RaSoR + TR + LM (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		77.58384.163

31

Dec 21, 2017		Jenga (ensemble)

Facebook AI Research

77.23784.466

31

Nov 06, 2017		Conductor-net (ensemble)

CMU

https://arxiv.org/abs/1710.10504		76.99684.630

31

Jan 23, 2018		MARS (single model)

YUANFUDAO research NLP

76.85984.739

32

May 14, 2018		VS^3-NET (single model)

Kangwon National University in South Korea

76.77584.491

32

Nov 01, 2017		SAN (single model)

Microsoft Business AI Solutions Team

https://arxiv.org/abs/1712.03556		76.82884.396

32

Sep 26, 2018		{gqa} (single model)

FAIR

77.09083.931

32

Dec 19, 2017		FRC (single model)

in review

76.24084.599

32

Oct 13, 2017		r-net (single model)

Microsoft Research Asia

http://aka.ms/rnet		76.46184.265

33

Oct 22, 2017		Conductor-net (ensemble)

CMU

76.14683.991

34

Sep 08, 2017		FusionNet (single model)

Microsoft Business AI Solutions team

https://arxiv.org/abs/1711.07341		75.96883.900

35

Oct 22, 2017		Interactive AoA Reader+ (single model)

Joint Laboratory of HIT and iFLYTEK

75.82183.843

35

Oct 18, 2018		KAR (single model)

York University

https://arxiv.org/abs/1809.03449		76.12583.538

36

Jul 14, 2017		smarnet (ensemble)

Eigen Technology & Zhejiang University

75.98983.475

37

Mar 15, 2018		AVIQA-v2 (single model)

aviqa team

75.92683.305

38

Aug 18, 2017		RaSoR + TR (single model)

Tel-Aviv University

https://arxiv.org/abs/1712.03609		75.78983.261

38

Mar 20, 2020		Kbs (single model)

Tsinghua University

75.03483.405

38

Oct 23, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		75.08783.081

38

Nov 01, 2017		Mixed model (ensemble)

Sean

75.26582.769

38

May 21, 2017		MEMEN (ensemble)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1707.09098		75.37082.658

38

Nov 17, 2017		two-attention-self-attention (ensemble)

guotong1988

75.22382.716

38

Jul 10, 2017		DCN+ (single model)

Salesforce Research

https://arxiv.org/abs/1711.00106		74.86682.806

38

Mar 09, 2017		ReasoNet (ensemble)

MSR Redmond

https://arxiv.org/abs/1609.05284		75.03482.552

38

Oct 31, 2017		SLQA (single model)

Alibaba iDST NLP

74.48982.815

38

Feb 06, 2018		Jenga (single model)

Facebook AI Research

74.37382.845

38

Jan 02, 2018		Conductor-net (single model)

CMU

https://arxiv.org/abs/1710.10504		74.40582.742

38

Aug 14, 2018		eeAttNet (single model)

BBD NLP Team

https://www.bbdservice.com		74.60482.501

39

Feb 13, 2018		SSR-BiDAF

ensemble model

74.54182.477

40

Jul 14, 2017		Mnemonic Reader (ensemble)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		74.26882.371

41

Dec 23, 2017		S^3-Net (ensemble)

Kangwon National University in South Korea

74.12182.342

42

Jul 29, 2017		SEDT (ensemble model)

CMU

https://arxiv.org/abs/1703.00572		74.09081.761

43

Jul 06, 2017		SSAE (ensemble)

Tsinghua University

74.08081.665

43

Jul 25, 2017		Interactive AoA Reader (single model)

Joint Laboratory of HIT and iFLYTEK Research

73.63981.931

43

Feb 22, 2017		BiDAF (ensemble)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		73.74481.525

43

Apr 22, 2017		SEDT+BiDAF (ensemble)

CMU

https://arxiv.org/abs/1703.00572		73.72381.530

43

Nov 06, 2017		Conductor-net (single)

CMU

https://arxiv.org/abs/1710.10504		73.24081.933

43

Dec 14, 2017		Jenga (single model)

Facebook AI Research

73.30381.754

43

Jan 24, 2017		Multi-Perspective Matching (ensemble)

IBM Research

https://arxiv.org/abs/1612.04211		73.76581.257

43

May 01, 2017		jNet (ensemble)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		73.01081.517

44

Oct 22, 2017		Conductor-net (single)

CMU

72.59081.415

44

Apr 12, 2017		T-gating (ensemble)

Peking University

72.75881.001

44

Nov 16, 2017		two-attention-self-attention (single model)

guotong1988

72.60081.011

44

Sep 20, 2017		BiDAF + Self Attention (single model)

Allen Institute for Artificial Intelligence

https://arxiv.org/abs/1710.10723		72.13981.048

44

Mar 03, 2018		AVIQA (single model)

aviqa team

72.48580.550

44

Dec 15, 2017		S^3-Net (single model)

Kangwon National University in South Korea

71.90881.023

45

Nov 06, 2017		attention+self-attention (single model)

guotong1988

71.69880.462

46

Nov 02, 2016		Dynamic Coattention Networks (ensemble)

Salesforce Research

https://arxiv.org/abs/1611.01604		71.62580.383

46

Apr 13, 2017		QFASE

NUS

71.89879.989

46

Jul 14, 2017		smarnet (single model)

Eigen Technology & Zhejiang University

https://arxiv.org/abs/1710.02772		71.41580.160

47

Jul 14, 2017		Mnemonic Reader (single model)

NUDT and Fudan University

https://arxiv.org/abs/1705.02798		70.99580.146

47

May 23, 2018		AttReader (single)

College of Computer & Information Science, SouthWest University, Chongqing, China

71.37379.725

47

Apr 22, 2018		MAMCN (single model)

Samsung Research

70.98579.939

47

Oct 27, 2017		M-NET (single)

UFL

71.01679.835

48

Mar 24, 2017		jNet (single model)

USTC & National Research Council Canada & York University

https://arxiv.org/abs/1703.04617		70.60779.821

48

Apr 02, 2017		Ruminating Reader (single model)

New York University

https://arxiv.org/abs/1704.07415		70.63979.456

48

Mar 14, 2017		Document Reader (single model)

Facebook AI Research

https://arxiv.org/abs/1704.00051		70.73379.353

48

Mar 08, 2017		ReasoNet (single model)

MSR Redmond

https://arxiv.org/abs/1609.05284		70.55579.364

48

Dec 29, 2016		FastQAExt

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		70.84978.857

48

May 13, 2017		RaSoR (single model)

Google NY, Tel-Aviv University

https://arxiv.org/abs/1611.01436		70.84978.741

48

Apr 14, 2017		Multi-Perspective Matching (single model)

IBM Research

https://arxiv.org/abs/1612.04211		70.38778.784

49

Aug 30, 2017		SimpleBaseline (single model)

Technical University of Vienna

69.60078.236

49

Feb 06, 2018		SSR-BiDAF

single model

69.44378.358

50

Apr 12, 2017		SEDT+BiDAF (single model)

CMU

https://arxiv.org/abs/1703.00572		68.47877.971

51

Jun 25, 2017		PQMN (single model)

KAIST & AIBrain & Crosscert

68.33177.783

52

Apr 12, 2017		T-gating (single model)

Peking University

68.13277.569

52

Jul 29, 2017		SEDT (single model)

CMU

https://arxiv.org/abs/1703.00572		68.16377.527

52

Dec 29, 2016		FastQA

German Research Center for Artificial Intelligence

https://arxiv.org/abs/1703.04816		68.43677.070

52

Jan 22, 2018		FABIR

Single Model

https://arxiv.org/abs/1810.09580		67.74477.605

52

Nov 28, 2016		BiDAF (single model)

Allen Institute for AI & University of Washington

https://arxiv.org/abs/1611.01603		67.97477.323

53

Oct 26, 2016		Match-LSTM with Ans-Ptr (Boundary) (ensemble)

Singapore Management University

https://arxiv.org/abs/1608.07905		67.90177.022

53

Sep 19, 2017		AllenNLP BiDAF (single model)

Allen Institute for AI

http://allennlp.org/		67.61877.151

54

Feb 05, 2017		Iterative Co-attention Network

Fudan University

67.50276.786

55

Jan 03, 2018		newtest

single model

66.52775.787

55

Nov 02, 2016		Dynamic Coattention Networks (single model)

Salesforce Research

https://arxiv.org/abs/1611.01604		66.23375.896

56

Oct 26, 2016		Match-LSTM with Bi-Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		64.74473.743

57

Sep 21, 2017		OTF dict+spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		64.08373.056

57

Feb 19, 2017		Attentive CNN context with LSTM

NLPR, CASIA

63.30673.463

58

Nov 02, 2016		Fine-Grained Gating

Carnegie Mellon University

https://arxiv.org/abs/1611.01724		62.44673.327

58

Sep 21, 2017		OTF spelling (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.89772.016

59

Sep 21, 2017		OTF spelling+lemma (single)

University of Montreal

https://arxiv.org/abs/1706.00286		62.60471.968

60

Sep 28, 2016		Dynamic Chunk Reader

IBM

https://arxiv.org/abs/1610.09996		62.49970.956

60

Nov 15, 2019		RQA+IDR (single model)

BUAA & MSRA

https://arxiv.org/abs/2005.02925		61.14571.389

61

Aug 27, 2016		Match-LSTM with Ans-Ptr (Boundary)

Singapore Management University

https://arxiv.org/abs/1608.07905		60.47470.695

62

Aug 27, 2016		Match-LSTM with Ans-Ptr (Sentence)

Singapore Management University

https://arxiv.org/abs/1608.07905		54.50567.748

62

Nov 15, 2019		RQA (single model)

BUAA & MSRA

https://arxiv.org/abs/2005.02925		55.82765.467

63

Aug 22, 2019		UQA (single model)

Anonymous

53.69864.036