Tasks

List of ParlAI tasks defined in the file task_list.py:

QA

bAbI 1k

Tag: #bAbI-1k

Full Path: babi:All1k

Group Tags: #All, #QA

Description: 20 synthetic tasks that each test a unique aspect of text and reasoning, and hence test different capabilities of learning models. From Weston et al. ‘16. Link: http://arxiv.org/abs/1502.05698

Notes: You can access just one of the bAbI tasks with e.g. ‘babi:Task1k:3’ for task 3.

bAbI 10k

Tag: #bAbI-10k

Full Path: babi:All10k

Group Tags: #All, #QA

Description: 20 synthetic tasks that each test a unique aspect of text and reasoning, and hence test different capabilities of learning models. From Weston et al. ‘16. Link: http://arxiv.org/abs/1502.05698

Notes: You can access just one of the bAbI tasks with e.g. ‘babi:Task10k:3’ for task 3.

MCTest

Tag: #MCTest

Full Path: mctest

Group Tags: #All, #QA

Description: Questions about short children’s stories, from Richardson et al. ‘13. Link: https://www.microsoft.com/en-us/research/publication/mctest-challenge-dataset-open-domain-machine-comprehension-text/

Movie Dialog QA

Tag: #MovieDD-QA

Full Path: moviedialog:Task:1

Group Tags: #All, #QA, #MovieDD

Description: Closed-domain QA dataset asking templated questions about movies, answerable from Wikipedia, similar to WikiMovies. From Dodge et al. ‘15. Link: https://arxiv.org/abs/1511.06931

Movie Dialog Recommendations

Tag: #MovieDD-Recs

Full Path: moviedialog:Task:2

Group Tags: #All, #QA, #MovieDD

Description: Questions asking for movie recommendations. From Dodge et al. ‘15. Link: https://arxiv.org/abs/1511.06931

MTurk WikiMovies

Tag: #MTurkWikiMovies

Full Path: mturkwikimovies

Group Tags: #All, #QA

Description: Closed-domain QA dataset asking MTurk-derived questions about movies, answerable from Wikipedia. From Li et al. ‘16. Link: https://arxiv.org/abs/1611.09823

Simple Questions

Tag: #SimpleQuestions

Full Path: simplequestions

Group Tags: #All, #QA

Description: Open-domain QA dataset based on Freebase triples from Bordes et al. ‘15. Link: https://arxiv.org/abs/1506.02075

SQuAD

Tag: #SQuAD

Full Path: squad

Group Tags: #All, #QA

Description: Open-domain QA dataset answerable from a given paragraph from Wikipedia, from Rajpurkar et al. ‘16. Link: https://arxiv.org/abs/1606.05250

TriviaQA

Tag: #TriviaQA

Full Path: triviaqa

Group Tags: #All, #QA

Description: Open-domain QA dataset with question-answer-evidence triples, from Joshi et al. ‘17. Link: https://arxiv.org/abs/1705.03551

Web Questions

Tag: #WebQuestions

Full Path: webquestions

Group Tags: #All, #QA

Description: Open-domain QA dataset from Web queries from Berant et al. ‘13. Link: http://www.aclweb.org/anthology/D13-1160

WikiMovies

Tag: #WikiMovies

Full Path: wikimovies

Group Tags: #All, #QA

Description: Closed-domain QA dataset asking templated questions about movies, answerable from Wikipedia. From Miller et al. ‘16. Link: https://arxiv.org/abs/1606.03126

WikiQA

Tag: #WikiQA

Full Path: wikiqa

Group Tags: #All, #QA

Description: Open domain QA from Wikipedia dataset from Yang et al. ‘15. Link: https://www.microsoft.com/en-us/research/publication/wikiqa-a-challenge-dataset-for-open-domain-question-answering/

InsuranceQA

Tag: #InsuranceQA

Full Path: insuranceqa

Group Tags: #All, #QA

Description: Task which requires agents to identify high quality answers composed by professionals with deep domain knowledge. Link: https://github.com/shuzi/insuranceQA

MS_MARCO

Tag: #MS_MARCO

Full Path: ms_marco

Group Tags: #All, #QA

Description: A Reading Comprehension Dataset for the Artificial Intelligence research community. Link: http://www.msmarco.org/dataset.aspx

Cloze

BookTest

Tag: #BookTest

Full Path: booktest

Group Tags: #All, #Cloze

Description: Sentence completion given a few sentences as context from a book. A larger version of CBT. From Bajgar et al., 16. Link: https://arxiv.org/abs/1610.00956

Children’s Book Test (CBT)

Tag: #CBT

Full Path: cbt

Group Tags: #All, #Cloze

Description: Sentence completion given a few sentences as context from a children’s book. From Hill et al., ‘16. Link: https://arxiv.org/abs/1511.02301

QA CNN

Tag: #QACNN

Full Path: qacnn

Group Tags: #All, #Cloze

Description: Cloze dataset based on a missing (anonymized) entity phrase from a CNN article, Hermann et al. ‘15. Link: https://arxiv.org/abs/1506.03340

QA Daily Mail

Tag: #QADailyMail

Full Path: qadailymail

Group Tags: #All, #Cloze

Description: Cloze dataset based on a missing (anonymized) entity phrase from a Daily Mail article, Hermann et al. ‘15. Link: https://arxiv.org/abs/1506.03340

Goal

Dialog Based Language Learning: bAbI Task

Tag: #DBLL-bAbI

Full Path: dbll_babi

Group Tags: #All, #Goal

Description: Short dialogs based on the bAbI tasks, but in the form of a question from a teacher, the answer from the student, and finally a comment on the answer from the teacher. The aim is to find learning models that use the comments to improve. From Weston ‘16. Link: https://arxiv.org/abs/1604.06045

Dialog Based Language Learning: WikiMovies Task

Tag: #DBLL-Movie

Full Path: dbll_movie

Group Tags: #All, #Goal

Description: Short dialogs based on WikiMovies, but in the form of a question from a teacher, the answer from the student, and finally a comment on the answer from the teacher. The aim is to find learning models that use the comments to improve. From Weston ‘16. Link: https://arxiv.org/abs/1604.06045

Dialog bAbI

Tag: #dialog-bAbI

Full Path: dialog_babi

Group Tags: #All, #Goal

Description: Simulated dialogs of restaurant booking, from Bordes et al. ‘16. Link: https://arxiv.org/abs/1605.07683

Movie Dialog QA Recommendations

Tag: #MovieDD-QARecs

Full Path: moviedialog:Task:3

Group Tags: #All, #Goal, #MovieDD

Description: Dialogs discussing questions about movies as well as recommendations. From Dodge et al. ‘15. Link: https://arxiv.org/abs/1511.06931

Personalized Dialog Full Set

Tag: #personalized-dialog-full

Full Path: personalized_dialog:full

Group Tags: #All, #Goal, #Personalization

Description: Simulated dataset of restaurant booking focused on personalization based on user profiles. From Joshi et al. ‘17. Link: https://arxiv.org/abs/1706.07503

Personalized Dialog Small Set

Tag: #personalized-dialog-small

Full Path: personalized_dialog:small

Group Tags: #All, #Goal, #Personalization

Description: Simulated dataset of restaurant booking focused on personalization based on user profiles. From Joshi et al. ‘17. Link: https://arxiv.org/abs/1706.07503

ChitChat

Cornell Movie

Tag: #CornellMovie

Full Path: cornell_movie

Group Tags: #All, #ChitChat

Description: Fictional conversations extracted from raw movie scripts. Link: https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html

Movie Dialog Reddit

Tag: #MovieDD-Reddit

Full Path: moviedialog:Task:4

Group Tags: #All, #ChitChat, #MovieDD

Description: Dialogs discussing Movies from Reddit (the Movies SubReddit). From Dodge et al. ‘15. Link: https://arxiv.org/abs/1511.06931

Open Subtitles

Tag: #OpenSubtitles

Full Path: opensubtitles

Group Tags: #All, #ChitChat

Description: Dataset of dialogs from movie scripts: http://opus.lingfil.uu.se/OpenSubtitles.php. A variant of the dataset used in Vinyals & Le ‘15, https://arxiv.org/abs/1506.05869.

Ubuntu

Tag: #Ubuntu

Full Path: ubuntu

Group Tags: #All, #ChitChat

Description: Dialogs between an Ubuntu user and an expert trying to fix issue, from Lowe et al. ‘15. Link: https://arxiv.org/abs/1506.08909

Visual

VQAv1

Tag: #VQAv1

Full Path: vqa_v1

Group Tags: #All, #Visual

Description: Open-ended question answering about visual content. From Agrawal et al. ‘15. Link: https://arxiv.org/abs/1505.00468

VQAv2

Tag: #VQAv2

Full Path: vqa_v2

Group Tags: #All, #Visual

Description: Bigger, more balanced version of the original VQA dataset. From Goyal et al. ‘16. Link: https://arxiv.org/abs/1612.00837

VisDial

Tag: #VisDial

Full Path: visdial

Group Tags: #All, #Visual

Description: Task which requires agents to hold a meaningful dialog about visual content. From Das et al. ‘16. Link: https://arxiv.org/abs/1611.08669

MNIST_QA

Tag: #MNIST_QA

Full Path: mnist_qa

Group Tags: #All, #Visual

Description: Task which requires agents to identify which number they are seeing. From the MNIST dataset.