Facebook has opened source ParlAI, a framework for conversational AI research

• All datasets look like natural dialogue: a single format / API.
• Both fixed datasets (conversation logs) and interactive (online/RL) tasks.
• Both real and simulated tasks.
• Supports other media, e.g. visual in VQA.
• Can use Mechanical Turk to run / collect data / evaluate.
• Python framework.
• Examples of training with PyTorch.
• Uses zmq to talk to other toolboxes not in Python, examples of Lua Torch given.
• Supports hogwild and batch training of models.

ParlAI currently requires Python3.

Dependencies of the core modules are listed in requirement.txt. Several models included (in parlai/agents) have additional requirements such as PyTorch or Lua Torch–any python requirements in these modules are listed in requirements_ext.txt.

The main concepts (classes) in ParlAI:

• world – defines the environment (can be very simple, just two agents talking to each other).
• Learner — an agent in the world, e.g. learner.
• Teacher — a type of agent that talks to the learner, implements one of the tasks listed before.

After defining a world and the agents in it, a main loop can be run for training, testing or displaying, which calls the function world.parley(). The skeleton of an example main is given in the left panel, and the actual code for parley() on the right.

Each of these fields are technically optional, depending on your dataset, though the ‘text’ field will most likely be used in nearly all exchanges.

For a fixed supervised learning dataset like bAbI, a typical exchange from the training set might be as follows (the test set would not include labels):

Teacher: { 'text': 'Sam went to the kitchen\nPat gave Sam the milk\nWhere is the milk? ', 'labels': ['kitchen'], 'label_candidates': ['hallway', 'kitchen', 'bathroom'], 'episode_done': False } Student: { 'text': 'hallway' } Teacher: { 'text': 'Sam went to the hallway\nPat went to the bathroom\nWhere is the milk? ', 'labels': ['hallway'], 'label_candidates': ['hallway', 'kitchen', 'bathroom'], 'episode_done': True } Student: { 'text': 'hallway' } Teacher: { ... # starts next episode } ...Copy the code

The core library contains the following files:

• agents.py: this file contains a few basic agents which can be extended by your own model
  • Agent: base class for all other agents, implements the act() method which receives an observation table and returns a table in response
  • Teacher: child of Agent, also implements the report method for returning metrics. Tasks implement the Teacher class
  • MultiTaskTeacher: creates a set of teachers based on a “task string” passed to the Teacher, creating multiple teachers within it and alternating between them
  • create_task_teacher: instantiate a teacher from a given task string (e.g. ‘babi:task:1’ or ‘squad’)
• build_data.py: basic utilities for setting up data for tasks. you can override if your filesystem needs different functionality.
• dialog_teacher.py: contains a base teacher class for doing dialog with fixed chat logs, along with a data class for storing the data
• dict.py: contains code for building general NLP-style dictionaries from observations
  • DictionaryAgent: agent which tracks the index and frequency of words in a dictionary, and can parse a sentence into indices into its dictionary or back
• fbdialog_teacher.py: contains a teacher class which implements a function setup_data which parses data in the FB Dialog data format
• metrics.py: computes evaluation metrics for dialog, e.g. ranking metrics, etc.
• params.py: uses argparse to interpret command line arguments for ParlAI
• thread_utils.py: utility classes/functions for use in Hogwild multithreading (multiprocessing)
  • SharedTable: provides a lock-protected, shared-memory, dictionary-like interface for keeping track of metrics
• worlds.py: contains a set of basic worlds for tasks to take place inside
  • World: base class for all other worlds, implements parley.shutdown.__enter__, and __exit__
  • DialogPartnerWorld: default world for turn-based two-agent communication MultiAgentDialogWorld: round-robin turn-based agent communication for two or more agents HogwildWorld: default world for setting up a separate world for every thread when using multiple threads (processes)

This directory contains a few particular examples of basic loops.

• base_train.py: very simple example shows the outline of a training/validation loop using the default Agent parent class
• display_data.py: uses agent.repeat_label to display data from a particular task provided on the command-line
• display_model.py: shows the predictions of a provided model on a particular task provided on the command-line
• eval_model.py: uses the named agent to compute evaluation metrics data for a particular task provided on the command-line
• build_dict.py: build a dictionary from a particular task provided on the command-line using core.dict.DictionaryAgent
• memnn_luatorch_cpu: shows a few examples of training an end-to-end memory network on a few datasets
• drqa: shows how to train the attentive LSTM DrQA model of Chen et al. on SQuAD.

See here for the current complete task list.

Choosing a task in ParlAI is as easy as specifying it on the command line, as shown in the above image (right). If the dataset has not been used before, ParlAI will automatically download it. As all datasets are treated in the same way in ParlAI (with a single dialog API), a dialog agent can in principle switch training and testing between any of them. Even better, one can specify many tasks at once (multi-tasking) by simply providing a comma-separated list, E.g. the command line “-t babi,squad”, to use those two datasets, or even all the QA datasets at once (-t #qa) or indeed every task in ParlAI at once (-t #all). The aim is to make it easy to build and evaluate very rich dialog models.

Each task folder contains:

• build.py file for setting up data for the task (downloading data, etc, only done the first time requested, and not downloaded if the task is not used).
• agents.py file which contains default or special teacher classes used by core.create_task to instantiate these classes from command-line arguments (if desired).
• worlds.py file can optionally be added for tasks that need to define new/complex environments.

To add your own task:

• (optional) implement build.py to download any needed data
• implement agents.py, with at least a DefaultTeacher (extending Teacher or one of its children)
  • if your data is in FB Dialog format, subclass FbDialogTeacher
  • if not…
    • if your data is text-based, you can use extend DialogTeacher and thus core.data.TextData, in which case you just need to write your own setup_data function which provides an iterable over the data according to the format described in core.data
    • if your data uses other fields, write your own act() method which provides observations from your task each time it’s called

The mturk library contains the following directories and files:

• core: this directory contains the core code for setting up AWS backend that supports the MTurk chat interface, and code for HIT creation and approval.
• tasks: this directory contains two sample MTurk tasks that are provided in the first release.
  • qa_data_collection: get questions and answers from turkers, given a random paragraph from SQuAD.
  • model_evaluator: evaluate the information retrieval baseline model on the Reddit movie dialog dataset.
• run_mturk.py: file for calling mturk core code with user-specified task module, dialog model agent, number of HITs, and reward for each HIT.

To run sample MTurk task and agent:

• In run_mturk.py, uncomment the task module and the agent class you want to use
• For create_hits method, change num_hits and hit_reward if needed. Set is_sandbox to True if you want to run the sample in MTurk sandbox only, or set it to False to allow turkers to work on it and potentially get paid for it.
• Run python run_mturk.py

To add your own MTurk task and dialog model:

• create a new folder within the mturk/tasks directory for your new task
• implement task_config.py, with at least the following fields in the task_config dictionary:
  • hit_title: a short and descriptive title about the kind of task the HIT contains. On the Amazon Mechanical Turk web site, the HIT title appears in search results, and everywhere the HIT is mentioned.
  • hit_description: a description includes detailed information about the kind of task the HIT contains. On the Amazon Mechanical Turk web site, the HIT description appears in the expanded view of search results, and in the HIT and assignment screens.
  • hit_keywords: one or more words or phrases that describe the HIT, separated by commas. On MTurk website, these words are used in searches to find HITs.
  • worker_agent_id: a short name indicating the turker’s role in the conversation.
  • task_description: a detailed task description that will be shown on the HIT task preview page and on the left side of the chat page. Supports HTML formatting.
• implement agents.py, with at least an agent class that extends from Agent
  • write your own __init__() method that wraps your dialog model agent. (Please see mturk/tasks/model_evaluator/agents.py file for a concrete example.)
  • write your own act() method that returns your dialog model’s response as well as helpful text to the turker for what action they should take next.
• import your task module and agent class in run_mturk.py file, and then run python run_mturk.py.

ParlAI is BSD-licensed. We also provide an additional patent grant.