Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader's background knowledge. We formalise this task and develop a crowd-sourcing strategy to collect 32k task instances.
In our task, the goal is to answer questions by possibly asking follow-up questions first.
We assume that the question is often underspecified, in the sense that the question does not provide enough information to be answered directly. However, an agent can use the supporting rule text to infer what needs to be asked in order to determine the final answer. In the example in the figure, a reasonable follow-up question is "Have you been working abroad 52 weeks or less?"
More explanation on the task and the dataset can be found in the paper.
Kindly note that the official leaderboard is now closed to future submissions. To help support future work using this resource, the test set has now been released publicly (as of the 9th February, 2024) and is available to download through the data page or Hugging Face Datasets. Thank you for your participation!
marzieh.saeidi [at] gmail.com
maxbartolo [at] gmail.com
patrick.s.h.lewis [at] gmail.com
s.riedel [at] cs.ucl.ac.uk