Instructions

Key:

Outcome
Intervention
Comparator

Example prompt:

With respect to frequency of contractions, characterize the difference between antibiotics and placebo.

Task description:

This task involves generating questions (or "prompts", see example above) that are answerable on the basis of a given full-text article describing a randomized control trial (RCT). More specifically, such prompts will concern the comparison of specific interventions with respect to particular outcomes. Essentially, these should ask whether a given treatment was found (in the RCT) to be significantly more effective than a comparator treatment with respect to a defined outcome of interest. Trials typically report results for multiple outcomes (and often more than two interventions), and the results for these will of course differ. Therefore, you will be asked to generate multiple unique prompts for each full-text article. Taken together, the answers to these prompts should summarize the main findings reported in a given article regarding the treatment effectiveness of the interventions studied. A high-level description of the task is as follows.

For each article:

Find an outcome, comparator, and intervention.
Add these, along with the relationship between each, to the table.
If there are no more "prompts," then submit. Otherwise, repeat the process.

The following example shows an example of an outcome, intervention, and comparator taken from an article's abstract:

Key:

Outcomes
Interventions
Comparators

Results: Clinical characteristics including gestational age at enrollment, frequency of contractions, cervical Bishop's score, and white blood cell count on admission were similar in the 2 groups. Antibiotic therapy was well tolerated. No significant differences in the interval to delivery, birth weight, and neonatal outcomes were observed between the 2 groups. Women in the antibiotic group had a significantly lower incidence of postpartum infections compared with women in the placebo group.

Adding a single prompt:

To begin generating prompts, click the "add" button.

Detailing a prompt:

You will be required to fill in 5 slots: outcome, intervention, comparator, answer, and reasoning. The answer describes the effectiveness of the intervention with respect to an outcome; the comparator will be used as a baseline. The reasoning is a quote from the text to justifies your answer. Examples of the outcome, intervention, and comparator were given in a previous slide. Note that when entering the outcome, intervention, and comparator, you should try to not use the exact same words as the article.

Supporting Evidence and Answers:

As discussed, you will be required to find a quote that supports your response to the "answer" section. You must click the "Select Text" button to retrieve your evidence.

Selecting Supporting Evidence.

After you press the "Select Text" button, you will be brought to the original article. You will then have to highlight your evidence, and then release. Afterwards, you will be brought back to the original form.

Submitting the Prompt:

Our quote suggests that there is no difference between placebo and antibiotics with respect to frequency of contractions. After filling out each entry slot, press the submit button.

Finishing:

We have entered our prompt into the table. We will continue adding prompts until we can no longer find anymore. After adding in all possible prompts, we will press the submit button located in the lower-right hand corner.

Differentiating between 1 and multiple prompts:

The outcome, intervention, and comparator should each be 1 item (3 total). For example, suppose that the the article states the following:

"... iburprofen and tylenol proved to be significantly more effective for treating headaches and pain, in comparison to placebo..."

This span of text can be used to generate 4 prompts, as opposed to 1 single prompt. The table below details the different prompts that we would look for.

Intervention	Comparator	Outcome
Ibuprofen	Placebo	Headaches
Ibuprofen	Placebo	Pain
Tylenol	Placebo	Headaches
Tylenol	Placebo	Pain

"Exhaustive annotations" differs from the original task in 2 main ways:

Rather than labeling only 5 prompts, the goal of the task is to exhaustively find every prompt in the article.
Whenever we come across a span of text that contains an ICO frame (i.e. a prompt), we must mark it even if we have already marked this same Intervention, Outcome, and Comparator before.

NOTE: if we come across 2 different spans of text that gives rise to the same ICO frame, we must make sure that when entered into the system, that the ICO elements are exactly identical. To ensure this, make sure to use the copy button provided. But, also make sure to change the answer and reasoning.

Prompt generation guidelines