↓ Skip to main content

Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial

Overview of attention for article published in Trials, January 2018
Altmetric Badge

About this Attention Score

  • In the top 25% of all research outputs scored by Altmetric
  • High Attention Score compared to outputs of the same age (87th percentile)

Mentioned by

33 tweeters


15 Dimensions

Readers on

65 Mendeley
You are seeing a free-to-access but limited selection of the activity Altmetric has collected about this research output. Click here to find out more.
Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial
Published in
Trials, January 2018
DOI 10.1186/s13063-017-2405-6
Pubmed ID

Sara T. Brookes, Katy A. Chalmers, Kerry N. L. Avery, Karen Coulman, Jane M. Blazeby


Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery. A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey. In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]). In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias. The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.

Twitter Demographics

The data shown below were collected from the profiles of 33 tweeters who shared this research output. Click here to find out more about how the information was compiled.

Mendeley readers

The data shown below were compiled from readership statistics for 65 Mendeley readers of this research output. Click here to see the associated Mendeley record.

Geographical breakdown

Country Count As %
Unknown 65 100%

Demographic breakdown

Readers by professional status Count As %
Researcher 8 12%
Student > Master 7 11%
Student > Ph. D. Student 7 11%
Student > Bachelor 5 8%
Student > Doctoral Student 4 6%
Other 16 25%
Unknown 18 28%
Readers by discipline Count As %
Medicine and Dentistry 19 29%
Nursing and Health Professions 7 11%
Pharmacology, Toxicology and Pharmaceutical Science 4 6%
Psychology 3 5%
Social Sciences 3 5%
Other 9 14%
Unknown 20 31%

Attention Score in Context

This research output has an Altmetric Attention Score of 19. This is our high-level measure of the quality and quantity of online attention that it has received. This Attention Score, as well as the ranking and number of research outputs shown below, was calculated when the research output was last mentioned on 23 February 2018.
All research outputs
of 21,470,856 outputs
Outputs from Trials
of 5,420 outputs
Outputs of similar age
of 296,863 outputs
Outputs of similar age from Trials
of 1 outputs
Altmetric has tracked 21,470,856 research outputs across all sources so far. Compared to these this one has done particularly well and is in the 92nd percentile: it's in the top 10% of all research outputs ever tracked by Altmetric.
So far Altmetric has tracked 5,420 research outputs from this source. They typically receive more attention than average, with a mean Attention Score of 8.0. This one has done particularly well, scoring higher than 91% of its peers.
Older research outputs will score higher simply because they've had more time to accumulate mentions. To account for age we can compare this Altmetric Attention Score to the 296,863 tracked outputs that were published within six weeks on either side of this one in any source. This one has done well, scoring higher than 87% of its contemporaries.
We're also able to compare this research output to 1 others from the same source and published within six weeks on either side of this one. This one has scored higher than all of them