Study Design
A two-round modified virtual Delphi process was planned and conducted in Feb-April 2020. We followed the Recommendations for the Conducting and Reporting of Delphi Studies (CREDES). See Additional file 1: Appendix A.
Participant recruitment
As a first step, funding was obtained to initiate a new network of researchers to promote the development of new research proposals and identify parameters for potential implementation proposals (see Figure 1). Potentially interested researchers and implementation experts from Canada, the United States, United Kingdom and Australia were identified from personal contacts and from discussion at two conferences in 2019: Diabetes Canada and North American Primary Care Research Group. In addition, all researchers active in Canada who had published on primary care and/or lifestyle interventions in diabetes, pre-diabetes and metabolic syndrome over the past 10 years were identified through PubMed and CINAHL and contacted by email for possible interest in the network. Two teleconferences and a webinar on knowledge translation had been completed prior to initiating the Delphi process. Invitation emails were sent to 53 network members with a link to the survey and an attached document describing the purpose and process, as well as shared access (through Dropbox) to six key background papers (18-23). They were also encouraged to send an invitation to interested others.
Questionnaire Development
Evaluation Criteria and Research Priorities
A preliminary list of research priorities was generated through feedback from network members from the first two teleconferences (n=53). The list was highly diverse and included a mixture of specific and general topics, reflecting the many ways research can be described. The literature was therefore searched for possibly relevant implementation research frameworks. The work of the Child Health and Nutrition Research Initiative (CHNRI) was notable as they had used systematic and transparent methods to set child-focused health systems research priorities in low-income and middle-income countries (24). Multiple health research prioritization exercises had been undertaken over several years using CHNRI methods and two innovations were adapted to the present project. The first was an explicit list of priority-setting criteria that might apply to judgments of different stakeholders in a research priority-setting context. These 20 evaluation criteria were rated for relevance to our topic area (23). The second innovation was the use of the 4D framework for describing research focus (“description”, “delivery”, “development” and “discovery” research), which we adapted to describe 22 specific research areas for assessment (23). See Additional File 1: Appendix B for 20 options related to Evaluation Criteria and 22 research areas included under Research Priorities.
Indicators and Measures
Measures from two different consensus processes addressing obesity management were categorized, combined and adapted by one researcher (PB) to create an initial list. Some measures were identified by both groups, but many were not. The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) is a US-based effort to better understand and deliver tailored obesity treatments, based on behavioural, biological, environmental, and psycho-social domains. Part of this work involved the development of a Core List of Constructs and Measures, an initial list of high-priority constructs based on the current state of science (21). The OBEsity Diverse Interventions Sharing group (OBEDIS) developed a set of European guidelines for a minimal set of variables to include in future clinical trials on obesity, regardless of the specific endpoints (19). In addition, other possible measures were identified from previous implementation study guides (20, 22). The final seven categories were:
- Demographic/Environment/Context (15 indicators/measures)
- Diet and Physical Activity (8 indicators/measures)
- Formative and Process Measures (6 indicators/measures)
- Psycho-social Constructs (21 indicators/measures)
- Body Composition and Energy Estimation (6 indicators/measures)
- Biological/Clinical/History measures (33 indicators/measures)
- Social Environment (4 indicators/measures)
The finalized online Delphi survey was divided into three sections – Evaluation Criteria, Research Priorities, and Measures/Constructs/Outcomes. Each statement was rated on a 9-point scale, which ranged from not relevant/ not priority/not core (1-3), to uncertain/ equivocal (4-6), to highly relevant/priority/core (7-9). The survey avoided making any forced responses and provided forward-backward buttons. Additional suggestions and comments were requested at the end of each section. The survey was reviewed and pilot-tested by the research group. The round 1 and round 2 surveys were very similar, with 132 and 135 statements, respectively. In particular based on feedback from round 1, in round 2, two indicators were added to the Demographic/Environment/Context (i.e. marital status and material hardship (e.g. difficulty paying bills, rent etc.) and one indicator was added to Diet and Physical Activity (i.e. type / extent of food processing as indicator of diet quality). See the Additional File 1: Appendix B for the Final Delphi Questionnaire.
Data collection
Each round of the Delphi process was delivered as a self-administered survey (developed and delivered through Qualtrics®) and debriefing virtual meetings (delivered through ZOOM). Six relevant background papers were made available to all participants by Dropbox Sharing feature (18-23). For round 1, participants had about 2 weeks to complete the survey (Feb 25 - Mar 12,2020). Before round 2, they received an Excel summary listing each statement, the group results (median and 25-75%ile), and their own answers. Two virtual meetings were conducted to allow for differences in participant schedules. Both meetings had the same format of reviewing the overall results and most contentious statements, as well as a general discussion of the process. Meeting notes were reviewed by the research team and sent to participants after each meeting. For round 2, the survey was deployed with a 2-week turn-around (Mar 28 - Apr 12, 2020) and results were again sent back to participants. A final online conference meeting was held in April 2020 to review results and discuss next steps for the development of the network.
Analysis
Initial review of group results after round 1 indicated that all but one statement (Evaluation Criteria: Likelihood of generating patents/lucrative products) was at least partially supported, so scores were highly left skewed. Despite encouragement to use the full scoring range, scores were still highly skewed after round 2. To report results, we therefore used the most strict definition of strong agreement, as described by the original developers (15), which required that all three indicators (median, 25th and 75th percentile) fell within the 1-3 or 7-9 range. A second set of statements had the median and 75%ile fall within 7-9 range, but the 25%ile indicated more variable scoring. These statements achieved weaker agreement. The third group of uncertain statements had lower medians and diverse ranges.