The reporting of this protocol follows the criteria of the standard protocol items: recommendations for interventional trials (SPIRIT) (29) and the UK Medical Research Council (MRC) Framework for the development and evaluation of complex interventions (30). For the completed SPIRIT checklist, see additional file 1.
The trial uses a superiority randomised control group design, comparing two groups of equal sizes. In addition, the trial includes a formative evaluation seeking to fully understand the involved mediating mechanisms as well as the barriers towards the implementation process.
Setting and participants
The intervention is made available on-site at providers of public available health information (e.g. health insurances, self-help associations or foundations, health portals, hospitals, rehabilitation clinics, nursing facilities and physicians) in the German-speaking area, mainly in Germany and Austria. Providers are defined as institutions or working groups rather than individuals and may therefore comprise several individuals (e.g. up to ten).
Eligibility and recruitment
Providers will be eligible if they are responsible for the production and publishing of health information. The use of external services (e.g. counselling by experts, graphic design, external performing of pilot tests or external literature searches) is not a reason for exclusion, as long as the provider is designated as the responsible publisher/editor. Providers should have regularly and currently published information. Therefore, they are eligible if they have published any information in the last 18 months. In addition, at least a single information produced in the last three years has to fulfil the following criteria: The information…
has to inform a health related decision,
has to address patients or medical laypersons,
has to discuss different options regarding one specified health problem,
does not inform about a single option, procedure or healthcare system and
it does not give general advice on health and wellbeing.
It is essential that the participating providers assent to comply with the training programme and the production of one item of health information. Above all, the providers will be notified about the purpose and schedule of the intervention and their obligation to produce the new health information within ten months.
Members of the working groups on the guideline evidence-based health information or the GPGI are not eligible. The same applies to providers producing exclusively (drug) fact(s) boxes or offering counselling services such as medical online consulting websites, health related blogs, forums or communities or (online) encyclopaedias.
The providers themselves will choose the participants for the training programme. No other criteria are defined for these participants. If possible, all the individuals in the working group, who are closely involved in the development of the health information, should participate.
Providers will be identified by means of internet searches. We assume that most providers have a web appearance, even if they issue printed material. We will search via Google and MetaGer with different terms for online health information and webpages that refer to information (print or pdf). Data of the providers will be collected via these information and webpages (e.g. contact details). In addition, data of well-known providers will be listed. The identified providers will be assessed for their eligibility (membership in the working groups guideline evidence-based health information and/or GPGI, the amount and kind of information published by them in the last three years). Contact details and the persons in charge will be identified online or by telephone. We will continue the search until 100 eligible providers are identified. These will receive a cover letter informing them briefly about the study and the guideline evidence-based health information as well as an invitation to take part in this project (see study flow, figure 1). Providers who do not reply within two weeks will be contacted personally by telephone with an offer of further information. If providers are interested, a telephone or online meeting will be scheduled to assess further eligibility criteria and to give an overview of the course of the study.
Afterwards, a personal meeting will be set up with eligible providers who intend to participate. The meeting is thought to give further information, to obtain informed consent, to assess baseline data and to perform the allocation. Further information about the study will be presented with special reference to the relevance of EBHI, the intervention and the primary outcome. The providers have to know about the requirements and advantages associated with their participation. Requirements are for example: releasing employees from work for the training programme, providing time and financial resources to develop a single health information in the study period, possibly with more effort in the intervention group in order to realise the guideline recommendations. In return, a final symposium with the presentation of the best information will be scheduled after completion of the project. With their consent, the providers’ logos will be published on the project webpage after completing the study. In addition, the health information produced in the study may be published on the webpage along with the results of the quality rating. After receiving notification about requirements and incentives, the providers must assent to comply with the training programme and to produce one item of health information informing a health-related decision within ten months. In addition, written informed consent will be obtained from the individual participants and baseline data will be assessed by the researchers. Recruitment and randomisation will be performed consecutively.
Randomisation and blinding
Concealed allocation of participating provider groups to either the intervention or the control group will be determined by randomisation using a computer-generated list with randomly permuted blocks of length 2, 4, 6 or 8. An independent external person will prepare sealed opaque envelopes. After baseline assessment of the respective provider group, researchers will open the sealed opaque envelope and reveal the centre’s allocation on-site.
Due to the nature of the intervention, blinding of the participating providers and researchers conducting the training programme is not possible. However, assessment of the primary endpoint and all analyses will be blinded towards group allocation.
Intervention group (IG)
Providers in the intervention group receive the guideline evidence-based health information and an adjunctive training, which is intended to facilitate the appropriate use of the guideline.
Guideline evidence-based health information
The guideline defines the quality criteria of EBHI. It comprises general and ethical requirements regarding the development process, target group orientation and content of EBHI as well as 21 evidence-based recommendations assigned to the topics: presentation of frequencies, application of graphics, pictures and drawings, narratives, value clarification tools, formats and involvement of the target group. Table 1 gives an overview of the guideline and its recommendations.
Workshop for persons in charge for provided health information (module 0)
Even though the actual target group of the training programme are people directly involved in developing and producing health information, our intervention also needs to address the providers on the level of responsible decision-makers and leaders in the organisation. Attitudes of these persons can be both facilitators and/or barriers to the implementation of the EBHI criteria. Therefore, they will be invited to a workshop with the purpose to discuss and reflect on the benefits of using the EBHI criteria. In particular, the systemic structures required for developing EBHI will be addressed. If possible, the workshop (up to two hours) should take place on-site directly after randomisation into the intervention group. If this is not possible, a second personal or online meeting will be scheduled.
Training programme (module 1 and 2)
The free training programme is based on a problem-based instructional design that is learner-centred and enables learners to link theory with practice and to apply knowledge and skills to develop EBHI to a defined health problem for special target groups (31). Case-based learning links theory to practice through the application of knowledge about the cases and by using inquiry-based learning methods (32). We set up a case about smoking cessation. This was chosen since it has relevance for several health professions and smoking is still a widely discussed topic. Moreover, (passive) smoking concerns everyone in daily life. Good health information is needed because most people in Germany try to quit smoking on their own, which is the less promising option (33). People need to be informed about evidence-based methods for smoking cessation, such as counselling, medications (e.g. antidepressants) and nicotine replacement therapy. Therefore, it is also a relevant topic for providers of health information.
The training programme comprises module 1 "ebm training" and module 2 "application of the guideline". It is designed in a blended learning format (34) where face-to-face and web-based learning activities alternate (see figure 2).
Figure 2: Modular training structure
The ebm training module is divided into five sub-modules: introduction, treatment studies, evidence syntheses and guidelines, diagnostic studies and systematic literature search. The teaching goals are broadly defined (table 2).
The first module (two days of face-to-face training followed by one day of web-based training) aims to impart competences in searching for, selecting, critically appraising and extracting relevant literature according to the principles of ebm.
The second module is designed as a converted classroom scenario (one day of web-based training followed by one day of face-to-face training) (35). In preparation for the second module, participants become acquainted with the guideline recommendations for the development of EBHI. Guideline recommendations are presented in an audio-visual form with illustrating examples. Participants can then intensify and improve their understanding of the criteria for EBHI by critically appraising an existing health information.
During the second face-to-face training, online tasks are discussed and further group exercises are done in order to improve the understanding of the development and pilot testing of EBHI.
The participants are invited to reflect their current practice concerning the development of health information, taking into consideration the methods of evidence-based medicine and the guideline recommendations for EBHI.
For the web-based learning scenario, a learning management system (ILIAS) is used to provide web-based content. Features of the learning management system are slideshows and text resources combined with online tasks and (video) tutorials. Further information on the course content is provided. Participants are encouraged to upload their results and receive feedback during the face-to-face training. The group size for training should not exceed 20 participants. The training programme was pilot tested and revised according to the pilot study results. The study protocol is available online (28). The publication of the results is being prepared.
Control group (CG)
Providers in the control group do not get access to the training programme or materials exceeding the guideline itself. They are offered the possibility of participating in the training programme after data collection is completed.
The baseline characteristics of the health information providers will be assessed on both the institutional and individual level. On the institutional level, the size of the organisation (number of employees in total and number of employees developing health information), number of published health information in the last three years, main target group(s) and topic(s) of health information, funding of health information and the management of conflict of interests will be assessed. The responsible contact has to fill in a standardised questionnaire. On the level of individual participants, their sex, age, education status and English language skills will be assessed, along with their qualifications for the development of health information, duration of their current position, and previous experiences and attitudes regarding ebm and EBHI. These participants also have to fill in a standardised questionnaire. In addition, a declaration about conflict of interests is required.
Quality of health information
Each provider’s most recent health information, which meets the criteria of materials giving information towards a health-related decision, will be rated regarding quality using the MAPPIN Health Information Quality (MAPP’INFO) instrument.
Critical health literacy
Critical health literacy will be assessed using the Critical Health Competence Test (CHC test) (36). The test is based on a 4 by 4 facets design. The first facet consists of four different and relevant content areas of healthcare. The second facet is built up with four subareas of competence, representing the underpinning theoretical structure of the critical health literacy construct. These categories are: A. Understanding medical concepts; B. Skills of searching literature; C. Basic statistics; and D. Design of experiments and sampling. Reliability of the test for the Rasch model (ANOVA) was 0.91 and for the single scenarios 0.71 (scenario 1); 0.78 (scenario 2); 0.75 (scenario 3) and 0.80 (scenario 4).
The CHC test comprises four different scenarios which allows testing at different time points using different scenarios. The test will be carried out at baseline and as part of the process evaluation after the training programme in IG and CG. Critical health literacy will be aggregated on group level using the maximum of individual values of the CHC test.
Outcomes and data collection
The primary outcome measure is the quality of health information produced by the participating providers. Quality is operationalised as the extent of adherence to the guideline’s recommendations. The quality of this information will be rated using the MAPP’INFO instrument.
By that time, MAPP’INFO will have passed all the validation steps and the study protocol for the validation study will be in the process of publication. Information on reliability, internal consistency, construct validity, criterion validity and divergent validity will be available. The instrument has been developed to score the extent of compliance with the guideline criteria, observable based on the pure information or the directly linked sources; this means the transparency may be established by a report or documentation of the methods. MAPP’INFO is supposed to be the best possible estimate for the overall quality of health information. The instrument comprises 21 to 23 test items (according to the nature of the addressed decision) in four categories. Adherence to the criteria checked by MAPP’INFO is essential but not sufficient to provide material enhancing informed decision-making. Non-adherence with one of them may already challenge the entire information process. Therefore, all the criteria should be met. Nevertheless, the instrument provides percentage scores for each category and for each criterion. In this study, scores based on MAPP’INFO will be expressed as a percentage of the fully met criteria. Scores for information quality will be generated. MAPP’INFO coding works without previous training. The information will be coded by two independent and blinded raters.
Due to the pronounced importance of single aspects of the quality concept, it is planned to use a selection of single criteria as secondary outcomes. For this purpose, those single criteria are chosen following strong recommendations based either on empiric effects or on ethical considerations:
CONTENT 5 / PRESENTATION 2: „Information about possible benefits is provided in an appropriate manner.”
CONTENT 6 / PRESENTATION 3: “Information about possible harms is provided in an appropriate manner.”
CONTENT 7 / PRESENTATION 4: „In the case of diagnostic problems: Information about the reliability and safety of the test are provided in an appropriate manner.”
PRESENTATION 5: „The health information uses a neutral wording / language throughout.“
DEVELOPMENT: “The health information target group has been involved in the development process using appropriate methods.”
All these secondary outcomes are binary.
The sample size was calculated for a t-test comparing IG and CG regarding guideline compliance to EBHI as measured by MAPP’INFO. It is assumed that MAPP’INFO is normally distributed with the same standard deviation σ of 10% in both groups. Based on a sample size of 26 provider groups (13 in each study group) a quality improvement of 15% from 20% to 35% adherence can be detected with 90% power using a two-sided significance level of 5% after excluding at most 10% dropouts.
Sample size calculation is based on the assumption of σ = 10% from pretest data for MAPP’INFO, and the clinically relevant difference of 15% was chosen taking into account some pragmatic considerations starting from Cohen’s d corresponding to a strong effect. The power was calculated using the statistical software package SAS version 9.4 (PROC POWER).
During recruitment, both groups will be informed about the requirement to produce a health information within ten months, according to the defined criteria.
In one personal meeting, information on the requirements for study participation will be given and the providers must agree to comply with these requirements. Written informed consent will be obtained from individual participants. Afterwards, baseline data will be assessed (t-1) and randomisation will be performed (t0) (see SPIRIT-Figure, figure 3). In the IG, the workshop for the persons responsible (module 0) will be conducted and a timely convenient appointment for the training programme (module 1 and 2) will be made (t1). The training should take place on-site at the provider’s institution. If possible in terms of time and place, two or more groups may attend the training together. When the training is completed, they can begin with their production of the health information. The providers in the CG can start directly after the randomisation.
The information, produced within the study period, has to inform a decision in healthcare (e.g. therapy, diagnostic procedure, screening, prevention or rehabilitation). It addresses patients or medical laypersons and discusses at least two options regarding a specified health problem. Material regarding procedures (e.g. how to perform peritoneal dialysis at home) or the healthcare system, resources giving general advice for health and wellbeing as well as (drug) fact(s) boxes without additional text are not appropriate. Despite these requirements, the providers are free to choose the topic and target group of the information. Therefore, they can further develop already planned information. The providers may use external services (e.g. counselling by experts, graphic design, external performing of pilot tests or external literature searches), but they are responsible for the methods applied and for the content and design of the final information.
During the time of production, we will contact the providers every two months to keep ourselves informed about the progress and to ask about any unexpected difficulties. The study team will be accessible and will answer upcoming questions regarding requirements and the course of the study. The IG will have access to the training material on the online learning platform, including resources such as updated search strategies. We realise that the lack of access to relevant references is an important barrier in the development process and therefore the providers in the IG and CG will equally receive support regarding the acquisition of research articles. No further support will be provided.
Completed health information can be provided to the study team at any time. After ten months, the development process should be concluded and the providers will be asked for the material. A provider who cannot deliver after ten months may get a 2-month extension of the deadline. He should give a justification for the delay and a reasonable timeline for the following two months.
After twelve months (t2), even unfinished material must be submitted. The information will be rated with regard to the primary outcome. Single criteria (or else the entire information) that do not meet the MAPP’INFO requirements, are coded as zero. Providers of both groups (IG and CG) who completed their information will be asked to give further information on the development process (t3). Interviews will be conducted in the IG to explore barriers and facilitators of implementation (cf. process evaluation). The providers in the CG will be invited to make appointments for the training programme (t3). After the provided information has been rated, providers may receive an individual feedback on their material. This is optional, but this way especially providers in the CG may benefit from such feedback.
Data management, protection and analyses
Data survey and data processing are used for the scientific purpose of the study only and are appropriate and indispensable for that purpose. Figure 4 shows data flow. Risk assessment according to §35 General Data Protection Regulation (DS-GVO) revealed a very low risk for the planned study. Special categories containing personal data will not be collected. In addition, the assessment using the Black List / White List of the competent supervisory authority of Saxony-Anhalt did not reveal the necessity for a privacy impact assessment. The remaining very low risk can be further minimised by the planned procedures (e.g. data storage using password-protected servers at the computer centre, no disclosure of data to third parties).
Withdrawal from the study is possible at any time without giving reasons and without disadvantages for the participants. In the case of withdrawal, data can be deleted only before anonymization (after completion of the analysis). Privacy and data protection will be taken into account according to the German Data Protection Act (37). A code number will be assigned to every participant. All data will be marked with this code number and thus pseudonymized. Identification of single participants is only possible with the coding key, which is stored on password-protected servers at the Martin Luther University Halle-Wittenberg. Only the project team has access to the coding key which is protected securely from third parties. Once data analysis has finished, the coding key will be deleted. From then on, data can no longer be deleted in the case of study withdrawal.
All data provided by the study participants are treated as absolutely confidential. Researchers adhere to the data protection rules. All data will be stored on password-protected servers and there will be no disclosure to third parties. Publication and storage of data will only be performed in an anonymised form. Audio recordings will be deleted after transcription. Data will be stored at the Martin Luther University Halle-Wittenberg for 10 years.
The primary analyses will be performed from the intention-to-treat perspective. Provider’s characteristics (institutional and individual level) will be analysed descriptively. Baseline variables of individuals will be described on individual level by means, standard deviations, percentiles, and frequency tables depending on their distributions. The IG and CG will be described separately.
The primary outcome will be analysed on provider level assuming approximately normal distribution by comparing means between the IG and CG using the t-test. The distribution of MAPP’INFO will be described graphically. In case of deviations from the normal distributions, transformations or alternative tests will be discussed as secondary sensitivity analyses. Frequencies of missing values will be described in detail. In the primary analysis, multiple imputations will be performed if enough data are available. Unfinished information will not be regarded as missing values. For the binary secondary outcomes, relative frequencies will be described in both provider groups and compared using Fisher’s exact test. A secondary analysis per protocol will be performed as a complete case analysis (without multiple imputations for the primary outcome).
A data monitoring committee will not be necessary, as the trial does not involve a high-risk intervention and participants do not belong to a vulnerable population. We do not expect adverse events or other unintended effects of the intervention. During the entire study period, participants will have the possibility to contact the study centre.
To support future implementation of the guideline evidence-based health information and the training programme, a comprehensive analysis of the underlying processes of this complex intervention is indispensable (38). Barriers and facilitators of implementation should be assessed. Additionally, high quality of the training programme should be ensured. We will focus on parameters such as recruitment, reasons for participation or non-participation, intervention fidelity, structure and process-related factors, attitudes toward the intervention, response of individuals and organisations, and unintended consequences (38, 39). Mixed methods will be applied (40) according to the MRC framework for process evaluation of complex interventions (38).
Structured documentation will be used to assess data of recruitment and intervention fidelity (e.g. recruitment process; numbers of institutions invited, responses and participants in each training session; location, time and duration of the training; completeness of modules and online tasks and reasons for deviations; and unexpected difficulties). Feasibility and acceptance of the training programme will be assessed at the end of the training sessions using structured feedback and all statements will be documented. Critical health literacy will be assessed using the CHC test (36) after the IG and CG have completed the training programme.
To map the development process and the methodological quality of the produced health information, all the providers (IG and CG) will be asked to provide the MAPP’INFO self-declaration after completing the health information. The self-declaration comprises free-text questions on the selection of the reported options and outcome measurements, search strategies, methods of data extraction and critical appraisal, reasons for the use of pictures, graphics, fact boxes, value clarification tools, animations and interactive tools. Experts will review the results and if there are outstanding issues, interviews will be conducted. In addition, to identify influencing factors, resources and support used in the development process as well as any training attended during the study period will be assessed.
In the IG (theoretical sampling), semi-structured interviews will be conducted with single participants as well as with persons in charge in order to explore the implementation barriers and facilitators. Relevant factors for acceptance and usability of the educational contents and materials may be further assessed. In particular, the use of educational contents in daily routines and barriers against putting the guideline recommendations into practice will be explored. If the participant consents, interviews will be audio recorded and then transcribed.
Data will be collected at various points of time. An iterative process of collecting and analysing qualitative data will allow exploring unexpected aspects in further interviews (38). Data will be analysed in accordance with the method of collection (40). Descriptive statistics will be used for quantitative data. For qualitative data, a qualitative content analysis according to Mayring will be performed (40).