The development of the online intervention “Understanding ADHD in primary care” was conducted over eight months and included multiple steps, with input from various stakeholders at each point. A brief overview of the development process is shown in Figure 1.
Initial development
Selection of topics
In order to guide the development of the content of the intervention, specific ADHD topics were selected that targeted previously identified unmet needs of GPs when managing ADHD.
The selection of these topics was guided by the findings of three previously conducted studies in this area (7,18,19). These studies highlighted specific barriers in GPs understanding of ADHD, a summary of the main barriers are presented in Table 1.
Table 1-Summary of themes relating to awareness and understanding of ADHD in primary care
Systematic review (French et al, 2018)
|
Pilot study
(French, unpublished thesis chapter)
|
Semi-structured interviews (French et al, 2020)
|
Need for education
|
Negative connotation of ADHD
|
Lack of identification in primary care
|
Misconceptions and stigmas
|
Parenting
|
Lack of clear diagnostic pathway and services
|
Constraints with recognition, management and treatment
|
Social background
|
GPs’ knowledge of ADHD and misconceptions
|
Multidisciplinary approach
|
Lack of experience/knowledge
|
Difficult communication between multiple stakeholders
|
|
Diagnosis / consultation procedure
|
Impact of diagnosis and the risks linked to no diagnosis
|
The collective findings from these three studies can be broadly categorised into two main concepts – (1) issues around knowledge and (2) issues around the complexity of the diagnostic process. As such, these two concepts were the focus of the intervention. Based on this, the team developed videos with the aim of capturing the lived experiences of patients around these two themes.
Videos
Two sets of videos were used for this intervention. One set of videos filmed five adults (male and female) talking about their lived experiences and the daily impact of ADHD. The second set of videos filmed eight children with ADHD and their parents talking about their experiences. The video participants were recruited from support or charity groups and signed consent (or parental consent in the case of the children) for the use of their videos. The videos were co-designed by a consultant psychologist and the research team.
Two lead researchers (BF and DD) reviewed the footage and condensed the content down to two 5-minute video clips which contained the most important information pivotal to the themes shown in Table 1.
Following the video development, an initial outline of the intervention emerged which centred on the video discussions pertinent to the two main barriers (lack of knowledge and complexity of diagnosis).
As such, the proposed content for the online intervention focussed on:
- Understanding the different roles held by different stakeholders
- Understanding the role of the GP
- Understanding the diagnosis pathway
- Improving general knowledge of ADHD
- Dispelling common myths on ADHD
- Socioeconomic status (SES), parenting and the child’s behaviour in the consultation
- Understanding and challenging common negative conations of ADHD
- Benefits of receiving an assessment and/or diagnosis
- Risks of untreated ADHD
Development process
The online intervention was developed in partnership with the HELM (Health E-Learning and Media) team from the University of Nottingham, UK. The HELM team specialises in media-based educational materials and intervention in health and were chosen for the development of the online resource due to their expertise in the area. The HELM development process has specific stages to ensure the most optimal final product and learning outcomes, which has established efficacy.
Stage one - a workshop is set up with service users or the population of interest (in our case GPs) in order to develop a targeted resource that is appealing and accessible to its users (20). This workshop creates a set of storyboards that informs the content of the resource. Specifications for the resource are then developed (by BF in this project) including but not limited to written content, exercises, interactive activities and assessment.
Stage two - A peer review process follows whereby the proposed content is reviewed by an expert on the topic that has not been involved in the development. The creation phase then starts and is solely conducted by the HELM team. Upon completion of the online resource, another review process is conducted where a reviewer and the team assessed the final product before dissemination.
The development process that we followed, paralleled this process while also incorporating an additional third Stage – a usability study to assess the intervention. Here, we outline each of these three stages.
Stage 1: Development workshops
To further develop the intervention and in line with the HELM process, workshops were conducted with 15 GPs and two other healthcare professionals.
In a variation from the HELM process, we held two separate workshops to enable ideas from the first workshop to be presented to the second workshop members for further development and validation of the original concepts.
Workshop members
The first workshop included the lead researcher (BF) and 11 GPs and the second workshop four GPs and two secondary care professionals specialising in ADHD assessment - one from child services and one from adult services. The two secondary care specialists were included to gain more specific input to the content of the resource.
The workshops lasted three hours and participants were compensated for their time. After a brief presentation of the research project and the HELM team, the participants were split into three groups and asked to work on storyboards for the resource. They were specifically asked to think about the format and appearance of the resource rather than specific content. Examples of online resources were presented in order to facilitate ideas. The three groups then presented their storyboards to the whole group.
The second workshop was run using a similar format however, a review element was added due to the smaller numbers. In addition to the storyboards, the participants in this workshop were presented with a summary of the suggestions from the first workshop and asked to review these suggestions. The participants were split into two groups, each tasked with creating a storyboard. These storyboards reinforced specific content suggested in the first workshop but also brought out some new ideas.
Table 2 presents the main ideas that emerged from the workshops. Table 2- main suggestions presented at the workshops
Suggestions from the workshops
|
Making two short online resources, one specific to ADHD (Symptoms, epidemiology…) and one specific to the GPs role in diagnosis and treatment
|
Including information on the benefits of diagnosis, what can happen without treatment (information on prison statistics, substance abuse, suicidality...)
|
Shorter videos of patients focusing on symptoms
|
Adding expert videos on symptomatology and secondary care pathways. What happens after a referral
|
Separating clearly child and adult pathways, having a child specific module and an adult specific module
|
Adding an assessment at the end in the form of a multiple choice questionnaire
|
Including information on comorbidities in the form of a diagram
|
Including information on ADHD at different ages
|
Adding access to resources for management and for patients’ information (Parenting websites, ADHD support groups, charities…)
|
Comprehensive information on treatments
|
What is the role of the GP?
|
Drag and drop activities on myth versus facts
|
Including an example of a consultation
|
Information on local pathways
|
Stage Two: Content development and review
Development group
The development group involved seven stakeholders with specific ADHD expertise. It included the lead research team and additional group members. The research team included: the lead researcher (BF) and two supervisors with ADHD-related expertise (DD, KS). The group also included two healthcare professionals working with adults and children with ADHD, one GP who was diagnosed with ADHD and one GP who carried out research on ADHD as part of their PhD. The role of this group was to act as a form of steering committee, overseeing all aspects from the developing intervention and making decisions on final content. Some of the group members facilitated and attended the workshops (BF, AG, JK) while others’ roles were more focused on reviewing the content.
Two members of the development group (BF, DD) synthesised the information from the workshop and developed a draft intervention. The group were mindful to include different activities within the intervention to keep the content entertaining and engaging. Examples of activities include drag and drop games, questionnaires on myths about ADHD and animated pictures of brain correlates. The draft was reviewed by the rest of the development group and sent to HELM to create the intervention. A summary of the changes made based on the workshop is provided below.
Integration of recommendations
Most recommendations from the workshop were integrated into the online resource. However, some recommendations had to be discarded. Table 3 presents these suggestions and the research team’s rationale for or against implementation.
Table 3- Implementation of the main suggestions presented at the workshops
Suggestions from the workshops
|
Implementation
|
Making two short online resources, one specific to ADHD (Symptoms, epidemiology…) and one specific to the GPs role in diagnosis and treatment
|
Instead of one module, we separated the content into two modules: “Understanding ADHD” and “The role of the GP in the diagnosis and treatment process”
|
Including information on the benefits of diagnosis, what can happen without treatment (information on prison statistics, substance abuse, suicidality...)
|
A page on the risks of undiagnosed and untreated ADHD was added with research statistic accentuating the importance of early intervention
|
Shorter videos of patients focusing on symptoms
|
The videos were changed to make them symptom specific. The patients’ testimonies were restructured and six shorter videos were developed focusing on features of hyperactivity, inattention and impulsivity in adults and in children
|
Adding expert videos on symptomology and secondary care pathways. What happens after a referral
|
Expert videos were added. Four ADHD experts were filmed to give a specialist opinion on specific topics. A GP with a diagnosis of ADHD, related her lived experience of being both a GP and a patient with ADHD. A lead researcher on ADHD (DD), discussed strategies to help support ADHD patients during the diagnosis process and non-pharmacological approaches. An advanced nurse practitioner and a consultant psychologist, explained the secondary care process following referral
|
Adding an assessment at the end in the form of a multiple choice questionnaire
|
A quick assessment on ADHD knowledge was added at the beginning and at the end of the modules to assess any changes in participants’ knowledge
|
Including information on comorbidities in the form of a diagram
|
The diagram idea was added to the page on comorbidities in order to improve understanding of overlapping conditions.
|
Including information of ADHD at different ages
|
An infographic was created to show the development of ADHD symptoms through the ages.
|
Adding access to resources for management and for patients’ information (Parenting websites, ADHD support groups, charities…)
|
A toolkit was created at the end of the module where many resources on management, support groups, screening etc. can be found
|
Comprehensive information on treatments
|
The pages on treatment were expanded to include pharmacological and non-pharmacological treatments with details on the specific types of medications
|
What is the role of the GP?
|
The first page of the second module included a concise summary of what the role of the GP is exactly, and what it isn’t.
|
Drag and drop activities on myth versus facts
|
An interactive drag and drop activity was created to address typical misconceptions about ADHD
|
|
Suggestions that could not be implemented
|
Including an example of a consultation
|
GPs suggested including a video of a mock consultation. While it would have been very interesting to implement this idea, adding an extra 10 minutes of videos to encompass a whole consultation felt too lengthy. Furthermore, identifying ADHD in patients is very different depending on many factors such as the type of ADHD, the age or the gender and it was felt that we couldn’t represent it all accurately in one mock consultation.
|
Separating clearly child and adult pathways, having a child specific module and an adult specific module
|
This suggestion was addressed to an extent by clearly specifying the differences in child and adult pathways when relevant. However, it seemed too repetitive to create separate modules for each as a lot of the information overlapped.
|
Information on local pathways
|
Information on local services and pathways was unanimously the one piece of information GPs wanted to receive the most. However, it is impossible to know the different pathways in each British locality as firstly, there are so many and secondly, services are constantly changing in response to commissioning decisions. However, a statement was added to explain that local services information needed to be sought by the GPs in order to offer best access to care.
|
Reviews
A thorough review process was implemented throughout the development process. The modified online intervention (see Table 3) that developed from the workshops and development group review then underwent three further reviews:
- A GP first reviewed the content to ensure it was appropriately targeted to GPs. The content specifications were also sent to a reviewer (KS) who had not taken part in the content development. Additionally, it was proof read by a professional proof reader.
- Following the online development, the final resource was produced. The lead researcher (BF) reviewed the content to ensure the resource was developed according to the original specifications. The resource was then sent to an external reviewer to assess time, accessibility, content and format.
- Finally, the resource was reviewed by the Royal College of General Practitioners (RCGP) in order to receive accreditation. Upon seeing the final version, a few details had to be addressed in order for the accreditation to be granted. This feedback was minor, easily addressed and accreditation was received in July 2019.
The developed intervention: “Understanding of ADHD in primary care”
The developed intervention was called “understanding of ADHD in primary care” and was delivered on an open source learning management system from a University of Nottingham server. The complete online intervention consists of two 25-minute modules undertaken sequentially. The two modules follow the same format of having text on the left hand side of the screen and interactive activities on the right. The activities varied and included patient testimonies, drag and drop games, videos and pictures.
Module 1: “Understanding Attention Deficit Hyperactivity Disorder” introduces the many aspects of ADHD. After a brief description of ADHD epidemiology and neuroscience, the core three symptoms are discussed with real life settings examples. Other symptoms, common misconceptions and key impacts on children and adults are also discussed. Finally, comorbidities and risks associated with ADHD are presented.
Module 2: “The role of General Practitioners in ADHD diagnosis and management” introduces in more detail the GP’s role in the ADHD diagnosis and treatment pathways. Clarifying the gatekeeping role held by GPs and the pathway to care in the UK, this module also expends on identification of ADHD, treatment options and the effect of gaining better ADHD knowledge on practice. Finally, an “ADHD toolkit” included with various downloadable forms such as screening tools, strategies or useful websites.
This resource can be found on: www.adhdinfo.org.uk
Stage Three – usability study
To determine the usability of the intervention, a pilot study was conducted with 10 GPs. The aims of the pilot study were to assess the intervention usability, to ensure that the intervention ran in a timely manner and that no technical errors occurred.
Participants
GPs who had registered consent to contact after taking part in the interviews and in the development workshops were contacted by a member of the research team to review the usability of the online resource. Fifteen GPs were approached and ten GPs (4 females) completed the study (66% response rate).
Measures
Three online measures were completed.
Usability questionnaire: A usability questionnaire was developed, containing 29 questions assessing key usability criteria such as learnability, efficiency and memorability. Question type varied from forced choice questions (“I will use this tool in the future”-agree, disagree, unsure) and free text questions (“Were any parts of the tool not helpful?”). This questionnaire was completed after engagement with the online intervention.
Knowledge of Attention Deficit Disorders Scale (KADDS) questionnaire: This 39-item self-report scale was originally developed to measure understanding and knowledge of ADHD in teachers (3). However, the itemised questions are not solely relevant to teachers but also pertinent to general knowledge and understanding of ADHD in GP’s. Twenty-seven questions from this questionnaire were used in this evaluation. As the aim of the questionnaire was to assess a change in understanding of ADHD after taking part in the online intervention, twelve questions from the original questionnaire were excluded as they were not answered by the online resource. This questionnaire was completed pre intervention (time 1) and immediately after completing the intervention (time 2).
GPs’ awareness of ADHD questionnaire: This questionnaire assesses GPs’ attitude and experience of ADHD (22). Some questions were excluded as they were not relevant to the UK healthcare system or they were similar to the ones asked by the KADDS. Thirteen questions from this questionnaire were added as they were specifically tailored to GPs’ experiences. This questionnaire was completed before and after engaging in the online intervention.
Procedure
GPs who had given consent to be contacted after taking part in previous studies were emailed details about the study and sent links to an online information sheet and consent form to complete in order to take part. Upon receiving consent, GPs who agreed to take part were then sent a link to the intervention with embedded outcome measures. While some participants had taken part in the initial development workshops, none of them were familiar with the final online intervention. GPs were advised to set aside 90 minutes to complete the study in one go. Although it was not encouraged, participants were able to stop the study at any point and come back to it at a later point. Participants completed time 1 measures (baseline) before commencing the intervention. Time 2 measures were completed immediately after finishing the intervention. Although this study did not seek to obtain effectiveness data of the learning resource, the ADHD knowledge questionnaires assessing the effectiveness of the learning resource were retained to assess the flow and length of the study design.
Upon completing all questions, participants were given an inconvenience allowance and a Continuing Professional Development (CPD) certificate from the RCGP.
Descriptive analyses were used to summarise the findings from this study.
Results
Usability and acceptability
Ten GPs took part in the usability study. Nine were aged between 25-35 years and one between 36-45 years. Years of practice since qualifying as a GP ranged from 10 months to 11 years (mean: 6y 7m).
The completion time (including the questionnaires and intervention) ranged from 45-72 minutes although it was not possible to assess the response time of two participants as they did not complete the intervention in one seating.
Results from the usability questionnaire are presented below. Participants were asked to rate some questions on a scale of 1 to 10 (Table 4) and others if they agreed or disagreed with specific statements. Free text questions on their overall interaction with the resource were also included.
Table 4 - Usability and acceptability evaluation on a scale of 1-10 (1: not at all and 10: a lot). Table values represent the number of responses for each scale point.
Scores
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
Total mean
|
How confident are you in your knowledge of ADHD
Pre intervention
|
|
|
1
|
|
5
|
1
|
1
|
2
|
|
|
6.2
|
Post intervention
|
|
|
|
|
|
1
|
2
|
4
|
3
|
|
7.9
|
How useful did you find the information in this programme?
|
|
|
|
|
|
|
|
3
|
2
|
5
|
9.2
|
Did you like using the tool?
|
|
|
|
|
|
|
|
6
|
3
|
1
|
8.5
|
Do you feel the tool impacted your knowledge on ADHD?
|
|
|
|
|
|
|
2
|
2
|
5
|
1
|
8.5
|
How likely is this information going to inform your practise?
|
|
|
|
|
|
|
|
4
|
5
|
1
|
8.7
|
Do you believe the content was relevant to your practice?
|
|
|
|
|
|
|
|
3
|
2
|
5
|
9.2
|
The participants reported a high degree of satisfaction with the content and layout of the online intervention. All participants were able to navigate through the resource easily, and only one suggestion was made to improve navigation. The wording and presentation of the content was well received, participants reported the content to be clear, interactive and easy to follow. All participants also felt that the resource was useful, increased their knowledge and was relevant to their practice and confirmed they would recommend the resource. While a few suggestions for improvement were made, the feedback was overall strongly positive.
Positive feedback:
All participants agreed that they will definitely recommend the resource and most felt that no parts of the resource were unhelpful or that anything was missing from the content. The additional comment section contained mainly positive comments where participants principally highlighted that they liked the interactivity and the structure of the resource. The participants especially liked the videos used to reinforce their learning.
“Great resource, videos help to give a true account” (P4)
“Good mix of bullet point text and short videos. Interplay between the two helped reinforce points” (P10)
While most agreed that the resource was the right length, a couple of participants that suggested the resource might be too long but acknowledged that despite feeling that it might be a bit lengthy, they wouldn’t know which part to cut out.
“It was (too long), hard to decide what was the least useful. All useful stuff” (P5)
Suggestions for improvement:
Only a few suggestions for improvement were made relating to the length and format of the intervention, the content, and navigation.
Length and format: While participants were mostly satisfied with the length of the intervention, one participant highlighted that it was important to advise participants of how long it will take beforehand. Another participant suggested highlighting the key points from each slide to make it quicker, with take home messages in bold.
Content: Two participants suggested improvement of content. One suggested inputting a bit more information on the difference between autism and ADHD. The other participant suggested including more information on treatments, management and monitoring.
Navigation: Finally, the last suggestion for improvement was in relation to the navigation of the resource. The participant suggested that the two modules would flow better in one module rather than two separate modules.
Questions on the usefulness of the resource in practice were also asked to ensure that the content did help to increase awareness of ADHD. All participants agreed that the resource will help them identify ADHD patients better, all believed that they will retain the knowledge acquired from the intervention and that it impacted on their attitude towards ADHD and ADHD patients.