Feasibility Assessment of a Health Preference Study in Autism Based on the Childhood Autism Rating Scale (CARS2): A Qualitative Study with Clinicians and Caregivers

doi:10.21203/rs.3.rs-1294995/v1

Download PDF

Research Article

Feasibility Assessment of a Health Preference Study in Autism Based on the Childhood Autism Rating Scale (CARS2): A Qualitative Study with Clinicians and Caregivers

https://doi.org/10.21203/rs.3.rs-1294995/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

In childhood autism spectrum disorder (ASD), the Childhood Autism Rating Scale–2nd edition (CARS2) is a condition-specific instrument to be filled out by clinicians, resulting in a score for diagnosis and severity. Our aim is to estimate a preference-based scoring of CARS2 to better understand the value of changes in the CARS2 score. Using caregivers and clinicians as proxies for autistic children, we assessed the feasibility of establishing preferences for CARS2-based attributes.

Methods

The 15 CARS2 items were assessed regarding their relevance as attributes and the appropriateness of their wording. Best-worst scaling (BWS) and discrete choice experiment (DCE) choice task designs were developed, as well as a mapping task between CARS2 and the EQ-5D-Y. Individual qualitative interviews with caregivers, clinicians, and autistic adults were conducted (N=10). A committee of experts advised on the study, including caregivers, clinicians, and specialists in CARS2, health technology assessment, and preference research methodology.

Results

Thirteen of the 15 CARS2 items were deemed appropriate attributes for the preference study. Caregivers and autistic adults identified with the attributes and perceived them as comprehensive. The attribute definitions and level descriptions were shortened and refined to ensure comprehension for non-experts. The choice tasks were challenging for participants; however, 9/10 respondents could make a choice. Interviewees favored a two-profile DCE over a three-profile BWS design. In a subsequent internal pilot (N=13), participants favored a stacked layout, in which attributes with overlapping levels were clustered. The CARS2/EQ-5D-Y mapping task was feasible using step-by-step instructions.

Conclusion

A preference study in childhood ASD based on the CARS2 instrument is feasible with caregivers and clinicians as proxies for children’s preferences.

Childhood Autism

Autism Spectrum disorder (ASD)

Childhood Autism Rating Scale–2nd edition (CARS2)

Health Preference Research

Discrete Choice Experiment (DCE)

Qualitative Research

Feasibility Assessment

Autism spectrum disorder (ASD) is a neurodevelopmental condition usually diagnosed during childhood, with an estimated worldwide prevalence of up to 2% (1-3). It is a heterogenous disorder characterized by persistent impairments in social communication and interactions (core domain one), as well as restrictive and repetitive behaviors (core domain two), which range from mild to severe (4). Current treatment options for ASD predominantly revolve around non-pharmaceutical interventions (5). No licensed pharmaceutical options are available that target the core symptoms, however, some agents are in development (6, 7).

Clinical trials in ASD often use the Childhood Autism Rating Scale–2^nd edition (CARS2) as a primary outcome measure to assess treatment efficacy (8-11). There are two versions of the condition-specific instrument: the standard version (ST) for children younger than six years and/or with intellectual disability (intelligence quotient <80) or notably impaired communication, and the high-functioning (HF) version for children six years or older without intellectual disability and with fluent language (12, 13). Both versions consist of 15 CARS2 questionnaire item descriptors (Table 2) and focus on the core symptoms of ASD (8). Each item is scored by a clinician from level 1 (“no evidence of difficulty or abnormality”) to level 4 (“severely abnormal”), and the total sum corresponds to the severity of the disorder (14). The clinician’s scoring is based on their assessment of the child and information from the CARS2–Questionnaire for Parents and Caregivers (CARS2-QPC), which is an unscored form completed by the parent/caregiver (12). After the diagnostic/baseline assessment, changes in the CARS2 score are used to assess improvements achieved through treatment. However, it is unknown what reduction in CARS2 score is perceived by autistic individuals as a clinically meaningful improvement, and how this perception aligns with the clinician’s assessment.

Understanding how patients value changes in disease/disorder attributes (or other health outcomes) is playing an increasingly important role in the registration of pharmaceuticals, reimbursement, and clinical decision-making (15). Patient preference studies aim to capture the value patients place on such attributes: Which attributes matter to patients, how much do they matter to patients, and how do patients make trade-offs between these attributes (15)? Study design is usually based on a choice task, in which patients choose from a number of hypothetical scenarios or profiles formed from relevant attributes set at varying levels. In discrete choice experiments (DCE) patients choose their preferred profile from a choice of two or more, whereas in multi-profile best-worst scaling (BWS), patients choose their preferred (best) and least-desired (worst) profile from three or more profiles (16).

A quantitative health preference study with clinicians and caregivers as proxies for children and adolescents with autism is planned to associate value with changes in the CARS2 score and provide insight into the relative importance of the CARS2 items. These findings will augment the interpretation of meaningful change on the CARS2 scale. The primary objective of this current qualitative study was to assess the feasibility of such a study, specifically to determine whether CARS2 items can be used as a basis for attributes and to optimize the choice task design. A secondary objective of this study was to determine the feasibility of anchoring both the ST and the HF version of CARS2 on a common, generic scale, i.e., EuroQoL’s EQ-5D-Y (Youth version) in combination with EQ VAS (visual analogue scale). The anchoring will facilitate comparisons between the ST and the HF preference outcomes, as well as to a broader health context, i.e., beyond autism. This paper reports on the qualitative research for the development and feasibility testing of a quantitative preference study in childhood autism based on CARS2.

2.1 Steering committee & ethical approval

A steering committee (SC) was formed to provide advice and input throughout the study. The SC consisted of various experts from several European countries, i.e., two caregivers of autistic children (EF, ZS), two clinicians working with autistic children (ASJ, TC)—of which one was a CARS2 expert (ASJ), an HTA/payer expert (OW), and a methodological expert in preference research (AM).

Ethical approval for this study was granted by SALUS IRB, US.

2.2 Preparatory reviews

Literature reviews were conducted to 1) verify the scope and relevance of attributes identified in ASD-specific instruments compared with CARS2 items, 2) identify preference research studies in ASD, 3) identify example cases of choice task designs that contain many attributes and/or many attribute levels, 4) inform the patient preference study design, and 5) to inform specific criteria to assess the feasibility of the CARS2 patient preference study.

2.3 Attribute assessment

The 15 items (each of them describing four levels of severity) of CARS2 were assessed in collaboration with the SC for their suitability to serve as attributes in a choice task setting for a preference study. Items were assessed for relevance and redundancy to identify those that could serve as attributes in order to form realistic profiles of hypothetical autistic children. The research team and the SC made the final decision on which items to exclude due to redundancy based on their experience and data from several published factor analyses (17-20).

Following the agreement of which CARS2 items were to be used as attributes, the complexity of the final list of attributes was assessed, specifically in terms of comprehensibility for non-expert users, the length of attribute descriptions, and the number of attributes.

2.4 Development of choice task design

DCE and BWS choice task designs were developed. Several versions of each task design were created to test formatting, length of attribute descriptions, and the use of color shading to indicate severity levels.

2.5 Qualitative interviews

Individual, one-hour interviews were conducted with participants to test if the attributes were understandable, if the choice tasks were doable, and to receive feedback on optimal choice task design. Since the pediatric population of interest includes very young autistic children and older children who are non-verbal or with intellectual disability, clinicians (n=4) and caregivers (parents of autistic children, n=4) were selected as the closest proxies. To avoid having to rely solely on the proxy perspective, two autistic adults were also included for interviews, as it was considered important to elicit the experiences of those living with autism and obtain reflections on their childhood. The semi-structured interviews were conducted via video-conferencing with screensharing. A professional recruitment agency (Global Perspectives) supported the recruitment, screening, and consent of participants. The interviews took place between the 11^th and the 30^th of June 2020.

During the interview, all participants read the attribute descriptions (“reading task”), and caregivers were asked to rate their own child based on the level descriptions. Participants then completed 10 BWS and DCE choice tasks with different layouts. Finally, if time allowed, the interviewees were asked to complete 2 mapping tasks with EQ-5D-Y and EQ VAS (online resource 1) (21, 22). For this mapping exercise, the profile of a hypothetical child for whom all attributes were set at the most severe CARS2 level (level 4) was presented. The EQ-5D-Y and EQ VAS tasks were completed for this severe CARS2 profile. Throughout the interview, participants were specifically asked to provide their feedback on the tasks to enhance the planned quantitative survey.

2.6 Feasibility assessment

Feasibility assessment criteria (Table 1) were developed based on the findings of the preparatory reviews regarding DCE methods (15, 23-25). The criteria were used to evaluate the findings from the qualitative interviews and to inform the decision whether to continue with the quantitative health preference study. The feasibility of mapping CARS2 to the EQ-5D-Y and EQ VAS (online resource 1) was also assessed.

Table 1 Feasibility Criteria for a Preference Study on Childhood Autism (15, 23-25)

Criteria	Definition
Capturing relevant patient experience	Do the attributes, level descriptions, and profiles resonate with caregivers and clinicians?
Choice task complexity	Is the choice task question phrased in an understandable manner? Does the proxy nature of the choice task make sense to participants?
Hypothetical nature of choice tasks	Is the hypothetical nature of the choice tasks clear and is enough information provided so that participants can make an informed choice in the hypothetical scenarios?
Intended meaning of attribute definitions and levels	Do the attributes and levels of the choice tasks retain the same meaning as when CARS2 is used within clinical practice?
Attribute dominance	Are any attributes likely to exhibit attribute dominance, i.e., do participants consistently choose those profiles where a specific attribute shows a better level?
Cognitive burden and time constraints	Can the questions be designed such that caregivers and clinicians can provide meaningful answers? Given the survey time available, can background information questions, warm-up explanation/exercises, and a sufficient number of choice tasks all be included within an online survey?
Choice task and experimental design feasibility	What type of choice task presentation can be used to minimize the cognitive burden subject to the constraint that it also provides relevant data to estimate preferences? Can an experimental design be developed that provides the data required to model preferences for the instrument considering the online survey duration and target sample size?
Mapping experiment with EQ-5D-Y and VAS	Can disorder-specific CARS2 profiles be mapped to a generic HRQoL scale? Does the mapping experiment facilitate creating a comparison between ST and HF?

CARS2, Childhood Autism Rating Scale–2^nd edition, EQ-5D-Y, EuroQoL’s Youth version, HF, high-functioning, HRQoL, health-related quality of life, ST, standard, VAS, visual analogue scale.

3.1 Preparatory reviews

An overview of the publications identified through the preparatory reviews is provided in online resource 1. In summary, publications reported that CARS2 adequately reflects the patient experience and the core symptom domains of autism (8, 17, 26). Moreover, it was considered to have satisfactory psychometric properties regarding internal consistency, sensitivity, specificity, reliability regarding in-between rating, test-retest reliability, and generalizability coefficient (27, 28). A total of four ASD-specific preference studies were identified (29-32). The preparatory literature review identified medication use as an additional attribute that is not covered in CARS2 (29), however, as our study does not look at specific treatment attributes, this was not considered to be relevant.

Numerous methods were identified from the literature review to reduce choice task complexity, such as color coding and partial profiling, i.e., only presenting a subset of attributes per choice task (24, 33). A method in-between full and partial profiling was constructed, instead of omitting overlapping attributes, these were grouped together and presented to the participant to provide the full picture of the profile (“stacked approach”). This method was considered highly relevant for this study and was incorporated in the final design of the choice tasks. Published methods on the use of color coding according to severity to provide visual support were also used to support choice task designs (24).

3.2 Attribute assessment and refinement

The 15 CARS2 items as displayed in Table 2 were reduced to 13 relevant attributes. The item “Level and Consistency of Intellectual Response” was not considered to be modifiable and thus not applicable for the current preference research aiming at valuing changes in CARS2 items from a patient perspective. The item “General Impressions” was also removed from the list of attributes. As it provides an overall assessment, it is therefore overlapping and potentially conflicting with other attributes presented in a hypothetical profile (i.e., it presented a high risk of illogical combinations). Based on factor analyses (17-20) and their own expertise, the SC did not identify any redundancy among the remaining 13 attributes, nor did they identify illogical level combinations, given the large heterogeneity seen among autistic individuals.

Table 2 CARS2 Questionnaire Item Descriptors for the Standard (ST) and High-Functioning (HF) Versions

No.	CARS2-ST	No.	CARS2-HF
Items included as relevant attributes
1	Relating to People	1	Social-Emotional Understanding
2	Imitation	2	Emotional Expression and Regulation of Emotions
3	Emotional Response	3	Relating to People
4	Body Use	4	Body Use
5	Object Use	5	Object Use in Play
6	Adaptation to Change	6	Adaptation to Change/Restricted Interests
7	Visual Response	7	Visual Response
8	Listening Response	8	Listening Response
9	Taste, Smell, and Touch Response and Use	9	Taste, Smell, and Touch Response and Use
10	Fear or Nervousness	10	Fear or Anxiety
11	Verbal Communication	11	Verbal Communication
12	Nonverbal Communication	12	Nonverbal Communication
13	Activity Level	13	Thinking/Cognitive Integration Skills
Items excluded as attributes
14	Level and Consistency of Intellectual Response	14	Level and Consistency of Intellectual Response
15	General Impressions	15	General Impressions

CARS2, Childhood Autism Rating Scale–2nd edition, HF, high-functioning, ST, standard.

For both CARS2 versions (ST and HF), minor editorial amendments were made to all 13 attribute descriptions to improve readability, while staying as close as possible to the original instrument (online resource 2). The refined CARS2 attributes were tested with each SC member to ensure adequate comprehensibility.

3.3 Development of choice task design

DCE and BWS choice task designs were developed using the 13 attributes and tested with SC members individually via video-conferencing. Feedback received was used to refine the choice tasks to 6 DCE and 4 BWS choice tasks, which varied in their design (e.g., use of severity color-coding, length of descriptions), for testing in qualitative interviews. Fig. 1 and Fig. 2 present examples of the BWS and DCE designs tested.

3.4 Qualitative interviews

Qualitative interviews were conducted with four caregivers, four clinicians, and two autistic adults. Demographic and background information is presented in Table 3.

Table 3 Demographic Characteristics and Background Information of Interviewees

Demographic characteristics/background information	Participants
Clinicians (N=4)
Range of professions	Child psychiatrist, pediatrician, occupational therapist
Country of origin: France/UK	1/3
Working in diagnosis/treatment/both areas	1/3/1
Years of experience in working with autistic children/adolescents	20-26 years
N of autistic children per month (average estimation)	12-175
Familiar with CARS2? Yes/No	4/0
Have you used CARS2? Yes/No	3/1
Version of CARS2 applied in interview (ST/HF)	2/2
Caregivers (N=4)
Age of the caregiver	40-48 years
Gender of the caregiver (m/f/non-binary)	0/3/1
Country of origin: UK	4
Level of education of the caregiver: higher education Yes/No	3/1
Familiar with CARS2? Yes/No	3/1
N of autistic children: 1/>1	4/0
Age of the child	5-15 years
Gender of the child (m/f/non-binary)	2/2/0
Age of the child at diagnosis	5-13 years
Version of CARS2 applied in interview (ST/HF)	2/2
Autistic adults (N=2)
Age	48/57 years
Level of education: higher education Yes/No	2/0
Country of origin: UK	2
Age at diagnosis	42/47 years
Familiar with CARS2? Yes/No/Somewhat	1/0/1
Version of CARS2 applied in interview (ST/HF)	0/2

CARS2, Childhood Autism Rating Scale–2nd edition, HF, high-functioning, ST, standard.

Two of the clinicians and two of the caregivers were interviewed based on the ST versions of the CARS2-based attributes, all remaining participants were interviewed using the HF version. Table 4 provides relevant excerpts from the interviews.

Table 4 Feedback from the Qualitative Interviews

Participant	Topic	Quotations
Caregiver	Attribute “Relating to People”	“[…] when it says here, 'difficult initiating or managing the interaction' I think that's certainly one of her struggles […] .”
Caregiver	Attribute “Listening Response”	“'Unusual over or underreactions to noise or sounds.' Yes. We've got construction [next to our house] and it's really, really bothering my daughter, […] she experiences it really intense, […] I really resonate with that. I think actually my daughter's a great listener I think she definitely doesn't ever really look towards a person when she's listening and I think, over time, her friends recognize that she still listens.”
Caregiver	Attribute “Emotional Expression and Regulations of Emotions”	“They [attribute and level descriptions] are clear to me. I would say that we go between one and three, you know, depending on the time.”
Caregiver	Attribute “Object Use in Play”	“My thoughts are that it describes quite accurately my son. Pokémon will always arrive in the game, you know, that's his current obsession. So, no matter what he's playing, Pokémon will become centred in some way in the game. […] he does restrict his world by what he wants to play with, and we move from object obsession to object obsession.”
Clinician	Attributes Overall	“Reading the attributes is very useful, you don't have to be trained to use the CARS instrument”
Autistic adult	Attribute “Social-Emotional Understanding”	“It sort of touches home a bit, because I have trouble understanding other people's body language, facial expressions, everything.”
Caregiver	Choice task complexity	• “So, this is for any child or my child?” • “I think these slides are particularly difficult […] At the moment I’m thinking I don’t know where to start, I’m thinking what things concern me most or give me the most kind of pressures from a caring point of view.”
Caregiver	Choice task complexity	• “[…] so from all these characteristics I think for me the age appropriate adaptation to change is more important than anything else.” • “I think it’s very hard to choose between the severely abnormal activity level and the severely abnormal fear and nervousness […]” • “It’s again, you know, the abnormal fear and nervousness for me would be quite problematic, so A would be the worst and C would be the best.”
Clinician	Choice task complexity	• “So, if I just try to visualize child A and child B, we've got child A who has some difficulties relating to other people, but he doesn't show flapping or rocking, so it's more subtle for other people although he gets upset with new situations […]. It's difficult to choose because every child has their strength and difficulties, but possibly A is slightly preferred from the parents' point of view.”
Clinician	Choice task complexity	• “That’s very difficult to answer. I guess… I will go for profile A because looking at profile B, severely abnormal body use, that’s of great concern to me.” • “So I’m just going to go straight into the severe… abnormal fear, anxiety… moderately abnormal… for me quickly it’s A but I could be wrong.” • “The speech. So, moderately abnormal verbal communication, speech may be absent. So no speech I can imagine what that means for that child. So for me, that’s the worst.”
Autistic Adult		• “I suppose it would be easier to answer if there's only a choice of two. When you've got a choice of three, especially when they're so subjective, I don't know, how do people trade these things off?” • “It's difficult. Let's pick. I'm not too bothered about object use in play, so I would score that one out and then, you know, I'd reduce my options in some way and then that would help me…”
Caregiver	EQ-5D-Y and VAS mapping task	• “A lot of these don’t fit.” • “The activities, I’d probably say abnormal activity level again, I have a lot of problems. The pain and discomfort, severely abnormal use of taste and smell, a lot of pain.” • “The severely abnormal emotional would be very worried”

EQ-5D-Y, EuroQoL’s 5 Dimensions Youth version, VAS, visual analogue scale.

3.4.1 Attribute Descriptions

The attributes were comprehensible and clear for all participants. Clinicians considered the attributes applicable, and caregivers and autistic adults identified with the CARS2 descriptions and judged them as relevant. Moreover, all caregivers and autistic adults related the experiences they had with their child or themselves as a child. One autistic adult commented that the items did not take into consideration the positive aspects of autism.

All the caregivers found it possible to rate their child per attribute. When rating, three out of four caregivers also considered feedback they received from teachers at school. However, sometimes caregivers indicated that the rating was difficult because the child showed behavior that fluctuated from day to day and over the course of years.

Most participants considered the attribute descriptions as lengthy, with the reading task requiring substantial time. However, only one participant expressed difficulties in processing the amount of information provided in the descriptions. Moreover, several comments on the wording used in the CARS2-based attributes descriptions were made across the participant groups. For example, the phrasings like “abnormal” and “severely impaired” were considered to have a somewhat judgmental connotation to describe an autistic child’s behavior.

3.4.2 Choice Task Complexity

Participants found the choice task exercises very challenging, however, all but one participant (an autistic adult) were able to make choices based on the hypothetical profiles presented. Participant questions about the tasks gave valuable insight into the difficulties they were facing and highlighted that the choice tasks needed to be accompanied by clear explanations and examples. Respondents were often unsure which perspective they should take—i.e., that of the caregiver, clinician, child, or society, or whether they should select the most realistic profile versus the preferred profile.

The choice tasks did not reveal any attribute dominance, but for caregivers and autistic adults, the attributes “Fear or Nervousness”/“Fear or Anxiety” and “Verbal Communication” stood out as important decision drivers in the choice tasks. While making choices, several caregivers considered the societal context, since “fitting into society” was considered an important factor for the child and caregiver.

3.4.3 Choice Task Design

Six out of the 10 participants were shown both the BWS and DCE approach (the remaining four were not shown both designs due to time constraints). Two caregivers preferred the DCE approach, because they perceived comparing and choosing between two profiles as easier. One clinician clearly favored the BWS approach over the DCE. Overall, the interviews revealed that digesting two profiles (DCE) was easier for participants than three profiles (BWS).

The different layout versions of the choice tasks—i.e., with or without color-coding, fading of overlapping attributes, and use of bold font to emphasize non-overlapping attributes—and the use of short and long text versions were tested with all participants except one caregiver. The use of color-coding to indicate the severity level of the attributes was considered helpful by all participants. The use of fading and bold font was not considered useful and was even judged to cause bias on the choice. The short attribute descriptions were preferred over the long versions by all caregivers and by one autistic adult. However, one clinician made a remark that the short text version might cause issues for respondents who were not familiar with CARS2, and another clinician also preferred the longer attribute description in the profiles.

3.4.4 EQ-5D-Y and EQ VAS Mapping Task

Three caregivers and one clinician had sufficient interview time to attempt the mapping task. Many questions were raised during the exercise, however, all four participants were able to execute the task. An important learning from the participants’ comments was that thorough explanation is needed to guide the participants through the exercise.

Regarding the EQ-5D-Y task, one caregiver had a number of questions around the interpretation of the terminology, in particular around “mobility” (physical mobility versus sensory overload), “usual activities” (what activities were standard for society versus those standard for an autistic child), and “pain and discomfort” (physical versus emotional feelings). The clinician stated that the mapping was possible, although not all EQ-5D-Y items were applicable to CARS2, e.g., “looking after yourself”.

Regarding the EQ VAS, one caregiver commented that it was unclear if the scale was about physical or mental health and questioned if the scale meant being “medically ill.” From a clinician’s perspective, rating the worst CARS2 profile on the EQ VAS posed difficulties, because “a child presents with strengths and difficulties.”

3.4.5 Subgroup analysis

The qualitative interviews revealed no specific differences between caregivers and clinicians in terms of comprehensibility and interpretation of the CARS2-based attributes or the completion of choice task and mapping exercises (Table 5). Both groups expressed similar difficulties regarding the choice tasks and attribute wording used in CARS2. Thus, the research team concluded that the same study can be applied to both groups, caregivers and clinicians.

Table 5 Comparison of Interview Outcomes Between Participant Groups

Topic	Caregivers	Clinicians	Autistic adults
Understanding of the attributes	Clear and understandable	Clear and understandable	Understandable, but challenging to process all the information
Identification with the attributes	Could relate/tell experiences	Not applicable	Could relate/tell experiences
Rating with the CARS2 scale*	No issues with rating	Not applicable	Not applicable
Accounting for fluctuating behavior of the child	Fluctuating behavior of child sometimes made the rating challenging	CARS2 accounts for fluctuating behavior with an observation over 2 weeks	Not mentioned
No accounting for positive aspects of autism	Not mentioned	Not mentioned	Positive aspects of autism not captured in the attributes
Parents’ vs. school’s view	Some caregivers considered feedback from school in their rating	Not mentioned	Not mentioned
Attribute dominance	No attribute dominance identified	No attribute dominance identified	No attribute dominance identified
Reading/length of the attribute descriptions	Text perceived as very lengthy	Text perceived as very lengthy	Text perceived as very lengthy
Wordings	Some wordings seemed not appropriate to use	Some wordings seemed not appropriate to use	Some wordings seemed not appropriate to use
Choice task complexity	Choice tasks perceived as challenging, but they were doable	Choice tasks perceived as challenging, but they were doable	Choice tasks perceived as challenging, one participant could not make a choice
Choice strategies	“Fitting into society” as important decision driver	Mixed strategies	Fear and anxiety were decision drivers for one participant
Formatting of the choice tasks	Colors and shorter text were preferred	Mixed answers, but coloring was preferred	Colors and shorter text were preferred
Question format	Having 2 profiles seemed to be easier	BWS format preferred by one participant	One participant preferred the DCE question format
EQ-5D-Y and EQ VAS	Difficult task and applicability to CARS2 was questioned	One clinician did the task and stated that the mapping was feasible	Not done

*How caregivers found rating/scoring their child against the attributes. This topic was not relevant for clinicians or adults with ASD.

CARS2, Childhood Autism Rating Scale–2^nd edition, EQ-5D-Y, EuroQoL’s 5 Dimensions Youth version, VAS, visual analogue scale.

3.5 Feasibility assessment and subsequent modifications

Interview findings were assessed against the pre-defined feasibility criteria (Table 1) and discussed with the SC. No major issues were identified which might impede the continuation of the main quantitative health preference study.

Comments from qualitative interview participants regarding the language and length of the text describing attributes were addressed by further refinements to optimize readability. Discussion of the interview findings with the SC revealed that appropriate phrasing of the choice task question is crucial to get the caregivers’/clinicians’ perspectives on patient preferences. Phrasing the question as “Which profile do you prefer for a child?” as opposed to “Which do you think is preferred for a child?” avoids any estimate or guess required by the caregiver/clinician and does not imply that there is a right or wrong answer. As caregivers found rating their child difficult due to fluctuations in behavior, it was agreed that caregivers should be asked to rate their child based on behavior observed in the previous two weeks. The interviews also revealed that it was important to know the age of the child whom participants had in mind. Feedback from the interview participants and the SC allowed for further improvements to the choice task presentation and led to the final DCE choice task design (online resource 1).

The qualitative interviews flagged an important time constraint: only four participants had sufficient time left to do the EQ mapping exercise within the hour-long interview. The quantitative study is to be conducted as an online survey that should ideally last no longer than 45 minutes. Several modifications were agreed with the SC to help reduce the time taken to complete the survey in particular: 1) providing an introductory video that better explains the choice task, thereby reducing time taken to complete the exercise, 2) making attribute descriptions readily available in the choice tasks through pop-up windows, 3) including a ranking task of all the attributes to help respondents prioritize attributes, thereby supporting them in making choices between profiles, and 4) implementing attribute overlap: In each choice task, seven attributes of the two profiles were set at the same severity level and six attributes at different levels. The overlapping attributes were clustered at the top of the profiles and non-overlapping attributes at the bottom, resulting in a “stacked” layout (Fig. 3). Overlapping attributes were not omitted in the choice tasks as they provide relevant background information on the complete profile of a child. The clustering aimed at minimizing the time and effort required to search for differences between the two profiles. With all these supportive measures, it was anticipated that respondents could complete approximately 14 choice tasks in a 45-minute online survey for the quantitative study phase. The research team tested the revised choice tasks in an internal pilot study (six members of the SC and seven colleagues) to confirm the modifications and reveal any remaining issues.

Finally, the research team decided to integrate the EQ-5D-Y and EQ VAS mapping tasks as an add-on component of the online survey which participants can choose to do optionally. Although the task was challenging and could not be done by all interviewees due to time constraints, the mapping is perceived as a unique experimental component that can provide an anchor for CARS2 to a generic measure of health-related quality of life.

To the best of our knowledge, no preference studies have been completed in childhood autism based on the CARS2 items. Our research assessed the feasibility of such a preference study and that of a novel approach of mapping CARS2-based profiles to the EQ‑5D‑Y questionnaire and EQ VAS.

CARS2 was selected as the basis of the preference study as it is considered the most comprehensive instrument covering the core symptoms of ASD (8, 27, 34, 35). Compared with other generic instruments used in ASD assessment, in particular the Vineland-II Adaptive Behavior Scales (VABS) and the Social Responsiveness Scale (SRS-2), CARS2 encompasses more of the core symptoms of ASD, which makes it better suited to reflect the patient experience (8). The French National Health Authority (Haute Autorité de Santé) states that CARS2 fits well with current practice according to the DSM-5, has good psychometric properties, and is used as a measure of change in clinical practice (27, 35). Moreover, the European Medicines Agency recommends CARS2 as an instrument to assess efficacy in clinical trials, making it a de facto tool able to capture improvements or worsening in the ASD symptomatology (34).

In a recent review and meta-analysis of nine studies, data from two suggest that CARS2 may have suboptimal specificity (0.79 and 0.75) and should be used alongside other tools for diagnosis (17). In our study, CARS2-based attributes are used as an outcome measure, and, as a result of suboptimal specificity, there may be heterogeneity in how the health states/profiles presented are interpreted. However, as this is a health preference study, it is neither feasible nor desirable to eliminate room for interpretation.

Our research found that although CARS2 is designed to be used by clinicians experienced in the management of autistic children/adolescents, the attribute descriptions were clear and understandable to both caregivers and clinicians. Caregivers could relate to the attributes and were able to rate their children based on the CARS2 level descriptions. Even though the choice tasks were perceived as challenging, the interviewees were able to make choices. Moreover, a mapping of CARS2 to EQ-5D-Y and EQ VAS is also feasible when thorough explanation of the task is provided.

The feasibility assessment also aimed to enhance the design of the planned quantitative preference study. A risk for the quantitative survey is that the cognitive burden of the survey might cause participants to tire or lose attention. This qualitative study explored and tested many avenues to reduce the cognitive burden and complexity of the experiment and demonstrated that interviewees were capable of processing the information, understanding the task at hand, and making choices. Feedback sought during the qualitative interviews (and during the subsequent internal pilot test) led to several survey modifications to reduce the complexity of the tasks (including using stacked DCE design with color coding) and to improve the readability of the attribute descriptions. An important finding from this qualitative research is that a preference study involving both caregivers and clinicians, using the same set of choice tasks, is feasible. In the quantitative phase of our research, this will allow us to elicit differences in perceptions and preferences between caregivers and clinicians.

This study has several limitations. First, this preference research was not conducted with autistic children and adolescents, but with caregivers, clinicians, and autistic adults as their proxies. However, conducting such research with very young children or children who are non-verbal or with intellectual disability is not a feasible option. While a subgroup of autistic adolescents with average cognitive ability and good language skills could have participated, this would have led to an unrepresentative sample. Including two autistic adults allowed us to understand how their lived experiences affected their perception of the attributes. Second, this study did not factor in information that is acquired from caregivers through the CARS2-QPC in a diagnostic setting. However, the CARS2-QPC does not provide different attributes but rather observations in a different setting, thereby informing the clinician’s assessment. Third, the sample size of the qualitative interviews was small with four caregivers, four clinicians, and two autistic adults. Finally, the complexity of the experiment was high, requiring participants to familiarize themselves with 13 attributes and the presentation of choice task profiles that contain these 13 attributes with varying severity levels. A fold-in fold-out (FiFo) design, in which attributes are grouped in overarching domains, was investigated to reduce respondents’ cognitive burden. However, folded in (or grouped) attributes should form an intuitive grouping (36). As no consistent domains were identifiable for the CARS2 ST and HF (as supported by the factor analysis), a FiFo design was not considered suitable for this study.

This study demonstrated that it is feasible for caregivers and clinicians to use the CARS2 attributes to provide their perspective on autistic children’s preferences. The relevance of CARS2 was confirmed by our study’s interview participants, specifically by caregivers and autistic adults who could identify with the attribute descriptions, either from their experiences with their child or their own experiences as a child. Our qualitative research demonstrated that caregivers and clinicians can complete a DCE choice task design based on CARS2 attributes and it informs the designing of a quantitative preference study that will enhance the interpretation of meaningful changes in the CARS2 score. Future efforts of this preference research aim to enhance the patient-relevance of CARS2 and provide relevant patient-centric insights for the evaluation of treatments in childhood autism.

ASD	Autism Spectrum Disorder
BWS	Best-Worst Scaling
CARS2	Childhood Autism Rating Scale–2^nd edition
CARS2-QPC	CARS2–Questionnaire for Parents and Caregivers
DCE	Discrete Choice Experiment
DSM-5	Diagnostic and Statistical Manual of Mental Disorders, 5th Edition
EQ	EuroQoL
EQ-5D-Y	EuroQoL’s 5 Dimensions Youth version
FiFo	Fold-in Fold-out
HF	High-Functioning version
SC	Steering Committee
SRS-2	Social Responsiveness Scale
ST	Standard version
VABS	Vineland-II Adaptive Behavior Scales
VAS	Visual Analogue Scale

Ethics approval: Ethical approvals for the study were granted by Salus IRB, US.

Consent to participate: Written informed consents were obtained from all the participants of the qualitative interviews.

Consent for publication: Not applicable.

Availability of data and materials: All data generated or analysed during this study are included in this published article [and its supplementary information files].

Funding: This study was funded by Les Laboratoires Servier.

Authors’ contributions: KH, ND, RS, MH, and OE contributed to the study design, planning, and conduct of the study, as well as the analysis and interpretation of results and manuscript writing. ZS, EF, OW, TC, ASJC, and AM were part of the study steering committee and provided advice on the study design and the interpretation of the results, as well as critically revising the manuscript. BVH, JB, and ES advised on the conception and design of the study and the analysis and interpretation of data as well as critically revising the manuscript.

Acknowledgements: The authors thank Evangelos Zormpas for his contributions to the pilot study. The authors thank Dr. Aruna Jeans, who provided medical writing support.

ASDEU. Autism Spectrum Disorders in the European Union. Executive Summary. 2018. http://asdeu.eu/wp-content/uploads/2016/12/ASDEUExecSummary27September2018.pdf. Accessed 4 Jan 2022.
Lyall K, Croen L, Daniels J, Fallin MD, Ladd-Acosta C, Lee BK, et al. The Changing Epidemiology of Autism Spectrum Disorders. Annu Rev Public Health. 2017,38:81-102.
Fuentes J, Hervás A, Howlin P. ESCAP practice guidance for autism: a summary of evidence-based recommendations for diagnosis and treatment. European Child & Adolescent Psychiatry. 2020, doi:10.1007/s00787-020-01587-4
American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-5: Fifth edition. Arlington, VA : American Psychiatric Association, 2013.
Levy SE, Frasso R, Colantonio S, Reed H, Stein G, Barg FK, et al. Shared Decision Making and Treatment Decisions for Young Children With Autism Spectrum Disorder. Acad Pediatr. 2016,16(6):571-8.
Politte LC, Howe Y, Nowinski L, Palumbo M, McDougle CJ. Evidence-Based Treatments for Autism Spectrum Disorder. Current Treatment Options in Psychiatry. 2015,2(1):38-56.
Díaz-Caneja CM, State MW, Hagerman RJ, Jacquemont S, Marín O, Bagni C, et al. A white paper on a neurodevelopmental framework for drug discovery in autism and other neurodevelopmental disorders. European Neuropsychopharmacology. 2021,48:49-88.
Payakachat N, Tilford JM, Kovacs E, Kuhlthau K. Autism spectrum disorders: a review of measures for clinical, health services and cost-effectiveness applications. Expert review of pharmacoeconomics & outcomes research. 2012,12(4):485-503.
Crutel V, Lambert E, Penelaud P-F, Albarrán Severo C, Fuentes J, Rosier A, et al. Bumetanide Oral Liquid Formulation for the Treatment of Children and Adolescents with Autism Spectrum Disorder: Design of Two Phase III Studies (SIGN Trials). Journal of Autism and Developmental Disorders. 2020, doi: 10.1007/s10803-020-04709-8
Dai Y, Zhang L, Yu J, Zhou X, He H, Ji Y, et al. Improved symptoms following bumetanide treatment in children aged 3−6 years with autism spectrum disorder: A randomized, double-blind, placebo-controlled trial. Science Bulletin. 2021.
Zhang L, Huang C-C, Dai Y, Luo Q, Ji Y, Wang K, et al. Symptom improvement in children with autism spectrum disorder following bumetanide administration is associated with decreased GABA/glutamate ratios. Translational Psychiatry. 2020,10(1):9.
Schopler E, Van Bourgondien MEW, Glenna Janette , Love SR. (CARS™-2) Childhood Autism Rating Scale™, Second Edition 2011. https://www.wpspublish.com/cars-2-childhood-autism-rating-scale-second-edition. Accessed 4 Jan 2022.
Schopler E. The childhood autism rating scale, second edition (CARS2) : manual. Los Angeles, Calif.: Western Psychological Services, 2010.
Vaughan CA. Test Review: E. Schopler, M. E. Van Bourgondien, G. J. Wellman, & S. R. Love Childhood Autism Rating Scale (2nd ed.). Los Angeles, CA: Western Psychological Services, 2010. Journal of Psychoeducational Assessment. 2011,29(5):489-93.
van Overbeeke E, Janssens R, Whichello C, Schölin Bywall K, Sharpe J, Nikolenko N, et al. Design, Conduct, and Use of Patient Preference Studies in the Medical Product Life Cycle: A Multi-Method Study. Frontiers in pharmacology. 2019,10:1395-.
Mühlbacher AC, Kaczynski A, Zweifel P. Experimentelle Präferenzmessung im Gesundheitswesen mit Hilfe von Best-Worst Scaling (BWS). PharmacoEconomics German Research Articles. 2013,11(2):101-17.
Moon SJ, Hwang JS, Shin AL, Kim JY, Bae SM, Sheehy-Knight J, et al. Accuracy of the Childhood Autism Rating Scale: a systematic review and meta-analysis. Dev Med Child Neurol. 2019,61(9):1030-8.
Moulton E, Bradbury K, Barton M, Fein D. Factor Analysis of the Childhood Autism Rating Scale in a Sample of Two Year Olds with an Autism Spectrum Disorder. J Autism Dev Disord. 2019,49(7):2733-46.
DiLalla DL, Rogers SJ. Domains of the Childhood Autism Rating Scale: relevance for diagnosis and treatment. J Autism Dev Disord. 1994,24(2):115-28.
Magyar CI, Pandolfi V. Factor structure evaluation of the childhood autism rating scale. J Autism Dev Disord. 2007,37(9):1787-94.
EuroQol Group. EQ-5D-Y (Youth) | About. https://euroqol.org/eq-5d-instruments/eq-5d-y-about/. Accessed 2 Dec 2021.
EuroQuol Group. EQ-5D is a family of instruments to describe and value health. Available at: https://euroqol.org/eq-5d-instruments/. Accessed 2 Dec 2021.
Janssen EM, Marshall DA, Hauber AB, Bridges JFP. Improving the quality of discrete-choice experiments in health: how can we assess validity and reliability? Expert review of pharmacoeconomics & outcomes research. 2017,17(6):531-42.
Jonker MF, Donkers B, de Bekker-Grob E, Stolk EA. Attribute level overlap (and color coding) can reduce task complexity, improve choice consistency, and decrease the dropout rate in discrete choice experiments. Health Economics. 2019,28(3):350-63.
Ryan M, Bate A, Eastmond CJ, Ludbrook A. Use of discrete choice experiments to elicit preferences. Qual Health Care. 2001,10 Suppl 1(Suppl 1):i55-60.
Thabtah F, Peebles D. Early Autism Screening: A Comprehensive Review. Int J Environ Res Public Health. 2019,16(18).
Haute Autorité de Santé (HAS). Trouble du spectre de l’autisme. Signes d’alerte, repérage, diagnostic et évaluation chez l’enfant et l’adolescent. Méthode Recommandations pour la pratique clinique. 2018. https://www.has-sante.fr/upload/docs/application/pdf/2018-02/trouble_du_spectre_de_lautisme_de_lenfant_et_ladolescent__recommandations.pdf. Accessed: 4 Jan 2022.
McConachie H, Parr JR, Glod M, Hanratty J, Livingstone N, Oono IP, et al. Systematic review of tools to measure outcomes for young children with autism spectrum disorder. Health Technol Assess 2015,19(41).
dosReis S, N'Dri L, Ross M, Camelo Castillo W, Reeves G, Butler B. Care Management for Youth With Comorbid Developmental and Mental Health Conditions: A Discrete Choice Experiment Pilot Study. Acad Pediatr. 2020,20(2):241-9.
Colver A, Rapley T, Parr J, al. e. Facilitating the transition of young people with long-term conditions through health services from childhood to adulthood: the Transition research programme. Southampton (UK): NIHR Journals Library, 2019. Contract No.: No. 7.4.
Lavelle TA, Weinstein MC, Newhouse JP, Munir K, Kuhlthau KA, Prosser LA. Parent Preferences for Health Outcomes Associated with Autism Spectrum Disorders. PharmacoEconomics. 2019,37(4):541-51.
Beidas RS, Volpp KG, Buttenheim AN, Marcus SC, Olfson M, Pellecchia M, et al. Transforming Mental Health Delivery Through Behavioral Economics and Implementation Science: Protocol for Three Exploratory Projects. JMIR Res Protoc. 2019,8(2):e12121.
Lim S, Jonker MF, Oppe M, Donkers B, Stolk E. Severity-Stratified Discrete Choice Experiment Designs for Health State Evaluations. PharmacoEconomics. 2018,36(11):1377-89.
European Medicines Agency E. European Medicines Agency (EMA), Committee for Medicinal Products for Human Use (CHMP). Guideline on the clinical development of medicinal products for the treatment of Autism Spectrum Disorder (ASD). 9 November 2017. EMA/CHMP/598082/2013. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-clinical-development-medicinal-products-treatment-autism-spectrum-disorder-asd_en.pdf. Accessed 4 Jan 2022.
Haute Autorité de Santé (HAS), Agence Nationale de l'Évaluation et de la Qualité des Établissements et Services Sociaux et Médico-sociaux (ANESM). Autisme et autres troubles envahissants du développement : interventions éducatives et thérapeutiques coordonnées chez l’enfant et l’adolescent. Recommendation de bonne pratique. Argumentaire scientifique. 2012. https://www.has-sante.fr/upload/docs/application/pdf/2012-03/argumentaire_autisme_ted_enfant_adolescent_interventions.pdf. Accessed 4 Jan 2022).
Goossens LMA, Jonker MF, Rutten-van Mölken MPMH, Boland MRS, Slok AHM, Salomé PL, et al. The Fold-in, Fold-out Design for DCE Choice Tasks: Application to Burden of Disease. Medical Decision Making. 2019,39(4):450-60.

Competing interest reported. RS was a full-time employee of Servier at the time of study conduction and writing this manuscript. KH, ND, and MH are full-time employees of OPEN Health. BVH is professor at the University of Sheffield and an employee as well as a shareholder of OPEN Health. OPEN Health received funding from Servier for conducting the study. OE is professor at the University of Liège in Belgium, and has received consulting fees from Servier through SERFAN Innovation, a company that he owns. ZS and EF are part of the steering committee of this study and members of Autism-Europe. OW is part of the steering committee and full-time employee of Medi-Qualité Omega. TC is member of the steering committee and in the past three years, TC has served as a paid consultant to F. Hoffmann-La Roche Ltd. and has received royalties from Sage Publications and Guilford Publications. ASJC is member of the steering committee and an active member of the bumetanide trials, which are conducted by Servier, and in the past, has served as a paid consultant to F. Hoffmann-La Roche Ltd. AM is part of the steering committee and professor at University of Neubrandenburg and executive director of GEB mbH. All steering committee members received honoraria from Servier. JB and ES are advisors of OPEN Health and received honoraria from OPEN Health.

Download PDF

Version 1

posted

You are reading this latest preprint version

Feasibility Assessment of a Health Preference Study in Autism Based on the Childhood Autism Rating Scale (CARS2): A Qualitative Study with Clinicians and Caregivers

Status:

Version 1

Abstract

Figures

1 Background

2 Methods

2.1 Steering committee & ethical approval

2.2 Preparatory reviews

2.3 Attribute assessment

2.4 Development of choice task design

2.5 Qualitative interviews

2.6 Feasibility assessment

3 Results

3.1 Preparatory reviews

3.2 Attribute assessment and refinement

3.3 Development of choice task design

3.4 Qualitative interviews

3.4.1 Attribute Descriptions

3.4.2 Choice Task Complexity

3.4.3 Choice Task Design

3.4.4 EQ-5D-Y and EQ VAS Mapping Task

3.4.5 Subgroup analysis

3.5 Feasibility assessment and subsequent modifications

4 Discussion

5 Conclusion

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1