Participants
Approximately 250 participants will be recruited into the study. They will be aged 14 and over, of all genders, and divided into two groups – clinical and non-clinical. Clinical status will be determined post-testing using EDE-Q cut-off scores. We aim to recruit across all DSM 5 diagnostic categories for eating disorders.
Participants will be recruited in three ways:
- Amazon’s Mechanical Turk (MTurk): ‘MTurk’ is a global internet crowdsourcing service used frequently in academic research to recruit a broad base of willing research participants who are compensated for completing surveys, questionnaires and tasks as assigned by the researcher [46]. MTurk does not permit people under 18 to use its service: thus, it will be used to recruit adults aged 18+ only, both clinical and non-clinical. Participants will be able to sign up from anywhere in the world, deeming it an international cohort.
- Community: Participants will be recruited via online advertisements on health websites and social media channels/paper advertisements. Advertisements will be aimed at individuals who self-report an eating disorder diagnosis, though those without an eating disorder are also welcome to participate. We are specifically looking to recruit adolescents via this method, due to MTurk’s restriction on underage participants, however adults will also be targeted.
- SONApsych: Students enrolled in undergraduate psychology units at the University of Sydney will be given the option to engage in the task for course credit, where an alternative assessment and/or full debrief is compulsorily offered. Research participation is a traditional component of learning in these units and must provide educational value.
The use of three methods of recruitment aims to increase generalisability across age, ethnicity, gender and Body Mass Index (all of which will be reported) – this is significant due to the broad-reaching nature of the internet and the importance of validating the questionnaire across populations.
Inclusion/Exclusion criteria
At this stage of validation, the screener will be conducted in the English language only, thus proficiency in reading the English language is required. Participants under the age of 14 are excluded from the study – separate validation studies are indicated for younger age groups due to differences in comparison measures, linguistic ability and potential impact on item comprehension. Also excluded are those with no reliable access to the internet. Otherwise, no further demographic or exclusion criteria will be set. Participants with a current diagnosis of an eating disorder as well as those with no current or former diagnosis are equally welcome to participate: these two groups (clinical and non-clinical) will be defined and compared post testing using EDE-Q cut-off scores.
Overall Study Design
This study will use a cross-sectional survey research design with follow up on a sub-sample to examine the 6-item IOI-S for statistical reliability and validity. Participants will be given access to a baseline survey package delivered on the secure web survey platform REDCap. Thirty percent of participants will be randomly selected to participate in a follow-up re-test involving completion of the IOI-S alone a second time two weeks post initial testing, for the purposes of examining test-retest reliability.
Measures
Four questionnaires make up the baseline survey package: The EDE-Q, the SCOFF Questionnaire, the IOI-S and either the Marlowe-Crowne Social Desirability Scale (MC-SDS) (adults) or Children’s Social Desirability Short Scale (CSD-S) (adolescents), described in full below. The four existing surveys against which the IOI-S will be measured were chosen as the relevant diagnostic tools most frequently used in clinical settings and research trials. These will be presented in randomised order to the participants.
Baseline Package
InsideOut Institute Screener (IOI-S) (InsideOut Institute for Eating Disorders, 2018)
The six items of the new scale are displayed in Table 1. Item development followed review of the scientific literature for existing instruments screening and assessing eating disorder symptomatology, lived experience and clinical and research expert consultation. Instruments reviewed included the EDE-Q, the Eating Disorders Inventory (Garner et al., 1983) [47], the SCOFF questionnaire, the ESP, and the Eating Attitudes Test (EAT; Garner et al., 1982) [48], from which researchers developed an initial pool of ten questions covering six facets of eating pathology: an individual’s relationship with food, body, the extent to which body weight and shape determines self-worth, loss of control over eating, binge eating and compensatory behaviour. This was further narrowed by an expert consultation team to six relevant items to evaluate eating pathology and eating disorder risk in a non-clinical population. The IOI-S is rated on a 5-point Likert scale, where 1 is “never” and 5 is “all the time”; except for Question 1, where 1 is “worry and stress free” and 5 is “full of worry and stress”. Responses are summed to yield a score between 6 and 30 points total, where 6 points is the lowest degree of risk and 30 points the highest degree of risk.
Table 1. 6-item InsideOut Institute Screener
Item
|
|
|
|
|
|
1. How is your relationship with food?
|
1
Worry and stress-free
|
2
A bit problematic
|
3
Moderately problematic
|
4
Very problematic
|
5
Full of worry and stress
|
2. Does your weight, body or shape make you feel bad about yourself?
|
Never
|
A little bit
|
Sometimes
|
Quite a bit
|
All the time
|
3. Do you feel like food, weight or your body shape dominates your life?
|
Never
|
A little bit
|
Sometimes
|
Quite a bit
|
All the time
|
4. Do you feel anxious or distressed when you are not in control of your food?
|
Never
|
A little bit
|
Sometimes
|
Quite a bit
|
All the time
|
5. Do you ever feel like you will not be able to stop eating or have lost control around food?
|
Never
|
A little bit
|
Sometimes
|
Quite a bit
|
All the time
|
6. When you think you have eaten too much, do you do anything to make up for it?
|
Never
|
A little bit
|
Sometimes
|
Quite a bit
|
All the time
|
Eating Disorder Examination Questionnaire (EDE-Q) (Fairburn & Beglin, 2008)
The EDE-Q is a 28-item self-report version of the EDE structured clinical interview which is regarded as the ‘gold-standard’ diagnostic tool in eating disorders and has been validated in multiple trials. Traditionally paper-based, it has been psychometrically validated for online delivery [49]. It includes additional measures of weight, height, and missed menstrual periods, the latter of which will be excluded from our study due to the DSM-5 removal of and consequent diagnostic irrelevance of missed menstruation in ED. It is comprised of a global score and four subscales: Restraint, Eating Concern, Weight Concern, and Shape Concern, and employs a 7-point forced-choice severity rating where 0 points is the lowest severity and 6 points is the highest severity.
Eating Disorder Examination Questionnaire Short (EDE-QS) (Gideon et al., 2016)
The EDE-QS is a 12-item short form of the longer EDE-Q, designed as an abbreviated outcome measure for use on a session-by-session basis. It is highly correlated with the EDE-Q, particularly on measures of specificity and the true identification of non-cases [28, 29]. Items are drawn directly from and are identical to correlating items in the EDE-Q, and in this instance will be extracted from the administered EDE-Q for independent analysis.
SCOFF (Sick, Control, One Stone, Fat, Food) Questionnaire (Morgan et al, 1999)
The SCOFF questionnaire is a short 5-item forced choice (true/false) screening tool designed to assess eating disorder symptomatology. A threshold of >2 positive answers indicates a ‘likely case’ of AN or BN. It has shown good internal consistency and concurrent validity with the EDE-Q [50]. The SCOFF is included in addition to the EDE-QS due to the frequency with which it has been employed in clinical and research settings and the comparative newness of the latter.
Marlowe-Crowne Social Desirability Scale (MC-SDS) (Crowne & Marlowe, 1960) [51]
The MC-SDS is a well-validated 33-item self-report questionnaire measuring social desirability in adults by evaluating concern with social approval. Items are rated true/false and balanced for positive and negative wording. It is frequently used as a measure of discriminant validity in instrument design [50] and has shown good divergent validity with measures of psychopathology (anxiety, depression) related to eating disorders [51]. The MC-SDS is not validated for use in children: thus, an adapted Children’s Social Desirability Scale (CSD-S) will apply for those participants aged under 18.
Children’s Social Desirability Short Scale (CSD-S) (Baxter et al, 2004) [53]
The CSD-S is an abbreviated 14-item version of the gold-standard 48-item Children’s Social Desirability scale developed by Crandall, Crandall and Katovsky in 1965 [55]. The original scale was modelled on the MC-SDS for adults and is validated for use in a child/adolescent population [53]. It will be assigned to our 14 to 17-year-old participants. The scale uses binary response (yes/no) and is regularly used for methodological validity to detect confounding from social desirability bias. The CSD-S has demonstrated adequate internal consistency and test-retest reliability and good external validity [56].
Follow-up Package
InsideOut Institute Screener (IOI-S) (InsideOut Institute for Eating Disorders, 2018)
Follow-up involves completion of the 6-item IOI-S only.
Sample size calculation
Estimated sample size is based on 2.5-3 times the minimum sample size required for power as the outcome measure (statistical validity) is incrementally increased with participant numbers. Therefore, we anticipate a total sample size of approximately 250 participants.
Approximately 30% of participants will need to engage in the second task for sufficient validity. Estimated minimum sample size is based on the following analysis: in each group 10 participants are required for a power of 0.80 to detect an intraclass correlation coefficient of 0.70; 80 participants are required for sufficient power to detect a Spearman’s correlation of 0.40 or higher; 80 participants are required for good power to detect an Area Under Curve of 0.70, and at least 50 participants for a correlation coefficient of r +/- 0.70 for convergent and discriminant validity. All power analyses were conducted using PASS version 15 software (2017) [57].
Data analyses
Reliability
Reliability is an important feature of survey efficacy indicating the overall replicability and internal stability of a measure [58]. Measure reliability of the IOI-S will be examined on two parameters: internal consistency and test-retest reliability. The internal consistency, that is, the degree to which the six items on the scale measure the same underlying dimension, will be calculated using a standardized Cronbach's alpha score, with a minimum acceptable threshold of > .80 [59] and an item-total correlation threshold of >.40 [60, 61]. Correlations between successive iterations of the test (test-retest reliability) will be reviewed using a 2-way mixed effects model Intraclass correlation coefficient with absolute agreement, with a minimum acceptable threshold of > 0.70 [62].
Criterion Validity
The degree to which the results on the IOI-S are able to predict an outcome on established measures of eating disorder symptomatology (concurrent validity) [63] will be assessed as concordance between the IOI-S, the EDE-Q, and the SCOFF using a Spearman’s product moment correlation. Following, an Area Under Curve (AUC) analysis of Receiver Operating Characteristic (ROC) curve will be performed as analysis of IOI-S sensitivity and specificity threshold where a high sensitivity indicates accurate identification of true positive cases and high specificity indicates accurate identification of true negative cases [64]. Area Under Curve dictates the probability of identifying true positives or negatives by chance, where a value of 0.50 would suggest the measure is no more effective than guessing [65]. AUC threshold will be set at >0.70 with a 95% confidence interval.
Construct Validity
Convergent validity, or the degree to which the IOI-S correlates with eating disorder constructs and variables on both the EDE-Q and the SCOFF [66] will be measured with a correlation coefficient threshold set at r > 0.70. A correlation coefficient will also be determined for discriminant validity between the IOI-S and MC-SDS/CSD-S with a threshold of r > 0.70. A correlation coefficient of +/-0.70 indicates a strong linear relationship between two variables [67] An exploratory factor analysis will be performed on the IOI-S items using principal axis factoring and Promax oblique rotation to examine factor loading.