Figure 1 summarizes the four phases of questionnaire development. The Figure also includes how each phase related to subsequent statistical analyses of the Profile’s factor structure, internal consistency, test-retest reliability, and convergent validity.
Phase 1: Profile Item Development
Consistent with guidelines for validating novel questionnaires [36], and based on our extensive work in PWS, the authors first generated an item-pool. We reviewed data, interviews and clinical consultation notes garnered from 325 families and individuals enrolled in our current and previous PWS research programs. Our goal was to identify frequently observed behaviors, as well as less frequent behaviors that were either very challenging to manage or of high clinical concern. Our team then developed 73 Profile items that included brief descriptions of each behavior. In doing so we aimed to establish both construct validity with items that reflected behavioral and emotional problems in PWS, and content validity with items that fully represented the major domains of behavioral or emotional dysfunction in PWS.
Items were pertinent for individuals with PWS aged 5 years through adulthood. We established the 5-year-old cut-point as behaviors that characterize PWS typically emerge between 5 to 6 years of age (e.g., needs for sameness, skin-picking, temper tantrums) [37].
Phase 2: Parental and Professional Feedback
The pool of 73 items was subsequently vetted by six parents of individuals with PWS. Based on their feedback, a revised version was then reviewed by 10 specialists in PWS, including 2 psychologists, 2 clinical geneticists, 1 school consultant, 1 social worker, 3 PWS group home administrators, and 1 FDA regulatory consultant.
Parents and professionals were asked to provide feedback on the face validity of items (e.g., do items on aggression reflect aggression?), clarity of wording and item redundancy. Individuals also assessed content validity, specifically if additional items were needed to accurately depict behavioral or emotional problems in PWS [38]. Two additional items were added by the group home administrators (Creates situations to get into the hospital or attention from the police or health professionals; Disrobes when faced with intense emotional state). The revised Profile was then re-examined by the six parents, with an eye toward clarity and understandability of items.
Informed Consent
Prior to Phases 3 and 4 (pilot and large-scale studies), IRB approval was obtained from Vanderbilt University’s IRB Integrated Science Committee. For Phase 3, written, informed consent was obtained from participants using the e-consent function of REDCap (Research Electronic Data Capture), a secure, web-based software platform, designed to support data capture for research studies [44]. After consenting, parents were then invited to complete the PWS Profile on REDCap. Separate IRB approval was obtained for additional data collected from pilot participants. As described below, in Phase 4 additional Profile data were collected via the Foundation for Prader-Willi Research (FPWR) Global PWS Patient Registry. Study approval was also obtained by FPWR’s Internal Review Board. All registrants in the Global PWS Patient Registry gave consent for their de-identified data to be used for research purposes.
Phase 3: Pilot Testing
In the pilot phase, the 75-item Profile was administered to 112 parents of children and adults with PWS (see Participants). Parents were first asked to rate how frequently the behavior occurred over the past six months using a 4-point scale (0 = Almost never, 1 = Sometimes, 2 = Often, 3 = Almost always). For scores > 0, parents were then asked to rate the severity of the problem: 0 = Not a problem; 1 = Mild (annoying but easily redirected); 2 = Moderate (troublesome, needs intervention, causes disruption); 3 = Severe (highly disruptive, very distressing to individual or family). A final open-ended question invited feedback on the Profile.
Phase 3: Additional Pilot Data
Four well-established measures were administered to pilot participants to assess convergent validity and correlates of the Profile. Parents completed the Child Behavior Checklist (CBCL) [26], a 113-item measure that assesses Internalizing Problems (consisting of 3 subdomains, Depressed/Anxious, Depressed/Withdrawn and Somaticizing) and Externalizing Problems (consisting of 2 subdomains, Oppositional-Defiant and Aggression). The CBCL also includes additional subdomains: Social, Thought and Attention Problems. Items are rated as (0) not true; (1) somewhat or sometimes true; and (2) very true or often true. Raw scores were used in data analyses.
Parents also completed the Restrictive and Repetitive Behavior Scale-Revised (RBS-R [39], which assesses these behaviors in persons with ASD and other developmental disabilities in 43 items rated on a 4-point scale (0 = behavior does not occur to 3 = behavior occurs and is a severe problem). The RBS-R yields a total score, and scores for 6 subdomains: Stereotyped, Self-Injurious, Compulsive, Ritualistic, Sameness, and Restricted Behaviors.
Parents were administered the Vineland Adaptive Behavior Scales-2 [40], a widely used, semi-structured interview that yields standard scores in three domains of adaptive functioning: Communication, Daily Living Skills, and Socialization, as well as a composite score. Pilot participants with PWS were also individually administered the Kaufman Brief Intelligence Test-2 (KBIT-2) [41], which yields Verbal, Nonverbal and Composite IQ standard scores.
Phase 3: Preliminary Pilot Data Analyses and Additional Parental and Expert Reviews
Preliminary descriptive analyses of pilot data revealed that 18 items were infrequently endorsed or did not correlate well with other items. These 18 items were deleted, resulting in 57 items. The deleted items are listed in Additional File 1.
Descriptive Profile pilot results were then presented to the International PWS Clinical Trials Consortium, a team of expert PWS clinicians, researchers and parents who facilitate clinical trials in this syndrome. Based on their feedback, we simplified the response scale, which some parents in the pilot study found unwieldly. Instead, we adopted a 3-point scale: 1 = Not true or rarely true; 2 = Sometimes true or somewhat true, and 3 = Very true or often true. Doing so facilitated data analyses, reduced administrative time burden for parents, and is consistent with research on optimal scaling [42].
Consortium members also endorsed keeping seven infrequently occurring items in the Profile. These items tapped clinically significant problems in need of further evaluation from mental health professionals (e.g., suicidality, psychosis). These “Red Flag” items were not included in subsequent factor analyses. We did, however, retain them to extend the utility of the Profile as a screener for more severe mental health problems.
Phase 4: Large-Scale Profile Administration
The final, 57-item version of the Profile was administered to parents of individuals with PWS via The Global PWS Patient Registry, a secure, web-based Registry sponsored by the Foundation for Prader-Willi Research (FPWR) and hosted on the National Organization for Rare Disorders “IAMRARE” registry platform [43]. The goal of the Registry is to identify the natural history of PWS, including medical, developmental, and behavioral features that can inform treatment and future clinical trials [43]. Registrants are asked to consider their offspring’s behavior, social or medical concerns over the last six months, and to complete questionnaires every six months. The six-month time frame was established by FPWR to enhance compliance and reduce parental burden of more frequent assessments. The Registry garnered 761 respondents, and 86% completed the Profile 6 months later.
Participants
Individuals recruited from Vanderbilt University for the pilot study (n = 112) did not differ from participants in the Global Patient Registry (n = 761) in age, genetic subtypes, race/ethnicity, parental income or education or Profile scores. As such, analyses combined Profile and demographic data from both sources. Ratings from the Vanderbilt pilot study were recoded to be compatible with the final 3-point scaling used in the large-scale study (i.e., 0 = 1, 1 = 2, 2 & 3 = 3).
Participants thus included a combined total of 873 parents of individuals with PWS (48% males, 52% females) aged 5 to 61 years of age (M = 18.00 years, SD = 10.87). Most respondents (92.1%) were from the U.S. or Canada, 6.7% were from Europe or Australasia, and 9 participants (1.2%) from other regions.
As shown in Table 1, participants were predominately White, resided with their families, and had relatively well-educated parents. Most individuals had paternal deletions (53.7%) or mUPD (33.2%), 3.2% had Imprinting Defects, and 2.0% had translocations. Those with unknown genetic subtypes (8.9%) all received clinical diagnoses of PWS, but parents did not provide genetic testing results. We offset this limitation by ensuring that those with unknown subtypes did differ from their counterparts on any demographic variables or scores on the PWS Profile.
Statistical Analyses
Factor Analyses of Phase 4 PWS Profile. Exploratory principal component analyses (PCA) were conducted to determine the underlying factor structure of the Profile [45]. Although we considered confirmatory factor analyses, this approach requires a priori designation of items that theoretically tap a latent construct. On face value, some Profile items clearly lend themselves to such theoretical groupings (e.g., items tapping aggressivity). Given the complexities of the PWS phenotype, however, it remained an open question of how most Profile items would cluster together, which is best addressed in PCA [46, 47].
All factor analyses adhered to well-established criteria [45]. These included Kaiser’s criteria with an eigenvalue > 1; visual inspection of the Scree Plot to confirm the number of factors; at least 3 items loading in factors that have a common conceptual meaning; nominal cross-loading across factors; factor loadings and communalities > .40; significant Bartlett’s Test of sphericity; and a Kaiser-Meyer-Olkin measure of sampling adequacy that was close to 1.
Separate PCA’s were conducted using orthogonal (i.e., varimax) and oblique rotations (i.e., equimax), allowing us to compare the results of both to determine the most parsimonious, conceptually meaningful solution. These analyses yielded the same number of factors, and similar percentages of variance and factor loadings. As results were similar across rotations, final analyses used the orthogonal solution [46].
Internal Consistency. As the Profile is multidimensional, an overall Cronbach’s alpha was not established for the entire instrument [48]. Instead, alphas were calculated for each factor that represented a common conceptual domain.
Convergent Validity. Pearson correlations were conducted between the total Profile mean score with the CBCL Internalizing and Externalizing Domains, and the total RBS-R score. If significant, we then followed up with correlations between Profile factors and subdomains of the CBCL or RBS-R. To correct for Type II errors, only correlations ≥ .40 and with p’s < .001 are reported.
Test-Retest Reliability. Most parents in the FPWR Patient Registry, 86%, completed the Profile at Time 1, and again 6 months later. To minimize test-retest measurement error, we ensured that raters were the same across assessments and that individuals with PWS were not enrolled in clinical trials aimed at attenuating their hyperphagia and related behavior problems.
Even so, the 6-month interval is longer than typically used in test-retest analyses. As such, we first computed Intraclass Correlation Coefficients (ICCs), which incorporate both the degrees of agreement and correlations between Time 1 and Time 2 [49]. ICC analyses were based on a single measurement and absolute agreement using a two-way, mixed-effects model in which participant effects were randomized and measure effects were fixed.
Second, to capture individual fluctuations over time, difference scores were calculated between Time 1 and 2 mean scores. We reasoned that on a 3-point scale, a change of +/- 0.5 point represented a noteworthy shift. To check this assumption, we also calculated the mean of each factor’s standard deviation at Time 1 and 2 and identified participants with difference scores that were higher or lower than ½ of each factors mean standard deviation.
Profile Factors and IQ, Adaptive Behavior and Demographics. Age in pilot participants was negatively correlated with KBIT-2 IQ and Vineland Adaptive Behavior scores. As such, partial correlations controlling for age were conducted between Profile factors and KBIT-2 and Vinland scores. Using the combined large data set, t-tests, Chi-Squares, or correlations were conducted to assess relationships between Profile factors and PWS genetic subtypes, age, gender, Body Mass Index (BMI), parental SES, and region of residence.
Predictors of Anxiety. Two regression analyses were conducted in pilot participants to address lingering questions related to anxiety in PWS. In the first regression, the dependent variable was the CBCL Depressed/Anxious subdomain, and predictors included five Profile factors: Depressed, Anxious; and Distorted Negative Thinking, as well as three other factors presumed to be indicative of anxiety in PWS; Rigidity, Insistence; Compulsive Behaviors; and Repetitive Questioning, Speech [9].
Because the CBCL subdomain includes symptoms of both depression and anxiety, we conducted a binary logistic regression to home in on anxiety disorders. The dichotomous dependent variable was the presence or not of Generalized Anxiety Disorder (GAD). These diagnoses were previously obtained in our research program using the DSM 5 version of the KSADS psychiatric interview [50]; 36% of pilot participants received this diagnosis. This logistic regression determined the effects of four Profile factors (Depression, Anxiety; Compulsive Behaviors; Rigidity, Insistence; and Repetitive Questioning, Speech) on the likelihood of having GAD [51].