Details of the consensus-based development of the CAPTURE-JIA PROM and PREM questionnaires have been described elsewhere. (23) In brief, the PROM comprises three core themes; physical, social and emotional wellbeing. All questions relate to a four-point response scale ranging from never (score = 0) to most of the time (score = 3). Questions relate to the past month. The PREM refers to the patient’s experience of the clinical encounter and encompasses the following themes; communication, information/education, environment and access/coordination of care. Response scales devised for questions 1 to 4 range from not at all (score = 0) to fully (score = 3). The response scale for question 5, addressing appointment delay, ranges from no unacceptable delay (score = 0) to unacceptable > 2 hour delay (score = 5). There are two versions of both PROM and PREM; questionnaires for CYP aged < 11 years are completed by the parent/carer whilst questionnaires for CYP aged 11 years or over are completed by the patient.
Validation included both quantitative and qualitative approaches in accordance with the OMERACT filters. Part One of the study consisted of cognitive interviews eliciting opinion from study participants and Part Two included participant completion of the CAPTURE-JIA PROM and PREM questionnaires. The Child Health Utility 9D (CHU 9D), (25) a validated measure capturing similar themes to those included on the PROM, was used as a reference measure for the PROM. The PREM is a unique tool, there are no JIA-relevant patient experience tools for comparison.
The study complies with the Declaration of Helsinki, the locally appointed ethics committee approved the research protocol [National Research Ethics Committee East Midlands-Leicester IRAS 212656] and informed consent was obtained from all subjects (or their legally authorized representative).
Part One
Study population
A convenience sample of CYP with a confirmed diagnosis of JIA attending paediatric rheumatology clinic between September and November 2017 were invited to participate in the study. Children < 1 year of age and families not fluent in English were excluded.
Data collection
Three rounds of cognitive interviews were conducted in a private room in the paediatric rheumatology clinic by an experienced female qualitative research assistant (post PhD). Interviews lasted an average of twenty minutes and sampling continued until data saturation was achieved. Think aloud techniques were used to elicit opinions on the PROM and PREM questionnaires, with parents/patients (as relevant depending on their age) invited to read each question aloud, explain their understanding of the question and describe any areas which they felt lacked clarity. Areas identified as unclear were probed in detail and parents/patients asked to suggest improvements. At the end of the interview, families were encouraged to identify any additional and relevant topics or issues for discussion. Interviews were audio-recorded with the participants consent, transcribed (and edited to ensure anonymity of respondent), and transcripts formed the data subjected to formal analysis. Data were analysed qualitatively by one experienced researcher and conducted according to the standard procedures of rigorous qualitative analysis, (26) using procedures from first-generation grounded theory (coding, constant comparison, memoing), (27) from analytic induction (deviant case analysis) (28) and constructionist grounded theory (mapping). (29) Data collection and analysis occurred concurrently, so that issues raised in earlier rounds of fieldwork could be explored subsequently.
Reflexivity was maintained by the research team throughout analysis and writing, by recording, discussing and challenging established assumptions. Joint first author NS conducted and analysed all interviews. Although she has a wide range of experience with JIA families, she was not known to the participants of this research prior to undertaking the study and was based in an external setting. This ensured she held no preconceptions in relation to health service delivery and gave participants the opportunity to discuss their thoughts without any potential influence from their care team.
Part Two
Study population
CYP with a confirmed diagnosis of JIA attending paediatric rheumatology clinic at the Great North Children’s Hospital in Newcastle or Royal Manchester Children’s Hospital in Manchester between September 2017 and February 2018 were eligible for inclusion. Children < 1 year of age and families not fluent in English were excluded.
Data collection
Participants were asked to complete the PROM and CHU 9D in the hospital waiting area before the clinical consultation and to complete the PREM after the consultation had taken place. A subset of recruited participants were asked to complete the PROM and PREM one week later at home, returning the completed forms to the research team in a stamped addressed envelope. Participants were recruited over a period of six months. All data were stored at the University of Manchester in accordance with data governance regulations.
Statistical analyses
The OMERACT filter was applied to assess three core domains of measurement validation; truth, discrimination and feasibility. (30, 31) The PROM was validated against the CHU 9D at each stage of validation with the exception of ‘medication side effects’ which is not captured on the CHU 9D. For the majority of the validation techniques, raw scores of each measure were used. For sensitivity and specificity assessments, outcome scores were dichotomised to high (‘often’ and ‘most of the time’ on the PROM; ‘quite’ and ‘very’ or, ‘many’ and ‘I can’t’ on the CHU 9D) and low scores (‘never’ and ‘sometimes’ on the PROM; ‘I don’t’, ‘a little bit’ and ‘a bit’ or, ‘no problems’, ‘a few problems’ and ‘some problems’ on the CHU 9D). Since the PREM is a unique tool, assessment against the OMERACT filter was limited.
Truth domain
This first domain of the OMERACT filter assesses whether each criterion is measuring what it is intended to measure, in an unbiased way. It encompasses face, content, criterion, and construct validity.
Face validity (PROM and PREM)
aims to provide evidence the criteria included on the measure is sensible, relevant and comprehensive. Can the CYP/parents completing the measure understand the criteria and what is being asked? Are the themes important and do they address areas of important relevance to the CYP/parent?
Content validity (PROM and PREM)
do CYP/parent understand the questions correctly and provide answers using suitable rating scales? Qualitative analysis of the cognitive interview transcripts provided insight into what CYP/parents understood by the questions. Within the cognitive interviews, CYP/parents were further given the opportunity to comment on the relevance of the themes within the questions and identify items important to them personally.
Criterion validity (PROM)
investigates whether patients are classified in the same way by the new measure as a previously validated measurement tool capturing the same or similar constructs. This was tested by assessing the sensitivity of the PROM in identifying high symptom levels in each domain versus high symptom levels in corresponding CHU 9D domains.
Construct validity (PROM)
to assess how well each PROM criterion measures the intended underlying constructs, we evaluated convergence with similar criteria on the CHU 9D. (Spearman correlations.)
Discrimination domain (PROM and PREM)
Classification validity
Can each criterion on the PROM identify whether or not a patient has the symptom of interest? Specificity of low scores on the PROM were tested against low scores on the CHU 9D. Area under the curve (AUC) analyses using receiver operating characteristics enabled determination of levels of distinction for classifying high from low symptom levels.
Reproducibility of results
Test re-test reliability used linear-weighted kappa coefficients to assess the strength of agreement of the ordinal scores completed one week apart.
Feasibility domain (PROM and PREM)
This element of the OMERACT filter assesses how easily the measures can be applied in the intended environment. (A hospital waiting area before and after the clinical consultation.) Time taken to complete the PROM, PREM and CHU 9D was recorded using a stopwatch and the proportion of participants completing each item on the PROM, PREM and CHU 9D was calculated. A cut-off of 80% completion was selected for data items to be considered feasible in the clinical environment.