A mixed method design was used, integrating both quantitative and qualitative data collection. A focal point of this follow-up study was the evaluation of the content validity, including item reduction, of the KOPEQ. According to COSMIN guidelines, content validity covers the aspects of relevance, comprehensiveness and comprehensibility of a questionnaire. The steps of the process are described below in successive phases, using qualitative and quantitative methods within an integrated approach.
The study used a cross-sectional design and was embedded in the clinical practice of an orthopedic department of the cantonal hospital in Winterthur, Switzerland. Patients with severe knee OA, scheduled for a KR operation, were offered an educational intervention as part of the hospital’s routine care. This routine intervention was developed in our previous study (9). The aim of the educational intervention is to impart practical knowledge to the patient through providing the following information: information and illustrative material on the anatomy of the knee and adjacent functional structures; recommendations on activities with a prothesis; information on postoperative pain management; and, details of the postoperative rehabilitation phase. Several authors recommend the use of targeted, easily understandable educational material to improve patients’ self-management, such as worksheets, handouts, presentations, photos and videos. The information provided must be prioritized according to importance, illustrated, conveyed in plain language and use short sentences without medical terms (15-19). The educational intervention applied in this study consisted of two sessions (with one-week interval) and incorporated these recommendations.
Inclusion criteria were: proficiency in both spoken and written German: scheduled for KR surgery based on OA; under treatment at the cantonal hospital in Winterthur; and, a Mini-Mental State Examination score of greater than 24/30 (20). During the first educational session, eligible participants were informed about the research project and, in addition, handed written documentation. One week later, following the second educational session, patients were asked whether they would agree to participate in the research study.
Those eligible patients agreeing to participate in the study completed the revised KOPEQ some six to ten weeks after their KR surgery, having previously attended the two patient education sessions. During these same sessions, participants were also asked to complete the patient-testing (interview) protocol (IP) and to undergo the Short Test of Functional Health Literacy (S-TOFHLA) (21).
The intention of the KOPEQ is, from the patients’ perspective, to measure the understandability of a pre-operative educational intervention and to assess its actionability. Within the theoretical field of health literacy, patient education materials are described as understandable, “When consumers of diverse backgrounds (patients) and varying levels of health literacy can process and explain key messages” (22). Patient education materials are described as actionable, “When consumers of diverse backgrounds and varying levels of health literacy can identify what they can do based on the information presented” (22).
The further development of the KOPEQ was executed in two phases in this study: 1) a reduction in the number of items on the initial 16-item KOPEQ; and 2) psychometric testing of internal consistency, factor analysis and construct validity. Reduction in the number of KOPEQ items was considered necessary because some placed too much focus on practical aspects (such as the number of intervention sessions). This was seen as, potentially, an impediment to the implementation of the KOPEQ in other contexts.
Phase 1: Reduction in the number of items on the initial 16-item KOPEQ
We used the COSMIN methodology for assessing the content validity of patient-reported outcome measures (PROMs) as a methodological guide (23) in reducing the number of items. This guide includes ten criteria concerning the three aspects of content validity of relevance, comprehensiveness and comprehensibility. In this phase the number of items on the KOPEQ was reduced from 16 to 12.
The impact of the reduction of KOPEQ items was subsequently tested by an exploratory factor analysis (EFA).
Phase 2: Hypotheses testing of the 12-item KOPEQ
The construct validity of the reduced KOPEQ was compared to an alternative measure with a comparable construct. An interview protocol (IP), tailored to the same patient educational material, was specifically developed for this study, according to comparator measures presented in a study by Shoemaker et al. (22). The comparisons were based on a priori hypotheses regarding the strength and direction of the correlation of the constructs of the index and comparator measures. The primary correlation studied was between the sum score of the reduced KOPEQ to the sum score of the IP. The secondary correlations of interest were between the reduced KOPEQ and the Short Test of Functional Health Literacy (German version) and the Mini-Mental State Examination score.
- Interview protocol (IP)
A patient-testing IP tailored to the same patient educational material (22), based on a mixed methods approach.
- Before the interview, patients are asked to randomly read or view selected material from the patient educational material
- The interviewer then asks a set of questions to investigate the patient’s understanding of the content of the material (understandability) and the extent to which they know what action to take (actionability). Patients could refer to the material as much as needed before answering the questions.
- There are 4 types of questions in the protocol:
- Comprehension questions
- Numeric-scoring questions (scale 1-10, how easy was the material to understand and/or to action)
- Open-ended opinion questions
- Questions that asked patients to describe what information was given in each session, or what a visual aid was showing
- Short Test of Functional Health Literacy (S-TOFHLA)
The S-TOFHLA (21) measures a patient’s ability to read and understand health-related materials. It consists of 40 items: 36 testing reading comprehension and four testing numeracy. Three aspects are measured:
- Reading comprehension is assessed by a reading test using two health-related passages. Both passages have every 5th to 7th word deleted, and, for each blank word, the participant must select from a list of four words the ones that best complete the sentence. The numeracy test assesses the ability to read and understand numerical information in the form of prescription medication, appointment slips or other health-related material. The items are selected based upon their perceived importance and frequency of the task in the health-care setting.
- The total time needed to complete the S-TOFHLA
- The number of points after seven minutes.
The Swiss, German-language validated version was used.
- Mini-Mental State Examination (MMSE)
The MMSE (20) is a widely-used test for screening cognitive functions. It includes tests of orientation, attention, memory, language and visual-spatial skills.
It has been suggested that reading ability may deteriorate with age and have an influence on health literacy (24-28). Therefore, age (measured in years) might be a confounder of the association between the KOPEQ and the IP.
For research questions regarding validity, a minimum number of 50 persons for an appropriate sample size is recommended (29). In this study, we followed this recommendation in our choice of sample size. A formal sample size calculation was not performed.
Phase 2 Statistical analysis
- Comparison of the constructs of the reduced KOPEQ with the IP.
- Comparison of the constructs of the reduced KOPEQ with the S-TOPHLA and the MMSE.
Descriptive statistics were used to describe the population for all collected variables.
The internal consistency of the KOPEQ was investigated using Cronbach’s alpha (30) and a score above 0.70 was set as an indicator of sufficient reliability (14).
Exploratory factor analysis (EFA) was performed to assess the dimensionality of the KOPEQ.
To quantify the construct validity by testing the preset hypotheses, all pairwise correlations were estimated in an exploratory manner. Firstly, the KOPEQ sum score was correlated with the IP sum score. Secondly, the KOPEQ was correlated with the other comparators (S-TOFHLA 1-3 and MMSE). We computed Pearson’s correlation coefficient with bootstrapped 95% CI using Fisher transformation. Lastly, the correlations were adjusted for the potential confounder of age. In addition to the Pearson’s correlation, a Bayesian estimation of the main quantity of interest (the correlation between KOPEQ and IP) was performed by using uninformative priors. We used Gibbs to sample from the posterior distribution. A 95% Highest Posterior Density (HPD) interval for this correlation was constructed.
All analyses were performed using the R statistical software R version 3.5.2 (2018-12-20) (31). For Bayesian analysis, we used JAGS (Just Another Gibbs Sampler).
A post hoc analysis, based on the qualitative information from the patient interviews, was optional for practicability reasons.
Data storage and protection
Data was handled and stored confidentially on the ZHAW server and coded according to the rules of the Institute to protect the privacy of the participants. The principal investigator (PI) (EOH) kept the key of the code safeguarded. Only the PI had access to the code. The PI and two members of the project team (epidemiologist and statistician) had access to the data.