Design
The design of the current study was a sequential process with four-stages consisting of identifying, selecting, adapting and testing psychometric measures.
Stage 1: Review of HOP Outcome Measures
A review of Honest Open Proud (HOP) intervention studies was conducted to identify instruments that have been previously used in peer reviewed journal articles up until December 2018. Instruments were only included if: the focus of the measure was self-stigma as defined by (25); the instrument had been used as an outcome measure in the evaluation of a HOP intervention; intervention studies were published in peer reviewed academic journals. Instruments were excluded if the focus of the measure was on constructs not applicable to dementia, for example symptomatic recovery.
Stage 2: Measure Selection
Measures were selected using a combination of psychometric quality and research team appraisal.
Psychometric Quality Appraisal
The instruments identified were appraised for psychometric quality using guidance by (19), which has been used in previous research to establish the quality of psychometric instruments (26–28). A focussed search for journal articles describing the development of each identified measure was conducted and each was appraised across seven domains: a) content validity, b) internal consistency, c) construct validity, d) reproducibility (in two parts: agreement and reliability), e) responsiveness, f) floor and ceiling effects and g) interpretability.
Any instrument could score a maximum of 2 per domain and a minimum of 0. Further details of scoring procedures are described in (26). Overall quality appraisal scores were calculated by summing the scores for each domain, with a potential score range between 0 and 18. Labels were assigned to interpret the quality of the instruments based on (26) where instruments that scored 0-4 were categorised as ‘poor’ quality, 5-9 as ‘moderate’ quality, 10-14 as ‘good’ quality, and 15-18 as ‘very good’ quality.
Research Team Appraisal
In addition to the quality appraisal criteria, two experts also appraised instruments, one an expert in stigma and disability (KS), the other an older adult’s expert with specialist knowledge in the measurement of psychological constructs (GC). Collectively, decisions were made to include instruments if all three of the following criteria were satisfied:
- Instrument did not require significant changes to language that might invalidate previous psychometric findings (e.g. stereotypes and language used would be similar for a UK population);
- The instrument was deemed acceptable and relevant for a person living with dementia;
- The instrument could serve as a feasible outcome measure for an anti-stigma intervention for people living with dementia (“Who to tell, how and when?”)
Stage 3: Adaptation and Modification
Consultation with Experts
Research Experts. Five expert researchers in the field of dementia research (1- dementia prevention assessment and intervention, 2- behaviour change and intervention fidelity, 3 - positive psychological outcomes and psychometrics, 4 - mixed methods research understanding the impact of chronic health conditions, 5- psychological support for people living with dementia and family carers) were asked to review the instruments on an item by item basis. The items were sent to each expert in a word document with instructions to indicate which items were relevant to people living with dementia based on their suitability and acceptability. Once all research experts indicated their views, they were collated in an excel spreadsheet.
Lived Experience Experts. A second expert group was made up of lived experience experts (people living with dementia and carers) involved in a patient and public involvement (PPI) capacity. PPI members were split into three sub-groups of approximately 2-3, with each group supported by one researcher. The instructions were to perform a card-sorting task where all items of the selected instruments were presented on strips of paper in no particular order and had to be sorted into two envelopes labelled “acceptable” and “not acceptable”. PPI members were informed that, in order for an item to be deemed acceptable, they must feel that it is understandable, relevant and that a person living with dementia would be able to answer the question. A round-robin technique was used to elicit thoughts and discussions on items from each member of the sub-groups. This methodology outlined by (29) allows for all group members to communicate a position rather than the acceptability of items being determined by a dominant personality. The card-sorting task was designed so that each item was reviewed at least twice by two different groups.
Measure Modification Framework
A measure modification framework (30) was used to incorporate modifications from consultation with two expert groups (29,31). In the event that expert groups had conflicting feedback about the instruments, discussions between authors were used to resolve this until a conclusion was reached. The Modification Framework described by (30) increased the likelihood that adaptations to the psychometric measures would lead to items with comparable meanings, reliability and validity to that of the original measures. Three types of modifications were used based on the above expert consultation: (1) drop dimension (a dimension (subscale) is omitted), (2) drop items (items are removed from an existing scale) and, (3) modify items (substituting a term or modifying wording without changing meaning).
Stage 4: Pilot Testing
Participants
Participants were included if they: (1) were an adult over the age of 18, and (2) had a primary progressive diagnosis of dementia. Participants were excluded if: (1) they had a chronic, terminal medical condition of which they were in the later stages, (2) they had a significant sensory impairment that could not be compensated for and precluded participation, and (3) they lacked capacity to consent to the study according to established guidelines. (32,33) Ethical approval for this research was granted by the University College London Research Ethics Committee (Project: 11501/002).
Participants were recruited via three avenues: (1) researchers contacted participants who declared an interest or were matched to the study criteria on the Join Dementia Research (JDR) database, (2) self-identification where participants had heard about the research and expressed an interest in taking part (e.g. via social media and advertisements placed in local community buildings and shops), and (3) through outreach activities carried out by the researchers such as attending dementia groups (e.g. Alzheimer’s Society localities).
Measures
Selected and modified measures from steps 1 to 3 were administered alongside:
Stigma Impact Scale (SIS) to test for concurrent validity.All 21 items were rated from 1 (‘strongly disagree’) to 4 (‘strongly agree’) with the addition of 0 for ‘not applicable’ items across four subscales, namely, social rejection (9 items, e.g. “I feel others avoid me because of my impairment”), internalized shame (5 items, e.g. “I feel others think I am to blame for my impairment”) and social isolation (7 items e.g. “I feel set apart from others who are well”). As per previous research, the financial insecurity subscale was excluded (34–36).
Rosenberg Self-Esteem Scale (RSES) to test for convergent validity. This (37) consists of 10 items rated from 1 (‘strongly disagree’) to 4 (‘strongly agree’) measuring an individual’s beliefs and attitudes of themselves (e.g. “On the whole, I am satisfied with myself”).
Procedure
Potential participants were given a study information sheet and at least 24 hours to consider participating before consent was sought. Participation methods were either independently online or face-to-face data collection where one researcher (the lead author or an MSc student) conducted home visits. For the latter, participants completed the measures independently or adjustments were made if this was not possible, for example the researcher would support a participant by ticking their preferred response or providing hard copies with larger font format. Qualtrics (Qualtrics, Provo, UT) was used for online data collection, where a participant accessed the participant information sheet, screening questions, consent form and study measures through a survey link. During face-to-face data collection, these documents were presented to participants.
A subsample of participants were asked to complete the study instruments one to two weeks later (T2) in the same format in which they had completed them initially (T1).
Statistical Analysis
The Statistical Package for Social Sciences (SPSS, Version 26) was used for data input and analysis.
Acceptability and Suitability. Acceptability and suitability were ascertained using completion rates, time taken to complete T1 and floor and ceiling effects. This was due to the premise that a more acceptable and suitable instrument would yield high completion rates, have similar times of completion across measures and no floor or ceiling effects would be present. If 15% of participants achieved the highest or lowest possible scores, floor and ceiling effects were considered significant. Researchers who conducted home visits took field notes on their experience of completing the instruments to understand the acceptability and suitability of the instruments.
Reliability
Internal consistency. The internal consistency for each scale and subscales was assessed using Cronbach’s Alpha. A value for alpha ≥ 0.7 is considered acceptable (38).
Test Retest. Stability was assessed through an Intraclass Correlation Coeffcient (ICC) analysis using a two-way random effect model. ICC figures ≥.70 or above indicate stability (19,39).
Validity
Concurrent Validity. A Pearson Product-Moment Correlation Coefficient (Person’s r) was used to assess concurrent validity against the SIS. A correlation of ≥.70 was considered an indication of good concurrent validity (19).
Convergent Validity: The RSES was used to assess convergent validity as self-esteem has been previously negatively correlated with stigma experience (e.g. application of self-stigma and secrecy). It was hypothesised that a low to moderate negative correlation between self-stigma and self-esteem would be documented as per previous research (20). If at least 75% of the results are in accordance with this hypothesis, this demonstrates adequate convergent validity (19).