Quality Improvement Questionnaires – Nursing Homes (QIQ-NH): validation of questionnaires for measuring quality of care in nursing homes from various perspectives

doi:10.21203/rs.3.rs-70866/v1

Download PDF

Research article

Quality Improvement Questionnaires – Nursing Homes (QIQ-NH): validation of questionnaires for measuring quality of care in nursing homes from various perspectives

https://doi.org/10.21203/rs.3.rs-70866/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

This study aims to describe the validation and optimization of a new instrument specifically designed to measure and improve the quality of care in nursing homes; the Quality Improvement Questionnaires for Nursing Homes (QIQ-NH). This instrument comprises several questionnaires on the perceived quality of care for various perspectives (e.g. clients, family and professional caregivers) and covers eight themes of the national quality framework for nursing home care in the Netherlands.

Methods

Data was collected in six nursing homes between September 2017 and June 2018, among 359 residents, 48 family caregivers and 648 professional caregivers who completed a subgroup-specific questionnaire of the QIQ-NH. The construct and criterion validity of the three questionnaires were tested with item- and scale analyses. The content validity of the questionnaires was tested in cognitive interviews with 20 participants (7 residents, 5 family caregivers, 8 professional caregivers).

Results

Psychometric analyses confirmed the multidimensionality and reliability of the three questionnaires, and the cognitive interviews showed various possibilities for further optimization of the instruments. The construct, criterion and content validity of the three questionnaires ranged from acceptable to good. Cronbach’s alphas were > .70 for almost all scales. More than half of the items were candidate for optimization according to the cognitive interviews, mainly due to clarity or knowledge problems, and the questionnaires of the QIQ-NH were optimized accordingly.

Conclusions

The Quality Improvement Questionnaires for Nursing Homes (QIQ-NH) provides a solid basis to continuously measure and improve the quality of nursing home care, by covering the national quality themes and by integrating the various perspectives of all parties involved. With real-time feedback, the instrument enables the management and care teams to select possibilities or areas for improvement and to implement and continuously monitor the effects of quality improvement strategies in nursing homes.

Health Economics & Outcomes Research

Nursing

Quality of care

Nursing homes

Questionnaires

Validity

Cognitive interviews

Quality improvement

Quality assurance and improvement is a major and continuous challenge in nursing homes. These long-term care facilities aim to provide person-centered, safe and effective care to their residents, in order to maximize their quality of life. However the quality of care remains under pressure due to an increasing elderly population, limited resources and labour shortages that negatively affect staff capacity and responsive workforces [1]. Given these ongoing strains, the question on how to maintain or even improve the quality of care in nursing homes remains topical. In response, continuous learning and improvement is stated as the standard in the national quality framework for nursing homes in the Netherlands [2].

In order to continuously monitor and improve the quality of nursing home care, valid instruments are needed to assess relevant quality indicators and to capture the perceived quality of care from different perspectives. Besides the commonly registered safety and clinical outcome indicators (e.g. medication errors, pressure ulcer scores, urinary tract infections, injurious falls), other instruments are needed in order to provide insight into the experiences of all stakeholders involved in the caring relationship (i.e. the residents, their family and the professional caregivers) and to realize responsive care that fits the various needs.

Quality of nursing home care in the Netherlands

Recurrent questions regarding the access, quality and costs of nursing home care occupy the public debate in the Netherlands. These issues focus on which older persons will be admitted to the nursing homes, what care and living conditions will be offered, and at what costs. Budget cuts and recent reforms with the introduction of the Long-term Care Act (in Dutch: Wlz) in 2015, have resulted in more restricted access to nursing homes; only for those whose health problems are too severe and complex to remain living independently at home [3]. Eventually, nursing homes are likely to transform into hospices where terminal patients spend the final phase of their lives.

Previous research in the Netherlands showed that the care in nursing homes is often perceived as hurried, with residents who do not feel they have a say in decisions about the care they receive [3]. An earlier national study among residents and family caregivers showed that the availability of staff was a major concern, as well as the residents’ need for information, shared decision making, care for their well-being, autonomy and appropriate activities [4]. These findings are in line with international studies showing the importance of good care relationships and providing personalized care to yield positive experiences and high quality of care in long-term care settings [5–7].

National quality framework

A new national quality framework on nursing home care was introduced in the Netherlands in 2017 to provide standards and to serve as a guide for providing good quality of care in Dutch nursing homes [2], aimed at optimizing the quality of care and life of residents. This framework is based on existing literature and the experience and knowledge of various experts and stakeholders. The national standards for residential care settings for older people in Ireland [8] served as an example, though the names and operationalization of themes (i.e. underlying items) are somewhat different (e.g. ‘Learning and improving quality’ instead of ‘Effective services’). See Box 1 for the eight themes of the national quality framework of the Netherlands.

Box 1. Themes of the national quality framework for nursing home care in the Netherlands (2017)*

Theme:	Description:
1 Person-centered care	How the nursing home places residents in the centre of what they do.
2 Living and well-being	How nursing homes identify, support and promote the quality of live and well-being for residents and their informal caregivers.
3 Safety	How nursing homes guarantee basic safety for residents by following professional standards and guidelines, and through preventing and minimizing harm and learning from safety incidents.
4 Learning and improving quality	How nursing homes dynamically learn and improve to provide optimal care for residents, using best available evidence and information.
5 Leadership, governance and management	The arrangements put in place by the nursing home for accountability, decision-making, risk management and meeting its strategic, legal and financial obligations.
6 Responsive workforce	Adequate and sufficient staff with the necessary numbers, skills and abilities to respond to the needs of residents.
7 Use of resources	Using resources effectively and efficiently to deliver best achievable outcomes for the money and resources used.
8 Use of information	Actively using information as a resource for planning, delivering, monitoring, managing and improving services, and transparency of quality information for clients, family and society.
*Source: Zorginstituut Nederland, 2017 [2]

Each theme provides specific standards in order to guarantee and improve the quality and safety of services for people living in nursing homes. Four themes relate to the content or quality and safety of care (themes 1–4), and four themes relate to the preconditions for quality, such as management, resources, responsive workforce and information (themes 5–8). Learning and improving quality (theme 4) is a central theme of the quality framework. The nursing homes have the responsibility to continuously reflect and improve quality of care and measure and evaluate client experiences at least once a year in order to transparently report and work on personalized care and the well-being of their residents.

Measuring quality of care in nursing homes

Many quantitative instruments have been developed to register and monitor the quality of care and quality of life in nursing homes. A quick review of reviews highlights at least 75 internationally used instruments for measuring quality of life or satisfaction of nursing home residents as indicators of the quality of nursing home care [9]. Most of these instruments measure quality of life, either focussing on specific domains of functioning (e.g. psychological well-being or cognitive functioning) or specific conditions such as dementia. For example, widely used instruments are the dementia-specific observation instruments QUALIDEM (quality of life instrument for persons with dementia) and PGC-ARS (Philadelphia Geriatric Centre Affect Rating Scale). Roughly half of the internationally used instruments are also available or translated into Dutch, however, only a few are actually used in the Netherlands [9].

Just like other industrialized countries, the Netherlands has a long tradition of measuring the quality of care in nursing home care in a quantitative way. For example, with risk assessments for the national Prevalence Measurement of Care Problems in Care Homes [10], or by measuring client experiences in long-term care facilities with the Consumer Quality Index (CQ-index) [4]. The CQ-index for nursing homes is an instrument partly based on the American CAHPS® Nursing Home Survey [11]. Over the last decade, however, there has been a new trend in the measurement of quality of care in Dutch nursing homes. A lot of qualitative and narrative methods have been developed in order to more fully capture the construct of experienced quality of care, and to obtain more concrete clues for quality improvement. This resulted in a large number and wide variety of instruments and methods to assess and improve the quality of care in nursing homes in the Netherlands, including about 80% qualitative or mixed methods [9]. According to a national inventory in 2018, about 130 different instruments are currently being used in the Netherlands [12]. The growing awareness that quality is not just static but continuously needs discourse, maintenance and involvement of all stakeholders, has led to a wide variety of predominantly qualitative instruments. Furthermore, more dynamic instruments have been developed to invite relevant stakeholders involved to join in the measurement and improvement of quality. One of these new and frequently used instruments is the ‘Quality Improvement Questionnaires – Nursing Homes’ (QIQ-NH), which is the subject of this study.

Quality Improvement Questionnaires – Nursing Homes (QIQ-NH)

The QIQ-NH consists of several questionnaires for measuring the experiences with quality of care in nursing homes from different perspectives (e.g. residents, family and professional caregivers), with questions covering all eight themes of the national quality framework in the Netherlands (see Box 1). This instrument has been developed in 2015–2017 by Vilans, the national Centre of Expertise for Long-term Care in the Netherlands, and has not been published before. The instrument is also called the ‘Verbetermeting 360’, in Dutch, and is part of a total method for quality management called the ‘Kwaliteit Verbetercyclus’.

Considerations in developing this instrument were to design a relevant, comprehensive and useful instrument for providing real-time 360 degrees feedback to staff (i.e. care professionals and managers) in order to continuously monitor, evaluate and improve the quality of nursing home care in line with the PDCA-cycle (Plan, Do, Check, Act) for quality control of Deming [13]. With real-time feedback, the instrument enables management and care teams to select possibilities or areas for improvement, to integrate the various perspectives of different parties involved, and to implement and monitor the effects of quality improvement strategies in nursing homes.

The questionnaires have been developed for various stakeholders or different subgroups (e.g. managers and volunteers), of which residents, family and professional caregivers are the main categories of respondents. All stakeholders involved in nursing home care, including residents themselves, are invited to complete a digital questionnaire. The results are accessible in real-time via an online dashboard. Care professionals, teams and the management can subsequently use the feedback information to improve care and may conduct follow-up measurements in order to monitor and evaluate the implementation of their quality improvement efforts.

Aim of the study

This study aims to validate and optimize three QIQ-NH questionnaires that measure quality of nursing home care from three perspectives (i.e. residents, family and professional caregivers), by answering the following two research questions:

1. What is the construct, criterion and content validity of the QIQ-NH questionnaires?

2. How could the questionnaires be further optimized?

This study describes the validation and optimization of the Quality Improvement Questionnaires for Nursing Homes (QIQ-NH) for three subgroups: residents, family and professional caregivers. The questionnaires were extensively validated and optimized in a multistage approach, including both quantitative and qualitative testing, with psychometric testing and cognitive interviewing respectively. See Supplementary file 1 for the items of the questionnaires in the English language.

Quality Improvement Questionnaires for Nursing Homes (QIQ-NH)

The QIQ-NH questionnaires address six to eight themes of the national quality framework for nursing homes in The Netherlands. The themes and number of items addressed in each questionnaire depend on the applicability and the ability of each subgroup to reflect on a certain topic. Questions are formulated as propositions and answering scales range from 1 (completely disagree) to 5 (completely agree). See Table 1 for an overview of domains and number of items, and Supplementary file 1 for a more detailed description of the content of the three questionnaires. The questionnaires are not under license but part of the ‘Kwaliteit Verbetercyclus’ (in Dutch) and can be requested by contacting Vilans (third author).

Table 1. Domains and items of the Quality Improvement Questionnaires - Nursing Homes (QIQ-NH)^#

	Number of items per domain, per subgroup:
Domains:	Residents	Family caregivers	Professional caregivers
1 Person-centered care	8	8	9
2 Living and well-being	7	8	8
3 Safety	3	3	12
4 Learning and improving quality	0	0	5
5 Leadership, governance and management	1	1	5
6 Responsive workforce	3	3	8
7 Use of resources	0	0	5
8 Use of information	1	1	2

# Original versions of the three questionnaires for residents, family caregivers or professional caregivers (Vilans, 2017)

The Net Promoter Score (NPS) is also part of the QIQ-NH questionnaires. The NPS reflects the likelihood of residents, family and professional caregivers to recommend the nursing home to others (family, friends or colleagues), and is used as a global indicator for the quality of care in nursing homes in the Netherlands [2]. The NPS-question was formulated as ‘Would you recommend our nursing home to family, friends or colleagues?, with an answering scale from 0 to 10. Scores are categorized into promoters (9 or 10), passives (7 or 8), and detractors (0 to 6) [14]. To calculate the NPS, the percentage of detractors is subtracted from the percentage of respondents who are promoters: NPS = % Promoters (scores 9 or 10) - % Detractors (scores 0 to 6), resulting in a score between -100 and +100. Answers were also dichotomized into being or not being a promotor (‘yes’: NPS=9-10; ‘no’: NPS=0-8).

Data collection and study population

Data was collected with QIQ-NH questionnaires between September 2017 and June 2018, at the start of a quality improvement program in six nursing homes in the Netherlands. The questionnaires were distributed during a three-week period to convenience samples of residents, family caregivers and professional caregivers. Inclusion criteria were: living in a nursing home (residents), or providing informal or formal care to residents (family caregivers and professional caregivers, respectively). The questionnaires were distributed by e-mail to family caregivers and professional caregivers, and residents were asked to complete the questionnaire on an electronic device (i.e. laptop, notebook or tablet). At least 1,325 persons were enrolled: 458 residents, 64 family caregivers, and 803 professional caregivers received the digital questionnaire.

Analyses

Analysis of the data collected with the QIQ-NH questionnaires was performed with the statistical software of Stata 15 [15]. Subsequently, scale analyses (i.e. factor and reliability analyses, correlations) and regression analyses were used to assess the construct and criterion validity of the three questionnaires. Item-analyses were performed to assess the measurement properties of each item and to select problematic items for cognitive testing.

Construct validity: factor structure, scale reliability and inter-scale correlations

Principal Component Analysis with oblimin rotation was conducted. The factor structure of each domain was confirmed with an Eigenvalue of >1, a Kaiser-Meyer-Oklin value of >0.60 and when Bartlett’s test of sphericity reached significance (p<0.05). Factor loadings of items should exceed the threshold of >.40. To assess the reliability or internal consistency of the scales found, Cronbach’s alpha was calculated for all factors (domains or scales) of each questionnaire. In classical test theory, a Cronbach’s alpha of 0.7 or higher is indicated as cut-off for a reliable scale [16], but 0.6 is generally recommended as a cut-off value in exploratory analyses [17]. Inter-scale correlations were assessed to determine whether the scales of each questionnaire represent unique and independent constructs, with Pearson correlations being preferably between 0.3 and 0.7 [18].

Criterion validity: predictive value of each domain as an indicator for nursing home quality

Criterion validity was investigated by analyzing the relationship between the domains of the QIQ-NH questionnaires (i.e. scale scores) and the Net Promoter Score (NPS) as a global indicator for the perceived quality of nursing home care. Both univariate and multivariate regression analyses were performed with the NPS either as a continuous (0-10) or dichotomous (yes/no promotor) dependent variable. The analyses indicate whether and to what extent the scores on each domain of the QIQ-NH questionnaire determine the recommendations of nursing homes in each group. Regression coefficients (ß) were calculated and p-values of <0.05 were regarded as statistically significant.

Content validity: item analysis and cognitive testing

The content validity of the questionnaires was tested with item-analyses and cognitive interviews.

Item analyses and item selection for cognitive testing

Item- and scale analyses were determined in order to select those questionnaire items that required further testing and optimization with cognitive interviews. These analyses respectively focused on item non-response (% missings), skewness (extreme distribution of answers), overlap (inter-item correlations), and the contribution of an item to a scale.

The following criteria were used to select problematic items for cognitive testing:

Item non-response: >10% of the answers are either missing or not applicable;
Item skewness: >90% of the answers are in an extreme response category;
Item overlap: Pearson correlation between items is >0.70;
Item not fitting in or not contributing to a reliable scale (i.e. homogeneous set of items): factor loading <0.40 or Cronbach’s alpha increases if item is deleted from scale.

Cognitive interviews

Cognitive interviews were used to test and refine the problematic questionnaire items, in order to optimize the content validity of the QIQ-NH questionnaires. Cognitive interviewing is a valid and frequently-used technique in the development and testing of questionnaires [19]. It can be used to assess the participants’ comprehension of questions and to identify confusing or unclear wordings, in order to further improve the questionnaires. The interviews were conducted in July 2018, after a period of data collection, and focused on a selection of problematic items of the QIQ-NH questionnaires. Participants were recruited by four nursing homes, representing the three subgroups (i.e. residents, family caregivers, professionals), and they were interviewed in two rounds.

Two researchers (MT and JM) conducted the interviews by using a combination of two interviewing techniques: 1) the think-aloud technique, and 2) the verbal probing technique [20]. Participants were asked to read out loud each question and to describe their thoughts while answering the question. In addition, probes were used to structure the interview and to gain more information on the understanding of the questions. Examples of probes were: ‘Can you repeat the question in your own words?’ or ‘How did you get to this answer?’. All cognitive interviews were audio-recorded, transcribed verbatim, and content analyzed. The interviewers (MT and JM) coded problems independently by using the coding system of Willis [20]. This system distinguishes seven categories of questionnaire problems, see Box 2. For each problematic item, the most prominent underlying problem was coded, and sometimes multiple codes were used for one item. All data was entered in a spreadsheet to get an overview of problems per item of each questionnaire; including both qualitative (i.e. verbatim quotations, problem codes, and researchers’ comments) and quantitative data (i.e. number of problems and respondents).

Box 2. Coding system for classifying questionnaire problems of Willis (1999) [20]

Problem category:	Subcategories:
Clarity: problems with the intent or meaning of a question	wording, technical term, vague, lack of reference periods
Knowledge: likely to not know or have trouble remembering information	knowledge, recall, computation
Assumptions: problems with assumptions or underlying logic	inappropriate assumptions, assuming constant behavior, double-barreled
Response categories: problems with the response categories	missing, mismatch question-answer, vague, open-ended questions, overlapping, illogical order
Sensitively: sensitive nature or wording/ bias	sensitive content (general), sensitive wording (specific), socially acceptable
Instructions: problems with introductions, instructions or explanations
Formatting: problems with lay out or question ordering

The QIQ-NH questionnaires were optimized, based on the results of the interviews, suggestions of participants and deliberations of the research team. After the first interview round, problem codes were compared and discussed between the two interviewers (MT and JM). Subsequently, questions were adjusted and re-tested in a second interview round, with other participants, to see if these questions indeed had improved. If necessary, items were further optimized after the second interview round. All modifications were discussed and finalized after consensus was reached in the research team (MT, JM and BvdB).

Informed consent

The nursing homes gave their consent to use the anonymized data for our research purposes. All participants of the cognitive interviews provided a written informed consent, after they received written and verbal information about the cognitive testing. The study did not fall under the remit of the Dutch Medical Research Involving Human Subjects act, and ethical approval was not necessary.

Out of the 1,325 persons who were enrolled for the study, a total of 1,055 participants completed the questionnaires (response rate 80%), including 359 residents, 48 family caregivers and 648 professional caregivers.

Construct and criterion validity of the QIQ-NH questionnaires

Factor structure and scale reliability

Scale analysis of the QIQ-NH questionnaires revealed four factors with acceptable to good scale reliability for the residents (Person-centered care: α=0.87, Living and well-being: α=0.69, Safety: α=0.70, Responsive workforce: α=0.71), and the same homogenous item sets with slightly different Cronbach’s alphas for the family caregivers (Person-centered care: α=0.85, Living and well-being: α=0.77, Safety: α=0.64, Responsive workforce: α=0.83). Furthermore, factor analysis of the QIQ-NH for professionals revealed seven factors with acceptable to good reliability (Cronbach’s alpha: 0.73 to 0.88). Table 1 shows the number of items and Cronbach’s alpha for each questionnaire scale.

Table 1 Number of items and Cronbach’s alpha (α) of the questionnaires’ scales for the three groups

	Residents		Family caregivers		Professional caregivers
Domains (scales):	No. of items	α	No. of items	α	No. of items	α
1 Person-centered care	8	.87	8	.85	9	.84
2 Living and well-being	7	.69	8	.77	8	.73
3 Safety	3	.70	3	.64	12	.88
4 Learning and improving quality	0	N/A	0	N/A	5	.79
5 Leadership, governance and management	1	N/A	1	N/A	5	.79
6 Responsive workforce	3	.71	3	.83	8	.78
7 Use of resources	0	N/A	0	N/A	5	.75
8 Use of information	1	N/A	1	N/A	2	N/A

N/A=not applicable (questions did not have to be filled in by this group or Cronbach’s alpha could not be calculated)

Inter-scale correlations

Inter-scale correlations for the residents’ and professionals’ questionnaire was good, with Pearson correlations between 0.3 and 0.7 (see Supplementary file 2: Tables 1.1-1.3), confirming independent scales and unique constructs. For the family caregivers’ questionnaire, three scale scores were strongly correlated (Person-centered care and Living and well-being = 0.76; Living and well-being and Responsive workforce = 0.78; p<0.01), showing some scale overlap for the domains 1, 2 and 6.

Predictive value of each domain as indicator for nursing home global quality rating

In order to test the criterion validity of the QIQ-NH, it was assessed how well the eight scales contribute to the Net Promoter Score (NPS, i.e. recommendations by residents, family and professionals as a global quality indicator of the perceived quality of nursing home care). Univariate regression analysis, with the NPS as a continuous dependent variable (0-10), showed that all scales were significantly and strongly (ß≥0.5) related to recommendation of the nursing home in each group (see Supplementary file 2: Table 2.1).

In the multivariate model, as presented in Table 2, Person- centered care (ß=0.34, p<0.001) and Living and well-being (ß=0.25, p=0.001) remained major significant predictors of the recommendation by residents, while Safety (ß=0.49, p=0.02) appeared to be the main predictor of the recommendation by family caregivers. Furthermore, Responsive workforce (ß=0.32, p<0.001), Use of information (ß=0.20, p<0.001) and Leadership, governance and management (ß=0.18, p<0.001) were significant predictors of the recommendations by professionals, with Responsive workforce being the strongest predictor. The three multivariate regression models explained 53%, 39% and 35%, respectively.

Table 2 Multiple regression analysis of QIQ-NH scale scores^# as potential predictors of NPS (0-10); standardized regression coefficient (ß) and adjusted R2 for each multivariate regression model

		Residents (n=239)		Family caregivers (n=35)		Professional caregivers (n=579)
Domains (scales):	ß		ß		ß
1 Person-centered care		0.34***		0.17		-0.03
2 Living and well-being		0.25**		0.24		-0.02
3 Safety		0.09		0.49*		-0.04
4 Learning and improving quality		N/A		N/A		-0.01
5 Leadership, governance and management		0.11		-0.11		0.18***
6 Responsive workforce		0.04		-0.17		0.32***
7 Use of resources		N/A		N/A		0.08
8 Use of information		0.05		0.11		0.20***
Adjusted R2		0.53		0.39		0.35

^#Mean scores per scale, with different number of items and respondents per subgroup

N/A=not applicable (questions did not have to be filled in by this group)

* Significant relation with NPS (continuous score, 0-10), p<0.05

** Significant relation with NPS (continuous score, 0-10), p<0.01

*** Significant relation with NPS (continuous score, 0-10), p<0.001

Results of logistic regression analyses (see Supplementary file 2: Table 3.1 and 3.2), with the NPS as a dichotomous dependent variable (yes/no promoter), revealed similar results for each subgroup. Again, the univariate analyses showed that almost all scales were significant predictors, with Person-centered care as the strongest predictor of promotership in both residents (OR=13.68, p<0.001) and family caregivers (OR=5102.98, p=0.010). In professional caregivers, Responsive workforce was the strongest predictor of being a promoter (OR=5.35, p<0.001), followed by Person-centered care (OR=4.17, p<0.001). In the multivariate logistic model, Person-centered care was the only independent significant predictor of being a promoter in residents (OR=6.46, p<0.01), whereas none of the scales significantly predicted promotership in family caregivers. In professional caregivers, Responsive workforce (OR=3.50, p<0.001) and Use of information (OR=1.57, p<0.05) were significant independent predictors of promoting the nursing home.

Content validity and questionnaire optimization

Item analyses and item selection for cognitive testing

Results of the construct validation analyses and the item non-response analyses were used to select items for further testing with cognitive interviews. Seven out of 23 items (30%) of the residents questionnaire, 11 out of 24 items (46%) of the family questionnaire, and 12 out of the 54 items (22%) for professional caregivers had more than 10% missing values. In addition, six items (in themes 1, 2 and 6) of the questionnaire for family caregivers showed item overlap (Pearson correlation: 0.70 to 0.78). Nevertheless, no item skewness was present in all three questionnaires and the questionnaires for residents and professional caregivers showed no excessive item overlap. Eventually, 11 items (48% of 23 items) of the QIQ-NH for residents, 17 items (71% of 24 items) of the questionnaire for family caregivers and 18 items (33% of 54 items) of the questionnaire for professional caregivers met one or more criteria for further testing, and these items were selected for the cognitive interviews.

Participants and interviews in two rounds

A total of 20 participants (7 residents, 5 family caregivers, 8 professional caregivers) were interviewed in two rounds; 11 persons in the first round and nine in the second round. The mean duration of interviews was 26 minutes (range 11-48 minutes). The mean age of participants was 84 years (SD=4.9) for residents, 63 years (SD=6.3) for family caregivers, and 41 years (SD=11.8) for professional caregivers. Most participants were female (65%) and the educational level varied from low (25%) to high (35%) (see Supplementary file 2: Table 4.1).

Number and type of problems

Table 3 presents the number and type of questionnaire problems in the two interview rounds. The first round (n=11) showed a total of 114 problems for 46 questions (i.e., on average 0.23 problems per question per participant); 18 problems regarding 11 questions in three residents, 31 problems regarding 17 items in three family caregivers, and 65 problems regarding 18 items in five professional caregivers. Considering the various number of participants and questions in each group, the mean number of problems per question per participant was 0.55 for residents, 0.61 for family caregivers, and 0.72 for professionals. Problems in the first interview round mainly concerned clarity (67), and sometimes problems like (a lack of) knowledge (15), assumptions (8), sensitivity (1), or various other problems (23).

In the second round (n=9), after refining problematic questions, the total number of problems declined to 32 problems: 14 problems in 4 residents, 5 problems in two family caregivers, and 13 problems in three professionals. The mean number of problems per question per participant in the second round was 0.32 for residents, 0.15 for family caregivers, and 0.24 for professional caregivers respectively. The remaining problems concerned clarity (18), knowledge (7), assumption (1) and some other problems (6). Other problems were for example ambiguous items actually containing two questions, or items with abstract or difficult words that needed more explanation (e.g. by adding examples) or rephrasing (with concrete or simple words).

Table 3 Number and type of questionnaire problems* in the two interview rounds

	Round 1
	Residents (n=3)	Family (n=3)	Professionals (n=5)
Number of items tested	11 items	17 items	18 items
Type of problem
Number of clarity problems	14	9	44
Number of knowledge problems	4	8	3
Number of assumptions problems	0	2	6
Number of sensitively problems	0	1	0
Number of other problems	0	11	12
Total number of problems	18	31	65
Mean number of problems per item per respondent	0.55	0.61	0.72
	Round 2
	Residents (n=4)	Family (n=2)	Professional (n=3)
Type of problem
Number of clarity problems	8	0	10
Number of knowledge problems	4	2	1
Number of assumptions problems	0	0	1
Number of other problems	2	3	1
Total number of problems	14	5	13
Mean number of problems per item per respondent	0.32	0.15	0.24

* categorized according to the scoring system of Willis (1999) [20]

Questionnaire optimization

Based on the feedback from the cognitive interviews, more than half of the total number of items of the QIQ-NH were adapted; 15 of the 23 items (65%) for residents, 18 of the 24 items (75%) for family caregivers, and 21 of the 54 items (39%) for the professional caregivers (see Table 4). In addition, one item on Safety (domain 3) was deleted and one item on Leadership, governance and management (domain 5) was added to both questionnaires for residents and family caregivers. See Supplementary file 2, Box 1.1 for three examples of questions that have been adapted. The revised versions of the QIQ-NH questionnaires (October 2018) consist of 23 items for residents, 24 items for family caregivers, and 54 items for professional caregivers (see Supplementary file 2: Table 4.1).

Table 4 Results of cognitive interviewing per questionnaire: number of items adapted or deleted

QIQ-NH questionnaire for:	Residents		Family caregivers		Professional caregivers
Domains:	No. of items adapted	No. of items in final version	No. of items adapted	No. of items in final version	No. of items adapted	No. of items in final version
1 Person-centered care	3	8	5	8	1	9
2 Living and well-being	4	7	4	8	4	8
3 Safety	3^x	2	3^x	2	5	12
4 Learning and improving quality	N/A	N/A	N/A	N/A	1	5
5 Leadership, governance and management	2⁺	2	2⁺	2	4	5
6 Responsive workforce	2	3	3	3	2	8
7 Use of resources	N/A	N/A	N/A	N/A	2	5
8 Use of information	1	1	1	1	2	2
Total (% of total items)	15 (65%)	23	18 (75%)	24	21 (39%)	54

^x 2 items were adapted and 1 item was deleted
⁺1 item was adapted and 1 item was added

This study aimed to validate and optimize the QIQ-NH, a Dutch instrument to measure the quality of care in nursing homes from different perspectives, i.e., residents, family caregivers, and professional caregivers. The instrument assesses experiences with the quality of care in nursing homes for eight themes of the national quality framework [2]. The QIQ-NH appears to be a solid, comprehensive and promising instrument with an acceptable to good construct, criterion, and content validity. Nonetheless, about half of the items of the three questionnaires needed some adaptations and some items have been deleted after cognitive testing, particularly concerning the questionnaire for family caregivers. This optimization is likely to have yielded an even better, valid and reliable instrument for measuring, improving and monitoring the quality of nursing home care.

Acceptable to good psychometric properties

Testing the three questionnaires showed robust scales and independent or unique constructs, with Cronbach’s alphas varying between .69 and .88 (except for Safety in the questionnaire for family caregivers, which is no longer a scale as an item has been deleted), and Pearson inter-scale correlations between .30 and .70 (except for three domains in the questionnaire for family caregivers showing some overlap, with correlations of .76 and .78). Furthermore, each scale of the questionnaires was significantly related to the generic quality measure, the Net Promoter Score, proving criterion validity. With respect to the content validity, however, about half of the items (11 to 18 items per questionnaire) were further tested, because of excessive missing values or redundancy due to item overlap. In the cognitive interviews, these problematic items particularly showed clarity and knowledge problems, and consequently have been adapted.

The psychometric analysis of the QIQ-NH showed good internal consistency or reliability for almost all scales (Cronbach’s alpha >0.7). The internal consistency of two scales was somewhat lower, i.e. safety in the QIQ-NH for family caregivers (α=0.64) and living and well-being in the questionnaire for residents (α=0.69), but the Cronbach’s alphas were still acceptable [16-17].

The psychometric properties of the questionnaire for family caregivers were the least optimal and this questionnaire also needed most adaptations because of item-overlap, many missing values and various problems shown by the cognitive interviews. Family caregivers reported the most ‘knowledge problems’ during the cognitive interviews, compared to the other two groups, which relate to difficulties in answering a question due to a lack of information [20]. A possible explanation for this could be that family caregivers are less involved in the daily care for their relatives, and that they might be relatively unaware of the specific care process and activities in the nursing home. Consequently, family caregivers are likely to have most difficulty in rating the quality of nursing home care and it is hard to capture the experiences or perspective of family caregivers well. Nevertheless, the cognitive interviews offered many clues for item optimization and seemed to have resulted in a more valid and feasible questionnaire since the second interview round showed far less problems.

Monitoring and improving quality from different perspectives

In the past years, many instruments have been developed to assess quality of care in nursing homes in the Netherlands. However, most instruments are not validated and either focus on specific quality domains or on specific groups of residents [9]. The QIQ-NH is a relatively new instrument for evaluating the quality of care comprehensively and multi-dimensionally, from different perspectives. It enables the monitoring and evaluation of the various domains of quality of care in nursing homes, for all themes of the national quality framework in the Netherlands, and for all types of residents. It can be applied in an everyday care setting in order to continuously monitor and improve the care in nursing homes, based on real-time feedback. The instrument is increasingly being used for nationwide quality evaluations or ‘scans’ in nursing homes (in Dutch: ‘Scan Kwaliteitskader Verpleeghuiszorg’) [21]. Recent national data in particular shows potential areas of quality improvement concerning the preconditions ‘Staff composition’ and ‘Leadership’, as well as for the themes ‘Safety’ and ‘Learning and improving’ in nursing home care [21].

The results of our study confirm the capacity of the instrument to show potential areas for quality improvement. Nonetheless, residents, family caregivers and professional caregivers might differ in their evaluations and opinions regarding ‘good quality of care’. Therefore, it is important to be aware of different perspectives or preferences and to take into account the different views and opinions in the management and improvement of care. It is also important to carefully interpret the questionnaires’ outcomes, as participants might overrate or underrate the quality of care. For example, residents might overrate the quality of care due to dependence or factors such as not feeling safe nor being able to speak freely [22]. Additional conversation and in-depth interviews with the different stakeholders could help to determine how to further improve the quality of nursing home care.

Various stakeholders rate quality differently

The results of our study show that residents, family caregivers and professional caregivers are generally likely to differ in their evaluation of the quality of care. Interestingly, a varying impact of the eight themes was found in predicting the overall perceived quality of care for the three perspectives as measured by the NPS (reflecting the likelihood of residents, family and professional caregivers to recommend the nursing home to others i.e. family, friends or colleagues). Residents rated person-centered care as the most important theme for a good quality of care, whereas safety was most predictive in family caregivers and leadership and a responsive workforce were the most important predictors of quality of care. These findings do not only imply that various stakeholders might perceive or interpret quality differently, but also suggest that the experiences or assumptions underlying the overall perceived quality of care actually differ between subgroups. In other words, the concept quality of care is likely to be defined differently by various stakeholders. Future studies with the QIQ-NH could reveal whether the perceived quality of care in nursing homes and the key areas for quality improvement actually differ between subgroups.

Literature confirms the different views of various stakeholders on the quality of nursing home care. For example, results of a Dutch nationwide improvement program for the elderly showed that older adults evaluated the quality of care less positively than professionals, particularly in evaluating the person-centeredness of care [23]. This difference particularly concerned person-centered care, which seems to be of high priority for elderly when receiving care. We also found the QIQ-NH theme person-centered care to be the most important predictor of the perceived quality of care in residents. In contrast, professionals rated preconditions such as having a responsive workforce and leadership, governance and management as the most important predictors of quality of care. Other research shows that factors related to the work environment, such as team climate and collaboration between team members, contribute to the quality of care in nursing homes [24-25]. Thus, in order to improve care from a residential and professional perspective, it seems crucial to focus both on person-centered care and on preconditions of care, as these factors have been proven to be strongly related to the overall quality of care in nursing homes.

For family caregivers, the QIQ-NH theme safety was most important in predicting good quality of care. This finding is in line with the policy of the health inspectorate in the Netherlands to warrant safety as a basic prerequisite for good quality of care in nursing homes. Previous research also showed that family caregivers particularly emphasize the need to improve the safety of the living environment [4]. However, it is important to notice that the sample size of family caregivers who filled in the questionnaire was relatively small (n=48). Also, previous research indicated that assessments of nursing home care by family caregivers should be interpreted with caution as their judgement may be influenced by factors that are not related to the caregiving situation, i.e., a general opinion on nursing homes or emotional feelings such as guilt or sadness because of their loved ones living in nursing homes [26-27]. Therefore, the perspective of family caregiver on the quality of care in nursing homes must be interpreted carefully and is also a subject for further study.

Study limitations

The present study has some limitations. Firstly, the participating nursing homes also took part in a quality improvement program in the Netherlands, which means that these nursing homes had distinct quality issues. Secondly, the participants who filled in the questionnaire were not randomly selected and a selection bias might have been introduced by professionals who invited residents to participate and who distributed the questionnaires among family caregivers. In addition, since the background and response characteristics of participants were not registered, we were not able to describe the response and demographic features of participants, nor able to verify the representativeness of the data. Thus, the study setting is not likely to be representative for all nursing homes in the Netherlands and the respondents might have under- or overrated the quality of care in the nursing homes. However, irrespective of the level of quality in the participating nursing homes, this study presents important information on the underlying mechanisms of quality of care that can be used to understand and improve quality of care in Dutch nursing homes nationwide.

Future research

We expect the optimized instrument to show even more relevant, valid and reliable quality scores in future measurements. Nonetheless, future research with the QIQ-NH is necessary to determine whether the psychometric quality of the optimized and final versions of the questionnaires has actually been improved. These future studies should ensure more representative study settings and participants, with acceptable sample sizes for all subgroups, and could also confirm the wider usability and feasibility of the instrument. Finally, future research is needed to actually measure and improve the quality of care in many nursing homes, for example with comparative research or nationwide benchmark studies, and to further investigate the differences in views and ratings of various stakeholders in order to improve the quality of care from various perspectives.

In conclusion, the Quality Improvement Questionnaires for Nursing Homes (QIQ-NH) appears to be a promising and valid instrument for measuring and improving the quality of nursing home care. This comprehensive instrument is designed for measuring, improving and monitoring the quality of care in nursing homes from various perspectives, by covering the national quality themes and the views of all stakeholders involved. This study shows that the QIQ-NH has overall good psychometric quality, with an acceptable to good construct, criterion and content validity. Nonetheless, cognitive testing showed several problems in answering the questionnaires and future research should prove whether the optimization resulted in better, more valid and reliable questionnaires. Results also show that residents, family caregivers and professional caregivers are likely to differ in their values and evaluations of the quality of care in nursing homes. Future research should actually demonstrate the improved psychometric properties of the optimized questionnaires and the wider usability and feasibility of the QIQ-NH for implementing and monitoring the effects of quality improvement strategies in nursing homes.

QIQ-NH: Quality Improvement Questionnaires for Nursing Homes

Ethics approval and consent to participate: All data are collected and handled in accordance with the relevant privacy protection guidelines. According to Dutch law, the study did not fall under the remit of the Medical Research Involving Human Subjects policy and no approval by a medical ethical committee was necessary. Verbal consent was obtained from respondents who completed a questionnaire anonymously, and their response expressed their willingness to participate. Written consent was obtained from participants to the cognitive interviews.

Consent for publication: All participants to the cognitive interviews completed a consent form, stating that they were well-informed about the interviews and that they agreed upon the publication of anonymized data.

Availability of data and materials: The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: Not applicable.

Funding: This research was funded by Vilans, Netherlands Centre of Expertise for Long-term Care, Utrecht, The Netherlands. Vilans provided data for the study but had no interference in the design and analysis of the data. The third author (BvdB) is a senior advisor of Vilans and she was involved because of her specific knowledge on the application of the instrument, her responsibility for data recruitment and contacts with the participating nursing homes.

Authors' contributions: MT and JM conceived and designed the study. BvdB was responsible for data collection with the QIQ-NH and she arranged the dataset and the recruitment of subjects for the cognitive interviews via the participating nursing homes. MT and JM analyzed and interpreted the quantitative data, and also conducted and analyzed the cognitive interviews. All authors were involved in the interpretation of data of the cognitive interviews, in order to reach consensus for optimizing the instrument. MT and JM drafted the manuscript and all authors were responsible for writing the manuscript and the final approval of the version to be published.

Acknowledgements: None.

Colombo F, Llena-Nozal A, Mercier J, Tjadens F. Help wanted? Providing and paying for long-term care. OECD Health Policy Studies: OECD publishing; 2011. (https://doi.org/10.1787/9789264097759-en)
Zorginstituut Nederland. Kwaliteitskader verpleeghuiszorg. Samen leren en verbeteren. [in Dutch] Diemen: Zorginstituut Nederland; 2017. https://www.zorginstituutnederland.nl/binaries/zinl/documenten/publicatie/2017/01/13/kwaliteitskader-verpleeghuiszorg/Kwaliteitskader+Verpleeghuiszorg.pdf (Accessed 7 Aug, 2020)
Van Campen C, Verbeek-Oudijk D. Happy in a nursing home? Perceived quality of life and care of older residents of nursing and care homes. The Hague: Netherlands Institute for Social Research (SCP); 2017.
Triemstra M, Winters S, Kool T, Wiegers TA. Measuring client experiences in long-term care in the Netherlands. A pilot study with the Consumer Quality Index Long-term Care. BMC Health Serv Res. 2010;10:95.
Scheffelaar A, Bos N, Hendriks M, Van Dulmen S, Luijkx K. Determinants of the quality of care relationships in long-term care: a systematic review. BMC health services research. 2018;18(1):903.
Sion KYJ, Verbeek H, Zwakhalen SMG, Odekerken-Schröder G., Schols JMGA, Hamers JPH. Themes Related to Experienced Quality of Care in Nursing Homes From the Resident’s Perspective: A Systematic Literature Review and Thematic Synthesis. Gerontology & Geriatric Medicine. 2020;6:1-16. (https://doi.org/10.1177%2F2333721420931964)
Soklaridis S, Ravitz P, Nevo GA, Lieff S. Relationship-centred care in health: A 20-year scoping review. Patient Exp. J. 2016;3:130–145.
Health Information and Quality Authority (HIQA). National standards for residential care settings for older people in Ireland. Dublin/Cork: Health Information and Quality Authority (HIQA); 2016. https://www.hiqa.ie/reports-and-publications/standard/national-standards-residential-care-settings-older-people-ireland (Accessed 7 Aug, 2020)
Triemstra M, Francke A. Kwaliteit van leven en zorg meten. Literatuurstudie en overzicht van instrumenten. Onderdeel van het kennisprogramma Waardigheid en trots. [in Dutch] Utrecht: Nivel; 2017. https://www.waardigheidentrots.nl/wp-content/uploads/2017/04/Literatuurstudie-kennisprogramma-kwaliteit-.pdf (Accessed 7 Aug, 2020)
Halfens RJG, Meesterberends E, Rijcken S, Schols JMGA, Wolters S. Report on the results of the national Prevalence Measurement of Care Problems in Care Homes 2015 [in Dutch: Rapportage Resultaten Landelijke Prevalentiemeting Zorgkwaliteit 2015] Maastricht: Maastricht University; 2016.
Sangl J, Buchanan J, Cosenza C, Bernard S, Keller S, Mitchell N, Brown J, Castle N, Sekscenski E, Larwood D. The development of a CAHPS® instrument for nursing home residents (NHCAHPS). Journal of aging & social policy. 2007;19(2):63-82.
ActiZ (2018). Instrumenten Cliëntervaringsonderzoek: geen waaier, wel set van eisen. [in Dutch] Utrecht: ActiZ; April 2020. https://www.actiz.nl/nieuws/web/ouderenzorg/open/2018/04/instrumenten-clientervaringsonderzoek-geen-waaier-wel-set-van-eisen (Accessed 7 Aug, 2020)
Deming WE. Out of the Crisis. Cambridge, MA: MIT Press; 1986. p. 88.
Krol, MW, De Boer D, Delnoij DM, Rademakers JJ. The Net Promoter Score. An asset to patient experience surveys? Health Expec. 2015;18(6):3099-109.
(2017). Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC.
Nunnally J, Bernstein I. Psychometric theory. New York: McGraw Hill; 1994.
George D, Mallery P. SPSS for Windows Step by Step: A Simple Guide and Reference. 0 Update. Boston, MA: Allyn & Bacon; 2003.
Ferketich, S. Focus on psychometrics. Aspects of item analysis. Res Nurs Health. 1991;14(2):165-8.
Buers C, Triemstra M, Bloemendal E, Zwijnenberg NC, Hendriks M, Delnoij DM. The value of cognitive interviewing for optimizing a patient experience survey. Int J Soc Res Methodol. 2014;17(4):325-340.
Willis, G. Cognitive interviewing a ‘how to guide’. Reducing survey error through research on the cognitive and design processes in surveys. Short course presented at the 1999 meeting of the American Statistical Association. Rockville: Research Triangle Institute; 1999.
Voortgangsrapportage Programma Waardigheid en trots op locatie. [in Dutch] Utrecht: Vilans; 2020. https://www.waardigheidentrots.nl/wp-content/uploads/2020/06/Vierde_voortgangsrapportage_Thuis_in_het_Verpleeghuis.pdf (Accessed 7 Aug, 2020)
Bomhoff M, Paus N, Friele R. Niets te klagen. Onderzoek naar uitingen van ongenoegen in verzorgings- en verpleeghuizen. [in Dutch] Utrecht: Nivel; 2013.
Verweij LM, Wehrens R, Oldenhof L, Bal R, Francke AL. Perspectives of nursing professionals and older adults differ on aspects of care for older people after a nationwide improvement program. BMC Health Serv Res. 2018;18:321.
Anderson RA, Issel LM, McDaniel RR. Nursing homes as complex adaptive systems: relationship between management practice and resident outcomes. Nurs Res. 2003;52(1):12-21.
Backhaus R, Rossum EV, Verbeek H, Halfens RJ, Tan FE, et al. Work environment characteristics associated with quality of care in Dutch nursing homes: A cross-sectional study. Int J Nurs Stud. 2017;66:15-22.
Grau L, Teresi J, Burton B, Chandler B. Family members’ perception of the quality of nursing home care. Int J Geriatr Psychiatry. 1995;10(9):787-96.
Finnema E, De Lange J, Dröes RM, Ribbe M, Van Tilburg W. The quality of nursing home care: do the opinions of family members change after implementation of emotion-oriented care? J Adv Nurs. 2001:35(5):728-40.

Supplementary files 1 and 2 were not provided with this version of the manuscript.

Download PDF

Review #2 received at journal
19 Dec, 2020
Editorial decision: Major revision
19 Dec, 2020
Review #1 received at journal
13 Dec, 2020
Reviewer #2 agreed at journal
17 Nov, 2020
Reviewer #1 agreed at journal
17 Nov, 2020
Reviewers invited by journal
22 Oct, 2020
Editor assigned by journal
15 Oct, 2020
Submission checks completed at journal
14 Oct, 2020
Editor invited by journal
14 Oct, 2020

You are reading this latest preprint version

Quality Improvement Questionnaires – Nursing Homes (QIQ-NH): validation of questionnaires for measuring quality of care in nursing homes from various perspectives

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Background

Quality of nursing home care in the Netherlands

National quality framework

Measuring quality of care in nursing homes

Quality Improvement Questionnaires – Nursing Homes (QIQ-NH)

Aim of the study

Method

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Supplementary Information

Status:

Version 1