The Physical Therapy Efficacy Index and Chart: a stimulus for value-based healthcare using real-world data. A mixed methods study

doi:10.21203/rs.3.rs-4696768/v1

Download PDF

Research Article

The Physical Therapy Efficacy Index and Chart: a stimulus for value-based healthcare using real-world data. A mixed methods study

https://doi.org/10.21203/rs.3.rs-4696768/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background
Health care costs are rising rapidly in Western societies. Understanding the benefits and costs of care is crucial to maintaining or improving existing health care systems. We propose an instrument that provides a clear overview of both the costs and returns of a treatment to improve the quality of care while keeping the costs affordable.

Methods

First, a general value-based healthcare concept was developed as an efficacy index. Second, a Physiotherapy-specific Efficacy Index (PE-_Index) for musculoskeletal disorders was formulated based on pain and functional improvement, treatments, and episode duration. The PE-_Index 's discriminative value was assessed using a linear mixed model with physiotherapy practices as a random effect in real-world data from a national registry. Variation attributed to practices was quantified by an intraclass correlation coefficient. Separate linear mixed models and a radar plot (PE-_Graph) visualized individual PE-_Index components. Lastly, stakeholders evaluated the PE-_Index and PE-_Graph for internal quality improvement and external transparency through surveys and advisory board meetings.

Results

In total, 95.805 episodes treated in 370 practices were included in the linear mixed models. The PE-_Index demonstrated an adequate discriminative ability with an ICC of 0.118.

Stakeholders agree that the PE-_Index and the PE-_Chart are appropriate for improvement of quality of care and enhancing the current system for external transparency. Nevertheless, because of concerns about a too hasty implementation and the risk of strategic gaming, both were not considered suitable for external transparency right now.

Conclusions
The PE-_Index and PE-_Graph are adequate instruments to discriminate between practices and can be used for internal quality improvement, however, are not yet suitable for external transparency purposes.

In the Western world, healthcare costs are rising rapidly. In the years prior to the COVID-19 pandemic, the total costs of healthcare in the Netherlands, as calculated according to the international definition of the System of Health Accounts, experienced a 15% increase from 2015 to 2019, reaching a total of 82 billion euros (1)

To effectively manage increasing healthcare costs without compromising the quality of care, the application of cost-effectiveness analyses can be helpful. However, cost-effectiveness analyses typically compare interventions, and they do not adequately consider the extensive number of and variation between health care providers (2). These analyses encompass crucial factors that impact the quality of care, such as the physical environment of the care setting (e.g., cleanliness and comfort), the availability of medical equipment, the organization of interprofessional care, and the expertise of the treating therapist or doctor.

Given these issues, stakeholders, including patient organizations, therapists, health insurers, and policymakers, have recognized the need for more comprehensive tools, in line with the principles of value-based health care (3), that enable a thorough analysis of costs and outcomes in care delivery. Such a tool could enhance the quality of care in relation to costs and facilitate peer learning among practices and therapists.

One prominent factor contributing significantly to escalating healthcare expenditures is the widespread prevalence of musculoskeletal complaints. Globally, these complaints are widely acknowledged as the foremost drivers of disability-adjusted life years (4). Given that musculoskeletal complaints are predominantly managed within private physical therapy settings, this setting emerges as the most suitable context for initiating the development of a tool designed to assess the cost-effectiveness ratio.

Consequently, we conducted this study aimed at

a) Developing a physical therapy efficacy index (PE-_Index) and an associated chart (PE-_Chart). These tools will evaluate the quality of physical therapy in relation to costs in private physical therapy practices, initially focusing on musculoskeletal complaints in the Netherlands. These tools will incorporate treatment outcomes, treatment frequency and duration, and patient characteristics. We will assess the discriminability of the PE-_Index tool among Dutch physical therapy practices.
b) Furthermore, we evaluate the appropriateness of these tools for internal quality improvement and external transparency among relevant stakeholders. The stakeholders included patient associations, physical therapists, professional bodies, and health insurers.

This study was structured as an explanatory sequential mixed methods design.

Part 1. Data collection

The data utilized for investigating the feasibility of a PE-_Index (Physical therapy Efficacy Index) and PE-_Chart (Patient Efficacy Chart) were extracted from the Dutch National Data Registry (LDK) of the Association for Quality in Physical Therapy (SKF), which contains real-world data provided via electronic health records. These electronic health records included patient and episode characteristics, such as the course, duration, and number of treatments, as well as the results of tests to measure and monitor the severity of the problems.

To construct the PE-_Index and PE-_Chart, Two Patient Reported Outcome Measures (PROMs) were employed, namely the Numeric Pain Rating Scale (NPRS) and Patient-Specific Functioning Scale (PSFS). The NPRS measures pain intensity on a scale from 0 (no pain) to 10 (worst imaginable pain) (5), while the PSFS assesses the ability to complete an activity at a level experienced prior to injury or change in functional status using an 11-point scale (6). These PROMs have good measurement properties, are frequently used in clinical practice and are recommended in the Dutch guidelines for physical therapy for musculoskeletal disorders (7).

The data were collected by physical therapists during standard care through their electronic health records and classified using a four-digit coding system based on body location and pathology (8)(Table A1 in the appendix). Under Dutch law, the use of electronic health records for research purposes is permitted under certain conditions. When these conditions are met, neither obtaining informed consent from patients nor approval by a medical ethics committee is required for observational studies containing no directly identifiable data (art. 24 GDPR Implementation Act jo art. 9.2 sub j GDPR; Dutch Civil Law, Article 7:458).

Part 2. Development of the physical therapy efficacy index and chart

The PE-_Index and PE-_Chart were developed and tested through a four-step process that included 1. conceptualization, 2. development of the tool, 3. discriminative value between practices, and 4. calculation of individual practice scores and charts.

1. Conceptualization

First, a general conceptualization of the value-based health care principle of Porter & Teisberg (2006) was formulated as an efficacy index. In this index, the health outcomes are divided by the cost of the episode. In this division, health outcomes were defined as meaningful patient-reported outcome measures (Eq. 1).

$$\:EI:\:\frac{\varDelta\:\:Meaningful\:patient\:reported\:outcome\:measures}{Episode\:cost}\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:1)\:\:$$

2. Development

Second, a physical therapy-specific efficacy index (PE-_Index) was formulated for patient episodes of a musculoskeletal nature. Musculoskeletal disorders comprise a diverse range of conditions (appendix 1). The outcome measures were defined as the difference in PROMs (NPRS and PSFS) between baseline and at the end of every individual episode. The episode costs were considered to consist of two parts, namely, the financial costs (episode price) and the costs expressed in the burden for the patient (duration of the treatment episode). The treatment episode price was defined as the number of treatments (N treatments) instead of the actual treatment cost since the price of all treatments was approximately equal. The treatment episode duration was defined as the total duration of the trajectory per two weeks (intake-end).

The constants C1, C2, C3, and C4 are added to ensure that the PE-_Index ranges between 0 and 1, and the outcome is multiplied by 100 for easier interpretation (Eq. 2). The PE-_Index formula is as follows:

$$\:\frac{-\varDelta\:\:NPRS-\varDelta\:\:PSFS+\text{C}1\:\left(20\right)}{\left(N\:treatments-\:\text{C}2\:\left(3\right)\right)+\:\left(Episode\:duration-\:\text{C}3\:\left(1\right)\right)+\text{C}4\:\left(40\right)\:}\text{*}100\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:2)$$

where:

∆ NPRS: the change in the Numeric Pain Rating Scale (NPRS) score from baseline to the end of the episode
∆ PSFS: the change in Patient-Specific Functioning Scale (PSFS) score from baseline to the end of the episode
N treatments: the number of treatments received during the episode.
Episode duration: the total duration of the treatment episode, per two weeks, rounded up.
C1: the summation of the maximal outcomes of ∆ pain reduction and ∆ function improvement (set to + 20 to ensure a positive numerator)
C2: the summation of C1 and the maximum outcome of ∆ pain reduction and ∆ function improvement (set to 40 to ensure a PE-_Index score between 0 and 1)
C3: set to 3, based on the minimal value of N treatments.
C4: set to 1 based on the minimal value of episode duration.

3. Discriminative value

In the next step, the discriminative value of the PE-_Index between physical therapy practices was evaluated using linear mixed models. This value was compared to the discriminative value of the four components of the PE-_Index, namely, the delta NPRS, delta PSFS, number of treatments and episode duration. Additionally, the effect of adding explanatory variables was assessed. In total, ten linear mixed models were created, consisting of five intercept-only models and five adjusted models. The physical therapy practices were added as a random effect.

In the adjusted models, the following variables were included as main effects: age, sex, socioeconomic status, chronicity, classification code, number of treatments, and episode duration (in weeks). The patient's socioeconomic status was calculated using their postal code (9). Chronicity was expressed as the duration of the complaint before the patient sought help. The classification code is an administrative coding of the location of the complaint, e.g., low back pain, shoulder pain, etc. The continuous characteristics (age, baseline NPRS and PSFS) were converted to z scores. The characteristics were tested for collinearity by calculating Pearson’s correlation coefficient. If the correlation coefficient exceeded 0.8, only one of the characteristics was included in the analysis.

Episodes were included if 1) the disorder was classified as musculoskeletal, 2) the trajectory started after January 2018 and ended between January 2020 and January 2021, or 3) the episode consisted of at least three treatments. Episodes were excluded when sex, age, postal code or the NPRS or PSFS measurements at the start or at the end of the episode were missing. Moreover, episodes were excluded in the case of a new treatment for a recurrent condition in the same practice. Finally, outliers in the number of treatments or episode duration were identified and excluded by converting the outcomes of these variables into z scores, where a score greater than three or less than three was marked as an outlier (10).

The outcome of interest of the linear mixed models was the amount of variance caused by the random effect and thus the variance in outcome caused by the practices. This amount was estimated with the intraclass correlation coefficient (ICC), which can be calculated by dividing the variance of the random effect by the total variance. The total variance is the sum of the variance between practices and within practices. An ICC > 0.1 can be interpreted as an adequate measure to discriminate between practices.

The data were processed using Python (version 3.7.4), and linear mixed models were created with the python module Statsmodels (v0.13.2).

4.Individual PE-_Index and PE-_Chart

In the final step we calculated the corrected PE-_Index and its four components, namely, the delta NPRS, delta PSFS, number of treatments and episode duration, for all episodes using adjusted linear mixed models and aggregated them per practice. Next, these outcomes were converted to z scores and visualized, thus creating a radar plot with 5 factors, known as the PE-_Chart. This method enabled comparison of all corrected outcomes of an individual practice to the average of all practices or to a specific practice.

Part 3: Evaluation of the PE-_Index and PE-_Chart with Stakeholders

In phase three, we evaluated the PE-_Index and associated chart’s appropriateness for ‘internal’ quality improvement in SKF’s learning health system, which includes peer learning sessions among physical therapists, management reports for practice executives, and practice visits, and ‘external’ transparency, such as reimbursement choices and patient decision aids, and as a comparison tool among care disciplines. We gathered data through stakeholder surveys and advisory board meetings, with input from physical therapists, patients, insurance companies, and statisticians.

Survey

The appropriateness for internal quality improvement was evaluated for the PE-_Chart, and the appropriateness for external transparency was evaluated for both the PE-_Index and PE-_Chart. Participants were asked to rate the index and chart with the following question: “To what degree do you think the PE-_Index and PE-_Chart are appropriate for this specific goal?”, using a 9-point Likert scale, in which 1 was “Absolutely not appropriate” and 9 was “Absolutely appropriate”. In this study, we categorized the questions into three distinct groups. The first category addressed the utilization of the PE-_Index for internal quality purposes, specifically focusing on peer learning sessions without any form of judgment. The second category pertained to external transparency and consisted of two subcategories: one for the PE-_Index and the other for the PE-_Chart. Within these subcategories, we explored the suitability of the tools for reimbursement choices, their effectiveness as decision aids for patients, and their potential to facilitate comparisons between different health care professions.

Insurance companies currently employ a "treatment index" as a guiding metric for their reimbursement policies. This index is based on the mean number of treatments stratified according to subgroups within the patient population (11). In the latter category of our study, we inquired explicitly whether the respondents perceived an added value of the PE-_Index in comparison to the existing treatment index.

Table 1

Survey questions.
To which degree do you think…	Abbreviated as
Internal quality Chart
…the result chart is an appropriate instrument to learn and improve during a practice assessment?	Practice assessment
…the result chart is an appropriate instrument to learn and improve during peer learning sessions?	Peer learning
External transparency Chart
…the result chart is appropriate for reimbursement choices by insurance companies?	Reimbursement
…the result chart is appropriate as decision aid for patients?	Decision Aid
…the result chart is appropriate to compare Health Care Professions?	Comparison
External transparency Index
…the result index is appropriate for reimbursement choices by insurance companies?	Reimbursement
…the result chart is appropriate as decision aid for patients?	Decision Aid
…the result chart is appropriate to compare Health Care Professions?	Comparison
…the result index is a good alternative for the existing treatment index?	Treatment index
An open-ended question was added after each of the three main topics, asking the respondent if they had any additional comments (for example, about the topic itself, the layout, or the patient classifications).

With a median score ≥ 7 on the Likert scale, the instrument was considered appropriate, with a median score ≤ 3 as not appropriate. We arbitrarily chose an interquartile range ≤ 3 to assess whether a consensus was reached between respondents. To correct for possible ceiling effects, we also calculated a disagreement index in accordance with the RAND/UCLA manual for consensus studies (12). An index > 1 indicates a disagreement. The survey was conducted online via Google Forms. The outcomes of the individual participants were anonymized.

Physical therapists

We aimed to include 21 physical therapists who had experience treating patients with musculoskeletal disorders. Each physical therapist provided data to the LDK and received an aggregated feedback report, including the PE-_Index and results chart, prior to the survey. Fifteen physical therapists were randomly selected from the LDK database, and 6 were purposefully selected for their experience with routinely collected data for quality improvement initiatives. The physical therapists had no personal relationship with the researchers. The answers given by the physical therapists during the interviews and in the survey were processed anonymously.

Knowledge brokers

We invited 18 knowledge brokers who were independent employees with experience in quality evaluation in primary care physical therapy. These knowledge brokers typically visit practices for physical therapy every two years to provide guidance on quality regulations. After receiving an anonymized report with the PE-_Index and PE-_Chart based on real-world data, the knowledge brokers participated in the survey. As was the case with the physical therapists, no personal relationship was present with the researchers, and all answers were processed anonymously.

Vektis statisticians

Vektis is an organization that collects, analyses, and manages healthcare data on behalf of health insurance companies. Three statisticians of the organization were consulted once about the methodological aspects of the PE-_Index and PEC and were invited to share their experiences with the existing treatment index. Due to the low number of statisticians, they did not participate in the survey.

Advisory board

We created an advisory board with one representative of the Netherlands Patient Association (PN), two representatives of the Royal Dutch Society for Physical Therapy (KNGF), one representative of the Dutch Society for Exercise Therapy (VvOCM) and two representatives of Dutch Health Insurers (ZN). After receiving an anonymized report with the result chart and PE-_Index based on real-world data, the members of the advisory board participated in the survey and four board meetings. During the final meeting, we presented and thoroughly examined the study's results.

In total, 7,433 Dutch physical therapists working in 609 primary care practices reported outcomes of 291,643 episodes of 265,674 patients with musculoskeletal disorders in 2020. First, 25,980 episodes were excluded because they were recurrent episodes of low back pain. Second, 163,928 episodes were excluded due to missing data, and mainly the second NPRS and PSFS measurements were absent. Third, 4,030 episodes were marked as outliers, with an exceptionally large number of treatments (> 51) or total episode duration (> 63 weeks). Finally, 1,900 episodes were excluded because they were obtained in a practice with fewer than 30 total episodes. The characteristics of the included and excluded patients are reported in Table 2.

Table 2

Characteristics of included and excluded episodes in the analysis.
			Included (N = 95.805)	Excluded (N = 195.838)
				Recurrent (N = 25.980)	Missing (N = 163.928)	Outliers (N = 4.030)	Insufficient data (N = 1.900)
Sex
		Male	40.2%	37.9%	40.2%	36.3%	36.5%
		Female	59.8%	62.1%	59.8%	63.7%	63.5%
SES
		Low	30.6%	32.0%	32.5%	31.5%	29.0%
		Medium	41.6%	39.5%	37.9%	44.0%	43.5%
		High	27.8%	28.1%	28.8%	24.4%	27.5%
		Unknown	-	0.4%	0.8%	-	-
Chronicity
		Acute	31.9%	35.7%	36.4%	43.6%	44.8%
		Subacute	39.4%	35.4%	27.9%	25.4%	32.3%
		Chronic	28.7%	21.4%	23.0%	30.9%	22.8%
		Unknown	-	7.5%	12.7%	-	-
Region
		Neck	18.8%	19.9%	18.2%	14.9%	19.2%
		Lower back	19.5%	19.3%	16.8%	12.4%	20.0%
		Knee	16.2%	16.7%	17.5%	27.3%	18.3%
		Shoulder	12.7%	10.9%	12.1%	12.4%	12.1%
		Hip	9.0%	9.5%	8.5%	9.9%	9.4%
		Unspecified	0%	7%	0.0%	11.9%	7.0%
		Upper back	6.6%	7.1%	5.8%	5.5%	4.9%
		Elbow/hand	5.2%	4.5%	6.2%	2.4%	4.1%
		Pelvis	2.7%	2.7%	3.3%	2.2%	3.2%
		Ankle/foot	2.4%	2.3%	2.9%	1.1%	1.7%
Age
		Mean (SD)	50.0 (19.4)	50.9 (19.4)	48.5 (20.0)	51.9 (19.8)	50.6 (20.9)
NPRS baseline
		mean (SD)	6.1 (1.9)	6.1 (1.9)	5.9 (2.0)	5.7 (2.1)	6.0 (2.0)
PSFS baseline
		mean (SD)	7.0 (2.1)	6.8 (2.0)	6.9 (2.2)	7.6 (2.3)	6.9 (2.2)
∆ NPRS
		mean (SD)	-4.2 (2.5)	-4.2 (2.4)	-3.7 (2.6)	-2.9 (2.6)	-4.0 (2.5)
∆ PSFS
		mean (SD)	-5.1 (2.9)	-5.0 (2.7)	-5.0 (3.1)	-4.5 (3.3)	-4.9 (2.9)
Number of treatments
		mean (SD)	9.2 (7.3)	7.3 (6.2)	9.0 (11.4)	56.1 (36.8)	10.2 (8.1)
Episode duration [weeks]
		mean (SD)	13.3 (12.7)	9.2 (8.6)	13.5 (16.7)	72.7 (26.1)	15.6 (14.3)
PE-_Index
		mean (SD)	58.5 (16.0)	-	-	-	-
	Abbreviations: SES = Social economic status; SD = standard deviation; NPRS = numeric pain rating scale; PSFS = Patient specific functioning scale; PE-_Index = Physical therapy efficacy index.

Discriminative value of the PE-_INDEX

In the linear mixed models, 95.805 episodes measured in 370 practices were included. The absolute correlation coefficient indicated no strong collinearity between the independent variables. Table 3 provides information on the ICCs per dependent variable for both the intercept-only and adjusted models. The models are presented in Table A2 in the appendix. Generally, the adjusted model resulted in an equal or greater ICC than did the intercept-only model. The PE-_Index adjusted model resulted in the highest ICC value of 0.118, indicating that approximately 11.8% of the variation can be attributed to the practice in which a patient was treated. This value exceeds 0.1, which is generally accepted as an adequate measure to differentiate between practices (13–15)

Table 3

Interclass correlation coefficient per linear mixed model.
		Practice level
		Intercept-only model	Adjusted model
	PE-_Index	0.100	0.118
	∆ NPRS	0.063	0.096
	∆ PSFS	0.088	0.093
	Number of treatments	0.069	0.068
	Episode duration [weeks]	0.063	0.068
Abbreviations: PE-_Index = Physical therapy efficacy index; NPRS = numeric pain rating scale; PSFS = Patient specific functioning scale;

Individual PE-_Index and PE-_Chart

Linear mixed models were used to calculate adjusted outcomes for the ∆NPRS, ∆PSFS, number of treatments, episode duration per episode and PE-_Index per episode. Figure 1 shows a PE-_Chart, in which the average z-transformed scores of the two practices in 2020 were compared to the average of all practices in 2020.

Part 3: Evaluation of the PE _Index with Stakeholders

Survey

A total of 13 physical therapists, with an average of 22 years of work experience, participated in the online survey, yielding a response rate of 62%. The survey results, presented in Table 4, reveal that the PE-_Index and PE-_Chart were perceived as appropriate for internal quality improvement, with a median rating of 7. From the answers to the open questions, it became evident that physical therapists had difficulty understanding the graph without explanation.

The PE-_Index and PE-_Chart were not regarded as appropriate for external transparency, i.e., reimbursement, decision aid, or to compare disciplines, with median ratings ranging from 4 to 6. Notably, the physical therapists acknowledged the PE-_Index as a preferable alternative to the treatment index, with a median rating of 7.

From the answers in the open text fields, it became clear that physical therapists consider use of the graph and index for external transparency undesirable due to concerns that insurers may utilize it as an additional instrument to impose stricter requirements and administrative burdens, without a fair tariff in return. The issue of gaming was also identified as a risk factor, referring to the manipulation of outcomes by physical therapists, driven by the fear of negative financial consequences or to enhance positive performance outcomes. Comparability between healthcare disciplines is recognized as a complex aspect.

Of the knowledge brokers, 11 (61% response rate) participated in the online survey. Table 4 shows that, similar to physical therapists, knowledge brokers agreed that the PE-_Chart was appropriate for internal quality improvement (median of 7). Most of the brokers indicated that the outcome graph is a valuable tool that provides rapid and accurate insights into the quality of care. The graph offers clear insight, with a short explanation even for physical therapists who are less experienced with data analysis.

Brokers also agreed with the physical therapists that the PE-_Index and the PE-_Chart were not appropriate for one of the three forms of external transparency. The brokers expressed that the new instruments represent a step in the right direction, although there are several areas that could be improved. Concerns have been raised about the burden of adopting these new instruments without corresponding compensation or time allocation. Moreover, the risk of gaming, influenced by financial consequences, was strongly emphasized.

Several methodological points were also identified. The variation in patient populations was repeatedly highlighted as an issue. Practices with a greater proportion of chronic patients appeared to score less favourably, and those with a specialized focus, such as paediatric physical therapy, lacked suitable comparators. Further differentiation between specific conditions was advised. Some reservations were expressed about the use of Patient-Reported Outcome Measures (PROMS) altogether, while the benefits of employing PROMIS-CAT were outlined. Additionally, concerns were raised about the clarity of patient coding.

A point of concern was the uniformity, or lack thereof, of the electronic health records.

Regarding the comparability between healthcare disciplines, substantial differences exist between primary and secondary care, and efforts should focus on finding indicators suitable for both settings.

Overall, it is asserted that the index represents a better instrument for external transparency than the current treatment index (median 7), but it requires some refinement to address the identified areas for improvement.

All six advisory board members (response rate 100%) completed the online survey. All advisory board members unanimously agreed with the text summarizing the study's outcomes. The consensus and appraisal were high among all respondents for the usability of the PE-_Chart in peer learning sessions and visitations (median 8/8 ½). According to the advisory board, the current PE-_Index and PE-_Chart are important steps towards ‘value-based healthcare’ in Dutch primary care physical therapy. They can be used directly within an environment safe for learning goals, such as peer learning sessions and visitations. In future initiatives, collaborative efforts with stakeholders can further enhance the development of the PE-_Index and PE-_Chart.

Regarding external transparency, they deemed the current PE-_Index and PE-_Chart appropriate for patient decision aids and for comparing care disciplines with each other, but although preferable to the current treatment index, at this stage, they are not suitable for healthcare reimbursement purposes. The main reasons for this were the negative connotations of current control instruments, which hinder implementation, and the risk of gaming. The development of external transparency should be performed with cautious and gradual careful implementation. However, representatives from insurance companies did see a financial incentive for the use of charts and indices in such a learning environment as a good alternative.

The advisory board also stressed the importance of increasing the knowledge of physical therapists about the use of patient-reported outcome measures (PROMs). To gain trust in physical therapists, it should be clear to users how both are calculated and composed. Furthermore, they emphasized the importance of a standardized approach to data recording in electronic health records and subsequent export to the database to maintain data integrity and quality. The reliability and robustness of the national data registry (LDK) should be optimal before the PE-_Index and PE-_Chart can be implemented on a national level.

Another suggestion to increase the validity of the chart and index was to provide the PROMs directly from patients, ensuring that they were not influenced by their physical therapist. In current Dutch primary care physical therapy, PROMs are recorded by physical therapists in their electronic health records. Future efforts should focus on enabling patients to independently complete their PROMs using an easily accessible and user-friendly interface.

Concerning the composition of the index and chart, the advisory board suggested incorporating patient recurrence as a fixed factor in the analyses of the PE-_Index and PE-_Chart.

Statisticians

During the consultation with the statisticians, they expressed a preference for the regression model used in the chart and index over the statistical method employed in the current treatment index. Additionally, they emphasized the significance of transparency and understanding of the applied methodology, particularly concerning the corrective factors.

Table 4

Results survey results for the question ‘to what degree do you think the index/chart is appropriate for:
	Advisory		Therapists		Brokers		All
Internal Quality Chart	Median	IQR	Median	IQR	Median	IQR	Median	IQR
Practice assessment	8	0,25	7	5	7	3	7	3
Peer learning	8,5	1	7	3	7	1,5	7	1,25
External Quality Chart
Reimbursement	3,5	2	4	2	5	3	4	4,25
Patient Decision Aid	7	1	5	5	5	2,5	5	3,25
Comparison	7	1,25	6	3	3	2	5	4
External Quality Index
Reimbursement	3,5	3,75	5	3	5	3,5	5	5
Patient Decision Aid	8	1,25	5	4	4	3,5	5	4
Comparison	6,5	3,75	5	3	4	3	5	4
Treatment index	6	2,25	7	3	7	1,50	7	3
See Table 1 for detailed questions. Purpose considered appropriate (≥ 7) with consensus (IQR ≤ 3) in bold. Purpose considered not appropriate (≤ 3) with consensus in italic. All disagreement indexes were < 1 (> 1 indicates disagreement).

In this study, we developed and evaluated the discriminative value of the physical therapy efficacy index (PE-_Index) and its associated chart (PE-_Chart) using real-world data. These tools integrate patient-reported outcomes, considering not only the intervention but also various other factors related to the treatment setting, along with treatment costs. The outcomes of the linear mixed models demonstrated that the PE-_Index is an appropriate metric for differentiating between practices.

Key stakeholders supported the direct implementation of the index and chart to enhance internal quality improvement through peer learning approaches. While the perspective holds promise, stakeholders recommend caution in the immediate adoption of the PE-_Index and PE-_Chart for external transparency objectives, such as their deployment as patient decision-support tools, their influence on reimbursement determinations, or their extension for utilization across diverse healthcare disciplines to facilitate interprofessional comparisons. Instead, it is advised to first familiarize physical therapists with the practical utilization of the PE-_Index and PE-_Chart, fostering trust in these instruments. This cautious stance is influenced by several factors, including concerns related to gaming, the use of real-world data, therapists' proficiency levels, and certain methodological considerations. These issues will be elaborated upon in the following discussion.

Gaming

The issue of gaming was raised by all stakeholders. Gaming involves health care providers who are incentivizing or manipulating data to achieve better outcomes. When financial incentives are directly linked to quality, this can negatively impact the reliability of the data. Currently, most physical therapists input patient-reported outcome measures (PROMs) into electronic health records, allowing them to add, remove, or adjust the measures. This finding is not exclusive to physical therapy; a similar phenomenon has been observed in the use of indicators based on PROMs in mental health treatment (16).

The importance of preventing gaming introduced a seeming discrepancy in the results. Despite evaluating the PE-_Index as more suitable for reimbursement policies compared to the current treatment index, stakeholders cautioned against replacing the treatment index with the PE-_Index at this stage. Representatives from insurance companies did, however, see a financial incentive in using the PE-_Index and PE-_Chart in a learning environment as a safe alternative.

Another initiative to address the gaming problem is having patients independently complete PROMs separate from physical therapists, online from their own homes, or in the waiting area, for instance, using patient portals. Initiatives such as KLIK (17) and OnlinePROMS (18) are already available.

Real-world data

Both the PE-_Index and PE-_Chart have been compiled and calculated using routinely collected data, offering significant advantages (19) while also presenting certain limitations. First, establishing a data collection infrastructure is crucial. Fortunately, this infrastructure is already in place for practices connected to the SKF and those affiliated with the data registry of the Royal Dutch Society for Physiotherapy (LDF), with approximately 80% of all physical therapy practices in the Netherlands exporting their patient data to a national database. However, when expanding the use of the PE-_Index and PE-_Chart to other healthcare disciplines, a national data collection system becomes essential to facilitate broader applicability.

Second, real-world data often present a challenge due to the frequent occurrence of missing data, primarily due to a lack of second measurements. This can potentially introduce bias and affect the representativeness of the results for the entire population. To address this, we carefully evaluated the differences between the patient characteristics of the included and excluded patients and found no major discrepancies. Nonetheless, enhancing the trustworthiness of data registration remains a priority. Notably, participation in the LDK requires practices to adhere to specific data collection standards, which is not mandatory for other practices. Encouraging physical therapists to improve data quality can be challenging, prompting discussions on financial reimbursement, care, and administrative burdens. In our view, a more favourable approach involves investing in fostering innovation within the profession and empowering physical therapists who demonstrate internal motivation to take the lead rather than enforcing mandatory participation for all.

A third issue that demands attention is the lack of uniformity in electronic health records. Commercial entities supply EHR systems and often do not prioritize standardized data collection and clinimetry. This diversity poses a significant challenge when attempting to collect data from different EHR providers. Establishing a nationwide consensus on data gathering involving the participation of all or most healthcare providers could help overcome these inconsistencies.

Finally, the number of treatments is used as a measure of costs in the denominator of the fraction. Although this provides a reasonable indication, these costs can vary depending on the type of contract. Additionally, this methodology does not allow for innovative financing models, such as DBC-like constructions. It is recommended to explore more direct measures for healthcare costs in the long term, but this exploration should not hinder the implementation of the PE-_Index and PE-_Chart.

Therapists' proficiency level

The respondents struggled to interpret the index and chart effectively without additional explanation. This was reinforced by remarks in the open-text fields by physical therapists. However, after providing a short introduction to brokers and the advisory board, both the index and the chart were understood. Therefore, it seems crucial to develop an introductory knowledge clip or course to address this issue.

Methodological issues

Some adjustments can be made to further increase the validity of the model. Future studies should consider incorporating additional patient characteristics into linear mixed models, as this might lead to a further increase in the discriminative value of the PE-_Index and its components. For example, including information about recurrences could explain some of the between-episode variations (20). Furthermore, expanding the results chart to include the experienced treatment effect, measured with the Global Perceived Effect (GPE), could provide valuable insights into the overall treatment outcome and patient experience.

Strengths and weaknesses of this study

The strengths of this study lie in its comprehensive approach, as we gathered information from all relevant stakeholders and developed and evaluated the tools within a real-world scenario. Brokers, insurance professionals, and statisticians, representing their respective professions, were actively engaged in the study through focus group interviews, which enhanced its credibility.

The study was undertaken in the Netherlands, potentially introducing a regional bias to the findings. Nevertheless, since the nature of the observed complaints exhibits minimal divergence across Western countries and the fundamental principles underlying the efficacy index are universally applicable, we posit that the clinimetric properties of the instrument remain valid across all Western countries. However, it is acknowledged that the commitment of stakeholders may be influenced by local or regional variations. The metrics were derived from practices that contributed to the LDK, where those practices integrated with the SKF system underwent a rigorous process of practice assessment, data collection, and individual peer coaching. Consequently, results from practices not affiliated with the SKF might differ from those that are affiliated with the SKF. However, these differences could, in fact, augment the tools' utility as valuable resources for understanding the results-cost ratio.

The PE-_Index and PE-_Graph can serve as valuable components within a broader quality system, offering insights into the care process and promoting ‘peer learning’. These tools can be readily implemented in a safe learning context.

Due to the present negative connotation of control tools by physical therapists and the fear of the so-called "gaming" effect brought about by financial consequences, the outcome of both tools should not be used for external transparency purposes in this stage; however, the use of these tools can be reimbursed. Nevertheless, the index and chart represent a promising initial step towards providing insights into the quality of care per invested euro, with subsequently large effects on quality of care and affordability. Therefore, we strongly recommend further development of these tools through pilot studies incorporating all relevant stakeholders with care trajectories that involve multiple health care professions.

GPE – Global Perceived Effect

ICC = Intraclass correlation coefficient.

KNGF = Royal Dutch Society for Physical Therapy.

LDF = Landelijke Database Fysiotherapie. (National Data Registry of the KNGF)

LDK = Landelijke Database Kwaliteit. (National data registry of the SKF).

NPRS = Numeric Pain Rating Scale.

PE-_Index and PE-_Graph.Physical Therapy Efficacy Index and Graph.

PN = Netherlands Patient Association.

PROMS = Patient Reported Outcome Measures (PROMs).

PSFS = Patient Specific Functioning Scale.

SES = Social economic status.

SKF = The Association for Quality in Physical Therapy.

VVOCM = Dutch Society for Exercise Therapy.

ZN = Zorgverzekeraars Nederland. The umbrella organization of health insurers in The Netherlands.

Ethics approval and consent to participate

Consent for publication

Not applicable.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due to Dutch law, as described in the Methods section. The use of electronic health records for research purposes is permitted under certain conditions, including informing patients about the use of their data for this specific registry and scientific goals. Patients can actively withdraw their participation or, if they do not object, participate automatically. Through this ‘opt-out’ procedure, patients agree to the use of their data for this registry and scientific goals. However, if other scientists have comparable goals, the data are available from the corresponding author upon reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This study was funded by the Dutch Ministry of Health, Welfare and Sport for a total of 39,930 euros, including taxes.

Authors' contributions

HK conceived the idea for an easily usable instrument to provide transparency about the outcomes and costs of physiotherapy. RF developed the concept for the index and graph and performed the necessary statistical analyses. HK was the primary writer of the background and discussion sections. RF was the primary author of the methods section. MH provided statistical advice. HK and RF were the primary authors of the results section, with KV as the secondary author for all sections. HK and KV managed all logistics for the surveys and advisory board meetings. All the authors have read and approved the final manuscript.

Acknowledgements

We would like to thank all participants, health insurers, physical therapists, representatives from KNGF and VVOCM, statisticians from Vektis, and representatives from patient organizations for their selfless participation in this study.

Authors' information (optional)

HK, RF, and KV work for the SKF, an association for quality in physiotherapy. In this association, we are dedicated to providing insights into healthcare outcomes as the start of a peer learning process in a safe environment, with the ultimate goal of improving healthcare for patients at socially achievable costs.

CBS Statistics Netherlands. Centraal Bureau voor de Statistiek. 2023 [cited 2023 Dec 27]. Zorguitgaven; kerncijfers. Available from: https://www.cbs.nl/nl-nl/cijfers/detail/84047NED?dl=926E6
Fatoye F, Wright JM, Gebrye T. Cost-effectiveness of physiotherapeutic interventions for low back pain: a systematic review. Physiotherapy. 2020 Sep;108:98–107.
Porter ME, Teisberg EO. Redefining Health Care: Creating Value-based Competition on Results. Harvard Business Review Press; 2006.
Cieza A, Causey K, Kamenov K, Hanson SW, Chatterji S, Vos T. Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Lond Engl. 2021 Dec 19;396(10267):2006–17.
Hartrick CT. A four-category verbal rating scale (VRS-4), an 11-point numeric rating scale (NRS-11), and a 100-mm visual analog scale (VAS) were compared in the assessment of acute pain after oral surgery. Clin J Pain. 2001 Mar;17(1):104–5.
Beurskens AJ, de Vet HC, Köke AJ, Lindeman E, van der Heijden GJ, Regtop W, et al. A patient-specific approach for measuring functional status in low back pain. J Manipulative Physiol Ther. 1999 Apr;22(3):144–8.
KNGF. Richtlijnen [Internet]. 2023 [cited 2023 Dec 27]. Available from: https://www.kngf.nl/kennisplatform/richtlijnen
Vektis, Business intelligence centrum for Health Care. Vektis - business intelligence centrum voor de zorg. 2023 [cited 2023 Dec 27]. Declaratie paramedische hulp. Available from: https://www.vektis.nl/standaardisatie/standaarden/PM304-3.2
Centraal Bureau voor de Statistiek. Centraal Bureau voor de Statistiek. 2023 [cited 2024 Feb 1]. Sociaal-economische status per postcode, 2020 en 2021. Available from: https://www.cbs.nl/nl-nl/maatwerk/2023/33/sociaal-economische-status-per-postcode-2020-en-2021
Kaliyaperumal S, Kuppusamy M, Arumugam S. Labeling Methods for Identifying Outliers. Int J Stat Syst. 2015 Oct 15;10:231–8.
Visser J, Bekker, de P, Koolman X, Drunen, van P, Stam P. Validatieonderzoek van de behandelindex. Den Haag: Equalis Strategy & Modeling B.V; 2017 Mar p. 67.
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA Appropriateness Method User’s Manual [Internet]. Santa Monica, CA: RAND Corporation; 2001. Available from: https://www.rand.org/pubs/monograph_reports/MR1269.html
Bryk AS, Raudenbush SW. Hierarchical Linear Models: Applications and Data Analysis Methods. 1992.
Hox JJ. Multilevel Analysis: Techniques and Applications. 2010.
Jöreskog KG, Sörbom D. Advances in Factor Analysis and Structural Equation Models. 1979.
Wolpert M. Uses and abuses of patient reported outcome measures (PROMs): potential iatrogenic impact of PROMs implementation and how it can be mitigated. Adm Policy Ment Health. 2014 Mar;41(2):141–5.
hetklikt.nl. KLIK [Internet]. 2023 [cited 2023 Dec 28]. Available from: https://www.hetklikt.nu/
onlineproms.nl. OnlinePROMS maakt PROMs / PREMs verzamelen makkelijk [Internet]. 2023 [cited 2023 Dec 28]. Available from: https://onlineproms.nl/
Jorm L. Routinely collected data as a strategic resource for research: priorities for methods and workforce. Public Health Res Pract. 2015 Sep 30;25(4):e2541540.
Deutscher D, Werneke MW, Hayes D, Mioduski JE, Cook KF, Fritz JM, et al. Impact of Risk Adjustment on Provider Ranking for Patients With Low Back Pain Receiving Physical Therapy. J Orthop Sports Phys Ther. 2018 Aug;48(8):637–48.

No competing interests reported.

Appendix.docx

Download PDF

Editorial decision: Revision requested
16 Jul, 2024
Editor assigned by journal
14 Jul, 2024
Submission checks completed at journal
14 Jul, 2024
First submitted to journal
06 Jul, 2024

You are reading this latest preprint version

The Physical Therapy Efficacy Index and Chart: a stimulus for value-based healthcare using real-world data. A mixed methods study

Status:

Version 1

Abstract

Figures

Background

Methods

Part 1. Data collection

Part 2. Development of the physical therapy efficacy index and chart

1. Conceptualization

2. Development

3. Discriminative value

4.Individual PE-_Index and PE-_Chart

Part 3: Evaluation of the PE-_Index and PE-_Chart with Stakeholders

Survey

Physical therapists

Knowledge brokers

Vektis statisticians

Advisory board

Results

Discriminative value of the PE-_INDEX

Individual PE-_Index and PE-_Chart

Part 3: Evaluation of the PE _Index with Stakeholders

Survey

Statisticians

Discussion

Gaming

Real-world data

Therapists' proficiency level

Methodological issues

Strengths and weaknesses of this study

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1

The Physical Therapy Efficacy Index and Chart: a stimulus for value-based healthcare using real-world data. A mixed methods study

Status:

Version 1

Abstract

Figures

Background

Methods

Part 1. Data collection

Part 2. Development of the physical therapy efficacy index and chart

1. Conceptualization

2. Development

3. Discriminative value

4.Individual PE-Index and PE-Chart

Part 3: Evaluation of the PE-Index and PE-Chart with Stakeholders

Survey

Physical therapists

Knowledge brokers

Vektis statisticians

Advisory board

Results

Discriminative value of the PE-INDEX

Individual PE-Index and PE-Chart

Part 3: Evaluation of the PE Index with Stakeholders

Survey

Statisticians

Discussion

Gaming

Real-world data

Therapists' proficiency level

Methodological issues

Strengths and weaknesses of this study

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1

4.Individual PE-_Index and PE-_Chart

Part 3: Evaluation of the PE-_Index and PE-_Chart with Stakeholders

Discriminative value of the PE-_INDEX

Individual PE-_Index and PE-_Chart

Part 3: Evaluation of the PE _Index with Stakeholders