Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale in a large and representative sample of Child and Adolescent Mental Health practitioners: Is the use of a total scale score justified?

doi:10.21203/rs.2.11062/v1

Download PDF

Research article

Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale in a large and representative sample of Child and Adolescent Mental Health practitioners: Is the use of a total scale score justified?

https://doi.org/10.21203/rs.2.11062/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: High-quality instruments are essential for implementation practitioners and researcher; only valid and reliable measures can confidently and consistently measure what they are intended to measure. Psychometrically sound and pragmatic measures have the potential to guide the tailoring and evaluation of an implementation program thus contributing to the knowledge base of implementation science. The Evidence-Based Practice Attitude Scale (EBPAS) is one of few implementation instruments with strong psychometric qualities and is widely used to measure attitudes toward Evidence-Based Practice (EBP) in implementation practice and research. The Multidimensionality of the EBPAS has repeatedly been supported. Whether all of the fifteen individual items contribute to the EBPAS total scale beyond their subscales (Appeal, Requirements, Openness and Divergence) has yet to be proven. In addition, the Divergence subscale has been questioned because of low correlation with the other subscales and low inter-item correlations. Methods: The factor structure of the EBPAS was investigated in a child- and adolescent psychiatry setting (N=925, 62%, n =570) using first- and second-order Confirmatory factor analysis, thus replicating Aarons' initial studies. In addition, the present study is the first to use a bifactor model in a large and representative sample. Results: The EBPAS was supported as a multifaceted instrument. The EBPAS had a rather strong general factor supporting the use of the inventory as a single measure of attitudes. There were still four dimensions of attitude after the general factor was accounted for. All items, including the items of the Divergence factor, contributed to general attitude as well as to their specific construct. Conclusions: The present study is the first to show that it is justified to use and interpret the EBPAS as an instrument measuring global as well as specific attitudes toward EBP. Sum score and subscales scores can be used in implementation research and practice.

Psychology

EBPAS

attitudes toward EBP

Implementation

Confirmatory factor analysis

bifactor model

Implementation science and practice depends on psychometrically strong instruments. The EBPAS is one of few, tested with proven multidimensionality, but it is unclear if the use of all of the subscales (including Divergence) and total scale is justified.

This is the first study to use bi-model-CFA in a large and representative sample, justifying the use of all of the subscales as unique and reliable indicators of specific EBP-attitude while supporting the EBPAS as a global measure.

The EBPAS Total scale and subscales are valid measures for general and specific attitudes and can be used when tailoring and evaluating implementation programs.

A current priority for governments, policymakers and planners across the world is to disseminate empirically supported treatments in order to improve health. Evidence- based practice (EBP) is a paradigm that aim to integrate best available research with clinical expertise and patient values (1). Multiple factors at different levels; community, organizational and individual are supposed to influence the implementation of EBP (2-7). There is growing evidence that implementation may be enhanced and better supported by stakeholders if tailored to the local circumstances (8, 9). Potential enablers and barriers for change need to be identified early in the implementation planning process on the basis of previous research and practical experience (10). Different strategies are then selected and matched to address these determinants for change (7, 9-12).

Health practitioners’ attitudes and values play an important role in implementing EBP in real world settings (13). Positive attitudes along with subjective norms and a person’s self-efficacy influence the decision on whether or not to implement a new practice (14, 15). According to Rogers, the innovation-decision process starts with knowledge and persuasion when the provider forms a favorable or unfavorable attitude toward the innovation (16). Both general and specific constructs play an important role in understanding and predicting behavior. General predictor variables are suggested as the best to predict general outcomes and specific predictor variables the best to predict specific outcomes (17). General attitudes may explain and predict broad patterns of behavior whereas a specific attitude is a better measure of specific behavior (18). Assessing both specific and general attitudes towards EBP may help the implementation tailoring process and the evaluation of an implementation program . Psychometrically sound and theory- based measures are also essential for the field of implementation science (19-21).

The Evidence-Based Attitude Practice Scale (EBPAS) is an instrument of high overall psychometric quality (20, 22). The EBPAS was developed by Aarons on the basis of a comprehensive literature review and consultation with mental health service providers and researchers. Four important domains of attitudes were identified; the appeal of EBP, the provider’s openness to change, who require the change and the perceived difference between current practice and EBP (13, 23). Thus the EBPAS has four dimensions Appeal, Openness, Requirements and Divergence. Subscale scores are used to obtain information about the specific domains of attitude and a sum score (where the Divergence items are reversed) is used to estimate the global attitude towards EBP. Previous studies suggest good internal consistency for the subscale scores except for the Divergence scale and adequate to good internal consistency for the EBPAS total scale (13, 23, 24). The construct validity has been studied using different sample sizes (n= 146 vs. n=1,273), in various settings (e.g., mental health, health, correctional service, social service, residential care, educational sector), with participants at different levels (front line providers, administrators, consultants) and diverse cultures (e.g. USA, Europe, Africa, Asia) (13, 23-30). Different factor models with and without correlated residuals have been fitted to the data: a) first-order models, b) second-order models, and c) a bifactor model.

First-order models

First-order models focus on the subdomains. According to recent standards (31), the four-factor model has yielded poor to acceptable model fit (26). The factor loadings have generally been moderate to strong with weakest loadings for Divergence. Correlations between factors have generally been moderate except between Divergence and some of the other sub- factors (13, 24). Attempts have been made to test a more complicated five-factor model with two facets of the Divergence scale (26), but subsequent research did not support this model (27). To summarize, there has been support for four, somewhat correlated, sub-factors in the EBPAS inventory. However, a problem with first-order models is that they do not justify composite scores.

Second-order models

In second-order models, individual items load on their specificity factors which in turn load on a higher order construct that accounts for the commonality shared by the facets. For the second-order model with one general factor common to all sub-factors, model fit has been acceptable with generally strong and significant first factor loadings and moderate second factor loadings. Again the Divergence scale showed the weakest first and second factor loadings (23). Several studies have supported the second-order model, with (28, 32) and without added covariance (25, 27). In brief, the second-order models add support for the total scale of the EBPAS. The problems with the Divergence scale that have been shown in first-order models are also seen in the second-order models.

Bifactor model

In bifactor models individual items load both on a general factor and a specific factor. Only bifactor models can separate specific factors from the general factor. Also, bifactor models can identify whether a facet still exists after partialling out the general factor (33, 34). A bifactor model has been tested in a sample of 270 practitioners (28). The model fit was good and even slightly better than the second-order model. First factor loadings were in general moderate to strong and significant. General factor loadings were moderate and significant except for Divergence, which were weak or non-significant. The results gave some support to the hypothesis that the variance in the individual items can be attributed to both a general factor and uniquely to their specific factors. However, the items in Divergence seem to have little in common with the other items.

To summarize, understanding the structural validity of tests is essential for evaluating interpretability of provided scores and their relation to outcome variables (35, 36). The construct validity of the EBPAs has been supported in different contexts using different models. Previous studies support the hypothesis that the EBPAS can be interpreted as a multidimensional scale with four correlated subscales. Whether all of the individual items contribute to general attitude (the EBPAS total scale) beyond their subscales has yet to be proven. In other words, is it meaningful to interpret the EBPAS as a sum score? And do all of the subscales, including Divergence, uniquely contribute to the EBPAS measure?

The primary aim of this study was to contribute to the evidence regarding the construct validity of the EBPAS by investigating the factor structure in a large and representative sample of 570 child mental health practitioners, thus replicating Aarons initial validation study (24). We used a series of confirmatory factor analysis (CFA) to test three models: a) a first-order model with four separate factors, b) a second-order model with an added general factor, and c) a bifactor model. A secondary aim was to validate the Swedish version of EBPAS.

Thus, the EBPAS will be tested based on previous literature for:

Whether a first-order, a second-order or a bifactor model best fits the data and corollary, that the fit for the best model is adequate or if error correlations previously shown among items are necessary?

Whether all of the different sub-factors contribute uniquely to the concept of EBP, in other words, whether the sub-factors are uniquely supported when the general attitude has been accounted for?

Whether items contribute to both the general and specific factors of EBPAS?

Participants

The sample consisted of 570 participants from a group of 925 eligible professionals resulting in a response rate of 62%. The typical participant was female (84%), 35-45 years (28%) and a psychologist (38%). They took part of the Swedish Association of Child and Adolescent Psychiatry Implementation program “Deplyftet” for youth depression (37). Eleven of 31 eligible clinics in children and adolescent mental health have participated so far serving about 712 000 /2 000 000 (36%) of Swedish children. They represent all types of publicly owned and funded county council clinics serving 65 000 (41 000- 125 000) children in their catchment area. Swedish clinics serves an average of 64 000 (29000- 450 000) children in their catchment area.

Measures

The EBPAS was translated to Swedish (38). The EBPAS consists of 15 items rated on a Likert scale ranging from 0 (not at all) to 4 (to a very great extent) (13). Previous studies report Cronbach alpha, ranging from .76 - .91 except for the Divergence scale (.59 - .66) (13, 23). Mean values were high for most of the positively phrased EBPAS items, see table 1. For negatively worded items, the mean values were generally lower, Skewness was generally negative (J-shaped) for positively phrased items, but above 1.0 only for one item. Three items had a kurtosis value above 1, one; Item 15 “enough training” was extremely kurtotic (2.6) and also had a positive skew above 1.0. This item had also a very obvious ceiling effect with 54 % of the ratings in the highest response category. Reliability ranged from .58 to .88 (table 1). Test of item discrimination for the Divergence scale yielded values between .27 and .53 suggesting that these items were not very consistent with the EBPAS total scale or the subscale. However, item analyses showed no significant improvement in alpha values after removing single items.

Insert table 1 about here

Procedure

Data were collected as a baseline assessment as part of a larger trial from October 2014 to February 2017. A web-based survey was distributed per email to outpatient child mental health front-line professionals. Two to five reminders were sent if necessary.

Analyses

Five participants were excluded; three because of missing data for all items, one because all responses were the same and one for being multivariate outlier, (through Mahalanobis distance with p <0.001) leaving 565 cases for analysis. Two univariate outliers (extremely low z scores < 3.0) were replaced with their neighbors (39). SPSS statistical software version 24 was used for descriptive statistics (40). A series of CFA were conducted to test the factor structure since the factor structure of the EBPAS has been thoroughly examined in previous studies using both exploratory and confirmatory methods (23, 24).

The CFA models were estimated with MPLUS 8 (41). The weighted least squares -robust mean and variance adjusted (WLSMV) estimator was used since the items were ordinal (Likert scale). Cases with missing data were included in the CFA because the WSLW estimator permits their inclusion. Several different fit indices were used to summarize the model fit; chi-squared index (²), the comparative fit index (CFI), the Root mean square error of approximation (RMSEA), and the Weighted root mean square residual (WRMR). For categorical models Yu has recommended .95 for CFI and TLI (Tucker Lewis index), 0.05 for RMSEA and 1.00 for WRMR (42-44). The MPLUS program provides a likelihood ratio ²test for difference testing (DIFFTEST) for model evaluation.

Model Comparison using CFA

We tested three models, first the one suggested by Aarons et al. with four sub-factors based on the suggested scales of the inventory. The second model tested a hierarchical model with one general factor on the level above the four sub-factors. The third model was a bifactor model measuring a general factor defined to be unrelated to the sub-factors; this model was used to test for unique variance of the four scales to the general attitudes to EBP construct.

First-order models

In line with Aarons, Glisson et al. (2010), we tested the four-factor model first with correlation between factors and found adequate fit (see Model 1a, Table 2), especially CFI suggested that most of the covariance was represented in this model. To make the model feasible we fixed both item 11 (“Supervisor required”) and 12 (“Agency required”) in the indicators of the Requirements factor to 1. In line with Aarons et al., the strongest correlation was between 9 and 10 from the Appeal subscale (the items about “Intuitively appealing” and “Making sense” ) and adding this error correlation increased fit significantly (see Model 1b, Table 2). The estimated (error) correlation between item 9 and 10 was .65. The standardized loadings of Model 1b are depicted in Table 3, generally all loadings were high (above .5), with the exception of Divergence’s loading on item 3 “Know better than researcher” (0.383). Factor inter-correlations were generally medium and significant.

Insert table 2 about here

Insert table 3 about here

Even if the original four-factor models (Model 1a and 1b) revealed adequate model fit, there are a large number of unanalyzed correlations in the model, e.g. all correlations between factors. Previous studies have suggested rather high correlations between the factors, i.e. the model is oblique. A second-order model with a general factor above the four factors or a bifactor model where common variance is represented by a separate factor is two possible ways of representing this correlation between the factors.

Second-order models

Model 2a and 2b (shown in Figure 1) tested the second-order model. Model 2a did not include the correlation between item 9 and 10, that was added to Model 2b which had two degrees of freedom more than Model 1b and is therefore more parsimonious than Model 1b. The fit for this model was also very good (at least according to the CFI), even if the fit for this model was not as good as for Model 1b. The standardized factor loadings were similar to the four-factor model. First-order factor loadings were generally strong and statistically significant. All of the first-order factors loaded significantly and strongly on the General Attitude factor (see table 2, and figure 1).

Modification indices were difficult to interpret, for example they suggested additional correlation between Requirements and Appeal, but also between Openness and Divergence. Adding one of these increased model fit to a level comparable to Model 1b. Because this was not an expected correlation, we refrained from adding it to the final hierarchical model.

Insert figure 1 about here.

Bifactor model

The bifactor models take another strategy and gather all the common variance of all the items into a single factor while estimating the unique contribution of the factors orthogonal from this common factor. Because the factors include the unique common variance of the observed variables of each sub-factor, the model can be used to test whether the data supports the unique contribution of each of the four scales of the EBPAS. To succeed with the estimation of the model, a number of parameters were fixed due to problems with convergence and multi-collinearity. One of the items of Divergence was fixed to 1 and for the Openness factor we needed to fix the error variance of two items otherwise the estimation did not converge. All factors were uncorrelated in the first model (Model 3b, see Table 3 and Figure 2); with these restrictions the model converged and showed a very good fit. The only difference between Model 3a and 3b was the added correlation between item 9 and 10. All factor loadings were statistically significant (p<0.001). The average item to general factor loading was .49 (.32 - .71), see Table 3 and Figure 2. First-order factor loadings were generally weaker in the bifactor model, but loadings were still generally moderate. In addition, 9/15 items had a higher loading on the group factor than the general factor, further supporting multidimensionality.

We tested the unique information in the factors by deleting them one at a time from Model 3b. The CFI decreased from .987 to .938, .962, .965, and .968, for the four factors, (Requirement, Appeal, Openness, and Divergence) respectively. RMSEA increased from .073 to .154, .119, .115, and .109, respectively. All changes were clearly significant (p <.001) based on the DIFFTEST. From these estimations, it is obvious that all factors had a unique contribution, and also that the Requirement factor was clearly the one with the strongest unique contribution.

Insert Figure 2 about here

This large and representative study of the EBPAS, utilizing first-order, second-order and bifactor CFA-models, strongly supports the proposed structure of a higher order factor and four specific factors. The EBPAS had a rather strong general factor supporting the use of the inventory as a single measure of attitudes toward evidence-based practice. Furthermore, there were still four distinct dimensions of attitude after the general factor was accounted for. All items, including the items of the Divergence factor, contributed to general attitude as well as to their specific construct. In other words, the subscales can be scored and used as indicators of specific aspects of EBP attitude when a more precise and specific analysis is needed.

Although these four dimensions seem to be distinct, they address a common theme of attitude to EBP. Two possible ways of representing this information are a by a second-order model with a general factor above the four factors explaining common variance between factors or by a bifactor model where the common variance of the items are estimated by a general factor as well as separate group factors. The results indicate that both models had very good model fit. The second-order model did not have the best fit, but was clearly the most parsimonious model. The pattern of item to factor loadings, identified in prior analyses, were found in our analysis as well, with strong item to factor loadings across all factors except for Divergence and generally strong second-order factor loadings (23). More important, Divergence loading on Global attitude was strong and significant. In previous studies Divergence to general factor loading has been non-significant (27, 28) , small (23), and moderate in one study, (32) . However, the finding of moderate factor loadings in the study of Egeland was compromised by an out of range second-order factor loading for Appeal, indicating model misfit. The present study clearly supports the second-order model with four first-order factors that load onto a higher-order factor reflecting global attitude.

An advantage of the bifactor model is that it allows direct exploration of the extent to which subscales reflect a common target domain and the extent to which they reflect a primary subdomain. Items with larger first factor loading than general factor loading are seen as a relatively better measure of the specific construct than of the general construct. The bifactor model had good fit (33). All factor loadings were significant, in contrast with previous results (28). General factor loadings were moderate to strong. All items loaded substantially on the general factor and the average item to general loading was moderate, suggesting that items in the EBPAS are all affected by attitude toward EBP. For the Requirement factor, the factor loadings on individual items were still strong when the common variance was partialized out. For the other factors the results was mixed, but all except one item (item 3 “Know better than researcher”) exhibited more than a substantial loading on their specific factor, indicating that all factors had unique contribution after adjustment for the impact on items by the general factor. This finding was further supported by the fit statics after removing one factor at a time, which resulted in significant decrease in model fit for each factor. The analysis supports the model with four facets of willingness to adopt evidence-based practice above and beyond the general attitude toward evidence-based practice.

Consistent with previous studies, the Divergence factor is the most problematic one with the weakest loadings in the first-order model (13, 26). In contrast with these studies, inter-correlations between Divergence and the other factors were moderate and significant and item loadings were generally moderate. The same pattern of loadings was found in the second-order model but with strong and significant loadings on the second-order factor. In addition, the Bifactor analysis suggests that all items of the Divergence subscale contribute to the EBPAS total scale and most of them contribute to their subscale after the global attitude has been accounted for. Although loadings to the specific factors were somewhat weaker for Divergence, the average item on factor loading was still moderate. In line with previous studies, Item 3 “Know better than researchers” was the most problematic item, with the weakest factor to item loading in all three models (13, 25, 26). In the bifactor analysis, however, the general factor loading on item 3 was strong; indicating that item 3 contributes to the homogeneity of the EBPAS. To sum, the items of the Divergence subscale are definitely the most heterogeneous ones, but the results support Divergence. All items contribute to general attitude and most of them are indicators of the specific Divergence construct beyond general attitude.

To summarize, the estimations clearly supported the four-factor model suggested by Aarons et al. (13). After controlling for the single error correlation, previously reported by Aarons and others, there was not much room for more factors (23). In addition, it was found that the EBPAS has a rather strong general factor that supports the use of the inventory as a single measure of attitudes toward evidence-based practice. The bifactorial analysis added support to the model with four facets of the EBPAS, suggesting it to be worthwhile separating these factors when more fine-grained analyses are asked for.

Strengths and limitations

Some limitations should be noted. We could not compare respondents with non-respondents to examine potential bias. However, the sample representativeness seems to be good compared to available national data describing the child mental health service workforce (45). In addition, the analysis did not take in account the clustering of employees by service unit. However, previous studies have shown that the variability seems to be between persons rather than workgroups and clinics (23) .

There are some strengths to mention. The response rate of 62% is quite high for being a web-based survey. The sample represents practitioners from a geographically diverse area covering more than a third of Sweden’s child and adolescent psychiatric services from more than half of the Swedish regions which add to generalizability. All responders were front-line practitioners. The sample size of 565 was robust for the Confirmatory factor analyses.

Additional research is needed to further establish the factor structure of the EBPAS in different contexts. More research is also needed to investigate the relation between the EBPAS’s subscales, total scale and different outcome variables, and the EBPAS’s ability to predict adoption, fidelity and sustainment of EBP.

Implications

This study added to the knowledge about the EBPAS in general and for the Swedish version in particular. Most important, it provides support for the EBPAS being a psychometric valid measure for both general and specific attitudes towards Evidence-based practice. The results have implications for both clinical practice and future research. Assessing attitudes may be a starting point when tailoring implementation of EBP. Specific and general attitudes may be related to specific and general outcome variables important for the area of implementation science.

Conclusion

The factor structure of the Swedish version of the EBPAS is supported by the factor structure of the international versions while the reliability and construct validity of the Swedish version add further support to them. Managers and researchers can use the EBPAS total score as an estimate of overall attitude and subscale scores, including the Divergence, as indicators of specific attitudes towards EBP when tailoring and evaluating implementation programs.

EBP: Evidence-based Practice, EBPAS: Evidence-based Practice Attitude Scale, CFA Confirmatory Factor Analysis, WLSMV: weighted least squares -robust mean and variance adjusted, RMSEA: root mean square error, CFI comparative fit index, TLI Tucker-Lewis index, Weighted root mean square residual (WRMR). DIFFTEST: difference testing

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The study was approved by the Regional Ethical Review Board in Umeå, dep for medical research, (Regionala etikprövningsnämnden i Umeå avdelningen för medicinsk forskning); EPN 2015/186-31 and EPN 2016/502-32. In Sweden the Regional ethical boards used to be a part of the faculty of medicine of the regional University until 2004. Between 2004 and 2019 the Regional ethical boards were independent authorities. From 1 January 2019, applications for ethical examination of research are scrutinized by the new Swedish Ethical Review Authority.

Informed consent was obtained from all individual participants included in the study. The respondents were informed about the research project and that completion of the web-based survey was accepted as consent.

Consent for publication

Not applicable

Availability of data and materials

The dataset supporting the conclusions of this article is available in the Halland Hospital Halmstad repository. The dataset used and/or analyzed during current study are available from the corresponding author on reasonable request

Competing interests

The authors declare that they have no competing interests

Funding

This study was funded by grants from the Halland county council.

Authors´ contributions

AS and MB conceptualized the specific research questions and the analytic approach for this manuscript. AS, HJ, RH and SP were responsible for the translation of the EBPAS into Swedish. Analysis were conducted by AS and MB. AS wrote the first draft of the manuscript; all authors (AS, HJ, RH, SP and MB) edited and revised the manuscript and provided critical commentary. All authors (AS, HJ, RH, SP and MB) approved the final manuscript.

Acknowledgements

Not applicable

Authors’ information

Anna Helena Elisabeth Santesson, MD. PhD student¹; Håkan Jarbin, MD. PhD¹; Robert Holmberg, PhD, Senior Lecturer²; Sean Perrin, PhD, Professor²; Martin Bäckström, PhD, Professor²

¹Department of Clinical Sciences, Faculty of Medicine, Lund University

²Department of Psychology, Faculty of Social Sciences, Lund University

Correspondence regarding this article should be addressed to Anna Santesson, Department of Clinical Sciences, Faculty of Medicine, Lund University¸ BMC F12, S- 221 84 Lund, Sweden; E-mail [email protected]; [email protected]; Fax+4635128207; Phone: +467027077253

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71-2.

2. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581-629.

3. Cochrane LJ, Olson CA, Murray S, Dupuis M, Tooman T, Hayes S. Gaps between knowing and doing: understanding and assessing the barriers to optimal health care. J Contin Educ Health Prof. 2007;27(2):94-102.

4. Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q. 2007;85(1):93-138.

5. Nilsen P. Making sense of implementation theories, models and frameworks. Implementation Science : IS. 2015;10:53.

6. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

7. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.

8. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003;362(9391):1225-30.

9. Baker R, Camosso‐Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to address determinants of practice. Cochrane Database of Systematic Reviews. 2015(4).

10. Grol R, M W, M E. Improving Patient Care, the Implementation of Change in Clinical Practice: Elsevier; 2013, 2005.

11. Lewis CC, Scott K, Marriott BR. A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implementation Science. 2018;13(1):68.

12. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010(3):CD005470.

13. Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004;6(2):61-74.

14. Ajzen I. The theory of planned behavior. Organizational Behavior and Human Decision Processes. 1991;50(2):179-211.

15. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191-215.

16. Rogers E. Diffusion of innovation. New York: Free Press; 1995.

17. Swann Jr WB, Chang-Schneider C, Larsen McClarty K. Do people's self-views matter? Self-concept and self-esteem in everyday life. American Psychologist. 2007;62(2):84.

18. Ajzen I, Fishbein M. The influence of attitudes on behavior. The handbook of attitudes. 2005;173(221):31.

19. Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science : IS. 2013;8:22-.

20. Lewis CC, Stanick CF, Martinez RG, Weiner BJ, Kim M, Barwick M, et al. The Society for Implementation Research Collaboration Instrument Review Project: a methodology to promote rigorous evaluation. Implement Sci. 2015;10:2.

21. Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, et al. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11:42.

22. Kelly B, Perkins DF. Handbook of implementation science for psychology in education: Cambridge University Press; 2012.

23. Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychol Assess. 2010;22(2):356-65.

24. Aarons GA, McDonald EJ, Sheehan AK, Walrath-Greene CM. Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale in a geographically diverse sample of community mental health providers. Adm Policy Ment Health. 2007;34(5):465-9.

25. Melas CD, Zampetakis LA, Dimopoulou A, Moustakis V. Evaluating the properties of the Evidence-Based Practice Attitude Scale (EBPAS) in health care. Psychol Assess. 2012;24(4):867-76.

26. Wolf DAPS, Dulmus CN, Maguin E, Fava N. Refining the evidence-based practice attitude scale: an alternative confirmatory factor analysis. Social work research. 2014;38(1):47-58.

27. Keyser D, Harrington D, Ahn H. A confirmatory factor analysis of the evidence-based practice attitudes scale in child welfare. Children and Youth Services Review. 2016;69:158-65.

28. van Sonsbeek MA, Hutschemaekers GJ, Veerman JW, Kleinjan M, Aarons GA, Tiemens BG. Psychometric properties of the Dutch version of the Evidence-Based Practice Attitude Scale (EBPAS). Health Res Policy Syst. 2015;13:69.

29. Cook CR, Davis C, Brown EC, Locke J, Ehrhart MG, Aarons GA, et al. Confirmatory factor analysis of the Evidence-Based Practice Attitudes Scale with school-based behavioral health consultants. Implementation Science. 2018;13(1):116.

30. Kim M. Assessing attitudes towards evidence-based practice among social workers in Hong Kong. Journal of Social Work. 2018:1468017318784077.

31. Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of psychological research online. 2003;8(2):23-74.

32. Egeland KM, Ruud T, Ogden T, Lindstrom JC, Heiervang KS. Psychometric properties of the Norwegian version of the Evidence-Based Practice Attitude Scale (EBPAS): to measure implementation readiness. Health Res Policy Syst. 2016;14(1):47.

33. Reise SP. The rediscovery of bifactor measurement models. Multivariate behavioral research. 2012;47(5):667-96.

34. Chen FF, West SG, Sousa KH. A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research. 2006;41(2):189-225.

35. Hughes FPITBDJ. The Wiley handbook of psychometric testing : a multidisciplinary reference on survey, scale, and test development. First edition ed:

Hoboken, NJ ; Chichester, West Sussex : John Wiley & Sons, ; 2018.

36. American Educational Research Association.; American Psychological Association. NCoMiEJCoSfEaPTUS. Standards for educational and psychological testing. . 6th ed.) ed: Washington, DC : American Educational Research Association, [2014]. ; 2014 2014.

37. Santesson AJH. The Deplyftet Implementation Program http://www.sfbup.se/deplyftet/: The Swedish Association of Child and Adolescent Psychiatry 2019 [cited 2019.

38. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross‐cultural health care research: a clear and user‐friendly guideline. Journal of evaluation in clinical practice. 2011;17(2):268-74.

39. Tabachnick BG, Fidell LS. Using multivariate statistics: Allyn & Bacon/Pearson Education; 2007.

40. IBM Corp. Released 2016. IBM SPSS Statistics for Windows VA, NY: IBM Corp. . 2016

41. Muthén L, Muthén B. Mplus. The comprehensive modelling program for applied researchers: user’s guide. 2015;5.

42. Yu C-Y. Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes: University of California, Los Angeles Los Angeles; 2002.

43. Jackson DL, Gillaspy Jr JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological methods. 2009;14(1):6.

44. Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist. 2018;73(1):3.

45. Uppdrag Psykisk Hälsa S. Kartläggning Barn- och ungdomspsykiatrin 2016 https://www.uppdragpsykiskhalsa.se/assets/uploads/2017/11/Kartla%CC%88ggning-2016-%E2%80%93-Barn-och-ungdomspsykiatri.pdf: SALAR Swedish Association of Local Authorities and Regions/ SKL Sveriges kommuner och landsting; 2017.

Table 1

EBPAS item and scale Means, Standard Deviations, Reliability estimates and Item-total correlation

Total

scale

Sub-

scale

EBPAS subscales and Total

Requirements

2.70

0.82

.88

Agency required

2.61

0.92

.49

.88

Supervisor required

2.51

0.97

.44

.81

State required

2.98

0.84

.48

.63

Appeal

3.24

0.52

.74

Make sense

3.24

0.66

.43

.61

Intuitively appealing

3.15

0.75

.44

.58

Colleagues happy

with therapy

3.11

0.72

.43

.46

Enough training

3.47

0.66

.53

.48

Openness

2.88

0.58

.76

Will follow a treatment manual

2.98

0.78

.58

.68

Like new therapy types

2.90

0.72

.31

.52

Therapy different than usual

2.58

0.80

.39

.47

Divergence

1.20

0.59

.60

Research-based treatments not useful

0.92

.30

.42

Would not use manualized therapy

0.67

0.88

.53

.36

Clinical experience more important

1.90

0.84

.34

.42

Know better than researchers

0.28

0.93

.35

.27

EBPAS Total score

2.92

0.42

.81

Note: N= 565

Total, subscales and item means scores range from 0 to 4.

r= corrected item total

Table 2 Model fit information
Model	c²	Df	p	CFI	RMSEA	WRMR
1a	555.1	85	<.001	.973	.099	1.641
1b	399.0	84	<.001	.982	.081	1.363
2a	687.5	87	<.001	.965	.110	1.922
2b	558.5	86	<.001	.973	.098	1.704
3a	653.7	78	<.001	.967	.114	1.869
3b	582.9	77	<.001	.971	.108	1.760


Table 3. Standardized factor loadings from model results
	Model
	1b		2b		3b
EBPAS sub-factors and items					Sub-factor		General
Requirements			0.475
Agency required	0.960^a		0.960^a		0.872^b		0.449^b
Supervisor required	0.960^a		0.960^a		0.872^b		0.366^b
State required	0.779		0.780		0.597		0.492
Appeal			0.863
Make sense	0.638		0.643		0.389		0.514
Intuitively appealing	0.625		0.630		0.303		0.532
Colleagues happy	0.702		0.692		0.523		0.513
Enough training	0.837		0.840		0.724		0.467
Openness			0.739
Will follow a treatment manual	0.912		0.916		0.359		0.675
Like new therapy types	0.628		0.623		0.653		0.337
Therapy developed by researchers	0.821		0.817		0.328		0.707
Therapy different than usual	0.607		0.613		0.653		0.490
Divergence		-0.647
Research-based treatments not useful	0.594	0.591		0.610		-0.372
Would not use manualized therapy	0.827	0.841		0.392		-0.584
Clinical experience more important	0.514	0.494		0.466		-0.325
Know better than researchers	0.383	0.393		0.176		-0.467
Note: 1b= First-order added error correlation (item 9 and 10), 2b = Second-order added error correlation (item 9 and 10). 3b= Bifactor model with error correlation (item 9 and10). Sub-factor= item loading on sub-factor, General=item loading on general factor For model 2b the loadings to the general factors are on the rows of the factor labels. For model 3b, to highlight the items providing the best discrimination on the general factor, items loading greater than .50 on the general factor are in boldface type. Items with larger loadings on group factor than general factor are also in bold face type. ^aresidual covariance=0.65, *p<.001. ^bresidual covariance =0.67 p<.001. All factor loadings were statistically significant ** p<.001

Title and legend: Additional file 1. Reporting guidelines

Caption Additional file 1 Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross‐cultural health care research: a clear and user‐friendly guideline.

Title and legend: Additional file 2. Reporting guidelines

Caption Additional file 2 Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist. 2018;73(1):3

Title and legend: Additional file 3. Reporting guidelines

Caption Jackson DL, Gillaspy Jr JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological methods. 2009;14(1):6.

Download PDF

Version 1

posted

You are reading this latest preprint version

Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale in a large and representative sample of Child and Adolescent Mental Health practitioners: Is the use of a total scale score justified?

Status:

Version 1

Abstract

Figures

Contributions to the literature

Background

First-order models

Second-order models

Bifactor model

Method

Participants

Measures

Procedure

Analyses

Results

Model Comparison using CFA

First-order models

Second-order models

Bifactor model

Discussion

Strengths and limitations

Implications

Conclusion

List of abbreviations

Declarations

References

Tables

Additional file Legend

Supplementary Files

Status:

Version 1