Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale in a large and representative sample of Child and Adolescent Mental Health practitioners: Is the use of a total scale score justified?

Background: High-quality instruments are essential for implementation practitioners and researcher; only valid and reliable measures can confidently and consistently measure what they are intended to measure. Psychometrically sound and pragmatic measures have the potential to guide the tailoring and evaluation of an implementation program thus contributing to the knowledge base of implementation science. The Evidence-Based Practice Attitude Scale (EBPAS) is one of few implementation instruments with strong psychometric qualities and is widely used to measure attitudes toward Evidence-Based Practice (EBP) in implementation practice and research. The Multidimensionality of the EBPAS has repeatedly been supported. Whether all of the fifteen individual items contribute to the EBPAS total scale beyond their subscales (Appeal, Requirements, Openness and Divergence) has yet to be proven. In addition, the Divergence subscale has been questioned because of low correlation with the other subscales and low inter-item correlations. Methods: The factor structure of the EBPAS was investigated in a child- and adolescent psychiatry setting (N=925, 62%, n =570) using first- and second-order Confirmatory factor analysis, thus replicating Aarons' initial studies. In addition, the present study is the first to use a bifactor model in a large and representative sample. Results: The EBPAS was supported as a multifaceted instrument. The EBPAS had a rather strong general factor supporting the use of the inventory as a single measure of attitudes. There were still four dimensions of attitude after the general factor was accounted for. All items, including the items of the Divergence factor, contributed to general attitude as well as to their specific construct. Conclusions: The present study is the first to show that it is justified to use and interpret the EBPAS as an instrument measuring global as well as specific attitudes toward EBP. Sum score and subscales scores can be used in implementation research and practice. Implementation science and practice depends on psychometrically strong instruments. The EBPAS is one of few, tested with proven multidimensionality, but it is unclear if the use of all of the subscales (including Divergence) and total scale is justified. This is the first study to use bi-model-CFA in a large and representative sample, justifying the use of all of the subscales as unique and reliable indicators of specific EBP-attitude while supporting the EBPAS as a global measure. The EBPAS Total scale and subscales are valid measures for general and specific attitudes and can be used when tailoring and evaluating implementation programs.


Contributions To The Literature
Implementation science and practice depends on psychometrically strong instruments. The EBPAS is one of few, tested with proven multidimensionality, but it is unclear if the use of all of the subscales (including Divergence) and total scale is justified. This is the first study to use bi-model-CFA in a large and representative sample, justifying the use of all of the subscales as unique and reliable indicators of specific EBP-attitude while supporting the EBPAS as a global measure.
The EBPAS Total scale and subscales are valid measures for general and specific attitudes and can be used when tailoring and evaluating implementation programs. Background A current priority for governments, policymakers and planners across the world is to disseminate empirically supported treatments in order to improve health. Evidence-based practice (EBP) is a paradigm that aim to integrate best available research with clinical expertise and patient values (1). Multiple factors at different levels; community, organizational and individual are supposed to influence the implementation of EBP (2)(3)(4)(5)(6)(7).
There is growing evidence that implementation may be enhanced and better supported by stakeholders if tailored to the local circumstances (8,9). Potential enablers and barriers for change need to be identified early in the implementation planning process on the basis of previous research and practical experience (10). Different strategies are then selected and matched to address these determinants for change (7,(9)(10)(11)(12).
Health practitioners' attitudes and values play an important role in implementing EBP in real world settings (13). Positive attitudes along with subjective norms and a person's self-efficacy influence the decision on whether or not to implement a new practice (14,15). According to Rogers, the innovation-decision process starts with knowledge and persuasion when the provider forms a favorable or unfavorable attitude toward the innovation (16). Both general and specific constructs play an important role in understanding and predicting behavior. General predictor variables are suggested as the best to predict general outcomes and specific predictor variables the best to predict specific outcomes (17). General attitudes may explain and predict broad patterns of behavior whereas a specific attitude is a better measure of specific behavior (18).
Assessing both specific and general attitudes towards EBP may help the implementation tailoring process and the evaluation of an implementation program . Psychometrically sound and theory-based measures are also essential for the field of implementation science (19)(20)(21).
The Evidence-Based Attitude Practice Scale (EBPAS) is an instrument of high overall psychometric quality (20,22). The EBPAS was developed by Aarons on the basis of a comprehensive literature review and consultation with mental health service providers and researchers. Four important domains of attitudes were identified; the appeal of EBP, the provider's openness to change, who require the change and the perceived difference between current practice and EBP (13,23). Thus the EBPAS has four dimensions Appeal, Openness, Requirements and Divergence. Subscale scores are used to obtain information about the specific domains of attitude and a sum score (where the Divergence items are reversed) is used to estimate the global attitude towards EBP. Previous studies suggest good internal consistency for the subscale scores except for the Divergence scale and adequate to good internal consistency for the EBPAS total scale (13,23,24). The construct validity has been studied using different sample sizes (n= 146 vs. n=1,273), in various settings (e.g., mental health, health, correctional service, social service, residential care, educational sector), with participants at different levels (front line providers, administrators, consultants) and diverse cultures (e.g. USA, Europe, Africa, Asia) (13,(23)(24)(25)(26)(27)(28)(29)(30). Different factor models with and without correlated residuals have been fitted to the data: a) first-order models, b) second-order models, and c) a bifactor model.

First-order models
First-order models focus on the subdomains. According to recent standards (31), the fourfactor model has yielded poor to acceptable model fit (26). The factor loadings have generally been moderate to strong with weakest loadings for Divergence. Correlations between factors have generally been moderate except between Divergence and some of the other sub-factors (13,24). Attempts have been made to test a more complicated fivefactor model with two facets of the Divergence scale (26), but subsequent research did not support this model (27). To summarize, there has been support for four, somewhat correlated, sub-factors in the EBPAS inventory. However, a problem with first-order models is that they do not justify composite scores.

Second-order models
In second-order models, individual items load on their specificity factors which in turn load on a higher order construct that accounts for the commonality shared by the facets. For the second-order model with one general factor common to all sub-factors, model fit has been acceptable with generally strong and significant first factor loadings and moderate second factor loadings. Again the Divergence scale showed the weakest first and second factor loadings (23). Several studies have supported the second-order model, with (28,32) and without added covariance (25,27). In brief, the second-order models add support for the total scale of the EBPAS. The problems with the Divergence scale that have been shown in first-order models are also seen in the second-order models.

Bifactor model
In bifactor models individual items load both on a general factor and a specific factor.
Only bifactor models can separate specific factors from the general factor. Also, bifactor models can identify whether a facet still exists after partialling out the general factor (33,34). A bifactor model has been tested in a sample of 270 practitioners (28). The model fit was good and even slightly better than the second-order model. First factor loadings were in general moderate to strong and significant. General factor loadings were moderate and significant except for Divergence, which were weak or non-significant. The results gave some support to the hypothesis that the variance in the individual items can be attributed to both a general factor and uniquely to their specific factors. However, the items in Divergence seem to have little in common with the other items.
To summarize, understanding the structural validity of tests is essential for evaluating interpretability of provided scores and their relation to outcome variables (35,36). The construct validity of the EBPAs has been supported in different contexts using different models. Previous studies support the hypothesis that the EBPAS can be interpreted as a multidimensional scale with four correlated subscales. Whether all of the individual items contribute to general attitude (the EBPAS total scale) beyond their subscales has yet to be proven. In other words, is it meaningful to interpret the EBPAS as a sum score? And do all of the subscales, including Divergence, uniquely contribute to the EBPAS measure?
The primary aim of this study was to contribute to the evidence regarding the construct validity of the EBPAS by investigating the factor structure in a large and representative sample of 570 child mental health practitioners, thus replicating Aarons initial validation study (24). We used a series of confirmatory factor analysis (CFA) to test three models: a) a first-order model with four separate factors, b) a second-order model with an added general factor, and c) a bifactor model. A secondary aim was to validate the Swedish version of EBPAS.
Thus, the EBPAS will be tested based on previous literature for: Whether a first-order, a second-order or a bifactor model best fits the data and corollary, that the fit for the best model is adequate or if error correlations previously shown among items are necessary?
Whether all of the different sub-factors contribute uniquely to the concept of EBP, in other words, whether the sub-factors are uniquely supported when the general attitude has been accounted for?

Measures
The EBPAS was translated to Swedish (38). The EBPAS consists of 15 items rated on a Likert scale ranging from 0 (not at all) to 4 (to a very great extent) (13). Previous studies report Cronbach alpha, ranging from .76 -.91 except for the Divergence scale (.59 -.66) (13,23). Mean values were high for most of the positively phrased EBPAS items, see table 1. For negatively worded items, the mean values were generally lower, Skewness was generally negative (J-shaped) for positively phrased items, but above 1.0 only for one item. Three items had a kurtosis value above 1, one; Item 15 "enough training" was extremely kurtotic (2.6) and also had a positive skew above 1.0. This item had also a very obvious ceiling effect with 54 % of the ratings in the highest response category. Reliability ranged from .58 to .88 (table 1). Test of item discrimination for the Divergence scale yielded values between .27 and .53 suggesting that these items were not very consistent with the EBPAS total scale or the subscale. However, item analyses showed no significant improvement in alpha values after removing single items.
Insert table 1 about here Procedure Data were collected as a baseline assessment as part of a larger trial from October 2014 to February 2017. A web-based survey was distributed per email to outpatient child mental health front-line professionals. Two to five reminders were sent if necessary.

Analyses
Five participants were excluded; three because of missing data for all items, one because all responses were the same and one for being multivariate outlier, (through Mahalanobis distance with p <0.001) leaving 565 cases for analysis. Two univariate outliers (extremely low z scores < 3.0) were replaced with their neighbors (39). SPSS statistical software version 24 was used for descriptive statistics (40). A series of CFA were conducted to test the factor structure since the factor structure of the EBPAS has been thoroughly examined in previous studies using both exploratory and confirmatory methods (23,24).
The CFA models were estimated with MPLUS 8 (41). The weighted least squares -robust mean and variance adjusted (WLSMV) estimator was used since the items were ordinal (Likert scale). Cases with missing data were included in the CFA because the WSLW estimator permits their inclusion. Several different fit indices were used to summarize the model fit; chi-squared index ( 2 ), the comparative fit index (CFI), the Root mean square error of approximation (RMSEA), and the Weighted root mean square residual (WRMR). For categorical models Yu has recommended .95 for CFI and TLI (Tucker Lewis index), 0.05 for RMSEA and 1.00 for WRMR (42)(43)(44). The MPLUS program provides a likelihood ratio 2 test for difference testing (DIFFTEST) for model evaluation.

Results
Model Comparison using CFA We tested three models, first the one suggested by Aarons Table 2), especially CFI suggested that most of the covariance was represented in this model. To make the model feasible we fixed both item 11 ("Supervisor required") and 12 ("Agency required") in the indicators of the Requirements factor to 1. In line with Aarons et al., the strongest correlation was between 9 and 10 from the Appeal subscale (the items about "Intuitively appealing" and "Making sense" ) and adding this error correlation increased fit significantly (see Model 1b, Table 2). The estimated (error) correlation between item 9 and 10 was .65. The standardized loadings of Model 1b are depicted in Table 3, generally all loadings were high (above .5), with the exception of Divergence's loading on item 3 "Know better than researcher" (0.383). Factor inter-correlations were generally medium and significant.
Insert table 2 about here  Insert table 3  Second-order models Model 2a and 2b (shown in Figure 1) tested the second-order model. Model 2a did not include the correlation between item 9 and 10, that was added to Model 2b which had two degrees of freedom more than Model 1b and is therefore more parsimonious than Model 1b. The fit for this model was also very good (at least according to the CFI), even if the fit for this model was not as good as for Model 1b. The standardized factor loadings were similar to the four-factor model. First-order factor loadings were generally strong and statistically significant. All of the first-order factors loaded significantly and strongly on the General Attitude factor (see table 2, and figure 1).
Modification indices were difficult to interpret, for example they suggested additional correlation between Requirements and Appeal, but also between Openness and Divergence. Adding one of these increased model fit to a level comparable to Model 1b.
Because this was not an expected correlation, we refrained from adding it to the final hierarchical model.
Insert figure 1 about here.

Bifactor model
The bifactor models take another strategy and gather all the common variance of all the items into a single factor while estimating the unique contribution of the factors orthogonal from this common factor. Because the factors include the unique common variance of the observed variables of each sub-factor, the model can be used to test whether the data supports the unique contribution of each of the four scales of the EBPAS.
To succeed with the estimation of the model, a number of parameters were fixed due to problems with convergence and multi-collinearity. One of the items of Divergence was fixed to 1 and for the Openness factor we needed to fix the error variance of two items otherwise the estimation did not converge. All factors were uncorrelated in the first model (Model 3b, see Table 3 and Figure 2); with these restrictions the model converged and showed a very good fit. The only difference between Model 3a and 3b was the added correlation between item 9 and 10. All factor loadings were statistically significant (p<0.001). The average item to general factor loading was .49 (.32 -.71), see Table 3 and Figure 2. First-order factor loadings were generally weaker in the bifactor model, but loadings were still generally moderate. In addition, 9/15 items had a higher loading on the group factor than the general factor, further supporting multidimensionality.
We tested the unique information in the factors by deleting them one at a time from Model 3b. The CFI decreased from .987 to .938, .962, .965, and .968, for the four factors, (Requirement, Appeal, Openness, and Divergence) respectively. RMSEA increased from .073 to .154, .119, .115, and .109, respectively. All changes were clearly significant (p <.001) based on the DIFFTEST. From these estimations, it is obvious that all factors had a unique contribution, and also that the Requirement factor was clearly the one with the strongest unique contribution.
Insert Figure 2 about here Discussion This large and representative study of the EBPAS, utilizing first-order, second-order and bifactor CFA-models, strongly supports the proposed structure of a higher order factor and four specific factors. The EBPAS had a rather strong general factor supporting the use of the inventory as a single measure of attitudes toward evidence-based practice.
Furthermore, there were still four distinct dimensions of attitude after the general factor was accounted for. All items, including the items of the Divergence factor, contributed to general attitude as well as to their specific construct. In other words, the subscales can be scored and used as indicators of specific aspects of EBP attitude when a more precise and specific analysis is needed.
Although these four dimensions seem to be distinct, they address a common theme of attitude to EBP. Two possible ways of representing this information are a by a secondorder model with a general factor above the four factors explaining common variance between factors or by a bifactor model where the common variance of the items are estimated by a general factor as well as separate group factors. The results indicate that both models had very good model fit. The second-order model did not have the best fit, but was clearly the most parsimonious model. The pattern of item to factor loadings, identified in prior analyses, were found in our analysis as well, with strong item to factor loadings across all factors except for Divergence and generally strong second-order factor loadings (23). More important, Divergence loading on Global attitude was strong and significant. In previous studies Divergence to general factor loading has been nonsignificant (27,28) , small (23), and moderate in one study, (32) . However, the finding of moderate factor loadings in the study of Egeland was compromised by an out of range second-order factor loading for Appeal, indicating model misfit. The present study clearly supports the second-order model with four first-order factors that load onto a higher-order factor reflecting global attitude.
An advantage of the bifactor model is that it allows direct exploration of the extent to which subscales reflect a common target domain and the extent to which they reflect a primary subdomain. Items with larger first factor loading than general factor loading are seen as a relatively better measure of the specific construct than of the general construct.
The bifactor model had good fit (33). All factor loadings were significant, in contrast with previous results (28). General factor loadings were moderate to strong. All items loaded substantially on the general factor and the average item to general loading was moderate, suggesting that items in the EBPAS are all affected by attitude toward EBP. For the Requirement factor, the factor loadings on individual items were still strong when the common variance was partialized out. For the other factors the results was mixed, but all except one item (item 3 "Know better than researcher") exhibited more than a substantial loading on their specific factor, indicating that all factors had unique contribution after adjustment for the impact on items by the general factor. This finding was further supported by the fit statics after removing one factor at a time, which resulted in significant decrease in model fit for each factor. The analysis supports the model with four facets of willingness to adopt evidence-based practice above and beyond the general attitude toward evidence-based practice.
Consistent with previous studies, the Divergence factor is the most problematic one with the weakest loadings in the first-order model (13,26). In contrast with these studies, inter-correlations between Divergence and the other factors were moderate and significant and item loadings were generally moderate. The same pattern of loadings was found in the second-order model but with strong and significant loadings on the secondorder factor. In addition, the Bifactor analysis suggests that all items of the Divergence subscale contribute to the EBPAS total scale and most of them contribute to their subscale after the global attitude has been accounted for. Although loadings to the specific factors were somewhat weaker for Divergence, the average item on factor loading was still moderate. In line with previous studies, Item 3 "Know better than researchers" was the most problematic item, with the weakest factor to item loading in all three models (13,25,26). In the bifactor analysis, however, the general factor loading on item 3 was strong; indicating that item 3 contributes to the homogeneity of the EBPAS. To sum, the items of the Divergence subscale are definitely the most heterogeneous ones, but the results support Divergence. All items contribute to general attitude and most of them are indicators of the specific Divergence construct beyond general attitude.
To summarize, the estimations clearly supported the four-factor model suggested by Aarons et al. (13). After controlling for the single error correlation, previously reported by Aarons and others, there was not much room for more factors (23). In addition, it was found that the EBPAS has a rather strong general factor that supports the use of the inventory as a single measure of attitudes toward evidence-based practice. The bifactorial analysis added support to the model with four facets of the EBPAS, suggesting it to be worthwhile separating these factors when more fine-grained analyses are asked for.

Strengths and limitations
Some limitations should be noted. We could not compare respondents with nonrespondents to examine potential bias. However, the sample representativeness seems to be good compared to available national data describing the child mental health service workforce (45). In addition, the analysis did not take in account the clustering of employees by service unit. However, previous studies have shown that the variability seems to be between persons rather than workgroups and clinics (23) .
There are some strengths to mention. The response rate of 62% is quite high for being a web-based survey. The sample represents practitioners from a geographically diverse area covering more than a third of Sweden's child and adolescent psychiatric services from more than half of the Swedish regions which add to generalizability. All responders were front-line practitioners. The sample size of 565 was robust for the Confirmatory factor analyses.
Additional research is needed to further establish the factor structure of the EBPAS in different contexts. More research is also needed to investigate the relation between the EBPAS's subscales, total scale and different outcome variables, and the EBPAS's ability to predict adoption, fidelity and sustainment of EBP.   Note: 1b= First-order added error correlation (item 9 and 10), 2b = Second-order added error correlation (item 9 and 10). 3b= Bifactor model with error correlation (item 9 and10). Sub-factor= item loading on sub-factor, General=item loading on general factor For model 2b the loadings to the general factors are on the rows of the factor labels. For model 3b, to highlight the items providing the best discrimination on the general factor, items loading greater than .50 on the general factor are in boldface type. Items with larger loadings on group factor than general factor are also in bold face type. a residual covariance=0.65, ***p<.001. b residual covariance =0.67 ***p<.001. All factor loadings were statistically significant *** p<.001

Implications
Additional File Legend