Evaluating the Implementation and Feasibility of a Guideline-Driven Process for Improving the Cultural Responsiveness of Non-Aboriginal Alcohol and Drug Treatment Services


 Background To improve Australian Aboriginal and Torres Strait Islander people’s access to, and experience of, healthcare services, including Alcohol and other Drug (AoD) treatment services, principles and frameworks have been developed to optimise cultural responsiveness. Implementing those principles in practice, however, can be difficult to achieve. This study has five aims: i) to describe a five-step process developed to operationalise improvements in culturally responsive practice in AoD services; ii) to evaluate the fidelity of implementation for this five-step process; iii) to identify barriers and enablers to implementation; iv) to assess the feasibility and acceptability of this approach; and v) to describe iterative adaptation of implementation processes based on participant feedback.Methods Participating services were 15 non-Aboriginal AoD services in New South Wales, Australia. Implementation records were used to assess the implementation fidelity of the project. Structured interviews with chief executive officers or senior management were conducted, and interview data were thematically analysed to identify project acceptability, and the key enablers of, and barriers to, project implementation. Quantitative descriptive analyses were performed on the post-implementation workshop survey data, and responses to the free text questions were thematically analysed. Results A high level of implementation fidelity was achieved. Key enablers were the timing of the introduction of the five-step process, the active interest of staff across a range of seniority and the availability of resources and staff time to identify and implement activities. Key barriers included addressing the unique needs of a range of treatment sub-groups, difficulty adapting activities to different service delivery models, limited time to implement change in this evaluation (three months) and the varied skill level across staff. The project was rated as being highly acceptable and relevant to services, with planned changes perceived to be achievable and important by CEOs/managers and staff. Based on staff feedback after the project was implemented at the initial services, several improvements to processes were made.Conclusion The operationalisation of the five-step process developed to improve cultural responsiveness was feasible and acceptable and may be readily applicable to improving the cultural responsiveness of a wide variety of health and human services.


Introduction
There is global concern about the impact of drug and alcohol use on health and wellbeing. The United Nations sustainable development goals include a target to strengthen the treatment of harmful substance use (1). In Australia, enduring impacts of colonisation and racism have included Aboriginal and Torres Strait Islander peoples (hereafter referred to as Aboriginal) experiencing a disproportionate amount of harm from substance use (2). To improve Aboriginal people's access to, and experience of, healthcare services, including substance treatment programs, principles and frameworks have been developed to optimise cultural responsiveness (e.g. 3). Cultural responsiveness is an ongoing process of adapting systems, services, and practice to t with user preferences (4). Although non-Aboriginal health services and clinicians may aim to provide health care that is culturally responsive to Aboriginal people, implementing its key principles can be di cult to achieve in practice (5).
Research to improve cultural responsiveness is in its infancy (6). Key principles of culturally responsive practice for alcohol and other drug treatment services have been identi ed, including promoting client choice, facilitating community engagement and providing person-centred practice, which facilitate both best practice and an organisational culture which is inclusive and culturally responsive to all clients' needs (7). Nevertheless, barriers to practice improvement have also been identi ed at both the organisational-level, including leadership and organisational processes (8,9) and the individual clinicianlevel, including a lack of knowledge and an attitudinal resistance to change (10). A key hypothesis is that these barriers to implementing the principles of cultural responsiveness exist because there is a lack of clarity about how these principles can be operationalised (11).
Operationalising the principles of culturally responsive practice will require best-evidence implementation strategies (12). Current evidence shows that since the provision of treatment guidelines, policies and training are ineffective on their own (13,14), active learning processes are required to meaningfully engage key services' staff in the process of change (15,16). The implementation process should be designed to engage staff in determining both what changes to make and how those changes should be implemented (16). That exibility is critical because there is evidence that different services will have different levels of cultural responsiveness at any one point in time (17) and because the approach to adopting more culturally responsive practice will need to be tailored to the individual circumstances of different services (16,(18)(19)(20)(21).
This study has ve aims: i) to describe a ve-step process developed to operationalise improvements in culturally responsive practice; ii) to evaluate the level of implementation delity for this ve-step process; iii) to identify barriers and enablers to implementation; iv) to assess the feasibility and acceptability of this approach; and v) to describe the process of iteratively adapting implementation processes based on participant feedback.

Study design
The project was developed and implemented using the principles of community-based participatory research, aiming to empower services to make changes relevant to their local context and priorities, while making use of their existing strengths (22). The project schedule allowed three months between baseline and follow-up audits. A mixed methods approach was used to evaluate the delity of project implementation and assess the feasibility of the project.

Participants
Non-Aboriginal non-government organisation (NGO) AoD treatment services from six Primary Health Network (PHN) districts in New South Wales (NSW), Australia, were invited to participate in the project by the commissioning PHNs. Of seventeen services expressing interest, fteen consented to participate (88%) (hereafter referred to as participating services). Participating services included a variety of AoD service types/delivery models including residential rehabilitation (n = 3), day programs (n = 2), centrebased counselling and support (n = 3), outreach counselling and support (n = 4), groupwork and phone support (n = 1) and group or individual youth services (n = 2).
Key staff working at participating services were nominated by management to attend the audits and implementation workshop and to participate in interviews and/or surveys. Services were encouraged to include chief executive o cers (CEOs) or senior managers in the audits and implementation workshops so that they could contribute their detailed knowledge about services' processes and policies, and so that staff with the capacity to decide and enact service level changes would be present to increase the likelihood that planned activities were implemented into services.
A project Aboriginal advisory group was established, comprising Aboriginal members selected by expressions of interest submitted to the Aboriginal Drug and Alcohol Network (ADAN), invited representatives of the Aboriginal Health and Medical Research Council of NSW (AHMRC), and service providers selected by the project team. Their role was to advise on the development of the guideline and implementation of the ve-step process for improving cultural responsiveness. They were encouraged to provide feedback on ways to improve the project throughout the ve-step process (23

Data collection and analysis
Implementation records. Throughout the project, a tracking document was maintained to establish the number and timing of services completing each component of the project. The essential components of the project to be delivered at each service included: baseline audit, provision of written audit feedback, implementation workshop and action plan and follow-up audit.
Interviews with CEO or senior management. Semi-structured telephone interviews regarding enablers of, and barriers to, implementation of practice change were conducted with CEOs/managers of participating services after the baseline and follow-up audit. An interview guide was followed, which aimed to capture interviewee perspectives of audit outcomes; priorities for the implementation workshop (in rst interviews only); feedback on the auditing process; preferences for ongoing development of cultural responsiveness, and perceptions about changes arising from the project (in second interviews only). Interviews were digitally recorded and transcribed verbatim. A report summarising the priorities identi ed by the CEO/manager for the implementation workshop was provided to the project team to assist with planning the implementation workshops. Interview data were thematically deductively analysed to identify enablers, barriers, and acceptability of the project using NVivo 12 (24). Implementation workshop feedback survey. At the end of the workshop, participants who attended the workshops were provided with a link to a brief online anonymous survey including 24 items about the participants' role, reasons for attending and perceptions of the audit and workshop (see Supplementary Appendix page 1 for a list of items). Simple descriptive analyses (e.g. frequencies) were performed using Excel (25), and responses to the free text questions were thematically analysed using NVivo 12 (24).
Iterative adaptation of the project components during the project implementation. To iteratively adapt the project in response to feedback from staff obtained via interviews, responses were reviewed to identify feedback or potential improvements around project processes, acceptability, and relevance.

Results
A ve-step process for operationalising improved cultural responsiveness A ve-step process to operationalise improvements in culturally responsive practice was devised, implemented, and evaluated using a program logic framework (see Supplementary Appendix page 2 for the program logic). First, a best-practice guideline that describes key elements of culturally responsive service delivery in non-Aboriginal AoD treatment services was developed (7). The development of this guideline was led by an Aboriginal researcher with experience working with AoD services (RW). Extensive consultation was undertaken with Aboriginal community members in NSW including AoD clients/consumers, and with the project Advisory Group (which included senior Aboriginal AoD clinicians), to identify how they wanted AoD services to be delivered to Aboriginal people. Next existing guidelines detailing ways of working with Aboriginal people from health, community services, education and natural resource sectors and government departments were reviewed (n = 80) and strategies consistently reported in each of them were selected. This information was synthesised into 6 themes and based on the details in the other guidelines, the consultation ndings and practice experience, 21 action areas were devised that describe key elements of culturally responsive service delivery in non-Aboriginal NGO AoD services. For example, Theme 1 Creating a welcoming environment, is operationalised in two action areas i) a welcoming greeting, which means that the service has a process to greet and welcome Aboriginal people when they arrive at the service that is consistently implemented; and ii) a welcoming physical environment, which refers to the physical environment of the service where Aboriginal people spend time being comfortable, having Aboriginal resources on display and being able to accommodate children and family members (7).
Second, a structured baseline audit of current practice, relative to the best-practice guideline, was designed and implemented in each participating service. Prior to the baseline audit, services were sent information explaining the audit process, what the audit is about, who should attend, how long it will take, information about the audit report and options for follow-up activities. Audits were conducted by two trained auditors who were external to the service, with at least one auditor who was Aboriginal. Audits were conducted in the setting where the service is delivered and took between 90 minutes to two hours. Service staff attending the audits included a service or program manager or team leader and service delivery staff. Having a range of staff attending assisted in gaining explanations of both direct practice and organisational approaches to working with Aboriginal people. Where possible, the same staff attended the follow-up audit. Only information collected on the day of the audit was included in the scoring.
Third, individualised written feedback from the audit ndings were provided to each of the participating services. This was provided as a report listing all guideline action areas and a rating for each area re ecting the level of evidence observed for each of the activities speci ed in the audit criteria (limited, some, good or excellent). An excerpt from an example baseline report is included in the Supplementary Appendix (page 3).
Fourth, implementation workshops were held with key staff from services to explain the guideline, review the written audit feedback, set goals for improvement, and develop an action plan (to operationalise their improvement goals). Speci cally, staff from each service were invited to select three priority action areas for their service (identi ed from their written audit feedback) to progress over the next three months. They then made a detailed action plan to implement change, tailored to their speci c service (an example action plan is included in the Supplementary Appendix, page 5).
Fifth, a follow-up audit of services was conducted after three months to assess change in culturally responsive practices. Services were again provided with individualised written feedback, similar to the baseline report and also including some discussion of any changes that had occurred. An excerpt from an example follow-up audit report is included in the Supplementary Appendix (page 7).

The delity of the implementation of the ve-step process
The ve-step process was predominantly implemented as planned, with a high level of delity achieved for each of the ve project components: a) development of guidelines; b) baseline audit of services; c) written feedback; d) staff attending implementation workshops; and e) follow-up audit of services. The guideline was published and provided to all participating services after the baseline audit. Twelve of the fteen participating services completed all four service-speci c project components (b to e). One service completed (b) to (d), one completed (b) only, and one did not complete any of the project components (see Table 1). The main reasons for services not completing all the project components included staff turnover during the project or the project team was unable to contact service staff to arrange subsequent components. Follow-up audits were scheduled to be completed 12 weeks after the baseline audit, however the average time between audits was 18 weeks (range 14-28 weeks) ( Table 1). Delays were predominantly due to scheduling commitments at services and staff availability. Of those with the longest delays, one service was re-scheduled due to a local bush re (Cluster 3) and another was delayed due to unavailability over the Christmas holiday period (Cluster 5).
Interviews with CEOs/managers at baseline (n = 14) and follow-up audit (n = 12) showed that audit reports were received by all services that completed an audit and an interview. Staff and CEOs/managers were actively engaged in the project, with staff from 13 services attending implementation workshops (Table 1). Six CEOs/managers reported some uncertainty among some of their staff about the project requirements before the baseline audit (about the project's background, expectations, scheduling, and next steps). These uncertainties were clari ed during discussions at baseline audits. In the implementation workshop feedback survey, most (33/35) staff reported that they were aware that the baseline audit had occurred at their service and most (26/35) were aware of the outcome of the audit.

Enablers and barriers to implementation
Thematic analysis of interview responses identi ed barriers and enablers to implementing cultural responsiveness activities (Table 2). Enablers included aligning the timing of the project with setting up new services, having multi-level buy-in for the project and having resources/staff time available to support project activities. Barriers included limited funding and time available to complete planned activities, challenges hiring Aboriginal or culturally responsive staff, the need to balance the needs of varied population groups, di culty adapting activities to different service delivery models and limited time to implement change. New services/programs or those undergoing internal changes (e.g. re-structuring, strategic planning, or developing/implementing Reconciliation Action Plans) were well positioned to implement changes to culturally responsive practice.
"The speci c service that was being looked at is actually a new program for us and we're still actually yet to o cially start the program. So, it was very useful to actually have [the auditor] come in actually just before we actually commenced service delivery and actually look at where we are in terms of our…cultural intelligence and cultural competence… and where we're at as a service, before we actually start commencing service delivery." Manager, Service J Interest in the project from multiple levels within services Buy-in from CEOs/managers and staff who attended project activities led to a productive environment which supported action around culturally responsive practice. Services often had clients from multiple ethnic, cultural, and religious backgrounds, which required them to be responsive. This resulted in some services having a limited capacity to tailor speci c work ows and processes to Aboriginal clients.
"I think for our organisation…we span over quite a diverse geography…so we actually have services on lots of different country. What we're nding to be somewhat di cult is how do we as an organisation support cultural competency from an organisational level, to then actually pay respect to the nuances of the different communities that we're in." Manager, Service L Di culty adapting activities to different service delivery models Services differed with respect to their delivery models, geographical locations, and organisational size, which meant that activities had to be adapted or in some cases, were not feasible for speci c settings. Some larger state and national organisations had internal processes which required longer timeframes to implement activities, and in some instances, proposed activities were not feasible because of these processes/policies (e.g. including Aboriginal board members).
"…the other activity that we'd planned was around trying to have a stronger connection with the local Aboriginal community. Again, that's challenging I think… because all of our services are outreach services." Manager, Service B Limited time (3 months) to implement change The timeframe was too short to show sustained change or implement activities, such as developing new relationships with Aboriginal representatives. The 3month follow-up audit was useful because it motivated staff to continue working towards achieving their planned activities before the follow-up audit.
"I had to manage all the staff leave and annual holidays, Christmas itself, so even though we had three months, it really, when you shook it down, it was more like two months but it was broken up over, it was all very disjointed." Manager, Service A Varied skill level across staff Some staff had extensive skills working with Aboriginal people, others required additional time to develop their skills and knowledge. Providing training to staff sometimes slowed down implementation.
"When you're getting these things, you don't know what you don't know, so it's really di cult. You know, they didn't know about…Land Councils, they didn't know about how to engage with elders, they didn't know that you can get ags from the local federal minister, from their o ce. So, the little things that you know about, they didn't know that there was a cycle of the Aboriginal model of the cycle of behaviour change. So, all those things that were sort of just bread and butter when you're working in AMS [Aboriginal Medical Service], when you step over to mainstream, people have just not been exposed to it." Manager, Service F Feasibility of the project -acceptability and staff perceptions

Acceptability to CEOs/Managers
The project was reported to be acceptable by all (100%) CEOs/managers who completed interviews after the baseline (n = 14) and follow-up audits (n = 12). Most reported that it had bene t to the services and to themselves, and that implementing cultural responsiveness activities was an important part of their work and a priority at their service. Many reported that their service had a focus on cultural responsiveness before the project began.

CEOs/Managers perspectives of the workshop and audits
Thematic analysis of interviews with CEOs/Managers at follow-up (n = 12) identi ed several major themes as outlined in Table 3. Services and staff built capability and skills around speci c activities involved with culturally responsive service delivery. Many staff reported that they found the project resources useful, particularly audit outcome reports, the Guideline, and the action planning tool (completed by staff in the implementation workshop to plan actions over the subsequent three months).
"The Guidelines are useful, but I would say the audit report was even more useful … having an organisation actually come in actually go, "this is where you're doing well. These are the areas you can improve on," I think that's really very valuable. So, moving forward, I would suggest that we're probably going to look at the recommendations in the audit report rather than the Guidelines." Manager, service J "It's been a really positive for us, and I think it's given us a really good framework of where we need to step up and what we can be doing a little bit more … and what things will be looking like for us to move forward to be working in a safe place for our clients." Manager, service L Acceptability and perceptions of other service staff Thirty-ve staff members attended implementation workshops and completed the survey, 71% of whom were in client-facing roles. Staff reported a high level of satisfaction with the processes surrounding the workshops, as well as the content and delivery of the workshops (Table 4). Approximately 40% reported that they perceived there had been improvements to the cultural responsiveness of their service since the baseline audit (Table 4). Iterative adaptation of the project components during the project implementation Participants were given the opportunity to provide suggestions for how the project could be improved in each interview. Suggestions were discussed among the team and the following components of the project were subsequently updated. Suggestions were mostly made by those interviewed from services audited early in the project. After these changes were made, no further substantive feedback was provided regarding improvements to the project.
1. Information provided to CEOs/managers at the beginning of the project emphasised the recommendation that a CEO/manager attend the audits, and clearly described the audit process to indicate that only information provided on the day of the audit would be included in the audit rating and report. This recommendation was made to ensure staff with knowledge around services' processes/policies attended the audit, so information they provided could be included in the audit outcome report and considered during rating allocation. Due to the scheduling commitments of the project, the audit only captured information provided at the time of audit, and staff were not able to comment and provide additional information after the audit outcome report was provided to services.
2. The audit report was restructured and developed during the project. Reports were shortened and wording was revised. Numerical ratings were removed from reports. The updated version used the words 'limited, some, good or excellent', replacing the rating of 0, 1, 2 or 3. Changing the reporting around the rating system underpinning the audit system provided service staff with encouraging feedback, with the aim of motivating them to take on and implement the audit feedback.

Discussion
Overall a high level of implementation delity was achieved, with most participating services completing all components of the project and staff demonstrating a high level of engagement with the project.
Delays in completing follow-up audits was the only signi cant departure from the planned project implementation. Feedback from CEOs/managers indicated that the short time between audits was a barrier to the realization of more complex change activities. Future work using the ve-step process devised for this project may bene t from extending the time between audits. This would provide more time for services to implement planned actions and for any changes implemented to have an impact on client outcomes.
The resources and processes developed through this project including the guidelines (7), audit tools, action plans and implementation workshops are feasible to use and highly acceptable to staff at services. While the guideline was initially thought to be the driving force underpinning the ve-step process, the importance of the other steps was quickly realised. The auditing of services, providing individualised feedback and assistance in creating action plans appeared to be key in helping services to improve cultural responsiveness. Staff rated the project resources (feedback reports and action plans) as particularly useful tools for implementing change activities. As has been previously identi ed (15,16), active learning processes in the implementation workshops appeared to be important in engaging staff and in operationalising the cultural responsiveness concepts into concrete activities. Discussions arising during audits and the timely feedback of outcomes also appeared to motivate immediate changes in many services; 40% of staff perceived there to have been changes to cultural responsiveness at their service prior to the implementation workshops. The recommendations in the audit reports personalised the guidelines for individual services, so rather than having to apply the guidelines rigidly, staff were able to focus on those change activities identi ed as important and achievable for their individual services. This approach of intervening using a balance of standardised and exible components is supported by existing evidence (16,18,19) and appears to be well suited to use in NGO AoD treatment services.
Staff commitment to and enthusiasm for the concept of cultural responsiveness was also identi ed as an enabler to change, consistent with previous research (16,26). The high level of participation and engagement from staff and CEOs/managers suggest that staff are actively aiming to enhance culturally responsive service delivery. In particular, engagement of senior staff with detailed knowledge about services' processes/policies related to cultural responsiveness and the capacity to decide and enact service level changes, appeared to be important in ensuring that planned activities were implemented into services.
A signi cant barrier to actioning changes was that services lacked resources including funding and staff time with which to implement changes. Establishing cultural responsiveness practice as a routine continuous quality improvement (CQI) cycle in services might help to overcome this, and other barriers, to improvement (27). Repeated CQI cycles have been shown to be an effective process for systems change in Aboriginal health services in a study examining diabetes care, with ow on improvements in key health indicators observed (28,29). Using this approach, services aim to incrementally improve practices over time, whilst tracking improvements through routinely collected data and regular auditing (29). Routine evaluation frameworks could be seamlessly embedded into service delivery at low-cost by better use of administrative data that are already routinely collected. This approach would allow services time to seek out funding or resources or to schedule change activities in line with funding or other service changes.

Strengths and limitations
Participating services were self-selected, so it is possible that they had pre-existing active interest in improving their cultural responsiveness and/or some resources to dedicate to the process. Such a high level of engagement may not be observed in other services, given that organisational climate and desire for change have previously been identi ed as enablers for change (16). However, the success of these 'early adopter' services could act as a catalyst for other services to implement this approach to improving their cultural responsiveness.
While a relatively small number of self-selected services were involved in the project, they represent a wide geographic and sociodemographic area of NSW and a variety of service types/delivery models. The resources and processes developed and implemented were highly acceptable to participating services across all the regions involved and the participatory research approach resulted in improvements in the delivery of the ve-step process. The lack of substantive changes suggested about how the project was run by the end of the project is further indication of satisfaction with the project by participating services.
While there is limited information on the acceptability to those services that did not complete all components, drop out appears to be related to staff turnover and scheduling issues, rather than dissatisfaction with the project. While this project does not directly address the feasibility of scaling-up standardised, best-evidence cultural responsiveness activities across all NGO-delivered AoD services in NSW, the project was successful in the goal of establishing resources that re ect current best-evidence practice and a ve-step process driven by independent Aboriginal auditors that could be expanded as a targeted and routinely utilised resource for all AoD services.
Further strategies could be developed to facilitate easy uptake of this approach, including the identi cation of minimal activities that could be shared across services or could be taken-up at negligible cost, such as identifying potential e cient and creative mechanisms by which previous services have successfully operationalised cultural responsiveness improvement activities which may help newer services to generate ideas. Though, an important strength of the project is the exibility with which the guidelines are implemented within individual services, an approach which is supported by previous research in Aboriginal health settings (19). The design of the current study allowed different types of services to prioritise different aspects of cultural responsiveness and to implement solutions that suited their unique needs and strengths.

Conclusion
This project developed a cultural responsiveness guideline following the principles of community engagement, choice, and person-centred practice. The results support the use of a pragmatic and participatory evaluation approach using a standardised core intervention and exible components that allowed different types of services to prioritise different aspects of cultural responsiveness and to implement solutions that suited their unique needs and strengths. The operationalisation of the ve-step process developed to improve cultural responsiveness was feasible and acceptable and is readily applicable to improving the cultural responsiveness of a wide variety of health and human services. Availability of data and materials

List Of Abbreviations
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Authors' contributions
The project was overseen by the Network of Alcohol and other Drugs Agencies (NADA), implemented by the project team (RW and JA), and the evaluation was completed by the National Drug and Alcohol Research Centre (SF, AS and AA). NADA, the project team, the Aboriginal Project Advisory Group and the evaluation team all worked closely to develop, implement and evaluate the ve-step process (RW, JA, SF, AS, AA), reporting to the coalition of Primary Health Networks (PHNs) who funded the project. RW, JA and SF acquired the data; SF conducted the data analysis; SF and AA wrote the methods and results, JA, AA, SF and AS wrote the introduction and discussion, with input from all authors. All authors provided substantial contributions to the interpretation of the results, and all critically revised the manuscript, and approved the nal manuscript as submitted.