Evaluation of an organisational performance measurement implementation framework for health charities

Background Non acute health charities form part of the global health services industry yet face concerns regarding performance and accountability. Organisational performance measurement (OPM) is enables performance however is under utilised in the non acute health charity sector. The Non-acute Health Charity Performance Implementation Framework (NCPI Framework) was developed to support OPM implementation in non acute health charities. The study goal was to assess whether NCPI Framework implementation could positively impact on the introduction of OPM to case study organisation. The hypothesis was that NCPI Framework implementation would positively impact on the introduction of OPM in a non acute health charity. Methods A convergent parallel design mixed methods case study evaluation was conducted of the twelve-month implementation of the NCPI Framework in a non acute health charity. Pre and post survey quantitative and qualitative techniques were used to assess the infrastructure and systems used to support the NCPI Framework’s program design and implementation in addition to participant perception of OPM usefulness. This study analysed quantitative data collected using identical pre and post Likert scales surveys. Response rates were 74% at pre and 64% at post data collection times. The valid percentage was used to exclude incomplete responses. Then, data were analysed using descriptive statistics such as modes, means and standard deviation. Wilcoxon Signed Rank Test was used to examine changes between pre and post data. Results Results support the hypothesis that the implementation of the NCPI Framework positively impacted on the introduction of organisational performance measurement in the case study organisation. The results were statistically signicant and demonstrated positive change in implementation infrastructure and participant perception of OPM usefulness. Board members (59%) reported a lower level of change in positive perception of usefulness than staff (93%) and salaried clinical advisors (93%). Results suggest of health

Address any conceptual barriers within the organisation · Communicate organisational strategy, purpose, the Implementation Plan and its status and ensure staff are aware OPM IS for strategic management not just performance measurement · Participation of staff in design and revisions · Facilitate periodic and systematic review, adjustment and improvement · Provide feedback to learn about and improve organisational strategy and Implementation Plan · Identify and support champions across the organisation · Skills and tools in data analysis and management, and implementing feedback and learning systems in-place · Use team-based collaborative approaches among disciplines that done regularly work together FACTOR 4: Alignment, integration and resourcing · Identify and align strategic initiatives · Designate implementation within the organisation's Business and Operational Plans · Acknowledge and prepare for deployment resourcing investment · Integrate within existing management processes, governance mechanisms, policies and reporting systems Methods A convergent parallel design mixed methods case study evaluation was conducted of the twelve-month implementation of the NCPI Framework in a non-acute health charity [17,18,19]. A convergent parallel design entails that the researcher concurrently conducts the quantitative and qualitative elements in the same phase of the research process, weighs the methods equally, analyses the two components independently, and interprets the results together [18]. The study was framed to evaluate the NCPI Framework's usefulness in supporting the introduction of OPM into a non-acute health charity over a twelve month period. Usefulness was de ned by utility -the level of participant satisfaction, and usability -the ease of functionality as perceived by participants [20]. The Gervais Program Evaluation Model [21] informed a quantitative pre-post survey design and data were collected using Likert scales. The participants surveyed included Board members, staff and salaried clinical advisors. Taking feedback from internal participants of a case study organisation was the chosen methodology so that real world, current and contextual understanding (Yin, 2017) could be attained and learnings could be identi ed to inform future practice [22].

Case study organisation
The participating organisation, referred to as the case study organisation in this article, was a non-acute health charity working in the area of rural and remote health within the geographical territory of New South Wales (Australia). The organisation pursued OPM as part of a broader organisational development program and utilised the NCPI Framework over a twelve-month implementation period. A sample of the organisation's implementation activities responding to the NCPI Framework's recommendations are listed in Table 2. embedding, of infrastructure and systems to support program design and implementation. It utilises ve distinct dimensions -structural, operational, strategic, systemic, and speci c to describe the "complex interaction that exists between a program, its environment, and the actors (sic. individuals) involved" [21, p. 166]. The ve dimensions are differentiated by the following de nitions -The Structural dimension includes program implementation procedures and resources such as physical, material, nancial, informational, or human. Gervais [21] identi es four topic areas within this dimension -resources; roles and responsibilities; exibility and adaptability; and communication; The Operational dimension integrates the program processes and activities with the behaviours of the participants. This includes the work environment and internal dynamics required to create and deliver services, such as -implementation method; feasibility; and personnel; The Strategic dimension encourages, and assesses, organisational ability to identify and solve problems, learn from experience and activate policies and management practice to support implementation; The Systemic dimension explores the activities of a program linked to its external environment. Its purpose is to ensure legitimacy, relevance and usefulness a program to its external environment. It encourages response to political, economic, legal, social, and other aspects that characterize the external environment at local, regional or national levels across two topic areas -complementary resources and collaboration; The Speci c dimension focuses on a program's impact and value. It reinforces what is necessary for a program to reach its target short-, medium-, and long-term results.

Participants
Purposive sampling was used and all Board members, staff and salaried clinical advisors were invited to participate in the study. Staff were located within the organisation's two separate o ces located two-hundred kilometres apart, while Board members and salaried clinical advisors were in multiple locations within the organisation's geographical boundaries. A total of 48 pre-surveys and 47 post-surveys were completed.

Recruitment
The process of recruitment followed the approved ethics methodology and was consistent for both survey rounds. An introduction to the study and invitation to participate in each survey round was emailed from the CEO to all potential participants with a direct weblink to the electronic survey. The weblink site included a welcome note from the second author who is independent from the organisation and the led the survey research. A plain language statement which con rmed respondent anonymity was also included and accepting the consent form was necessary to proceed with the survey. Participants were asked to complete the survey within two weeks. Submission of the completed survey indicated consent. A reminder email was sent after two weeks for both survey rounds in which participants were informed of an additional two-week window and revised closure date. The study did not require participation from anyone under 18-years of age, and there was no payment, reimbursement of expenses or incentives for participants. were completed before implementation of the NCPI Framework commenced in July 2017. The post-surveys (July 2018) were completed following the completion of the 12-month NCPI Framework implementation period.

Instrument development
A survey was developed speci cally for the study. Its purpose was to assess participant perception of OPM within the organisation's context, assess progress in the establishment of fundamental infrastructure and systems necessary to enable organisational performance measurement and determine participant satisfaction with OPM. The tool included three parts. Part A, titled "About You" included ve demographic questions. Part B titled "Organisational performance at the case study organisation" included fty-four questions divided into two sections to assess the two components of usefulness: utility and usability. Firstly, six introductory questions were developed by the authors to assess participant understanding, engagement and satisfaction of OPM (i.e. utility). This was followed by forty-eight questions to assess functional program elements (i.e. usability) through the ve dimensions of the Gervais Program Evaluation Model they had improved one or more points on the Likert scale, remained the same ('stable'), or 'declined' if they had declined one or more points on the Likert scale between the pre-and-post implementation surveys.
The Wilcoxon Signed Rank Test was utilised to assess whether or not there was statistical signi cance between the pre and post survey responses for the overall participants group responses. The p value was determined as .05. Signi cant difference would suggest that implementation of the NCPI Framework had an impact on the introduction of organisational performance measurement in the case study organisation.

Response rates and data validity
The survey response rates were strong with 74% (48/65) and 64% (47/73) of potential respondents completing the pre-and-post surveys respectively. The organisation saw growth during the twelve-month implementation period with total staff numbers increasing by nine (17%). As matched data was not collected it was not possible to determine response rates or feedback from existing or new staff members. Most notable was the reduction of Board member responses which reduced from 75% to 43% (six to three respondents). Table 3  These results allow for the study's hypothesis to be accepted and the null hypothesis to be con dently rejected. It indicates that the implementation of the NCPI Framework positively impacted on the introduction of OPM to the case study organisation. Further, it is unlikely that chance led to these differences, but rather the implementation activities brought about the changes.  As shown in Table 3, there was overall improvement in respondent perception of program utility between pre and post survey responses to the six questions of Section 1. This is further detailed in Table 4, which shows that ve of the six aspects reported overall modal score improvement when comparing pre and post survey results -awareness, belief in current bene t, participation, satisfaction, understanding. The sixth -future potential bene t, remained steady as 'good'.
Board members were the only participant group to record a variation to the overall modal scores in the perception of utility.

Gervais Program Dimension: Structural
As shown in Table 3, results of the eight structural dimension questions increased by one or more points on the Likert scales demonstrating overall modal score improvement from pre to post data collection times.

Gervais Program Dimension: Operational
As shown in Table 3, each of the eleven operational dimension questions demonstrated overall modal score improvement as indicated by Likert scale change from 'unsure' in the pre survey to either 'good' or 'acceptable' post implementation.
As indicated by Likert scale change, Board members recorded either a decline or stability in the three areas. In three areas Board members recorded a decline or stable result. There was a decline in the area which inquired about the level to which personnel involved with organisational performance measurement are consistently available (Question 3.8: p=.001), and the other two areas remained stable with an 'unsure' response for level of exibility and quality of the methods, activities and processes of organisational performance measurement (Question 3.2: p=.097) and the adequate use of program resources for organisational performance measurement (Question 3.1: p=.000).

Gervais Program Dimension: Strategic
As shown in Table 3, and indicated by Likert scale change, eleven of the fourteen questions demonstrated improvement. Of these, six reported three modal score improvement and included stability and growth of the organisational performance measurement program in the case study organisation (Question 4.1: p=.000), quality of program management (Question 4.4: p=.000), level of management's engagement with the program (Question 4.6:p=.000), effective management of program resources (Question 4.7: p=.000 ), resource optimisation (Question 4.10: p=.001) and the level of conformity to organisational values As shown in Table 3, four of the ve speci c dimension questions demonstrated overall modal score improvement. The overall responses relating to the quality and quantity of products and services generated from the program (Question 6.2 p=.000) remained stable at 'unsure'.
Board members were again the only group to report variation to staff and salaried clinical advisors. They reported a decline in two of the ve questions including the quality and quantity of products or services generated from organisational performance measurement (Question 6.2: pre=m1*, post=m1) and in perceived value and cost-effectiveness (Question 6.5: pre=m3*, post=2*).

Variation in Board member responses to other participant groups
As demonstrated in Table 4 have demonstrated that from the perspective of these participant groups, implementation of the NCPI Framework over a twelve-month implementation period was successful in supporting the introduction and establishment of OPM in a non-acute health charity. As such, the study's hypothesis that implementation of the NCPI Framework would positively impact on the introduction of OPM in a case study organisation has been accepted, and the null hypothesis rejected.
Elements of the NCPI Framework that could be adapted for enhanced usefulness were also identi ed.

Overcoming barriers to OPM implementation
Kaplan [23] and Mueller [24] argued that non-pro ts and charities should be held accountable and face similar governance standards as for-pro t entities especially considering they are bene ciaries of public and private funding. However, still today there is little consensus to non-pro t industry or sector-wide benchmarks on performance measurements that integrate accountability with sustainability, nor evaluated OPM implementation tools for speci c industry and sector types [4,5,24,25,26]. The outcomes of this study are signi cant because they have demonstrated the potential for the NCPI Framework to be utilised by non-acute health charities to facilitate the implementation of OPM. Concerns regarding the sector's level of accountability and governance standards compared to those of for-pro t and public service organisations could now be addressed by using this OPM implementation tool tailored speci cally for this sector. Adopters could position themselves as market leaders by demonstrating accountability and effectiveness to their stakeholders through the reporting mechanisms of the NCPI Framework.

Targeted engagement mechanisms for individual participant groups
The literature strongly suggests the need for tailored and exible OPM implementation approaches to suit the nuances of speci c industry types, sectors and organisations [27,28,29]. The evaluation has shown that the quantitative research design methodology utilised can identify the level of understanding and engagement with OPM by individual participant groups. Further, as this study showed understanding and engagement can vary across different participant groups, greater consideration of the speci c needs of each participant group within the OPM implementation could be bene cial.
The most notable variation in results over the twelve-month implementation period relates to Board member's perceptions. The Board's less positive responses in the post survey compared to both staff and salaried clinical advisors is notable, if not concerning. As in for-pro t organisations, the role of the Boards in not-for-pro ts and charities is important. An NFP charity Board is responsible for achievement of the organisation's mission, developing strategy to achieve that mission and ensuring the organisation has the necessary resources to be effective [30]. Ensuring the Board's understanding and continued commitment to organisational development and learning initiatives such as OPM is paramount from both long-term strategic and shorter-term operational perspectives.
The Board's feedback could be explained by the fact that the response numbers were considerably lower than the other participant groups. Further, in comparison to staff and salaried clinical advisors, they are most distant from program implementation within the organisation and therefore did not experience the changes rst hand. However, a review of the case study organisation's NCPI Framework implementation plan con rmed that there were minimal targeted messaging and communication mechanisms identi ed speci cally relating to Board members. Consistent with the case study ndings of Chavan [11] when investigating the OPM tool -The Balanced Scorecard, these results suggest that targeted engagement mechanisms for each individual participant group should be considered within the NCPI Framework implementation planning. Such mechanisms could include additional training and communication relating to organisational performance measurement and progress reporting.
Understanding the availability and relevance of resources Results from only four (7%) questions were not statistically signi cant. Three of these related to usage of available resources -staff's usage of available resources to support NCPI Framework implementation; the adequate use of program resources for organisational performance measurement and the level of partnership or engagement with existing programs to support NCPI Framework implementation. The fourth explored the level of exibility and quality of the methods, activities and processes of organisational performance measurement.
The commonality of these questions suggests a greater need for communication that is clear, consistent and frequent regarding availability of resources for OPM implementation. Potentially, the quantity of resources allocated to support implementation in the case study organisation was not adequate. However, considering the overall success of the NCPI Framework's implementation, it is likely that the availability of resources was not made clear to all participants or they may not have understood the importance or relevance of those resources. OPM implementation should be ongoing and is key to the engagement of staff and participating stakeholders and addressing any conceptual barriers they may have [31,32]. Stronger attention to communicating resource availability and relevance when tailoring the NCPI Framework activities may strengthen future applications.

Gervais Program Evaluation Model
This is the rst study to utilise the Gervais Program Evaluation Model [21] to evaluate the implementation of an OPM program. The Model is comprehensive and explores a diverse range of characteristics relating to program design and functionality under ve program dimensions. The successful use of the Model in this study adds value to understanding its potential in a range of evaluation environments, however some adaption for rapid or more user-friendly application in survey tools may assist participant involvement.
Limitations and opportunities for further study The authors identi ed limitations to the study that should be considered when interpreting the results. Expanding the study of the utility of the NCPI Framework to include multiple organisations would enable comparative analysis of results and learnings. Constructing the survey tool to capture matched data might assist in comparing results from existing and new respondents during the implementation period. Comparative analysis of the data collected through the participant interview qualitative evaluation tool may also provide opportunity for deeper interpretation of participant perspective and complement the quantitative data secured through pre-and-post surveys. Longitudinal studies to assess whether the introduction of OPM into non-acute health charities actually impacts organisational performance should also be a goal for future study.

Conclusion
Organisational performance measurement (OPM) is a recognised business tool however under-utilised or under-reported in the non-acute health charity sector.
The Non-acute Health Charity Performance Implementation Framework (NCPI Framework) was designed to support the introduction of OPM into the nonacute health charity sector. This study is the rst case study evaluation to assess the usefulness of NCPI Framework to enable the implementation of OPM within a non-acute health charity.
The study achieved its aim of assessing Board member, staff and salaried clinical advisor perception of the NCPI Framework in the two discrete areas of usefulness -utility (amount of user satisfaction) and usability (ease of the system's functionality). The study's hypothesis that implementation of the NCPI Framework would positively impact on the introduction of organisational performance measurement in a case study organisation has been accepted, and the null hypothesis rejected. As such, the NCPI Framework was successful in supporting the introduction and establishment of OPM in a non-acute health charity.
The pre and post survey tool informed by the Gervais Program Evaluation Model [21] was effective in supporting the evaluation. Adapting the activities of the NCPI Framework to include mechanisms for greater engagement of each participant group would be useful and likely support achieving greater impact in establishing organisational performance measurement within a non-acute health charity. Availability of data and material Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Competing interests
The authors declare that they have no competing interests.

Funding
No funding was received for this study. The research reported in this paper is the sole responsibility of the authors and re ects the independent work of the authors.

Authors' contributions
The study was designed by RC in collaboration with RR, KS and JT. RC led data collection and analysis. RR and KS supported data analysis. RC drafted manuscript. All authors critically reviewed the manuscript, provided signi cant editing of the article and approved the nal manuscript.