Evaluation of an organisational performance measurement implementation framework for health charities

doi:10.21203/rs.3.rs-27836/v1

Download PDF

Research article

Evaluation of an organisational performance measurement implementation framework for health charities

https://doi.org/10.21203/rs.3.rs-27836/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted 18 May, 2020

You are reading this latest preprint version

Background

Non acute health charities form part of the global health services industry yet face concerns regarding performance and accountability. Organisational performance measurement (OPM) is enables performance however is under utilised in the non acute health charity sector. The Non-acute Health Charity Performance Implementation Framework (NCPI Framework) was developed to support OPM implementation in non acute health charities. The study goal was to assess whether NCPI Framework implementation could positively impact on the introduction of OPM to case study organisation. The hypothesis was that NCPI Framework implementation would positively impact on the introduction of OPM in a non acute health charity.

Methods

A convergent parallel design mixed methods case study evaluation was conducted of the twelve-month implementation of the NCPI Framework in a non acute health charity. Pre and post survey quantitative and qualitative techniques were used to assess the infrastructure and systems used to support the NCPI Framework’s program design and implementation in addition to participant perception of OPM usefulness. This study analysed quantitative data collected using identical pre and post Likert scales surveys. Response rates were 74% at pre and 64% at post data collection times. The valid percentage was used to exclude incomplete responses. Then, data were analysed using descriptive statistics such as modes, means and standard deviation. Wilcoxon Signed Rank Test was used to examine changes between pre and post data.

Results

Results support the hypothesis that the implementation of the NCPI Framework positively impacted on the introduction of organisational performance measurement in the case study organisation. The results were statistically significant and demonstrated positive change in implementation infrastructure and participant perception of OPM usefulness. Board members (59%) reported a lower level of change in positive perception of usefulness than staff (93%) and salaried clinical advisors (93%).

Conclusion

This is the first methodological evaluation of OPM implementation in the non acute health charity sector. Results suggest the NCPI Framework has potential to support the establishment of infrastructure, systems and participant engagement to enable OPM introduction in non acute health charities.

Health Policy

Health Economics & Outcomes Research

Organisational performance measurement

implementation

non-acute

maintenance care

non-profit

charity

Organisations exist so that people can coordinate their actions and create more social, health, economic and other value to individuals or groups of stakeholders than working on their own [1]. Success and sustained competitive advantage by outperforming others are the ultimate aim of organisations [1, 2, 3]. The process of accounting and tracking achievement is known as organisational performance measurement (OPM). OPM provides operational and benchmarking data that can guide improvement and allow organisations to demonstrate performance. It is different to the sum of an organisation’s program evaluations which provide point in time assessments and do not accurately represent an overall capability to fulfil and sustain an organisation’s purpose and mission [4, 5]. OPM capability is "essential to the survival and success of the modern business" [6, p. 719] as it adds value through evaluation, control, budgeting, motivating, promoting, celebrating, learning and improving [7].

Non acute health charities deliver a broad range of non-hospital and maintenance care services however form a homogenous sector as part of the not-for-profit, or third-sector, health industry [4, 8]. The not-for-profit industry has been slow to embrace contemporary evidence-based business management practices such as OPM [5], and non acute health charities are no different. There is a dearth of literature relating to OPM in the sector which suggests it is under-utilised, or at best, under-reported. To this point there has not been a tailored OPM implementation guide, however as part of the broader research program investigating OPM in the sector, the authors [9] developed The Non-acute Health Charity Performance Implementation Framework (NCPI Framework). The NCPI Framework (Table 1) was developed with consideration of processes for organisational renewal through experience, culture and learning as described in Organisational Learning Theory [10, 11, 12] and action implementation theories [13]. The Framework recommends five implementation factors which incorporate 30 operating elements - 1) Implementation Plan (9 elements); 2) Commitment (5 elements); 3) Organisation Understanding and Learning (8 elements); 4) Alignment, Integration and Resourcing (5 elements); and 5) Measures and Indicators (3 elements). Its purpose is to act as an OPM implementation checklist for non acute health charity Boards and executive leaders to turn an OPM vision into concrete operations.

Table 1. Non-acute Health Charity Performance Implementation Framework (NCPI Framework)

(authors withheld, 2020)

FACTOR 1: implementation Plan

· Existence of a formal organisation strategy

· Development and endorsement of a formal implementation plan which utilises evidence

· Translate organisational vision and strategy into tangible objectives and measures

· Articulation of organisational strategy and Implementation Plan in a Strategy Map

· Go beyond short-term agendas and pay attention to medium and long-term objectives

· Acknowledge and plan for deployment over extended time period

· Ensure model is adapted to meet unique organisational realities and demands

· Confirm who is responsible for which actions and activities

· Undertake a test experience to improve the balanced scorecard measures and processes

FACTOR 2: Commitment

· Secure full organisational support for implementation

· Demonstrated and continued Executive and Senior Management support

· Organisational culture is appropriate and receptive to implementation

· Creation of an Implementation Steering Committee – potentially with mixed representation across staff

· Appointment of an Implementation Coordinator as a sole role or within existing role

FACTOR 3: Organisation Understanding & Learning

· Address any conceptual barriers within the organisation

· Communicate organisational strategy, purpose, the Implementation Plan and its status and ensure staff are aware OPM IS for strategic management not just performance measurement

· Participation of staff in design and revisions

· Facilitate periodic and systematic review, adjustment and improvement

· Provide feedback to learn about and improve organisational strategy and Implementation Plan

· Identify and support champions across the organisation

· Skills and tools in data analysis and management, and implementing feedback and learning systems in-place

· Use team-based collaborative approaches among disciplines that done regularly work together

FACTOR 4: Alignment, integration and resourcing

· Identify and align strategic initiatives

· Designate implementation within the organisation’s Business and Operational Plans

· Acknowledge and prepare for deployment resourcing investment

· Integrate within existing management processes, governance mechanisms, policies and reporting systems

· Cascading accountability in departmental and personal goals throughout organisation

FACTOR 5: Measures and indicators

· Measures and indicators customized to the organisation to represent all dimensions of the organisation

· Ensure shared vision amongst staff cohort (esp. clinicians and managers) regarding priority measures and indicators

· Ensure major objectives are set for each measure and measures and indicators are meaningful

Science based implementation models are recommended for the effective application of evidence-informed practices to support successful program implementation [14, 15]. The NCPI Framework is influenced by Organisational Leaning and Action Implementation theories and was tailored with sector specificities in mind as replicating methods from other industries and sectors does not guarantee OPM success [9, 16]. A nuanced OPM implementation approach is likely to engage and encourage like-minded people and organisational leaders to undertake and embrace OPM.

This study is the first to methodologically evaluate OPM implementation in the non acute health charity sector. Its aim was to test the NCPI Framework’s impact on the introduction of OPM in real-life case study environments. The hypothesis was that the NCPI Framework supports the introduction of OPM.

A convergent parallel design mixed methods case study evaluation was conducted of the twelve-month implementation of the NCPI Framework in a non-acute health charity [17,18, 19]. A convergent parallel design entails that the researcher concurrently conducts the quantitative and qualitative elements in the same phase of the research process, weighs the methods equally, analyses the two components independently, and interprets the results together [18]. The study was framed to evaluate the NCPI Framework’s usefulness in supporting the introduction of OPM into a non-acute health charity over a twelve month period. Usefulness was defined by utility - the level of participant satisfaction, and usability - the ease of functionality as perceived by participants [20]. The Gervais Program Evaluation Model [21] informed a quantitative pre-post survey design and data were collected using Likert scales. The participants surveyed included Board members, staff and salaried clinical advisors. Taking feedback from internal participants of a case study organisation was the chosen methodology so that real world, current and contextual understanding (Yin, 2017) could be attained and learnings could be identified to inform future practice [22].

Case study organisation

The participating organisation, referred to as the case study organisation in this article, was a non-acute health charity working in the area of rural and remote health within the geographical territory of New South Wales (Australia). The organisation pursued OPM as part of a broader organisational development program and utilised the NCPI Framework over a twelve-month implementation period. A sample of the organisation’s implementation activities responding to the NCPI Framework’s recommendations are listed in Table 2.

Table 2. Sample activities of the case study organisation’s implementation using the NCPI Framework
Implementation Factor	Sample Activities undertaken by the case study organisation
Implementation Plan	· Design and articulation of 12-month Implementation Plan which was approved by the working party, communicated to staff and then updated with regular progress reports to staff · Articulation of roles and responsibilities within the Implementation Plan · A test-bed trial for the development of organisational targets was conducted in the six-months prior to the Implementation Plan’s deployment
Commitment	· Board and Executive group endorsed instigation of organisational performance program and the Implementation Plan · Establishment of a Performance Program working party made of up representatives from each operational unit · Inclusion of costed Implementation Plan within annual operating plan and budget · Appointment of a dedicated Performance Program coordinator role to drive delivery of the Implementation Plan
Organisation Understanding and Learning	· Workshops with individual teams to support design of cascading staff target plans · Implementation Plan progress updates were provided by CEO to Board meetings and staff meetings throughout the 12-month period · Development of an internal technology application to enable dashboard reporting of progress against targets · Embedding of Implementation Plan evaluation to compliment the deployment program · Inclusion of Performance Program briefing materials in new staff on-boarding processes and materials
Alignment, Integration and Resourcing	· Organisational targets cascaded to annual Unit targets and individual staff member targets. · Implementation Plan progress reports and target reports included as standing agenda items for the Executive and Management Groups · Investment in a technology application to enable reporting of progress against targets
Measures and Indicators	· Six measures were developed as the overarching domains for the development of organisational targets. These measures were adapted from those recommended by Colbran, Ramsden, Stagnitti and Adams (2017) for non-acute health charities - Purpose and Impact; Quality of Service; Stakeholder Relations and Engagement, Finance, Infrastructure and Systems; Governance and Management; People, Culture and Leadership. · Twelve-month organisational targets and indicators developed to respond to each of the six measurement domains. These targets were then approved by the Board, and Executive Group. · Twelve-month Unit targets developed for each operational Unit developed to respond to the organisational targets. Cascading targets were then also set for each individual staff member.

A project officer led the implementation with support from an internal organisational performance program working party that included the CEO. The Chair of Directors provided consent for the study which was conducted in-line with the organisation’s Research, Data and Privacy policy protocols overseen by the General Manager. Ethics approval was obtained from Deakin University, Australia (approval number HEAG-H 89_2017).

Gervais Program Evaluation Model

The Gervais Program Evaluation Model [21] was chosen to inform the quantitative pre-post survey design to assess the usability of the NCPI Framework. The Model enables evaluation of program effectiveness in the health field [21] by analytical consideration of the elements necessary for the establishment, and embedding, of infrastructure and systems to support program design and implementation. It utilises five distinct dimensions - structural, operational, strategic, systemic, and specific to describe the “complex interaction that exists between a program, its environment, and the actors (sic. individuals) involved” [21, p. 166]. The five dimensions are differentiated by the following definitions –

The Structural dimension includes program implementation procedures and resources such as physical, material, financial, informational, or human. Gervais [21] identifies four topic areas within this dimension – resources; roles and responsibilities; flexibility and adaptability; and communication;
The Operational dimension integrates the program processes and activities with the behaviours of the participants. This includes the work environment and internal dynamics required to create and deliver services, such as – implementation method; feasibility; and personnel;
The Strategic dimension encourages, and assesses, organisational ability to identify and solve problems, learn from experience and activate policies and management practice to support implementation;

The Systemic dimension explores the activities of a program linked to its external environment. Its purpose is to ensure legitimacy, relevance and usefulness a program to its external environment. It encourages response to political, economic, legal, social, and other aspects that characterize the external environment at local, regional or national levels across two topic areas – complementary resources and collaboration;
The Specific dimension focuses on a program’s impact and value. It reinforces what is necessary for a program to reach its target short-, medium-, and long-term results.

Participants

Purposive sampling was used and all Board members, staff and salaried clinical advisors were invited to participate in the study. Staff were located within the organisation’s two separate offices located two-hundred kilometres apart, while Board members and salaried clinical advisors were in multiple locations within the organisation’s geographical boundaries. A total of 48 pre-surveys and 47 post-surveys were completed.

Procedure

Recruitment

The process of recruitment followed the approved ethics methodology and was consistent for both survey rounds. An introduction to the study and invitation to participate in each survey round was emailed from the CEO to all potential participants with a direct weblink to the electronic survey. The weblink site included a welcome note from the second author who is independent from the organisation and the led the survey research. A plain language statement which confirmed respondent anonymity was also included and accepting the consent form was necessary to proceed with the survey. Participants were asked to complete the survey within two weeks. Submission of the completed survey indicated consent. A reminder email was sent after two weeks for both survey rounds in which participants were informed of an additional two-week window and revised closure date. The study did not require participation from anyone under 18-years of age, and there was no payment, reimbursement of expenses or incentives for participants.

Data collection

Data collection was achieved by administering two identical anonymous electronic pre and post 5-point Likert scale surveys. The pre-surveys (June 2017) were completed before implementation of the NCPI Framework commenced in July 2017. The post-surveys (July 2018) were completed following the completion of the 12-month NCPI Framework implementation period.

Instrument development

A survey was developed specifically for the study. Its purpose was to assess participant perception of OPM within the organisation’s context, assess progress in the establishment of fundamental infrastructure and systems necessary to enable organisational performance measurement and determine participant satisfaction with OPM. The tool included three parts. Part A, titled “About You” included five demographic questions. Part B titled “Organisational performance at the case study organisation” included fifty-four questions divided into two sections to assess the two components of usefulness: utility and usability. Firstly, six introductory questions were developed by the authors to assess participant understanding, engagement and satisfaction of OPM (i.e. utility). This was followed by forty-eight questions to assess functional program elements (i.e. usability) through the five dimensions of the Gervais Program Evaluation Model [21]. All questions were answered using the same 5 point Likert scales (1=unsure, 5=excellent). Finally, Part C titled “Wrap-Up” included five open-ended questions for feedback. The analysis of Part C is not included in this paper.

Risk of coercion

Coercion risks were discussed between the organisation’s Executive group and study team and mitigating steps were workshopped and agreed upon. For example, all internal correspondence relating to the organisational performance measurement program came from the CEO and where necessary all research study related invitations and interactions were led by the second author as independent research lead. Anonymity and confidentiality was also guaranteed through the survey process for both quantitative and open-ended questions.

Data Analysis

The valid percentage was used to exclude missing data. Descriptive statistics analysis of the mean, mode and standard deviation were applied to all Likert scales questions. Standard deviation was used to measure the level of variation between the averages for each question in the pre-and-post survey data. Where multiple modes existed, the smallest value was used and is represented in the results using (*). Comparisons of mean and mode were also considered for each participant group. To determine the level of change for the overall results of each question, data was categorised into three groups – ‘improved’ if they had improved one or more points on the Likert scale, remained the same (‘stable’), or ‘declined’ if they had declined one or more points on the Likert scale between the pre-and-post implementation surveys.

The Wilcoxon Signed Rank Test was utilised to assess whether or not there was statistical significance between the pre and post survey responses for the overall participants group responses. The p value was determined as .05. Significant difference would suggest that implementation of the NCPI Framework had an impact on the introduction of organisational performance measurement in the case study organisation.

Response rates and data validity

The survey response rates were strong with 74% (48/65) and 64% (47/73) of potential respondents completing the pre-and-post surveys respectively. The organisation saw growth during the twelve-month implementation period with total staff numbers increasing by nine (17%). As matched data was not collected it was not possible to determine response rates or feedback from existing or new staff members. Most notable was the reduction of Board member responses which reduced from 75% to 43% (six to three respondents).

Table 3 lists the summary of results for all survey participants and questions, including primary modal score, standard deviation, mean and Wilcoxon Signed Rank Test calculations. Data were grouped according to the six dimensions of the Gervais Program Evaluation Model and the Introductory Questions category. There was little standard deviation in results with responses to only 2 of the 54 questions indicating a standard deviation greater than σ0.100. Results demonstrate an overall increase in means between pre and post data collection and a majority of results were statistically significant. More specifically, results of 50 out of the 54 questions (93%) asked in the pre and post surveys generated significant results.

All results in the Introduction Questions category, and two of the five dimensions of the Gervais Program Evaluation Model (Strategic and Specific) were statistically significant confirming change between results pre and post data collection. Of the eight Structural dimension questions, only one result (Question 2.5) which related to participant usage of available resources to support NCPI Framework implementation, did not return significant results (p = 0.358). Results from two of the eleven questions in the Operational dimension were not significant. The first (Question 3.2: p=.097) related to the level of flexibility and quality of the methods, activities and processes of organisational performance measurement while the second (Question 3.7: p=.396) related to the adequate use of program resources for organisational performance measurement. Finally, only one of the ten questions (Question 5.5: p=.076) in the Systemic dimension which related to the level of partnership or engagement with other programs to support organisational performance measurement in the organisation did not return statistically significant results.

These results allow for the study’s hypothesis to be accepted and the null hypothesis to be confidently rejected. It indicates that the implementation of the NCPI Framework positively impacted on the introduction of OPM to the case study organisation. Further, it is unlikely that chance led to these differences, but rather the implementation activities brought about the changes.

Table 3. Pre and Post Survey Results Data based on a Likert Scale of 1 to 5 with 1 = unsure, 2 = poor, 3 = acceptable, 4 = good, 5 = excellent *Multiple modes exist. The smallest value is shown.
Variable	Pre Survey 1			Post Survey 2			p value
	Mean	(SD)	Mode	M	(SD)	Mode
Section 1: Introductory Questions
1.1 The existence of organisational performance measurement.	1.8642	.73170	2.00	3.8846	.99305	4.00	.000
1.2 The level that you believe organisational performance measurement currently benefits the organisation	2.1154	.81618	2.00	3.5385	1.13950	4.00	.000
1.3 The degree to which you are encouraged to personally participate in organisational performance measurement	2.4231	.90213	2.00	4.0000	.97980	4.00	.000
1.4 Your satisfaction with organisational performance measurement	2.0385	.72004	2.00	3.6538	1.19808	4.00	.000
1.5 The degree to which you believe organisational performance measurement could benefit the organisation	3.7692	1.21021	4.00	4.3846	.75243	4.00	.043
1.6 Please rate your level of knowledge and understanding of organisational performance	2.6923	.88405	3.00	3.7308	.82741	4.00	.000
Section 2: Gervais Structural Dimension
2.1 The quantity of resources available to support organisational performance measurement.	1.6154	.89786	1.00	3.0385	1.24838	3.00	.000
2.2 The quality of resources, including competence of personnel, to support organisational performance measurement.	1.9231	1.01678	1.00	3.2308	1.21021	4.00	.001
2.3 The adequacy of resources to provide personnel with information and training to support organisational performance measurement.	1.6923	.73589	1.00	3.0000	1.46969	3.00	.002
2.4 The degree of staff acceptability of resources available to support organisational performance measurement.	1.5385	.76057	1.00	2.8077	1.26552	3.00	.001
2.5 The degree of staff usage of resources available to support organisational performance measurement.	2.1923	.80096	1.00	2.5000	1.27279	3.00	.358
2.6 Clarity of roles and responsibilities of different personnel in relation to organisational performance measurement.	2.2692	1.07917	2.00	3.4231	1.06482	4.00	.002
2.7 The level of flexibility and adaptability of organisational performance in order to solve a problem or barrier.	2.1538	.88056	3.00	3.1538	1.34736	4.00	.005
2.8 The adequacy of information and communication channels to organisational performance measurement.	1.9231	.89098	2.00	3.4231	1.02657	4.00	.000
Section 3: Gervais Operational Dimension
3.1 The degree of fairness of methods, activities and processes for organisational performance measurement.	1.8462	.92487	1.00	3.8462	1.15559	4.00	.000
3.2 The level of flexibility and quality of the methods, activities and processes of organisational performance measurement.	2.6923	1.37896	1.00	3.4231	1.30148	4.00	.097
3.3 The feasibility of organisational performance measurement.	1.8077	.98058	1.00	4.1154	.90893	4.00	.000
3.4 The level of conformity to existing norms and standards of organisational performance measurement.	1.9231	1.12865	1.00	2.8462	1.31734	4.00	.031
3.5 The organisational performance measurement program’s usefulness to support delivery of services and programs.	1.4615	.64689	1.00	4.0000	1.01980	4.00	.000
3.6 The ease of organisational performance measurement. (i.e. the fluidity of its processes and mechanisms of regulation).	1.5385	.76057	1.00	3.1154	1.03255	3.00	.000
3.7 The adequate use of program resources for organisational performance measurement.	2.6923	1.37896	1.00	3.0385	1.21592	3.00	.396
3.8 The level to which personnel involved with organisational performance measurement are consistently available.	1.8077	.98058	1.00	3.3077	1.31967	3.00	.001
3.9 The level to which personnel are empowered to take a creative and constructive approach to organisational performance measurement.	1.9231	1.12865	1.00	3.8077	1.20064	4.00	.000
3.10 The productivity of the personnel involved with organisational performance measurement.	1.4615	.64689	1.00	3.3077	1.40767	4.00	.000
3.11 The level of perceived satisfaction of the personnel involved with organisational performance measurement.	1.5385	.76057	1.00	3.0000	1.29615	4.00	.001
Section 4: Gervais Strategic Dimension
4.1 The level of stability and growth of organisational performance measurement.	1.8077	1.05903	1.00	3.5385	1.30325	4.00	.000
4.2 The level of organisational performance linkage between politics, policies and practices.	2.1154	.86380	1.00	3.4615	1.44861	3.00	.001
4.3 The affordability of organisational performance measurement.	1.4231	.70274	1.00	2.9615	1.53573	1.00*	.003
4.4 The quality of management of organisational performance measurement.	1.4231	.70274	1.00	3.8462	.83390	4.00	.000
4.5 The degree to which decisions are consistent with actions for organisational performance measurement.	1.8077	.98058	1.00	3.6154	1.09825	3.00	.000
4.6 The level of management’s formal and apparent engagement with organisational performance measurement.	1.7308	.91903	1.00	3.7308	1.15092	4.00	.000
4.7 The level of effective resources management for organisational performance measurement.	1.3846	.85215	1.00	3.0769	1.29377	4.00	.000
4.8 The level of efficient resources management for organisational performance measurement.	1.9231	.93480	1.00	3.0385	1.31090	1.00	.003
4.9 The degree to which resources to embed organisational performance are established.	1.5385	.76057	1.00	3.1923	1.44275	3.00	.000
4.10 The level that resources, means and methods for organisational performance measurement are optimised to attain objectives.	1.6923	.88405	1.00	3.0000	1.41421	4.00	.001
4.11 The level of conformity to the organisation’s values and program principles established for organisational performance measurement.	1.5000	.64807	1.00	4.0385	.99923	4.00	.000
4.12 The level of risk management for organisational performance measurement.	1.4615	.64689	1.00	2.8846	1.53172	1.00	.000
4.13 The level of change management processes utilised for organisational performance measurement.	1.6538	.84580	1.00	3.2692	1.31325	3.00	.000
4.14 The level of knowledge management processes utilised for organisational performance measurement.	1.3846	.63730	1.00	3.2308	1.33589	3.00	.000
Section 5: Gervais Systemic Dimension
5.1 The level of ability to build up resources for organisational performance measurement.	1.6923	1.04954	1.00	3.3846	1.23538	4.00	.000
5.2 Availability of resources and services for organisational performance measurement.	1.7308	1.00231	1.00	3.3846	1.26734	4.00	.000
5.3 Accessibility of resources and services for organisational performance measurement.	1.3462	.62880	1.00	3.3077	1.28901	4.00	.000
5.4 The complementary nature of activities to support organisational performance measurement.	1.6154	.85215	1.00	3.1538	1.46130	4.00	.000
5.5 The level of partnership or engagement with other programs to support organisational performance measurement.	2.5769	1.50128	1.00	3.2692	1.37281	3.00	.076
5.6 The level of partnership or engagement with other organisations to support organisational performance measurement.	1.46130	.97744	1.00	3.0769	1.26248	4.00	.001
5.7 The level of satisfaction with partnership and exchanges that support organisational performance measurement.	1.9231	.93480	1.00	2.9615	1.24838	4.00	.000
5.8 The level to which each party undertakes their role and responsibilities in supporting for organisational performance measurement.	1.5385	.76057	1.00	3.3077	1.31967	4.00	.000
5.9 The level of clarity in relation to organisational performance measurement and how to engage with it.	1.6923	.88405	1.00	3.3846	1.02282	3.00	.000
5.10 The level of collaboration between sections to ensure coordination and transfer of information.	1.5000	.64807	3.00	3.3462	1.19808	4.00	.000
Section 6: Gervais Specific Dimension
6.1 The degree to which objectives for organisational performance have been attained.	1.4615	.64689	1.00	3.3462	1.32491	4.00	.000
6.2 The quality and quantity of products or services generated from organisational performance measurement.	1.6538	.84580	1.00	2.9615	1.48272	1.00	.000
6.3 The degree of information generated, and use of that information to inform practice, as a result of organisational performance measurement.	1.3846	.63730	1.00	3.3846	1.41639	4.00	.000
6.4 The level of satisfaction expressed by personnel with implementation of organisational performance measurement.	1.6923	1.04954	1.00	3.3462	1.23101	4.00	.000
6.5 The level of perceived value and cost-effectiveness of organisational performance measurement.	1.7308	1.00231	1.00	3.2308	1.42289	3.00	.001

Demonstrated change

As indicated by pre and post implementation Likert scale reports and summarised in Table 4 (overall change results broken down by participant group), there was significant change within the case study organisation. Change was recorded in 93% of answers (50 of 54 questions), with 7% (4 questions) recording stable scores and 0% (0 of 54 questions) recording a decline.

Table 4. Level of Change – Overall Responses
Question Breakdown	Overall	Board	Staff	Salaried Clinical Advisors
Section 1: Introductory Questions (6)	5 / 6 (83%) Improved: 5 / 6 (83%) Stable: 1 / 6 (2%)	5 / 6 (83%) Improved: 4 / 6 (66%) Decline: 1 / 6 (17%) Stable: 1 / 6 (17%)	6 / 6 (100%) Improved: 6 / 6 (100%)	5 / 6 (83%) Improved: 5 / 6 (83%) Stable: 1 / 6 (17%)
Section 2: Structural Questions (8)	8 / 8 (100%) Improved: 8 / 8 (100%)	3 / 8 (38%) Improved: 3 / 8 (38%) Stable: 5 (62%)	8 / 8 (100%) Improved: 8 / 8 (100%)	8 / 8 (100%) Improved: 7 / 8 (88%) Decline: 1 / 8 (12%)
Section 3: Operational Questions (11)	11 / 11 (100%) Improved: 11 / 11 (100%)	8 / 11 (72%) Improved: 8 / 11 (73%) Decline: 1 / 11 (9%) Stable: 2 / 11 (18%)	11 / 11 (100%) Improved: 11 / 11 (100%)	11 / 11 (100%) Improved: 11 / 11 (100%)
Section 4: Strategic Questions (14)	11 / 14 (79%) Improved: 11 / 14 (79%) Stable: 3 / 11 (21%)	12 / 14 (86%) Improved: 10 / 14 (71%) Decline: 2 / 14 (14%) Stable: 2 / 14 (14%)	11 / 14 (79%) Improved: 11 / 14 (79%) Stable: 3 / 14 (21%)	14 / 14 (100%) Improved: 13 / 14 (93%) Decline: 1 / 14 (7%)
Section 5: Systemic Questions (10)	10 / 10 (100%) Improved: 10 / 10 (100%)	8 / 10 (80%) Improved: 5 / 10 (50%) Decline: 3 / 10 (30%) Stable: 2 / 10 (20%)	10 / 10 (100%) Improved: 10 / 10 (100%)	10 / 10 (100%) Improved: 9 / 10 (90%) Decline: 1 / 10 (10%)
Section 6: Specific Questions (5)	4 / 5 (80%) Improved: 4 / 5 (80%) Stable: 1 / 5 (20%)	4 / 5 (80%) Improved: 2 / 5 (40%) Decline: 2 / 5 (40%) Stable: 1 / 5 (20%)	4 / 5 (80%) Improved: 4 / 5 (80%) Stable: 1 / 5 (20%)	5 / 5 (100%) Improved: 5 / 5 (100%)
Summary Change Total (54)	49 / 54 (91%)	41 / 54 (76%)	50 / 54 (93%)	53 / 54 (98%)
Improved change	49 / 54 (91%)	32 / 54 (59%)	50 / 54 (93%)	50 / 54 (93%)
Decline change	0 / 54 (0%)	9 / 54 (17%)	0 / 54 (0%)	3 / 54 (6%)
Stable	5 / 54 (9%)	13 / 54 (24%)	4 / 54 (7%)	1 / 54 (2%)

Participant perceptions of NCPI Framework’s usefulness

As demonstrated by more detailed assessment of the pre and post implementation survey results, the participants perceived positive improvement in both the amount of satisfaction received from use of the OPM program (utility) and functionality and ease of which the OPM program was operated (usability).

Results relating to program utility

As shown in Table 3, there was overall improvement in respondent perception of program utility between pre and post survey responses to the six questions of Section 1. This is further detailed in Table 4, which shows that five of the six aspects reported overall modal score improvement when comparing pre and post survey results - awareness, belief in current benefit, participation, satisfaction, understanding. The sixth – future potential benefit, remained steady as ‘good’.

Board members were the only participant group to record a variation to the overall modal scores in the perception of utility. As indicated by the changes between Likert scale scores pre and post surveys, this group reported a modal score decline in terms of knowledge and understanding of the program (Question 1.6: pre=m3*, post=m2*). This result contrasted to Board members reporting the most significant improvement of all participant groups in awareness of the program (Question 1.1: pre=m2, post=m4).

Results relating to program usability

The comparisons to pre-and-post survey responses to the second section of Part B relating to program usability demonstrated improvement. Results increased by one or more points on the Likert scales in each of the Gervais Program Evaluation Model’s five dimensions. Again, these results are demonstrated in both Tables 3 and 4.

Gervais Program Dimension: Structural

As shown in Table 3, results of the eight structural dimension questions increased by one or more points on the Likert scales demonstrating overall modal score improvement from pre to post data collection times.

Salaried clinical advisors reported a modal score decline in in the level of information and communication relating to the program’s implementation (Question 2.8: pre= m3*, post=m2*. The Board member’s results remained stable at ‘unsure’ in questions relating to resources – quantity (Question 2.1: pre=1, post=1*), adequacy (Question 2.3: pre=m1, post m=1*), staff acceptability (Question 2.4: pre=1) and staff usage (Question 2.5: pre=1, post 1); plus adequacy of information and communication channels to organisational performance measurement (Question 2.8: p=.000).

Gervais Program Dimension: Operational

As shown in Table 3, each of the eleven operational dimension questions demonstrated overall modal score improvement as indicated by Likert scale change from ‘unsure’ in the pre survey to either ‘good’ or ‘acceptable’ post implementation.

As indicated by Likert scale change, Board members recorded either a decline or stability in the three areas. In three areas Board members recorded a decline or stable result. There was a decline in the area which inquired about the level to which personnel involved with organisational performance measurement are consistently available (Question 3.8: p=.001), and the other two areas remained stable with an ‘unsure’ response for level of flexibility and quality of the methods, activities and processes of organisational performance measurement (Question 3.2: p=.097) and the adequate use of program resources for organisational performance measurement (Question 3.1: p=.000).

Gervais Program Dimension: Strategic

As shown in Table 3, and indicated by Likert scale change, eleven of the fourteen questions demonstrated improvement. Of these, six reported three modal score improvement and included stability and growth of the organisational performance measurement program in the case study organisation (Question 4.1: p=.000), quality of program management (Question 4.4: p=.000), level of management’s engagement with the program (Question 4.6:p=.000), effective management of program resources (Question 4.7: p=.000 ), resource optimisation (Question 4.10: p=.001) and the level of conformity to organisational values and principles (Question 4.11: p=.000). The three questions that did not record overall improvement remained stable at ‘unsure’. These included, level of program affordability (Question 4.3: p=.003), the perceived efficient use of program resources (Question 4.8: p=.003) and the level of risk management for the program (Question 4.12: p=.000).

Board members were again the only participant group whose perceptions varied from the other groups. This occurred in relation to the Board’s perception of the affordability of organisational performance measurement (Question 4.3: pre=m1, post =m1*) and the level of risk management for organisational performance measurement (Question 4.12: pre=m1, post= m2*). Their response to two questions remained stable from the Board’s perspective relating to level of stability and growth of organisational performance measurement (Question 4.1: pre=m1, post=m2*) and level of efficient resources management for organisational performance measurement (Question 4.8: pre=m1, post=m1*).

Gervais Program Dimension: Systemic

As indicated in Table 3, the level of overall change for each of the ten systemic dimension questions was significant. Each question reported overall modal score improvement. This included seven of the ten questions recording a three modal score improvement.

Again, in contrast to the overall results and specific results from staff and salaried clinical advisors, Board members reported modal score declines in this dimension between the pre and post implementation. This occurred for questions relating to availability of resources and services to support the program (Question 5.2: pre m4, post=m2*), accessibility of resources services to support the program (Question 5.3 pre=3*, post=m2*) and the level of internal collaborations to support the program (Question 5.10: pre=m2*, post m=1*). The Board’s perception of the level of acceptance of roles and responsibilities by partners to the program remained stable as ‘poor’ (Question 5.8: pre=m2*, post=m2*).

Gervais Program Dimension: Specific

As shown in Table 3, four of the five specific dimension questions demonstrated overall modal score improvement. The overall responses relating to the quality and quantity of products and services generated from the program (Question 6.2 p=.000) remained stable at ‘unsure’.

Board members were again the only group to report variation to staff and salaried clinical advisors. They reported a decline in two of the five questions including the quality and quantity of products or services generated from organisational performance measurement (Question 6.2: pre=m1*, post=m1) and in perceived value and cost-effectiveness (Question 6.5: pre=m3*, post=2*).

Variation in Board member responses to other participant groups

As demonstrated in Table 4, despite reporting overall positivity and a high understanding of organisational performance measurement, the Board reported the lowest level of positive change at 59% (32 of 54 questions) compared to staff and salaried clinical advisors (50 of 54 questions, 98%) between pre-and-post implementation surveys. The Board also reported the highest level of unchanged modal scores from ‘unsure’ between pre and post surveys (8 of 25 questions, 32%) as compared to by staff (92%) and salaried clinical advisors (94%) both of which recorded change in questions that recorded ‘unsure’ in the pre-implementation survey. The Board perceived least change in structural dimension (3 of 8 questions, 38%), specific dimension (2 of 5 questions, 40%) and systemic dimension (5 of 10 questions, 50%). In the other three dimensions, the Board recorded over 70% improved change. In all but one dimension staff and salaried clinical advisors recorded over 80% positive change.

NCPI Framework’s impact on the establishment of organisational performance measurement

This study is the first to evaluate the NCPI Framework in a live operating environment. The study achieved its aim of assessing a case study organisation’s Board member, staff and salaried clinical advisor perceptions of the NCPI Framework in two discrete areas of usefulness – utility and usability. The results have demonstrated that from the perspective of these participant groups, implementation of the NCPI Framework over a twelve-month implementation period was successful in supporting the introduction and establishment of OPM in a non-acute health charity. As such, the study’s hypothesis that implementation of the NCPI Framework would positively impact on the introduction of OPM in a case study organisation has been accepted, and the null hypothesis rejected. Elements of the NCPI Framework that could be adapted for enhanced usefulness were also identified.

Overcoming barriers to OPM implementation

Kaplan [23] and Mueller [24] argued that non-profits and charities should be held accountable and face similar governance standards as for-profit entities especially considering they are beneficiaries of public and private funding. However, still today there is little consensus to non-profit industry or sector-wide benchmarks on performance measurements that integrate accountability with sustainability, nor evaluated OPM implementation tools for specific industry and sector types [4, 5, 24, 25, 26]. The outcomes of this study are significant because they have demonstrated the potential for the NCPI Framework to be utilised by non-acute health charities to facilitate the implementation of OPM. Concerns regarding the sector’s level of accountability and governance standards compared to those of for-profit and public service organisations could now be addressed by using this OPM implementation tool tailored specifically for this sector. Adopters could position themselves as market leaders by demonstrating accountability and effectiveness to their stakeholders through the reporting mechanisms of the NCPI Framework.

Targeted engagement mechanisms for individual participant groups

The literature strongly suggests the need for tailored and flexible OPM implementation approaches to suit the nuances of specific industry types, sectors and organisations [27, 28, 29]. The evaluation has shown that the quantitative research design methodology utilised can identify the level of understanding and engagement with OPM by individual participant groups. Further, as this study showed understanding and engagement can vary across different participant groups, greater consideration of the specific needs of each participant group within the OPM implementation could be beneficial.

The most notable variation in results over the twelve-month implementation period relates to Board member’s perceptions. The Board’s less positive responses in the post survey compared to both staff and salaried clinical advisors is notable, if not concerning. As in for-profit organisations, the role of the Boards in not-for-profits and charities is important. An NFP charity Board is responsible for achievement of the organisation’s mission, developing strategy to achieve that mission and ensuring the organisation has the necessary resources to be effective [30]. Ensuring the Board’s understanding and continued commitment to organisational development and learning initiatives such as OPM is paramount from both long-term strategic and shorter-term operational perspectives. The Board’s feedback could be explained by the fact that the response numbers were considerably lower than the other participant groups. Further, in comparison to staff and salaried clinical advisors, they are most distant from program implementation within the organisation and therefore did not experience the changes first hand. However, a review of the case study organisation’s NCPI Framework implementation plan confirmed that there were minimal targeted messaging and communication mechanisms identified specifically relating to Board members. Consistent with the case study findings of Chavan [11] when investigating the OPM tool – The Balanced Scorecard, these results suggest that targeted engagement mechanisms for each individual participant group should be considered within the NCPI Framework implementation planning. Such mechanisms could include additional training and communication relating to organisational performance measurement and progress reporting.

Understanding the availability and relevance of resources

Results from only four (7%) questions were not statistically significant. Three of these related to usage of available resources - staff’s usage of available resources to support NCPI Framework implementation; the adequate use of program resources for organisational performance measurement and the level of partnership or engagement with existing programs to support NCPI Framework implementation. The fourth explored the level of flexibility and quality of the methods, activities and processes of organisational performance measurement.

The commonality of these questions suggests a greater need for communication that is clear, consistent and frequent regarding availability of resources for OPM implementation. Potentially, the quantity of resources allocated to support implementation in the case study organisation was not adequate. However, considering the overall success of the NCPI Framework’s implementation, it is likely that the availability of resources was not made clear to all participants or they may not have understood the importance or relevance of those resources. OPM implementation should be ongoing and is key to the engagement of staff and participating stakeholders and addressing any conceptual barriers they may have [31, 32]. Stronger attention to communicating resource availability and relevance when tailoring the NCPI Framework activities may strengthen future applications.

Gervais Program Evaluation Model

This is the first study to utilise the Gervais Program Evaluation Model [21] to evaluate the implementation of an OPM program. The Model is comprehensive and explores a diverse range of characteristics relating to program design and functionality under five program dimensions. The successful use of the Model in this study adds value to understanding its potential in a range of evaluation environments, however some adaption for rapid or more user-friendly application in survey tools may assist participant involvement.

Limitations and opportunities for further study

The authors identified limitations to the study that should be considered when interpreting the results. Expanding the study of the utility of the NCPI Framework to include multiple organisations would enable comparative analysis of results and learnings. Constructing the survey tool to capture matched data might assist in comparing results from existing and new respondents during the implementation period. Comparative analysis of the data collected through the participant interview qualitative evaluation tool may also provide opportunity for deeper interpretation of participant perspective and complement the quantitative data secured through pre-and-post surveys. Longitudinal studies to assess whether the introduction of OPM into non-acute health charities actually impacts organisational performance should also be a goal for future study.

Organisational performance measurement (OPM) is a recognised business tool however under-utilised or under-reported in the non-acute health charity sector. The Non-acute Health Charity Performance Implementation Framework (NCPI Framework) was designed to support the introduction of OPM into the non-acute health charity sector. This study is the first case study evaluation to assess the usefulness of NCPI Framework to enable the implementation of OPM within a non-acute health charity.

The study achieved its aim of assessing Board member, staff and salaried clinical advisor perception of the NCPI Framework in the two discrete areas of usefulness – utility (amount of user satisfaction) and usability (ease of the system’s functionality). The study’s hypothesis that implementation of the NCPI Framework would positively impact on the introduction of organisational performance measurement in a case study organisation has been accepted, and the null hypothesis rejected. As such, the NCPI Framework was successful in supporting the introduction and establishment of OPM in a non-acute health charity.

The pre and post survey tool informed by the Gervais Program Evaluation Model [21] was effective in supporting the evaluation. Adapting the activities of the NCPI Framework to include mechanisms for greater engagement of each participant group would be useful and likely support achieving greater impact in establishing organisational performance measurement within a non-acute health charity.

CEO: Chief Executive Officer

NCPI Framework: Non-acute Health Charity Performance Implementation Framework

NFP: Not-for-profit organisation

OPM: organisational performance measurement

Ethics approval and consent to participate

The study obtained ethics approval from the Human Research Ethics Committee of Deakin University, Australia (approval number HEAG-H 89_2017) before commencement. Consent to participate was obtained in written format through direct weblink prior to participant access to the study’s electronic survey being granted.

Consent for publication

Not applicable.

Availability of data and material

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Competing interests

The authors declare that they have no competing interests.

Funding

No funding was received for this study. The research reported in this paper is the sole responsibility of the authors and reflects the independent work of the authors.

Authors' contributions

The study was designed by RC in collaboration with RR, KS and JT. RC led data collection and analysis. RR and KS supported data analysis. RC drafted manuscript. All authors critically reviewed the manuscript, provided significant editing of the article and approved the final manuscript.

Acknowledgements

Not applicable.

Jones G. Organizational theory, design, and change. Pearson; 2013.
Sharma A. Implementing Balance Scorecard for Performance Measurement. The Icfai University Journal of Business Strategy. 2009. VI(1): 7-16.
Colbran R, Ramsden R, Stagnitti K, & Toumbourou JW. Advancing towards contemporary practice: a systematic review of organisational performance measures for non-acute health charities. BMC Health Services Research. 2019;19; doi: 10.1186/s12913-019-3952-1
Kirk G. and Nolan S. Nonprofit mission statement focus and financial performance. Nonprofit Management and Leadership. 2010. 20(4):473-490.
Richard P, Devinney T, Yip S. and Johnson G. Measuring Organizational Performance: Towards Methodological Best Practice”. Journal of Management. 2009. 35(3):718–804; doi:1177/0149206308330560
Behn RD. Why Measure Performance? Different Purposes Require Different Measures. Public Administration Review. 2002; 63;5;586:606. Doi: 10.1111/1540-6210.00322
Campbell-Scherer D, Asselin J, Osunlana A, Fielding S, Anderson R, Rueda-Clausen C, Johnson J, Ogunleye A, Cave A, Manca D, & Sharma A. Implementation and evaluation of the 5As framework of obesity management in primary care: design of the 5As Team (5AsT) randomized control trial’, Implementation Science; 2014; 9;78.
Anonymous authors (2020) withheld until manuscript reviewed.
Argote L. Organizational learning: creating, retaining and transferring knowledge, Springer; 2013.
Chavan M. The balanced scorecard: A new challenge. Journal of Management Development. 2009; 28; 393:406; doi: 10.1108/02621710910955930.
Greenhalgh T, Robert G, MacFarlane F, Bate P, & Kyriakidou O. Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. The Milbank Quarterly; 2004; 82; 4; 581:629; https://doi.org/a0.1111/j.0887-378X.2004.00325.x
Zimmerman J. Using a balanced scorecard in a nonprofit organization. Nonprofit World; 2009; 27; 10:12.
Creswell J. Steps in Conducting a Scholarly Mixed Methods Study. In Creswell, John W., “Steps in Conducting a Scholarly Mixed Methods Study” (2013). DBER Speaker Series. 48. Retrieved October 28, 2017, from http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1047&context=dberspeakers (2013, November 14).
Creswell JW., Plano Clark V. Designing and conducting mixed methods research. Thousand Oaks, CA. SAGE; 2011.
Yin R. Case study research and applications. 6th ed. US: SAGE Publications Inc. 2017.
Tarkkanen K, Harkke V. and Reijonen P. Are We Testing Utility? Analysis of Usability Problem Types. [online] Information Systems Science, University of Turku, Turku, Finland: Kimmo Tarkkanen. 2015. Available at: https://www.researchgate.net/profile/Kimmo_Tarkkanen/publication/280704723_Are_We_Testing_Utility_Analysis_of_Usability_Problem_Types/links/55dd722008aeb41644aee7c8.pdf [Accessed 24 Nov. 2018].
Gervais M. A journey through five evaluation projects with the same analysis framework. CanadianJournal of Program Evaluation; 2008; 23; 2; 165:190.
Weir E, d’Entremont N, Stalker S, Kurji K. and Robinson V.Applying the balanced scorecard to local public health performance measurement: deliberations and decisions”. BMC Public Health. 2009. 9:127. https://doi.org/10.1186/1471-2458-9-127
Kaplan RS. Strategic Performance Measurement and Management in Nonprofit Organizations. Nonprofit Management & Leadership; 2001; 11;3;353:370.
Mueller J. When doing good is just the start to being good: a possible tool to improve the organizational effectiveness of non-profit health care organizations. J Hosp Mark Public Relations. 2007;17(2):45–60
Boateng A, Akamavi R. & Ndoro G. Measuring performance of non-profit organisations: evidence from large charities. Business Ethics: A European Review; 2016; 25; 1; 59:74.
Lowell S, Silverman L, & Taliento L. Not-for-profit management: The gift that keeps on giving. The McKinsey Quarterly, 2001; 147. Https://go.galegroup.com/ps/anonymous?id=GALE%7CA72524629&sid=googleScholar&v=2.1&it=r&linkaccess=abs&issn=00475394&p=AONE&sw=w (accessed 10 Oct 2018)
Nair M. Essentials of balanced scorecard’, Hoboken, NJ: Wiley. 2004.
Schalm C. Implementing a balanced scorecard as a strategic management tool in a long-term care organization. Journal of Health Services Research & Policy; 2008; 1;13; 8-14; doi: 10.1258/jhsrp.2007.007013

Download PDF

Version 1

posted 18 May, 2020

You are reading this latest preprint version

Evaluation of an organisational performance measurement implementation framework for health charities

Status:

Version 1

Abstract

Background

Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Status:

Version 1