Response rates and data validity
The survey response rates were strong with 74% (48/65) and 64% (47/73) of potential respondents completing the pre-and-post surveys respectively. The organisation saw growth during the twelve-month implementation period with total staff numbers increasing by nine (17%). As matched data was not collected it was not possible to determine response rates or feedback from existing or new staff members. Most notable was the reduction of Board member responses which reduced from 75% to 43% (six to three respondents).
Table 3 lists the summary of results for all survey participants and questions, including primary modal score, standard deviation, mean and Wilcoxon Signed Rank Test calculations. Data were grouped according to the six dimensions of the Gervais Program Evaluation Model and the Introductory Questions category. There was little standard deviation in results with responses to only 2 of the 54 questions indicating a standard deviation greater than σ0.100. Results demonstrate an overall increase in means between pre and post data collection and a majority of results were statistically significant. More specifically, results of 50 out of the 54 questions (93%) asked in the pre and post surveys generated significant results.
All results in the Introduction Questions category, and two of the five dimensions of the Gervais Program Evaluation Model (Strategic and Specific) were statistically significant confirming change between results pre and post data collection. Of the eight Structural dimension questions, only one result (Question 2.5) which related to participant usage of available resources to support NCPI Framework implementation, did not return significant results (p = 0.358). Results from two of the eleven questions in the Operational dimension were not significant. The first (Question 3.2: p=.097) related to the level of flexibility and quality of the methods, activities and processes of organisational performance measurement while the second (Question 3.7: p=.396) related to the adequate use of program resources for organisational performance measurement. Finally, only one of the ten questions (Question 5.5: p=.076) in the Systemic dimension which related to the level of partnership or engagement with other programs to support organisational performance measurement in the organisation did not return statistically significant results.
These results allow for the study’s hypothesis to be accepted and the null hypothesis to be confidently rejected. It indicates that the implementation of the NCPI Framework positively impacted on the introduction of OPM to the case study organisation. Further, it is unlikely that chance led to these differences, but rather the implementation activities brought about the changes.
Table 3. Pre and Post Survey Results
Data based on a Likert Scale of 1 to 5 with 1 = unsure, 2 = poor, 3 = acceptable, 4 = good, 5 = excellent
*Multiple modes exist. The smallest value is shown.
|
Variable
|
Pre Survey 1
|
Post Survey 2
|
p value
|
|
Mean
|
(SD)
|
Mode
|
M
|
(SD)
|
Mode
|
|
Section 1: Introductory Questions
|
1.1 The existence of organisational performance measurement.
|
1.8642
|
.73170
|
2.00
|
3.8846
|
.99305
|
4.00
|
.000
|
1.2 The level that you believe organisational performance measurement currently benefits the organisation
|
2.1154
|
.81618
|
2.00
|
3.5385
|
1.13950
|
4.00
|
.000
|
1.3 The degree to which you are encouraged to personally participate in organisational performance measurement
|
2.4231
|
.90213
|
2.00
|
4.0000
|
.97980
|
4.00
|
.000
|
1.4 Your satisfaction with organisational performance measurement
|
2.0385
|
.72004
|
2.00
|
3.6538
|
1.19808
|
4.00
|
.000
|
1.5 The degree to which you believe organisational performance measurement could benefit the organisation
|
3.7692
|
1.21021
|
4.00
|
4.3846
|
.75243
|
4.00
|
.043
|
1.6 Please rate your level of knowledge and understanding of organisational performance
|
2.6923
|
.88405
|
3.00
|
3.7308
|
.82741
|
4.00
|
.000
|
Section 2: Gervais Structural Dimension
|
2.1 The quantity of resources available to support organisational performance measurement.
|
1.6154
|
.89786
|
1.00
|
3.0385
|
1.24838
|
3.00
|
.000
|
2.2 The quality of resources, including competence of personnel, to support organisational performance measurement.
|
1.9231
|
1.01678
|
1.00
|
3.2308
|
1.21021
|
4.00
|
.001
|
2.3 The adequacy of resources to provide personnel with information and training to support organisational performance measurement.
|
1.6923
|
.73589
|
1.00
|
3.0000
|
1.46969
|
3.00
|
.002
|
2.4 The degree of staff acceptability of resources available to support organisational performance measurement.
|
1.5385
|
.76057
|
1.00
|
2.8077
|
1.26552
|
3.00
|
.001
|
2.5 The degree of staff usage of resources available to support organisational performance measurement.
|
2.1923
|
.80096
|
1.00
|
2.5000
|
1.27279
|
3.00
|
.358
|
2.6 Clarity of roles and responsibilities of different personnel in relation to organisational performance measurement.
|
2.2692
|
1.07917
|
2.00
|
3.4231
|
1.06482
|
4.00
|
.002
|
2.7 The level of flexibility and adaptability of organisational performance in order to solve a problem or barrier.
|
2.1538
|
.88056
|
3.00
|
3.1538
|
1.34736
|
4.00
|
.005
|
2.8 The adequacy of information and communication channels to organisational performance measurement.
|
1.9231
|
.89098
|
2.00
|
3.4231
|
1.02657
|
4.00
|
.000
|
Section 3: Gervais Operational Dimension
|
3.1 The degree of fairness of methods, activities and processes for organisational performance measurement.
|
1.8462
|
.92487
|
1.00
|
3.8462
|
1.15559
|
4.00
|
.000
|
3.2 The level of flexibility and quality of the methods, activities and processes of organisational performance measurement.
|
2.6923
|
1.37896
|
1.00
|
3.4231
|
1.30148
|
4.00
|
.097
|
3.3 The feasibility of organisational performance measurement.
|
1.8077
|
.98058
|
1.00
|
4.1154
|
.90893
|
4.00
|
.000
|
3.4 The level of conformity to existing norms and standards of organisational performance measurement.
|
1.9231
|
1.12865
|
1.00
|
2.8462
|
1.31734
|
4.00
|
.031
|
3.5 The organisational performance measurement program’s usefulness to support delivery of services and programs.
|
1.4615
|
.64689
|
1.00
|
4.0000
|
1.01980
|
4.00
|
.000
|
3.6 The ease of organisational performance measurement. (i.e. the fluidity of its processes and mechanisms of regulation).
|
1.5385
|
.76057
|
1.00
|
3.1154
|
1.03255
|
3.00
|
.000
|
3.7 The adequate use of program resources for organisational performance measurement.
|
2.6923
|
1.37896
|
1.00
|
3.0385
|
1.21592
|
3.00
|
.396
|
3.8 The level to which personnel involved with organisational performance measurement are consistently available.
|
1.8077
|
.98058
|
1.00
|
3.3077
|
1.31967
|
3.00
|
.001
|
3.9 The level to which personnel are empowered to take a creative and constructive approach to organisational performance measurement.
|
1.9231
|
1.12865
|
1.00
|
3.8077
|
1.20064
|
4.00
|
.000
|
3.10 The productivity of the personnel involved with organisational performance measurement.
|
1.4615
|
.64689
|
1.00
|
3.3077
|
1.40767
|
4.00
|
.000
|
3.11 The level of perceived satisfaction of the personnel involved with organisational performance measurement.
|
1.5385
|
.76057
|
1.00
|
3.0000
|
1.29615
|
4.00
|
.001
|
Section 4: Gervais Strategic Dimension
|
4.1 The level of stability and growth of organisational performance measurement.
|
1.8077
|
1.05903
|
1.00
|
3.5385
|
1.30325
|
4.00
|
.000
|
4.2 The level of organisational performance linkage between politics, policies and practices.
|
2.1154
|
.86380
|
1.00
|
3.4615
|
1.44861
|
3.00
|
.001
|
4.3 The affordability of organisational performance measurement.
|
1.4231
|
.70274
|
1.00
|
2.9615
|
1.53573
|
1.00*
|
.003
|
4.4 The quality of management of organisational performance measurement.
|
1.4231
|
.70274
|
1.00
|
3.8462
|
.83390
|
4.00
|
.000
|
4.5 The degree to which decisions are consistent with actions for organisational performance measurement.
|
1.8077
|
.98058
|
1.00
|
3.6154
|
1.09825
|
3.00
|
.000
|
4.6 The level of management’s formal and apparent engagement with organisational performance measurement.
|
1.7308
|
.91903
|
1.00
|
3.7308
|
1.15092
|
4.00
|
.000
|
4.7 The level of effective resources management for organisational performance measurement.
|
1.3846
|
.85215
|
1.00
|
3.0769
|
1.29377
|
4.00
|
.000
|
4.8 The level of efficient resources management for organisational performance measurement.
|
1.9231
|
.93480
|
1.00
|
3.0385
|
1.31090
|
1.00
|
.003
|
4.9 The degree to which resources to embed organisational performance are established.
|
1.5385
|
.76057
|
1.00
|
3.1923
|
1.44275
|
3.00
|
.000
|
4.10 The level that resources, means and methods for organisational performance measurement are optimised to attain objectives.
|
1.6923
|
.88405
|
1.00
|
3.0000
|
1.41421
|
4.00
|
.001
|
4.11 The level of conformity to the organisation’s values and program principles established for organisational performance measurement.
|
1.5000
|
.64807
|
1.00
|
4.0385
|
.99923
|
4.00
|
.000
|
4.12 The level of risk management for organisational performance measurement.
|
1.4615
|
.64689
|
1.00
|
2.8846
|
1.53172
|
1.00
|
.000
|
4.13 The level of change management processes utilised for organisational performance measurement.
|
1.6538
|
.84580
|
1.00
|
3.2692
|
1.31325
|
3.00
|
.000
|
4.14 The level of knowledge management processes utilised for organisational performance measurement.
|
1.3846
|
.63730
|
1.00
|
3.2308
|
1.33589
|
3.00
|
.000
|
Section 5: Gervais Systemic Dimension
|
5.1 The level of ability to build up resources for organisational performance measurement.
|
1.6923
|
1.04954
|
1.00
|
3.3846
|
1.23538
|
4.00
|
.000
|
5.2 Availability of resources and services for organisational performance measurement.
|
1.7308
|
1.00231
|
1.00
|
3.3846
|
1.26734
|
4.00
|
.000
|
5.3 Accessibility of resources and services for organisational performance measurement.
|
1.3462
|
.62880
|
1.00
|
3.3077
|
1.28901
|
4.00
|
.000
|
5.4 The complementary nature of activities to support organisational performance measurement.
|
1.6154
|
.85215
|
1.00
|
3.1538
|
1.46130
|
4.00
|
.000
|
5.5 The level of partnership or engagement with other programs to support organisational performance measurement.
|
2.5769
|
1.50128
|
1.00
|
3.2692
|
1.37281
|
3.00
|
.076
|
5.6 The level of partnership or engagement with other organisations to support organisational performance measurement.
|
1.46130
|
.97744
|
1.00
|
3.0769
|
1.26248
|
4.00
|
.001
|
5.7 The level of satisfaction with partnership and exchanges that support organisational performance measurement.
|
1.9231
|
.93480
|
1.00
|
2.9615
|
1.24838
|
4.00
|
.000
|
5.8 The level to which each party undertakes their role and responsibilities in supporting for organisational performance measurement.
|
1.5385
|
.76057
|
1.00
|
3.3077
|
1.31967
|
4.00
|
.000
|
5.9 The level of clarity in relation to organisational performance measurement and how to engage with it.
|
1.6923
|
.88405
|
1.00
|
3.3846
|
1.02282
|
3.00
|
.000
|
5.10 The level of collaboration between sections to ensure coordination and transfer of information.
|
1.5000
|
.64807
|
3.00
|
3.3462
|
1.19808
|
4.00
|
.000
|
Section 6: Gervais Specific Dimension
|
6.1 The degree to which objectives for organisational performance have been attained.
|
1.4615
|
.64689
|
1.00
|
3.3462
|
1.32491
|
4.00
|
.000
|
6.2 The quality and quantity of products or services generated from organisational performance measurement.
|
1.6538
|
.84580
|
1.00
|
2.9615
|
1.48272
|
1.00
|
.000
|
6.3 The degree of information generated, and use of that information to inform practice, as a result of organisational performance measurement.
|
1.3846
|
.63730
|
1.00
|
3.3846
|
1.41639
|
4.00
|
.000
|
6.4 The level of satisfaction expressed by personnel with implementation of organisational performance measurement.
|
1.6923
|
1.04954
|
1.00
|
3.3462
|
1.23101
|
4.00
|
.000
|
6.5 The level of perceived value and cost-effectiveness of organisational performance measurement.
|
1.7308
|
1.00231
|
1.00
|
3.2308
|
1.42289
|
3.00
|
.001
|
Demonstrated change
As indicated by pre and post implementation Likert scale reports and summarised in Table 4 (overall change results broken down by participant group), there was significant change within the case study organisation. Change was recorded in 93% of answers (50 of 54 questions), with 7% (4 questions) recording stable scores and 0% (0 of 54 questions) recording a decline.
Table 4. Level of Change – Overall Responses
|
Question Breakdown
|
Overall
|
Board
|
Staff
|
Salaried Clinical Advisors
|
Section 1: Introductory Questions (6)
|
5 / 6 (83%)
Improved: 5 / 6 (83%)
Stable: 1 / 6 (2%)
|
5 / 6 (83%)
Improved: 4 / 6 (66%)
Decline: 1 / 6 (17%)
Stable: 1 / 6 (17%)
|
6 / 6 (100%)
Improved: 6 / 6 (100%)
|
5 / 6 (83%)
Improved: 5 / 6 (83%)
Stable: 1 / 6 (17%)
|
Section 2: Structural Questions (8)
|
8 / 8 (100%)
Improved: 8 / 8 (100%)
|
3 / 8 (38%)
Improved: 3 / 8 (38%)
Stable: 5 (62%)
|
8 / 8 (100%)
Improved: 8 / 8 (100%)
|
8 / 8 (100%)
Improved: 7 / 8 (88%)
Decline: 1 / 8 (12%)
|
Section 3: Operational Questions (11)
|
11 / 11 (100%)
Improved: 11 / 11 (100%)
|
8 / 11 (72%)
Improved: 8 / 11 (73%)
Decline: 1 / 11 (9%)
Stable: 2 / 11 (18%)
|
11 / 11 (100%)
Improved: 11 / 11 (100%)
|
11 / 11 (100%)
Improved: 11 / 11 (100%)
|
Section 4: Strategic Questions (14)
|
11 / 14 (79%)
Improved: 11 / 14 (79%)
Stable: 3 / 11 (21%)
|
12 / 14 (86%)
Improved:
10 / 14 (71%)
Decline: 2 / 14 (14%)
Stable: 2 / 14 (14%)
|
11 / 14 (79%)
Improved: 11 / 14 (79%)
Stable: 3 / 14 (21%)
|
14 / 14 (100%)
Improved: 13 / 14 (93%)
Decline: 1 / 14 (7%)
|
Section 5: Systemic Questions (10)
|
10 / 10 (100%)
Improved: 10 / 10 (100%)
|
8 / 10 (80%)
Improved: 5 / 10 (50%)
Decline: 3 / 10 (30%)
Stable: 2 / 10 (20%)
|
10 / 10 (100%)
Improved: 10 / 10 (100%)
|
10 / 10 (100%)
Improved: 9 / 10 (90%)
Decline: 1 / 10 (10%)
|
Section 6: Specific Questions (5)
|
4 / 5 (80%)
Improved: 4 / 5 (80%)
Stable: 1 / 5 (20%)
|
4 / 5 (80%)
Improved: 2 / 5 (40%)
Decline: 2 / 5 (40%)
Stable: 1 / 5 (20%)
|
4 / 5 (80%)
Improved: 4 / 5 (80%)
Stable: 1 / 5 (20%)
|
5 / 5 (100%)
Improved: 5 / 5 (100%)
|
Summary Change Total (54)
|
49 / 54 (91%)
|
41 / 54 (76%)
|
50 / 54 (93%)
|
53 / 54 (98%)
|
Improved change
|
49 / 54 (91%)
|
32 / 54 (59%)
|
50 / 54 (93%)
|
50 / 54 (93%)
|
Decline change
|
0 / 54 (0%)
|
9 / 54 (17%)
|
0 / 54 (0%)
|
3 / 54 (6%)
|
Stable
|
5 / 54 (9%)
|
13 / 54 (24%)
|
4 / 54 (7%)
|
1 / 54 (2%)
|
Participant perceptions of NCPI Framework’s usefulness
As demonstrated by more detailed assessment of the pre and post implementation survey results, the participants perceived positive improvement in both the amount of satisfaction received from use of the OPM program (utility) and functionality and ease of which the OPM program was operated (usability).
Results relating to program utility
As shown in Table 3, there was overall improvement in respondent perception of program utility between pre and post survey responses to the six questions of Section 1. This is further detailed in Table 4, which shows that five of the six aspects reported overall modal score improvement when comparing pre and post survey results - awareness, belief in current benefit, participation, satisfaction, understanding. The sixth – future potential benefit, remained steady as ‘good’.
Board members were the only participant group to record a variation to the overall modal scores in the perception of utility. As indicated by the changes between Likert scale scores pre and post surveys, this group reported a modal score decline in terms of knowledge and understanding of the program (Question 1.6: pre=m3*, post=m2*). This result contrasted to Board members reporting the most significant improvement of all participant groups in awareness of the program (Question 1.1: pre=m2, post=m4).
Results relating to program usability
The comparisons to pre-and-post survey responses to the second section of Part B relating to program usability demonstrated improvement. Results increased by one or more points on the Likert scales in each of the Gervais Program Evaluation Model’s five dimensions. Again, these results are demonstrated in both Tables 3 and 4.
Gervais Program Dimension: Structural
As shown in Table 3, results of the eight structural dimension questions increased by one or more points on the Likert scales demonstrating overall modal score improvement from pre to post data collection times.
Salaried clinical advisors reported a modal score decline in in the level of information and communication relating to the program’s implementation (Question 2.8: pre= m3*, post=m2*. The Board member’s results remained stable at ‘unsure’ in questions relating to resources – quantity (Question 2.1: pre=1, post=1*), adequacy (Question 2.3: pre=m1, post m=1*), staff acceptability (Question 2.4: pre=1) and staff usage (Question 2.5: pre=1, post 1); plus adequacy of information and communication channels to organisational performance measurement (Question 2.8: p=.000).
Gervais Program Dimension: Operational
As shown in Table 3, each of the eleven operational dimension questions demonstrated overall modal score improvement as indicated by Likert scale change from ‘unsure’ in the pre survey to either ‘good’ or ‘acceptable’ post implementation.
As indicated by Likert scale change, Board members recorded either a decline or stability in the three areas. In three areas Board members recorded a decline or stable result. There was a decline in the area which inquired about the level to which personnel involved with organisational performance measurement are consistently available (Question 3.8: p=.001), and the other two areas remained stable with an ‘unsure’ response for level of flexibility and quality of the methods, activities and processes of organisational performance measurement (Question 3.2: p=.097) and the adequate use of program resources for organisational performance measurement (Question 3.1: p=.000).
Gervais Program Dimension: Strategic
As shown in Table 3, and indicated by Likert scale change, eleven of the fourteen questions demonstrated improvement. Of these, six reported three modal score improvement and included stability and growth of the organisational performance measurement program in the case study organisation (Question 4.1: p=.000), quality of program management (Question 4.4: p=.000), level of management’s engagement with the program (Question 4.6:p=.000), effective management of program resources (Question 4.7: p=.000 ), resource optimisation (Question 4.10: p=.001) and the level of conformity to organisational values and principles (Question 4.11: p=.000). The three questions that did not record overall improvement remained stable at ‘unsure’. These included, level of program affordability (Question 4.3: p=.003), the perceived efficient use of program resources (Question 4.8: p=.003) and the level of risk management for the program (Question 4.12: p=.000).
Board members were again the only participant group whose perceptions varied from the other groups. This occurred in relation to the Board’s perception of the affordability of organisational performance measurement (Question 4.3: pre=m1, post =m1*) and the level of risk management for organisational performance measurement (Question 4.12: pre=m1, post= m2*). Their response to two questions remained stable from the Board’s perspective relating to level of stability and growth of organisational performance measurement (Question 4.1: pre=m1, post=m2*) and level of efficient resources management for organisational performance measurement (Question 4.8: pre=m1, post=m1*).
Gervais Program Dimension: Systemic
As indicated in Table 3, the level of overall change for each of the ten systemic dimension questions was significant. Each question reported overall modal score improvement. This included seven of the ten questions recording a three modal score improvement.
Again, in contrast to the overall results and specific results from staff and salaried clinical advisors, Board members reported modal score declines in this dimension between the pre and post implementation. This occurred for questions relating to availability of resources and services to support the program (Question 5.2: pre m4, post=m2*), accessibility of resources services to support the program (Question 5.3 pre=3*, post=m2*) and the level of internal collaborations to support the program (Question 5.10: pre=m2*, post m=1*). The Board’s perception of the level of acceptance of roles and responsibilities by partners to the program remained stable as ‘poor’ (Question 5.8: pre=m2*, post=m2*).
Gervais Program Dimension: Specific
As shown in Table 3, four of the five specific dimension questions demonstrated overall modal score improvement. The overall responses relating to the quality and quantity of products and services generated from the program (Question 6.2 p=.000) remained stable at ‘unsure’.
Board members were again the only group to report variation to staff and salaried clinical advisors. They reported a decline in two of the five questions including the quality and quantity of products or services generated from organisational performance measurement (Question 6.2: pre=m1*, post=m1) and in perceived value and cost-effectiveness (Question 6.5: pre=m3*, post=2*).
Variation in Board member responses to other participant groups
As demonstrated in Table 4, despite reporting overall positivity and a high understanding of organisational performance measurement, the Board reported the lowest level of positive change at 59% (32 of 54 questions) compared to staff and salaried clinical advisors (50 of 54 questions, 98%) between pre-and-post implementation surveys. The Board also reported the highest level of unchanged modal scores from ‘unsure’ between pre and post surveys (8 of 25 questions, 32%) as compared to by staff (92%) and salaried clinical advisors (94%) both of which recorded change in questions that recorded ‘unsure’ in the pre-implementation survey. The Board perceived least change in structural dimension (3 of 8 questions, 38%), specific dimension (2 of 5 questions, 40%) and systemic dimension (5 of 10 questions, 50%). In the other three dimensions, the Board recorded over 70% improved change. In all but one dimension staff and salaried clinical advisors recorded over 80% positive change.