Participant enrolment and characteristics
The 43 reviewers assigned to review applications in the funding programme Health at the Norwegian Foundation Dam for the year 2018 were assessed for eligibility (Figure 2). One was excluded due to not being part of the review committee the previous year. The remaining 42 reviewers were included in the study and randomized to either the general feedback group (n=23) or the individual feedback group (n=19). Five reviewers served in both calls and were thus excluded from the Health fall call.
One participant in the general feedback group could not perform his reviews due to acute illness. Hence, he did not receive the allocated intervention. Follow-up data and data on compliance were retrieved for the remaining 41 participants, and none of them were excluded from the analyses. All participants in the general feedback group and 95 % of the participants in the individual feedback group confirmed that they had received and read the feedback report.
Participant characteristics were similar in the two intervention groups (Table 1).
Table 1. Baseline characteristics of the study participants.
Baseline measures
|
General feedback group (n=22)
|
Individual feedback group (n=19)
|
Total
(n=41)
|
Age, years
|
58 ± 11,4
|
49 ± 9,4
|
54 ± 11,4
|
Women, count (%)
|
14 (64 %)
|
9 (47 %)
|
23 (56 %)
|
Years of experience as reviewer for the foundation
|
2,6 ± 2,97
|
2,4 ± 3,17
|
2,5 ± 3,08
|
Reviews assessing the application as eligible, count (%)
|
651 (91 %)
|
484 (89 %)
|
1135 (90 %)
|
Application score (1-10)
|
6,3 ± 1,93
|
5,7 ± 2,14
|
6,1 ± 2,05
|
Average absolute difference
|
2,0 ± 1,54
|
2,2 ± 1,59
|
2,1 ± 1,56
|
Values are mean ± SD unless otherwise stated
Numbers analysed
A total of 2398 paired reviews were analysed (Table 2). In 2038 of these cases, the two reviewers agreed that the application was eligible, and the average absolute difference could be calculated.
Table 2. Numbers analysed at baseline and follow-up.
|
General feedback group (n=22)
|
Individual feedback group (n=19)
|
|
Baseline
|
Follow-up
|
Total
|
Baseline
|
Follow-up
|
Total
|
Total number of paired reviews
|
715
|
642
|
1357
|
545
|
496
|
1041
|
Number of applications included in analyses
|
511
|
450
|
961
|
409
|
376
|
785
|
Number of paired reviews included in analyses of average absolute difference
|
601
|
594
|
1195
|
434
|
409
|
843
|
Number of paired reviews included in eligibility agreement analyses
|
715
|
642
|
1357
|
545
|
496
|
1041
|
Outcomes and estimation
Main outcome: agreement
The results from the LMM analyses (Table 3) showed that the time×group interaction effect was not significant, indicating an equal decrease in the difference score over time for the individual feedback and the general feedback groups. There was an overall significant decrease in the difference score from baseline to follow-up (b = - 0.32, p =.004). At baseline, the reviewers within the general feedback group had a lower average absolute difference compared to the individual feedback group (b = - 0.24, p = .020).
Table 3. Linear mixed regression model analysis estimating the change in differences score over time by group.
|
b (se)
|
t-value
|
p
|
95 % CI
|
Intercept
|
2.20 (0.08)
|
28.593
|
.000
|
[2.05, 2.35]
|
Timea
|
-0.32 (0.11)
|
-2.924
|
.004
|
[-0.54, -0.11]
|
Groupb
|
-0.24 (0.10)
|
-2.329
|
.020
|
[-0.44, -0.04]
|
Time×Group
|
0.17 (0.15)
|
1.206
|
.228
|
[-0.11, 0.46]
|
A total of 2038 average absolute differences nested within 1500 applications were included in the analysis. At baseline, 434 difference scores were from reviewers in the intervention group, and 601 difference scores were from reviewers in the control group. At follow-up, 409 difference scores were from reviewers in the intervention group and 594 from reviewers in the control group. aBaseline=0, follow-up=1, bIndividual feedback group=0, General feedback group=1.
The ICC (one-way random, average measures) at baseline for the general feedback group and individual feedback group was 0.276 and 0.323, respectively. At follow-up, the values were 0.303 for the general feedback group and 0.401 for the individual feedback group (Figure 3).
The mean application score at baseline in the general feedback group was 6.3 (95 % CI from 6.19 to 6.49), and the mean score in the individual feedback group was 5.7 (95 % CI from 5.52 to 5.90) (Table 4). At follow-up, the mean score in the general feedback group was 6.6 (95 % CI from 6.46 to 6.76), and the mean score in the individual feedback group was 6.2 (95 % CI from 6.04 to 6.41).
Table 4. Application eligibility, application score and average absolute difference.
|
General feedback group (n=22)
|
Individual feedback group (n=19)
|
|
Baseline
|
Follow-up
|
Baseline
|
Follow-up
|
Reviews assessing the application as eligible, count (%)
|
651 (91 %)
|
613 (96 %)
|
484 (89 %)
|
442 (89 %)
|
Eligibility agreement, count (%)
|
612 (86 %)
|
599 (93 %)
|
453 (83 %)
|
417 (84 %)
|
Application score (1-10)
|
6,3 ± 1,93
|
6,6 ± 1,85
|
5,7 ± 2,14
|
6,2 ± 1,97
|
Average absolute difference
|
2,0 ± 1,54
|
1,8 ± 1,47
|
2,2 ± 1,59
|
1,9 ± 1,48
|
Values are mean ± SD unless otherwise stated
The results from an LMM analysis (Table 5) found no main effect of either time or group but a significant time×group interaction effect (b = 0.77, p = .006, OR = 2.17). The interaction effect indicated an increase in the proportion of eligibility agreement over time for the general feedback group only.
Table 5. Linear mixed binary logistic regression model analysis of eligibility agreement over time by group.
|
b (se)
|
t-value
|
p
|
Odds Ratio [95 % CI]
|
Intercept
|
1.65 (0.13)
|
12.6
|
< .001
|
5.19 [4.02, 6.70]
|
Timea
|
0.08 (0.19)
|
0.43
|
.667
|
1.09 [0.75, 1.58]
|
Groupb
|
0.19 (0.18)
|
1.05
|
.293
|
1.21 [0.85, 1.71]
|
Time×Group
|
0.77 (0.28)
|
2.74
|
.006
|
2.17 [1.25, 3.79]
|
A total of 2398 paired eligibility assessments nested within 1746 applications were included in the analysis. At baseline, 545 paired eligibility assessments were from reviewers in the intervention group and 715 paired eligibility assessments were from reviewers in the control group. At follow-up, 496 paired eligibility assessments were from reviewers in the intervention group and 642 from reviewers in the control group. aBaseline=0, follow-up=1, bIndividual feedback group=0, General feedback group=1.
Cohen’s kappa at baseline for the general feedback group and the individual feedback group was 0.097 and 0.197, respectively. At baseline, the values were 0.154 for the general feedback group and 0.082 for the general feedback group (Figure 4). The rate of eligibility agreement was high in both groups at baseline and follow-up, ranging from 83 % to 93 % (Table 4).
Secondary outcome: perceived usefulness
Table 6 displays the perceived usefulness of the interventions. The results show that 95 % (n=18) in the individual feedback group and 68 % (n=15) in the general feedback group responded to the question “To what degree did you find the feedback you received useful?” after finishing the reviews. An independent sample t-test of the mean scores showed that there was no significant difference between the groups (p=0.442).
Table 6. Perceived usefulness of the interventions.
|
General feedback group (n=15)
|
Individual feedback group (n=18)
|
“To what degree did you find the feedback you received useful?”, mean score (scale 1-5)*
|
3,5 ± 0,74
|
3,7 ± 0,75
|
“If you were offered this feedback next time, would you want it?”, the proportion of “Yes”
|
80 % (n=12)
|
94 % (n=17)
|
* The question was answered on a five-point Likert scale (To a very small degree, To a small degree, To some degree, To a large degree or To a very large degree) and coded from 1 (To a small degree) to 5 (To a very large degree).
Sensitivity analyses
The sensitivity analysis showed no difference between the two groups in the proportion of absolute agreement on application score, defined as a difference in score of 0 or 1 being similar (p=1.000), supporting the findings of the main analysis of average absolute differences.
There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding programme, with the general feedback group demonstrating a higher rate of eligibility agreement (p<.01) in the 2018 review.