Network meta-analysis (NMA)
The results from the NMA can be seen in Table 2, listing the relative effects of all interventions present in the network. The relative effectiveness of the interventions are presented as odds ratios (ORs) with 95% credible intervals. From Table 2, we can see that most interventions are more effective at increasing the uptake of the poison prevention behaviours for the safe storage of other household items than usual care, apart from the free/low-cost equipment intervention. Using the results of the NMA, we ranked the interventions according to which was the most effective at increasing the uptake of the poison prevention measures in the home. The results from the rankings can be seen in Table 3.
Table 2
Details of studies included in NMA for the safe storage of other household products outcome.
Intervention Comparison
|
Study Number
|
Study
|
Study quality
|
Safe storage of other household products
|
Usual care (1) vs.
Education (2)
|
1
|
Kelly (1987), RCT, USA
|
A=U,B=Y,F=N
|
43/54
49/55
|
2
|
Nansel (2002)a, RCT, USA
|
A=Y,B=U,F=Y
|
65/89
66/85
|
3
|
McDonald (2005), RCT, USA
|
A=Y,B=U,F=N
|
3/57
6/61
|
4
|
Gielen (2007), RCT, USA
|
A=Y,B=N,F=Y
|
44/62
57/73
|
5
|
Nansel (2008), Non-RCT, USA
|
A=U,B=N,F=N
|
59/73
117/144
|
Usual care (1) vs Education + Free/low cost Equipment (3)
|
6
|
Woolf (1992), Cluster-RCT, USA
|
A=U,B=Y,F=N
|
60/151
89/150
|
7
|
Clamp (1998), RCT, UK
|
A=U,B=N,F=Y
|
49/82
59/83
|
Usual care (1) vs.
Education + Equipment + Home Safety inspection (4)
|
8
|
Kendrick (1999), Cluster non-RCT, UK
|
B=N,F=N,C=Y
|
317/367
322/363
|
9
|
Swart (2008), Non-RCT, South Africa
|
A=U,B=Y,F=Y
|
46.86/57.96b
50.87/58.27b
|
10
|
Hendrickson (2002), USA, RCT
|
A=N,B=N,F=Y
|
14/40
34/38
|
Usual care (1) vs.
Education + Equipment (5)
|
11
|
Watson (2005), Cluster-RCT, UK
|
A=Y,B=N,F=Y
|
327/669
368/693
|
Education (2) vs.
Education + Equipment (3)
|
12
|
Posner (2004), RCT, USA
|
A=Y,B=Y,F=N
|
22/47
34/49
|
Education (2) vs.
Education + Equipment (5)
|
13
|
Sznajder (2003), RCT, France
|
A=Y,B=N,F=Y
|
32/41
40/48
|
Education+ equipment (3) vs.
Equipment only (7)
|
14
|
Dershewitz (1977), RCT, USA,
|
A=U,B=Y,F=N
|
1/101c
0/104c
|
Education + Equipment + home Safety inspection (4) vs.
Education + equipment + home safety inspection + Fitting (6)
|
15
|
King (2001), RCT, USA
|
A=Y,B=Y,F=Y
|
261/469
273/482
|
Last column includes the number of households with safe storage out of the total number of storage.
Abbreviations:
- A = adequate allocation concealment; B = blinded outcome assessment; C, the prevalence of confounders does not differ by more than 10% between treatment arms; CBA, controlled before-and-after study; F = at least 80% participants followed up in each arm; NMA, network meta-analysis; RCT, randomised clinical trial; U = unclear; Y= yes.
- a Two intervention arms were combined (tailored advice and tailored advice + care provider feedback)
- b Figures adjusted for the effect of clustering using ICC and method reported Kendrick et al. (2012)
- c Continuity correction applied by adding 0.5 and 1 to denominator and numerator to account for the zero events reported (no households that were assessed safely stored other household products).
Table 3
Table of the ranking of intervention for the safe storage of other household products outcome
Intervention
|
Ranking (95% Credible Interval)
|
Probability intervention is the best
|
1
|
Usual care (UC)
|
6 (4, 7)
|
0.00
|
2
|
Education (E)
|
5 (2, 7)
|
0.01
|
3
|
Education + Free/low cost Equipment
(E + FE)
|
3 (1, 6)
|
0.22
|
4
|
Education + Free/low cost Equipment + Fitting
(E + FE + F)
|
2 (1, 5)
|
0.22
|
5
|
Education + Free/low cost Equipment + Home safety inspection
(E + FE + HSI)
|
4 (1, 7)
|
0.05
|
6
|
Education + Free/low cost Equipment + Fitting + Home safety inspection
(E + FE + F + HSI)
|
2 (1, 7)
|
0.37
|
7
|
Free/low-cost equipment only
(FE only)
|
7 (1, 7)
|
0.13
|
From Table 3, we can see that the intervention with the highest probability of being the most effective is education + free/low-cost equipment + fitting + home safety inspection (E + FE + F + HSI), which is the most intensive intervention. This intervention was also ranked highest along with education + free/low-cost equipment + fitting (E + FE + F). The least effective interventions were usual care and free/low-cost equipment only. There was overlap between the 95% credible intervals for the rankings for all the interventions, indicating that no distinct intervention is optimal or worst.
Study level threshold analysis
The studies are sorted according to those with the smallest thresholds, with the intervention contrasts for each particular study identified in the brackets; the studies emphasised in bold represent those in which the 95% confidence interval for the effect estimate extends beyond the invariant interval. Where the 95% CI extends beyond the invariant interval, the invariant interval is coloured red rather than light blue. The optimal intervention for this NMA is intervention 6. The new optimal intervention is displayed for that particular study on either side of the invariant intervals. NT indicates "No threshold", meaning that no threshold exists in that particular direction, so no amount of adjustment in that particular direction would change the optimal intervention from intervention 6.
Figure 2 presents the results of the study level threshold analysis. We can see that of the 15 studies included in the network meta-analysis, 7 studies had 95% confidence intervals extending beyond the invariant interval (indicated in bold). This demonstrates that the intervention recommendations are sensitive to the amount of imprecision in the study estimates in studies: 6, 7, 9, 10, 12, 14, and 15. For example, for study 15, which compared interventions 4 and 6, the estimated log OR of 0.04 had an invariant interval of (0.00, NT). This indicates that a change of -0.04 in the log OR would change the optimal intervention recommendation from intervention 6 to intervention 4. The NT in the upper invariant interval represents "No threshold", which illustrates that no amount of change in this direction would change the optimal intervention recommendation. For study 10, which compared interventions 1 and 4, the estimated log OR of 2.76 has an invariant interval of (2.19, 50.88). This illustrates that a change in the log OR of -0.57 is substantial enough to change the intervention recommendation from intervention 7 to intervention 3. Therefore, a change in the log odds ratio of 0.82 would change the intervention recommendation to intervention 3 being the most optimal rather than intervention 6. However, for studies 6 and 12, the upper limits of the invariant intervals lie very close to the upper limits of the 95% confidence intervals. For the remaining 8 studies, their relative 95% confidence intervals fall within the invariant intervals, which indicates that no amount of change in the study estimates would change the intervention recommendation as the bias adjustment threshold is very large.
Contrast level threshold analysis
Figure 3 shows the results from the contrast level threshold analysis. Five of the intervention contrasts in the network have either upper or lower portions of their respective invariant intervals outside of the 95% credible intervals, indicating that the decision for these contrasts are sensitive to the level of imprecision in these estimates. For the other two contrasts in the network (2 vs 1, 5 vs 2), the invariant intervals are wide and contain the 95% credible interval for each estimate. This indicates that the average effectiveness estimates for these comparisons are robust to any changes in the evidence. The results from Figure 3 are consistent with those depicted in the study level threshold analysis (Figure 2).
It is important to note that when only one study observes a particular contrast in the network, the results of the threshold analyses at study level and contrast level must be consistent. From Figure 1, there are two two-arm studies in the network, which are single studies for comparisons 7 vs 3 and 6 vs 4. From Figure 3, we can see that the thresholds for the contrast 6 vs 4 are identical to those corresponding to study 15 in the study level analysis (as seen in Figure 2), as expected. However, we can see that the 95% credible interval for the effect estimate is wider in the contrast level analysis than the 95% confidence interval in the study level analysis. This is due to the combined NMA result being less precise than the study estimate due to the large level of heterogeneity in the NMA. However, for the 7 vs 3 contrast, both the effect estimates and thresholds are different at the study level and the contrast level. Despite the quantitative differences between the study level and the contrast level analyses for this comparison, the results for this particular contrast/study are consistent qualitatively. There is a lot of uncertainty around the effect estimate for this contrast/study, and the upper threshold (in favour of intervention 7) lies well within the confidence interval at study level and credible interval at contrast level.