Gerta Rücker

Adriani Nikolakopoulou

Theodoros Papakonstantinou

Georgia Salanti

Richard D. Riley

Guido Schwarzer

https://doi.org/10.1186/s12874-020-01075-y

Figure 1

Figure 2

Figure 3

Figure 4

The statistical importance of a study for a network meta-analysis estimate

Due to technical limitations, full-text HTML conversion of this manuscript could not be completed.&nbsp; However, the manuscript can be downloaded and accessed as a PDF.

BMC Medical Research Methodology

CC BY 4.0

Background: In pairwise meta-analysis, the contribution of each study to the pooled estimate is given by its weight, which is based on the inverse variance of the estimate from that study. For network meta-analysis (NMA), the contribution of direct (and indirect) evidence is easily obtained from the diagonal elements of a hat matrix. It is, however, not fully clear how to generalize this to the percentage contribution of each study to a NMA estimate. Methods: We define the importance of each study for a NMA estimate by the reduction of the estimate's variance when adding the given study to the others. An equivalent interpretation is the relative loss in precision when the study is left out. Importances are values between 0 and 1. An importance of 1 means that the study is an essential link of the pathway in the network connecting one of the treatments with another.&nbsp;Results: Importances can be defined for two-stage and one-stage NMA. These numbers in general do not add to one and thus cannot be interpreted as `percentage contributions'. After briefly discussing other available approaches, we question whether it is possible to obtain unique percentage contributions for NMA.&nbsp;Conclusions: Importances generalize the concept of weights in pairwise meta-analysis in a natural way. Moreover, they are uniquely defined, easily calculated, and have an intuitive interpretation. We give some real examples for illustration.

Recommendation: Accept after discretionary revisions
Form responses:
---

Comments to Author:
---
I again thank the authors for their thoughtful and detailed responses to the comments on the previous version. I do not agree with them on all points, but I respect their choices and explanations. I am glad that they found some of my suggestions helpful. They have produced an accessible paper that should make an important contribution.

I have a few comments and suggestions on the new version. Most of them are minor, aimed at improving the writing.

Page 2, line 28: I have never understood the justification for giving the name "the Bucher method" (or "Bucher's method") to a straightforward application of a standard result in introductory mathematical statistics. I do, however, recognize the contribution of Bucher and colleagues. How about changing "the Bucher method" to "the work of Bucher"?

Page 3, line 17: It would help to add a hyphen: "within- and between-study variances".

Page 3, line 29: "the direct and indirect evidence proportion" could be read as referring to a single proportion instead of "the direct evidence proportion and the indirect evidence proportion". To avoid this (perhaps unlikely) misreading, make "proportion" plural: "the direct and indirect evidence proportions". I think line 45 on page 3 also calls for the plural, and so do several other places in the paper. Line 36 on page 5 handles this correctly.

Page 4, line 47: Some readers may be puzzled by "Interpretation 3", which does not appear until the top of page 6. It may be possible to revise the sentence, so that it does not refer to Interpretation 3. Or at least insert "(below)".

Page 6, line 59: Instead of "what is not true in practice" it is customary to say "which is not true in practice".

Page 9, lines 24 to 26: I did not understand "the weights attribute the lower …". The sentence was clearer in the previous version, before the authors accepted the misguided suggestion of Reviewer 1! I suggest one way to make the sentence clearer: "This is because the contributions approach attributes lower weights to a study when the network distance between the treatments is greater." Another possibility: "The reason is that, when the network distance between the treatments is greater, the contributions approach attributes lower weights to a study."

Page 9, line 52: Change "both studies" to "the two studies". (The distinction is subtle. A detailed explanation is beyond the scope of these comments.)

Page 10, line 43: Remove "therein". It is not needed, and such words are archaic and stiff (except perhaps in legal documents). Similarly, "thereof" in line 44 on page 12.

Page 10, lines 53 and 56: Should "as when" be "than when"?

Page 11, line 40: "contribution" should be plural.

Page 11, line 41: Should "contributions" be singular?

Page 11, line 62: Instead of "abstain" I suggest "refrain".

Page 12, line 34: "connotation" should be plural.

Page 13, line 10: "aforementioned" belongs to the same category as "therein" and "therein". It might be all right to change  "the aforementioned" to "the same" or "similar". And at the end of the line "he" should be "the".

Page 13, line 48: How about removing "in" before "the focus of"?
* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **No**
* Are the methods appropriate and well described?: **Yes**
* Does the work include the necessary controls?: **Yes**
* Are the conclusions drawn adequately supported by the data shown?: **Yes**
* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**
* Quality of written English: **Needs some language corrections before being published**
* Declaration of competing interests: **I declare that I have no competing interests.**
* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please publish my name with my report.**


Recommendation: Accept after discretionary revisions
Form responses:
---

Comments to Author:
---
Summary: This paper propose a measurement to quantify the importance of a study for a comparison as the relative reduction of the variance of the estimate when adding the study to the network. The reviewer thinks the quality of the current manuscript has been significantly improved and is qualified for the publication in "BMC Medical Research Methodology". The followings are some minor comments to the authors:

Minor comments:

1. In page 2, the author said "Most of them come with some limitations; their results also do not in general agree". What do you mean by "general agree"? Would you please illustrate more details of this in the introduction?
2. In page 10, "This is because the weights attributed to a study by the contributions approach are the lower, the...", it would be better to be "This is because the weights attribute the lower to a study by the contributions approach, the..."
3. In page 13, "We noted that for the random effects model the omission of a study may decrease
some variances .... For example, for a circular network (Figure 2, left panel) leaving out a study always leads to zero inconsistency, because the resulting
network is free of loops. If inconsistency was large for the primary network, the variances of all estimates decrease for the subnetworks, resulting in negative
importances." Please pay attention to the tense in the whole paper, do not mix up the past tense and the present tense in one paragraph.
* Are the methods appropriate and well described?: **Yes**
* Does the work include the necessary controls?: **Yes**
* Are the conclusions drawn adequately supported by the data shown?: **Yes**
* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**
* Quality of written English: **Acceptable**
* Declaration of competing interests: **I declare that I have no competing interests**
* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please do not publish my name with my report. (default)**


Recommendation: Major revisions required
Form responses:
---

Comments to Author:
---
I thank the authors for their thoughtful and detailed responses to the comments on the previous version. Also, the document diff69.pdf, showing the changes, was helpful.

Some issues that I raised in my previous comments require further discussion.

In response to my reservations about the reliance of their approach on two-stage methods of meta-analysis (and particularly the use of weights based on inverse variances), the authors said that "our measure is completely independent of whether a two-stage or a one-stage method has been used for estimation." I did not appreciate that generality when I read the previous version. Unfortunately, it is easy to miss that important point in the new version. The Methods section begins, "This paper is based on the two-stage approach to network meta-analysis. …  We do not deal with one-stage modelling." Of course, those statements are not the same as saying that their approach cannot be used with one-stage methods. However, two-stage methods (and inverse-variance weights) play such a dominant role in the development that readers are likely to overlook the generality.

Some restructuring of the manuscript could, after appropriate motivation, introduce the definition of the importance of Study i for Comparison c:

p(i,c) = 1 - V_{all}(c) / V_{-i}(c) = (V_{-i}(c) - V_{all}(c)) / V_{-i}(c),

which appears on page 6 of the new version (for simplicity I have omitted NMA from the subscripts; in context it may not be needed). In that form the definition of p(i,c) does not depend on the method of NMA. Implementation of the definition can then deal with various details and special cases, such as situations in which removing Study i makes Comparison c impossible. It would be easier to handle that special case by assigning the appropriate value to p(i,c) than to use the rather artificial device (in Step 1 on page 6) of downweighting Study i by substituting an arbitrarily large value for the standard error of each of its treatment effect estimates (which relies on the inverse-variance weighting in the two-stage method).

The material on implementation can develop the details for the two-stage approach (perhaps more compactly than on the current page 4), as well as the extension to the random-effects model and the applications to pairwise meta-analysis and the one-stage approach based on the Mantel-Haenszel method for NMA.

As an aside, in my comments on the previous version I suggested using V_{all}(c) instead of V_{-i}(c) as the denominator in the expression for p(i,c) above. My motivation was to express change relative to the variance under the model before the change. Thus, if one starts with the model that excludes Study i and then adds Study i, the appropriate denominator in the second expression would be V_{-i}(c), which the authors use. If, on the other hand, one starts with the model that includes all studies and then removes Study i, the appropriate denominator would be V_{all}(c).

If the authors' definition of importance is fully general, it should be possible to develop the percentage of direct evidence for a comparison without relying on the two-stage approach.

I thank the authors for explaining, in their responses, the three versions of a hat matrix that netmeta offers. To avoid potential confusion, I wonder whether they can describe other authors' work on contributions without mentioning hat matrices at all. As they pointed out in their response, their approach is not based on the hat matrix.

I recognize that some of these suggestions may seem radical. I hope, however, that the authors will consider them carefully. The restructuring would clarify the generality of their approach to importance, and it would introduce the definition earlier.

In connection with the authors' understandable decision to concentrate on variances, an examination of the change in the pooled estimate of an effect associated with removing Study i would benefit from also examining the estimated variance of that effect in the analyses with and without Study i, as well as the difference between those variance estimates.

Since the variances involved in the definition of p(i,c) are actually estimates, it would be helpful to have some discussion of the resulting variability in p(i,c). For example, how might one obtain a confidence interval?

Page 13 (of 18), bottom: It may not be accurate to equate influential studies with outliers. The literature of regression diagnostics recognizes that an observation can be influential without being an outlier and, conversely, an observation can be an outlier without being influential.

Writing

The writing can be improved by some editing. I have some illustrative suggestions.

Abstract, Background: Change "is defined based on" to "is based on". Also, "a hat matrix" would be more accurate than "the hat matrix".

Abstract, Methods: Change "we want to question" to "we question" (you actually do it); "want to" also appears in line 17 on page 12.

Abstract, Results, line 3: Should "other" be "others"?

Page 2, lines 8 and 9: The wording is a bit too telegraphic. How about "for the estimate of a given treatment effect in the NMA"?

Page 2, line 21: "for a given treatment comparison" is not clear.

Page 2, line 26: Remove "will". The Methods section describes it. The future tense simply wastes space.

Page 2, line 37: Change "coined it" to "used".

Page 2, line 44: Change "underlied" to "underlies". (The net heat plot still exists, and the idea continues to underlie it. underlied is not a correct English word. If it were appropriate, the past tense of underlie is underlay. I wonder whether this error arose from a comment by the other reviewer.)

Page 2, line 48: Change "is associated with" to "has".

Page 2, line 50: Insert a comma after "concept".

Page 2, line 54: Change "impact on" to "affect".

Page 2, line 59: No need for "at hand"

Page 3, lines 6 and 7: Remove "into study-specific contributions" (not needed twice in this sentence).

Page 3, lines 19 and 20: Change "deviates from the idea of having `percentage contributions' to each network estimate summing up" to "does not require that the contributions to each network estimate sum".

Page 3, lines 24, 27, and 32: Remove "will".

Page 3, line 59: Remove "(NMA)". Page 1, line 58 already did this.

Page 4, line 5: Remove "all based on the common effect model". This is clear from line 60 on page 3.

Page 4, line 12: Avoid "Note that" (here and elsewhere). It is essentially a reminder to the author to write the statement that follows. When that statement has been written, "Note that" is no longer needed. (A colleague once described such language as "wheel-spinning words.")

Page 4, line 13: No need for "top of"

Page 6, line 46: Remove the "a" before "flooding".

Page 7, line 48: "Parkinson data" should be "Parkinson's data". Also at page 10, lines 28 and 29 (and elsewhere?).

Page 8, line 9: Insert "by" between "these treatments" and "A" (correct usage).

Page 10, lines 29 and 30 could be clearer. How about "and, for each of the seven studies, the effect of removing a single study"?

Page 10, line 49: Remove "To explain this, we note that".

Page 12, line 18: Insert a comma after "restricted to".

Page 12, line 42: Insert a comma after "linking studies".

Table 4, title: Remove "in a column".










* Are the methods appropriate and well described?: **Yes**
* Does the work include the necessary controls?: **Yes**
* Are the conclusions drawn adequately supported by the data shown?: **Yes**
* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**
* Quality of written English: **Needs some language corrections before being published**
* Declaration of competing interests: **I declare that I have no competing interests.**
* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please publish my name with my report.**


Recommendation: Major revisions required
Form responses:
---

Comments to Author:
---
A trend seems to be developing in meta-analysis, to make as little use as possible of the two-stage approach and inverse-variance weights (in which the estimated variances of the study-level estimates are a source of various shortcomings). That transition is likely to occur slowly. In the interim, the authors' measures of importance may be useful.

Caution is required when associating terms such as contribution and importance (which are already part of most readers' nontechnical vocabulary) with measures that have specific technical definitions. Misunderstandings may occur if the measure is not a close match with readers' nontechnical understanding.

The present manuscript would be more useful if it also discussed methods of assessing the impact of individual studies on the point estimates of the comparisons in a network. Most consumers of the results of a NMA will focus first on those point estimates. Estimates of the corresponding variances are of secondary interest (e.g., in constructing confidence intervals), but also important.

A brief review of the hat matrix (H) and its role in regression diagnostics would help to frame the approaches proposed by the authors and others for quantifying a study's importance or contribution in a network meta-analysis (NMA). In the usual multiple linear regression, multiplying the vector of observed y-values by H yields the vector of fitted y-values. The diagonal elements of H are usually interpreted as quantifying the leverage of each observed y-value on the corresponding fitted y-value. Regression diagnostics have been developed to quantify the actual influence of each observed y-value on various aspects of the fitted regression. Examples include the corresponding fitted y-value, each regression coefficient, and the residual variance. Those diagnostics are generally based on the results of removing each observation (one at a time). Applying the distinction between leverage and influence to NMA, it does not seem reasonable to expect a hat matrix for NMA to provide adequate information on a study's influence.

Applying the idea of the hat matrix directly in NMA leads to two versions. One maps the vector of observed contrasts in the studies to the vector of estimates of the basic contrasts. The other maps the vector of observed contrasts in the studies to the vector of NMA estimates of those contrasts. (One can obtain these from the specification of the fixed-effects model in Section 3.1 of Reference 10. On the other hand, the matrix H in Equation 2 of Reference 7 is more distantly related to the hat matrix; it maps the vector of direct estimates of contrasts to a vector of NMA estimates of all contrasts.) The elements of these two matrices have a natural interpretation as the contribution of each study-level estimate to the estimate of a basic contrast or to the NMA estimate of that contrast. Negative elements are informative, not a problem. In the usual way, one can examine the studies' influence by removing them one at a time.

The authors could simplify the process of estimating the importance of each study (page 6, top). First, inspect the network graph to identify the studies whose removal would disconnect the network. That information should play a prominent role in understanding and analyzing the network. If removing a study disconnects the network, further analysis should focus on the resulting subnetworks. By downweighting studies, the authors' approach seems likely to give that problematic feature of the network's structure less emphasis.

In that process, I understand that Step 3 is motivated by Equation (3), but it would be more natural and more appropriate to use V_{NMA_{all}}(c) as the denominator.

In the analogy to the network of roads (page 6, bottom), flooding is another, timely example of a cause of road closure.

Page 7, middle: Using \hat{\tau}^2 from the full pairwise MA may be appropriate, and it is definitely convenient. As in regression diagnostics, however, it is valuable also to examine the estimates of \tau^2 that result from removing each study. How similar are they?

The authors clearly explain the central role of inverse-variance weights in their definition of importance. At the bottom of page 3 they mention that they are using variance estimates. Omitting hats on various symbols does make the development more readable. Importantly, however, that simplification does not force readers to keep in mind the reliance on estimated variances. Because accurate estimation of variances often requires quite large sample sizes, the authors should include a careful discussion of the impact of using estimated variances of study-level treatment effects and the impact of the variability in estimates of \tau^2. It is not satisfactory to finesse this issue by following the common practice of using estimated variances as if they were the true variances. For example (page 7, top), when the w_j are based on estimated variances, one can no longer assume that 1/\Sigma w_j is a satisfactory estimate of the variance of the pooled common effect estimate. Abandoning or modifying that assumption would seriously disrupt the authors' derivations, but retaining it should be supported by empirical evidence.

Page 4, top: Referring to back-calculation as a principle seems an exaggeration when the calculation involves only a few steps of straightforward algebra, after expressing V_{nma}(c) as the inverse-variance-weighted average of V_{dir}(c) and V_{ind}(c). I suggest mentioning this fact and omitting mention of back-calculation.

Page 4, bottom: Before stating that "the values of p(c) are found on the diagonal of the hat matrix," the authors should introduce the hat matrix that they are referring to. That will avoid ambiguity about which elements are on the diagonal (the hat matrix in Reference 10 is square, but the matrix H in Equation 2 of Reference 7 is not).

Page 5, bottom: The statement that "adding a single study to a given direct comparison makes [sense only] if the enlarged network is connected" needs clarification. If the direct comparison is already in the network, adding a study that includes that direct comparison should not change the connectedness of the network.

Page 12, lines 13 and 14: The fact that each regression coefficient depends on the selected model is an important point, often overlooked.

The real data networks

For adoption of a new method, real-data examples play an important role by illustrating how readers could apply the method in their own work. I encourage the authors to include substantially more information. At present the two real-data networks seem lifeless and abstract. The description and discussion of them give the impression that the authors have not taken them seriously. Readers will want to see how the information on studies' importance affects substantive interpretation of the results of a NMA.

For example, in the network for the Parkinson's data, the active drugs are known only by their numbers (2 to 5), and in the network for the thrombolytic data the treatments are denoted by A through H. The text (or a table) should give the actual names of the treatments, and the caption of Figure 1 should include the names of the thrombolytic treatments. And a figure showing the network graph for the Parkinson's data would be a useful addition.

On page 7, the description of the thrombolytic data does not name the outcome (mortality up to 35 days)—a surprising omission.

In the context of the two networks, I was eager to see the payoff from calculating and examining the importances. Comparing them with the study weights in a pairwise meta-analysis is only a start.

The authors should be more careful and more transparent in reporting the sources for the specific data that they use. The paper should include the data for the Parkinson's example, and Franchini et al. (2012) (which includes the names of the treatments) is a more useful source than the supplementary material of Reference 23. The data for the thrombolytic example present an interesting challenge. Reference 25 explains that the analysis in Reference 24 excluded the data from the trials that involved either anistreptilase or urokinase (Studies 12-16 and 20-28 and the third arm of Study 2 in Additional File 3). (Those data do not appear in Reference 24, but it does give references to those trials.) Reference 25 does include those data, but it has an error in the data for Study 1: in the notation of Additional File 3, it gives n1 as 20,163 (Additional File 3 has the correct value: 20,173.). Thus, it would be safer to cite Reference 6 as the source of the data. (Some of the tables in Reference 24 have errors and inconsistencies.) Also, an example in Reference 9 uses the thrombolytic data, but with different denominators in three studies, an additional treatment (angioplasty), and a total of 50 studies.

Reference 13 should include k585 in addition to the volume number.

Reference 24 has three additional authors.

The caption for Figure 1 should explain the shading, and it could translate A through H into the names of the treatments.

In Table 3, the importance of Study 1 for comparison 1:4 is almost as large as that for comparison 2:3. Does this result have a sensible explanation?

The title of Table 6 says "common effects model," but the text (page 10, bottom) says "random effects model."



Reference

Franchini AJ, Dias S, Ades AE, Jansen JP, Welton NJ (2012). Accounting for correlation in network meta-analysis with multi-arm trials. Research Synthesis Methods 2012; 3:142_160.
* Are the methods appropriate and well described?: **Yes**
* Does the work include the necessary controls?: **Yes**
* Are the conclusions drawn adequately supported by the data shown?: **Yes**
* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**
* Quality of written English: **Needs some language corrections before being published**
* Declaration of competing interests: **I declare that I have no competing interests.**
* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: ** I agree to the open peer review policy of the journal**


Recommendation: Major revisions required
Form responses:
---

Comments to Author:
---
Summary: This paper propose a measurement to quantify the importance of a study for a comparison as the relative reduction of the variance of the estimate when adding the study to the network.

Major comments: 1). When we set investigation on certain measurement, we should first clarify what the underlying quantity we are estimating for. Like AIC, we are estimating the quantity proportional to the KL distance，which theoretically quantifies how well the model fits the data. I think the author should put more effort to explain where the proposed measurement comes from, and why this quantity is valid for defining the contribution of each study. Also, how to check whether your proposed measurement is proper or not seems vague and subjective to me. Thus, the author should make more solid argument to justify why this proposed importance indicator would theoretically work and better compared to existing methods.

2). It is at least not clear to me that, practically speaking, why we need to quantify the importance of each study in network meta-analysis. Will it further help to enhance the analysis, or for any design purpose? It would be much helpful if more practical motivation is provided to justify how we will be benefited if we correctly identify the importance of each study in the analysis, especially for the data application section.

3). The proposed method indeed is based on the idea of leave one out. So it is straight-forward to think what will happen if you leave several out instead of one. By doing so will help to quantify how several studies together contribute to the estimate. I would like to see at least some discussion on this potential extension.

Minor comments:1). It would be better to describe in details what is the NMA and what is the definition of direct and indirect effect in the framework of NMA at the beginning of this paper so that readers who are not very familiar to this topic will not be confused.

2). The author should pay attention to tense consistency in introduction. For example, the author wants to introduce Salanti's work in line 36-48 on page 2, but uses the past tense and present tense alternately, which would be better to be consistent with only one tense.

3).Please pay attention to the pauses in sentences:
Line 56 on page 5, it would be better to write as "Therefore, we focus...";
Line 61 on page 6, it would be better to write as "For pairwise MA, this leads...";
Line 10 on page 15, it would be better to write as "For each indirect comparison, we may...";
.......

4). Line 29 on page 13, it should be "This is different from the random effect model......"

5). All the references are missing, the author should revise the reference format.* Are the methods appropriate and well described?: **No**
* Does the work include the necessary controls?: **Yes**
* Are the conclusions drawn adequately supported by the data shown?: **Yes**
* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**
* Quality of written English: **Needs some language corrections before being published**
* Declaration of competing interests: **I declare that I have no competing interests**
* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: ** I agree to the open peer review policy of the journal**


The statistical importance of a study for a network meta-analysis estimate

Status:

Journal Publication

Version 4

Abstract

Figures

Full-text

Supplementary Files

Status:

Journal Publication

Version 4