Objectives: The aim of this study is to establish the inter-rater reliability of the CQS-2 and to test the null-hypotheses that it does not differ significantly from that of the first CQS version (CQS-1).
Methods: Four independent raters will be selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters will remain unaware regarding each other’s participation in this study until all rating is completed. Each rater will receive only one rating-template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters will complete each template and send these back to the principal investigator via email. Each rater will receive her/his next template two weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 will be established by use of the Brennan-Prediger coefficient. The coefficients of both CQS versions will be compared by use of the two-sample z-test. All data analysis will be carried out using SAS statistical software. A 5% significance level will be used. During secondary analysis, we will establish the inter-rater reliability for each single criterion and each corroboration level for both CQS versions.
Reporting: The results of this study will be reported in line with the Guidelines for Reporting Reliability and Agreement Studies (GRAAS). The final report will be made available online as preprint in one of the major preprint repositories and submitted to a peer-reviewed journal.