Development and Validation of the Study Quality Assessment of Design (SQUAD) Tool for Systematic Reviews

Despite consensus about the need to assess study design quality in systematic reviews, there remains a need for practical tools that can be used for quality assessment across diverse study designs. We developed the Study Quality Assessment of Design (SQUAD) tool by combining and streamlining the Cochrane tool for grading randomized controlled trials and the risk of bias criteria developed for Effective Practice and Organisation of Care (EPOC) reviews. We validated the tool and refined it through a four-iteration pilot. We then used the tool to evaluate the quality of studies in a systematic review of the effects of interpersonal interventions on Quadruple Aim outcomes (i.e., population health, cost, patient and provider experience).


Abstract
Background Despite consensus about the need to assess study design quality in systematic reviews, there remains a need for practical tools that can be used for quality assessment across diverse study designs.

Methods
We developed the Study Quality Assessment of Design (SQUAD) tool by combining and streamlining the Cochrane tool for grading randomized controlled trials and the risk of bias criteria developed for Effective Practice and Organisation of Care (EPOC) reviews. We validated the tool and refined it through a four-iteration pilot. We then used the tool to evaluate the quality of studies in a systematic review of the effects of interpersonal interventions on Quadruple Aim outcomes (i.e., population health, cost, patient and provider experience).

Results
During a pilot with 12 studies (8 randomized controlled trials and 4 observational studies), the ICC (1,1) improved from 0.41 to 0.89. In the systematic review of 77 studies (68 randomized control trials and 9 observational studies), the ICC (2,1) was 0.72.

Conclusions
SQUAD is a practical and reliable tool for assessing the quality of studies of various designs when synthesizing findings for systematic reviews. Standardized practices for quality assessment are critical to the reliability of systematic reviews. This pragmatic and standardized tool can facilitate efficient and high-quality assessments for a broad range of studies.

Background 3
Quality assessment of included studies is an integral component of rigorous systematic reviews as this step helps to ensure findings are externally valid. However, systematic reviews often include both randomized controlled trials (RCTs) and observational studies, and existing quality assessment tools are limited in the degree to which they allow for quality comparisons across these different study designs. Many existing quality tools tend to favor RCTs over other study designs, and while RCTs are among the most rigorous studies, observational studies often generate valuable evidence in situations where an RCT is not feasible. In this paper, we describe the development and validation of a novel Study Quality Assessment of Design (SQUAD) tool that can be used to assess the quality of multiple types of study designs within a systematic review.

Methods
The SQUAD tool was developed for a systematic review of studies examining the effects of clinician-patient interpersonal interventions on Quadruple Aim outcomes (i.e., population health, cost, patient and provider experience) 19 . First, we conducted a search for existing quality assessment tools using PubMed, Google, and systematic reviews in the literature.
For each of the 6 tools that we identified, 12,29,30,52,59,79 we examined the quality assessment criteria and the types of studies that could be assessed with the tool. Our search suggested that existing tools were unable to adequately handle diverse study designs. We determined that a tool combining the Cochrane tool for grading RCTs 52 and the risk of bias criteria developed for controlled observational studies developed by the Effective Practice and Organisation of Care (EPOC) reviews 29 could be used to compare quality across both RCTs and observational studies. Thus, both formed the foundation for the SQUAD tool, which incorporates elements of Cochrane and EPOC assessments, with 4 simplified language and processes for ease of use. The domains of the SQUAD tool are listed in Table 1.
To test and refine the tool, we performed a multi-phase pilot with 12 studies drawn from the systematic review (8 RCTs and 4 observational studies). 2,5,9,16,[20][21][22][23]32,37,53,75 For each phase, 3-4 studies were assessed by two raters (AT and SB). Ratings for the studies were reviewed by a third team member (MH), and discrepancies were resolved by consensus. The tool was revised after each of these meetings before additional studies were reviewed. Most differences between the two raters were due to missing or misplaced information in the text; discrepancies rarely arose due to different interpretations of the descriptions of the domains. We examined inter-rater reliability after each iteration by calculating the model 3 intra-class correlation coefficient for raters.
After refining the SQUAD tool, we tested it in an evaluation of all 77 studies for the systematic review 1-11,13- Table 2.

Discussion
The SQUAD tool offers a pragmatic approach to quality assessment for researchers evaluating RCTs and observational studies in systematic reviews. It is unique in that it can handle both types of studies without being biased towards a single study design, while maintaining the rigor necessary to help ensure external validity. It is also simple and accessible to researchers with varying levels of experience. A limitation of the design process is that it was tested in a single systematic review of RCTs and observational studies of clinician-patient interpersonal interventions. Additional testing is warranted to validate the tool across more study designs covering diverse content. However, our goal at this stage is to present a novel tool for use in systematic reviews of studies with a mixture of designs. The authors acknowledge that the tool could use further refinement, and is an avenue for future research.

Conclusions
SQUAD is a practical and reliable tool for assessing the quality of randomized trials and observational studies when synthesizing findings for systematic reviews. This tool will help ensure that results from systematic reviews maintain external validity when incorporating multiple study designs.   Each domain is given a score of 1 to 3 based on the level of adherence to the principle measured by it. An average of scores study is calculated to determine overall study score.