We needed four queries to develop a good SAP, and that took approximately 15 minutes. Researchers with less statistical experience may require more queries and time. The results are presented in two ways. First, we listed the queries without their responses to more easily show how the queries evolved with each iteration. After that, the queries and their responses were reported verbatim, including poor English grammar, as they were developed in real time for this report. The SAP validation is reported at the end of the results section.
Query 1 was,
Write the statistical analysis plan for a randomized controlled trial comparing two kinds of surgical interventions for cranial cruciate ligament rupture in dogs using ground reaction forces as outcome.
The response to query 1 was too vague to be useful, so we added more study details for query 2. Note that all the queries use information that is available before data collection from previously published studies. Query 2 was,
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. We want to account for baseline measures.
The response to query 2 mentioned analyzing missing data, and we were unsure how to do that, so we asked about it in query 3. We also needed more detail on how to incorporate baseline data into the statistical analysis. Query 3 was,
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. Include how we would adjust for baseline measures in the final analysis. Recommend tests for normality. How would we fix missing data?
We were reasonably satisfied with the response to query 3. However, we knew that ANOVA has assumptions that must be satisfied for it to be valid, so we asked about them. Query 4 was,
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. Include how we would adjust for baseline measures in the final analysis. Recommend tests for normality. How would we fix missing data? Include the model assumptions for the statistical tests.
The queries with their responses are below. Note that ChatGPT adds boilerplate material of no statistical value to each response. For completeness, we left that material in the text.
Query 1
Write the statistical analysis plan for a randomized controlled trial comparing two kinds of surgical interventions for cranial cruciate ligament rupture in dogs using ground reaction forces as outcome.
ChatGPT response to query 1
A Statistical Analysis Plan (SAP) is a critical document that outlines the planned statistical methods and procedures for analyzing data from a randomized controlled trial (RCT). Here’s an outline of a SAP for a RCT comparing two surgical interventions for cranial cruciate ligament rupture in dogs using ground reaction forces (GRF) as the outcome measure:
-
Introduction:
-
Provide a brief introduction to the study, including its objectives and rationale.
-
Mention the two surgical interventions being compared.
-
Highlight the importance of using GRF as the primary outcome measure.
-
Study Design:
-
Describe the study design, including randomization methods and blinding procedures.
-
Specify the duration of the study and the number of study participants (dogs) included.
-
Data Collection:
-
Detail the process of collecting GRF data, including the equipment used, measurement frequency, and any relevant quality control procedures.
-
Discuss any potential sources of bias in data collection.
-
Data Cleaning and Management:
-
Describe the procedures for data cleaning and management, including handling missing data and outliers.
-
Explain how data will be stored securely and confidentially.
-
Primary Outcome:
-
Define the primary outcome measure, which is the GRF data.
-
Explain why this outcome is chosen as the primary measure.
-
Statistical Analysis Objectives:
-
Clearly state the main statistical objectives of the analysis.
-
These may include comparing GRF outcomes between the two surgical intervention groups, assessing changes over time, and identifying any covariates of interest.
-
Statistical Methods: a. Descriptive Statistics:
-
Provide summary statistics for baseline characteristics of the study participants in each group. b. Comparison of Groups:
-
Specify the primary analysis method, such as an analysis of covariance (ANCOVA), mixed-effects model, or another appropriate statistical test.
-
Describe how adjustments for potential confounders will be made. c. Subgroup Analyses:
-
If planned, outline any subgroup analyses based on specific factors (e.g., dog breed, age, or preoperative severity). d. Handling of Missing Data:
-
Explain how missing data will be handled, including imputation methods if necessary. Sensitivity Analyses:
-
Describe any sensitivity analyses that will be conducted to assess the robustness of the primary findings. f. Significance Level:
-
Specify the alpha level for statistical significance (e.g., 0.05).
-
Sample Size Calculation:
-
Explain how the sample size was determined, including assumptions and considerations.
-
Confirm that the study is adequately powered to detect clinically meaningful differences.
-
Data Presentation:
-
Describe how the results will be presented, including tables, figures, and any supplementary materials.
-
Ethical Considerations:
-
Timeline:
-
Reporting:
-
Specify the intended audience for the study results (e.g., scientific journals, conferences, clinical guidelines).
-
Discuss plans for publication and dissemination of results.
-
References:
-
Appendices:
-
Attach any additional documents, such as the statistical code used for analysis, data dictionaries, or data collection forms.
It’s essential to develop this Statistical Analysis Plan before conducting the data analysis to maintain transparency, rigor, and adherence to the study’s objectives and methodology. Collaboration with a statistician or data analyst experienced in clinical trials is highly recommended to ensure the validity of the analysis plan.
Query 2.
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. We want to account for baseline measures.
ChatGPT response to query 2
Below is a Statistical Analysis Plan (SAP) for your randomized controlled trial (RCT) comparing TPLO (Tibial Plateau Leveling Osteotomy) to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as the primary outcome and LOAD scores as secondary outcomes. You mentioned having 50 dogs per group and accounting for baseline measures.
-
Introduction:
-
Study Design:
-
Describe the study design, including randomization methods, blinding procedures, and the duration of the study.
-
Confirm that you have 50 dogs allocated to each group.
-
Data Collection:
-
Explain the methods used to collect peak vertical force data and LOAD scores.
-
Discuss any quality control procedures, calibration, or standardization efforts for data collection.
-
Baseline Characteristics:
-
Provide summary statistics for baseline characteristics (e.g., age, breed, preoperative severity) for both treatment groups.
-
Explain how baseline imbalances, if any, will be addressed during the analysis.
-
Primary Outcome:
-
Define the primary outcome measure, which is the peak vertical force.
-
Justify why this outcome is chosen as the primary measure.
-
Secondary Outcomes:
-
Define the secondary outcome measure, which is the LOAD score.
-
Explain the relevance of LOAD scores in assessing treatment effects.
-
Statistical Analysis Objectives:
-
Statistical Methods: a. Primary Outcome Analysis:
-
Describe the primary analysis method, such as an analysis of covariance (ANCOVA) with baseline peak vertical force as a covariate.
-
Specify the alpha level for statistical significance (e.g., 0.05). b. Secondary Outcome Analysis:
-
Explain how LOAD scores will be analyzed, e.g., using mixed-effects models, repeated measures ANOVA, or another suitable method. c. Subgroup Analyses:
-
If planned, outline any subgroup analyses based on specific factors (e.g., age, breed, or preoperative severity). d. Handling of Missing Data:
-
Describe how missing data, if any, will be handled, including imputation methods. e. Sensitivity Analyses:
-
Mention any planned sensitivity analyses to assess the robustness of the primary findings. f. Adjustment for Multiple Testing:
-
If multiple secondary outcomes are analyzed, consider applying appropriate methods to control the familywise error rate (e.g., Bonferroni correction).
-
Sample Size Calculation:
-
Data Presentation:
-
Describe how the results will be presented, including tables, figures, and presentation of effect sizes and confidence intervals.
-
Ethical Considerations:
-
Timeline:
-
Reporting:
-
Specify the intended audience for the study results (e.g., scientific journals, conferences).
-
Discuss plans for publication and dissemination of results.
-
Appendices:
-
Attach any additional documents, such as the statistical code used for analysis, data dictionaries, or data collection forms.
It’s crucial to follow this analysis plan rigorously to ensure that the study’s objectives are met and that the analysis is conducted in a transparent and scientifically sound manner. Collaboration with a statistician or data analyst with experience in clinical trials is highly recommended to ensure the validity of the analysis plan.
Query 3.
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. secondary outcomes are LOAD scores. we have 50 dogs per group. include how we would adjust for baseline measures in the final analysis. recommend tests for normality. how would we fix missing data
ChatGPT response to query 3
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. Include how we would adjust for baseline measures in the final analysis. Recommend tests for normality. How would we fix missing data?
-
Introduction:
-
Provide a brief overview of the study, including its objectives, rationale, and the two surgical interventions being compared.
-
Study Design:
-
Describe the study design, including randomization methods, blinding procedures, and the number of dogs per group (50 dogs per group).
-
Data Collection:
-
Baseline Characteristics:
-
Provide summary statistics for baseline characteristics (e.g., age, breed, preoperative severity) separately for each treatment group.
-
Primary Outcome:
-
Secondary Outcomes:
-
Statistical Analysis Objectives:
-
Statistical Methods: a. Primary Outcome Analysis:
-
Perform an analysis of covariance (ANCOVA) to compare peak vertical force between the two treatment groups.
-
Include baseline peak vertical force as a covariate to adjust for baseline differences.
-
Check the normality of the residuals using visual inspection (e.g., histograms, Q-Q plots) and consider transformations (e.g., log transformation) if needed. b. Secondary Outcome Analysis:
-
Analyze LOAD scores using an appropriate statistical method (e.g., linear regression, mixed-effects model).
-
Include baseline LOAD scores as covariates to adjust for baseline differences.
-
Assess the normality of the residuals for the secondary outcome. c. Assessment of Normality:
-
Perform normality tests (e.g., Shapiro-Wilk, Kolmogorov-Smirnov) on both primary and secondary outcome variables.
-
Consider statistical transformations or non-parametric methods if the data are significantly non-normal. d. Handling Missing Data:
-
Describe the approach for handling missing data, such as multiple imputation, if applicable.
-
Report the proportion of missing data for each variable.
-
Sample Size Calculation:
-
Data Presentation:
-
Explain how the results will be presented, including tables, figures, and the presentation of effect sizes and confidence intervals.
-
Ethical Considerations:
-
Timeline:
-
Reporting:
-
Specify the intended audience for the study results (e.g., scientific journals, conferences).
-
Discuss plans for publication and dissemination of results.
-
Appendices:
-
Attach any additional documents, such as the statistical code used for analysis, data dictionaries, or data collection forms.
Ensure that this analysis plan is followed rigorously to maintain the integrity of the study. Collaboration with a statistician or data analyst experienced in clinical trials is highly recommended to ensure the validity of the analysis plan and adherence to statistical best practices.
Query 4.
Write the statistical analysis plan for a randomized controlled trial comparing tplo to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome. Secondary outcomes are LOAD scores. We have 50 dogs per group. Include how we would adjust for baseline measures in the final analysis. Recommend tests for normality. How would we fix missing data? Include the model assumptions for the statistical tests.
ChatGPT response to query 4
Certainly, here’s a detailed Statistical Analysis Plan (SAP) for your randomized controlled trial (RCT) comparing TPLO (Tibial Plateau Leveling Osteotomy) to lateral suture stabilization for cranial cruciate ligament rupture in dogs using peak vertical force as a primary outcome and LOAD scores as secondary outcomes. This plan includes methods to adjust for baseline measures, recommendations for assessing normality, handling missing data, and model assumptions for statistical tests.
-
Introduction:
-
Study Design:
-
Describe the study design, including randomization methods, blinding procedures, and the sample size (50 dogs per group).
-
Data Collection:
-
Baseline Characteristics:
-
Provide summary statistics for baseline characteristics (e.g., age, breed, preoperative severity) separately for each treatment group.
-
Primary Outcome:
-
Secondary Outcomes:
-
Statistical Analysis Objectives:
-
Statistical Methods: a. Primary Outcome Analysis:
-
Perform an analysis of covariance (ANCOVA) to compare peak vertical force between treatment groups.
-
Include baseline peak vertical force as a covariate to adjust for baseline differences.
-
Check the assumptions of ANCOVA, including normality of residuals and homogeneity of variances. b. Secondary Outcome Analysis:
-
Analyze LOAD scores using an appropriate method (e.g., linear regression or mixed-effects model).
-
Include baseline LOAD scores as covariates to adjust for baseline differences.
-
Assess the normality of residuals for the secondary outcome. c. Assessment of Normality:
-
Conduct normality tests (e.g., Shapiro-Wilk, Kolmogorov-Smirnov) on both primary and secondary outcome variables.
-
Consider transformations (e.g., log transformation) or non-parametric alternatives if data are significantly non-normal. d. Handling Missing Data:
-
Use multiple imputation or appropriate imputation methods to handle missing data, maintaining transparency in reporting the imputation process.
-
Report the proportion of missing data for each variable.
-
Model Assumptions:
-
Assumptions for ANCOVA include normality of residuals and homogeneity of variances. Verify these assumptions using diagnostic plots.
-
For regression or mixed-effects models, assumptions include linearity, independence of residuals, and homoscedasticity. These should be checked using residual plots.
-
Sample Size Calculation:
-
Data Presentation:
-
Describe how the results will be presented, including tables, figures, and the presentation of effect sizes and confidence intervals.
-
Ethical Considerations:
-
Timeline:
-
Reporting:
-
Specify the intended audience for the study results (e.g., scientific journals, conferences).
-
Discuss plans for publication and dissemination of results.
-
Appendices:
-
Attach any additional documents, such as the statistical code used for analysis, data dictionaries, or data collection forms.
It’s essential to adhere to this analysis plan rigorously to maintain the study’s validity. Collaboration with a statistician or data analyst experienced in clinical trials is highly recommended to ensure that the analysis plan aligns with best statistical practices and accounts for the specific characteristics of your data.
Validating the SAP
We validated the chatbot SAP in two ways. First, we compared the results of the chatbot to one developed by a human biostatistician (RE). Validation by humans is a common method of validating chatbot content in medical studies. [6] The two SAPs agreed on the statistical methods, but the chatbot SAP was longer, with extra nonstatistical material (e.g., information about ethics). Second, we compared the chatbot SAP analysis to two similar published inferential analyses from published clinical trials. [5, 7] While the published analyses were too brief for detailed comparison, the analyses looked about the same. If the SAP’s analysis is fundamentally different, the disparity should be investigated. Care is needed because statistical errors are sometimes perpetuated through many published articles. For example, post-hoc power calculations, testing normality, and the analysis of case-control studies are routinely reported with apparently incorrect analyses or interpretations.[8–10] These perpetuated errors may adversely affect the chatbot just as much as they affect researchers not familiar with the finer points of biostatistics.