See the published version of this preprint at Journal of Big Data.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research
Jenna Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background: The study design used to develop prediction models in observational healthcare databases (e.g., case-control and cohort) may impact the clinical usefulness. We aim to quantify how the choice of design impacts prediction model performance.
Aim: To empirically investigate differences between models developed using a case-control design and a cohort design.
Methods: Using a US claims database, we replicated two published prediction models (dementia and type 2 diabetes) which were developed using a case-control design, and also train models for the same prediction questions using cohort designs. We validated each model on data mimicking the point in time the models would be applied in clinical practice. We calculate the models’ discrimination and calibration-in-the-large performances.
Results: The dementia models obtained area under the receiver operating characteristics of 0.560 and 0.897 for the case-control and cohort designs respectively. The type 2 diabetes models obtained area under the receiver operating characteristics of 0.733 and 0.727 for the case-control and cohort designs respectively. The dementia and diabetes case-control models were both poorly calibrated, whereas the dementia cohort model achieved good calibration. We show that careful construction of a case-control design can lead to comparable discriminative performance as a cohort design, but case-control designs generally oversample the outcome leading to miscalibration.
Conclusion: Any case-control design can be converted to a cohort design. We recommend that researchers with observational data use the less subjective and generally better calibrated cohort design. However, if a carefully constructed case-control design is used, then the model must be prospectively validated using a cohort design for fair evaluation and be recalibrated.
Keywords
Prediction, classification, prognostic, case-control, cohort, patient-level prediction
Figure 1
Figure 2
Figure 3
This is a list of supplementary files associated with this preprint. Click to download.
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at Journal of Big Data.
Version 1
Posted 23 Mar, 2021
No community comments so far
Editorial decision: Major Revision
On 03 Jul, 2021
Review #3 received
Received 02 Jul, 2021
Review #5 received
Received 30 Jun, 2021
Review #4 received
Received 30 Jun, 2021
Reviewer #10 agreed
On 23 Jun, 2021
Reviewer #9 agreed
On 23 Jun, 2021
Reviewer #8 agreed
On 20 Jun, 2021
Reviewer #7 agreed
On 19 Jun, 2021
Reviewer #6 agreed
On 19 Jun, 2021
Reviewer #5 agreed
On 17 Jun, 2021
Review #2 received
Received 01 May, 2021
Review #1 received
Received 25 Apr, 2021
Reviewer #4 agreed
On 16 Apr, 2021
Reviewer #3 agreed
On 12 Apr, 2021
Reviews received
Received 12 Apr, 2021
Reviewer #2 agreed
On 12 Apr, 2021
Reviewer #1 agreed
On 08 Apr, 2021
Reviewers invited
Invitations sent on 08 Apr, 2021
Editor assigned
On 18 Mar, 2021
Submission checks complete
On 18 Mar, 2021
Editor invited
On 18 Mar, 2021
First submitted
On 17 Mar, 2021
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Theoretical Computer Science
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
See the published version of this preprint at Journal of Big Data.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research
Jenna Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at Journal of Big Data.
Version 1
Posted 23 Mar, 2021
No community comments so far
Editorial decision: Major Revision
On 03 Jul, 2021
Review #3 received
Received 02 Jul, 2021
Review #5 received
Received 30 Jun, 2021
Review #4 received
Received 30 Jun, 2021
Reviewer #10 agreed
On 23 Jun, 2021
Reviewer #9 agreed
On 23 Jun, 2021
Reviewer #8 agreed
On 20 Jun, 2021
Reviewer #7 agreed
On 19 Jun, 2021
Reviewer #6 agreed
On 19 Jun, 2021
Reviewer #5 agreed
On 17 Jun, 2021
Review #2 received
Received 01 May, 2021
Review #1 received
Received 25 Apr, 2021
Reviewer #4 agreed
On 16 Apr, 2021
Reviewer #3 agreed
On 12 Apr, 2021
Reviews received
Received 12 Apr, 2021
Reviewer #2 agreed
On 12 Apr, 2021
Reviewer #1 agreed
On 08 Apr, 2021
Reviewers invited
Invitations sent on 08 Apr, 2021
Editor assigned
On 18 Mar, 2021
Submission checks complete
On 18 Mar, 2021
Editor invited
On 18 Mar, 2021
First submitted
On 17 Mar, 2021
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Theoretical Computer Science
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published article at Journal of Big Data.
Background: The study design used to develop prediction models in observational healthcare databases (e.g., case-control and cohort) may impact the clinical usefulness. We aim to quantify how the choice of design impacts prediction model performance.
Aim: To empirically investigate differences between models developed using a case-control design and a cohort design.
Methods: Using a US claims database, we replicated two published prediction models (dementia and type 2 diabetes) which were developed using a case-control design, and also train models for the same prediction questions using cohort designs. We validated each model on data mimicking the point in time the models would be applied in clinical practice. We calculate the models’ discrimination and calibration-in-the-large performances.
Results: The dementia models obtained area under the receiver operating characteristics of 0.560 and 0.897 for the case-control and cohort designs respectively. The type 2 diabetes models obtained area under the receiver operating characteristics of 0.733 and 0.727 for the case-control and cohort designs respectively. The dementia and diabetes case-control models were both poorly calibrated, whereas the dementia cohort model achieved good calibration. We show that careful construction of a case-control design can lead to comparable discriminative performance as a cohort design, but case-control designs generally oversample the outcome leading to miscalibration.
Conclusion: Any case-control design can be converted to a cohort design. We recommend that researchers with observational data use the less subjective and generally better calibrated cohort design. However, if a carefully constructed case-control design is used, then the model must be prospectively validated using a cohort design for fair evaluation and be recalibrated.
Figure 1
Figure 2
Figure 3
This is a list of supplementary files associated with this preprint. Click to download.
Loading...