The results of this study demonstrate that the online version of the CHEERS tool presents good reliability, agreement, and utility to its paper version. Previous research demonstrates that the paper-based, educator-administered CHEERS survey is a reliable and valid audit tool for evaluating child care centre eating and activity environments (15,16). With billions of internet users worldwide, electronic health (eHealth) and mobile health (mHealth) innovations offer new avenues for health promotion (35). Online methods of health assessment and interventions have the advantages of convenience, ease of operation, adaptability, and accessibility. However, evidence of online reliability, validity, and effectiveness are an important component of building comprehensive evidence for the eHealth initiative (36).
The test-retest reliability of the online-based CHEERS overall score was strongly aligned with the paper-based administration (ICC = 0.91). These results align well with previously reported paper-based reliability data for the overall score (ICC = 0.81) (16). Psychometric properties of “reliability and agreement are estimates that vary based on interactions between a tool, its user, and the context of the assessment” (37). Considering that the current study was done with a new cohort of early childhood educators who had not previously participated and similar intra-rater results were found contributes to the overall psychometric evidence demonstrating reliability of the CHEERS audit tool.
The outcomes of this study are in accordance with outcomes in similar paper to online validation studies of child related health tools. For example, the NutriSTEP® questionnaire screens nutrition risk in preschoolers through a survey completed by primary caregiver/parents. An ICC of 0.91 for CHEERS overall score is in line with the ICC of 0.91 found in the NutriSTEP® adaptation from paper-based to online-based administration (38). There was similar alignment of ICC scores for the MiniPAQLQ, a quality of life questionnaire completed by caregiver’s of children with asthma, ICC of 0.89 (39). This is further supported by the results of a meta-analysis suggesting that computer and paper-based questionnaire administrations result in equivalent scores when comparing electronic and paper-based administrations (40).
The standard error of measurement (SEM) is a measure of stability, reflecting the precision of individual scores expressed in the same units as the original scores (22,28). While ICC reflects reliability among people, SEM quantifies precision within individuals and permits the calculation of the minimal detectable difference that can reliably meaningful difference to measure change that can be used in future research to determine if an intervention has a significant impact on creating change. The current study has provided evidence of stability as demonstrated by the SEM, MDC95, and MDC95% from paper-based to online-based administration.
Tool reliability and validity are important aspects of health measurement scales. However, a measure must also have utility and ease of use for respondents thus an important component of psychometric tool assessment is the usability, functionality, and availability of the tool (41,42). The literature clearly indicates a preference for digital over paper questionnaire completion (38,39,43). eHealth technologies can provide improved efficiency and accessibility for health promotion services for stakeholders. This study adds to the reliability and utility of the CHEERS tool through the addition of a user-friendly electronic version.
Potential limitations exist with this research. First, there is a possibility that participants remembered their responses from the first administration. The recommended interval between administrations appropriate for reliability studies is 2–3 weeks. Sufficiently short to ensure minimal opportunity for real behaviour change while sufficiently long enough to minimize recall and bias. However, some individuals may have remembered their responses and which may have allowed them to report similar answers in the second administration. A second limitation is the dropout numbers. Thirty-six participants were assigned to arms of the crossover study, however, approximately ten were lost to follow-up resulting in a 27% dropout rate which is higher than the 20% expected rate. This may have been due to characteristics of the survey (time demand) but may also be explained by high job turnover in the early childhood educator workforce (44,45) or losses resulting from lack of postal delivery (lost mail). However, it is reasonable to assume the missing data would not have meaningfully impact the study outcome, as reliability measures were similar within the literature.