CHEERS is reliable for online use: a randomized crossover paper versus online test-retest reliability study.

doi:10.21203/rs.3.rs-1658901/v1

Background

The creating healthy eating and active environments survey (CHEERS) is an audit tool used to assess the nutrition and physical activity environment in early childhood education and care (ECEC) centres. Availability of the tool has been limited to paper-based versions. Digital health initiatives offer improved reach and immediacy of support for community-based clients through novel technology products. In order to provide increased access to the CHEERS tool, an online version was developed. The objective of this study was to assess the reliability of an online version of CHEERS.

Methods

Utilizing a randomized crossover design, ECEC educators completed either a paper-based or online-based survey and then the opposite mode with a two to three-week interval. The intraclass correlation coefficient (ICC, with 95% confidence interval) was used to determine the reliability between test and retest. Absolute index of reliability in the original measurement was assessed through the standard error of measurement (SEM = SD × √1-ICC). The smallest amount of change not due to inherent variation was assessed by determining minimal detectable change at the 95% confidence level (MDC₉₅ = SEM × 1.96 ×√2; MDC₉₅% = MDC₉₅/mean ×100).

Results

Test-retest reliability was good to excellent for the online-based CHEERS total score (ICC = 0.86) and for each of the four subscales: food served (ICC = 0.82), healthy eating environment (ICC = 0.76), program planning (ICC = 0.76), and physical activity environment (ICC = 0.79). The SEM, MDC_95, and MDC₉₅% for the CHEERS overall score were 0.79, 2.19, and 9.6%, respectively.

Conclusions

The results of this study demonstrate that the online-based and paper-based versions of the CHEERS audit tool share comparable accuracy. The CHEERS tool can be reliably implemented in an online environment and this provides users an alternative means to complete the centre-based health assessment. The advantage of the online-based version includes user accessibility and the potential to develop a feedback response for participants using digitally collected data.

early childhood education

ECEC

eHealth

online

nutrition

healthy eating

physical activity

CHEERS

audit tool

reliability

The early years are a critical period in child development where the origins of lifestyle health behaviours can be identified. Patterns of eating and activity, as early as one to two years of age, have been demonstrated to track through childhood and into adulthood (1–6). Research suggests that Canadian preschool-aged children consume less than one-third of the vegetable and fruit intake recommended combined with low levels of physical activity (7–10). It is estimated that close to 60% of Canadian children 0–5 years participate in some type of childcare arrangement with over half of these attending centre-based care (11). This demonstrates that early childhood education and care (ECEC) environments represent rich opportunities for interventions that promote healthy eating and reduced sedentary behaviours aimed at preventing childhood obesity and the reduction of the long-term non-communicable chronic disease (12–14).

The creating healthy eating and active environments survey (CHEERS) tool is a community-based, educator-administered audit tool designed to offer ECEC centres an evaluative measure for eating and activity environments for their childcare context. The tool that has been assessed for reliability and validity with early childhood experts and educators. Content validity development for the CHEERS tool followed a structured process involving a panel ten nutrition, physical activity, early childhood, and public health professionals with a readability of grade 8.1 using Flesch–Kincaid (15) with evidence of good inter-rater reliability, intra-rater reliability, internal consistency, and concurrent validity (16).

The CHEERS tool was developed as a paper instrument that could be completed by an educator and returned by ground mail for evaluation. Paper-based questionnaires are human resource intensive and introduce the possibility of data omission and errors in data transcription to digital format. Electronic data collection is becoming more extensively used in healthcare research. There are many advantages of online-based questionnaires including remote community access, facilitated data entry time, reduced processing cost, and faster survey response return to stakeholder (17,18). Moreover, a key message from the early childhood community is the need for reciprocation. While data is collected from the ECEC centre, how can the centre benefit from completing the questionnaire? Educator awareness of policies and practices that support or impair healthy environments for children have been identified as a method of informing or reaffirming health promoting professional practice activities (19). An online-based survey can increase response time and support just-in-time learning for educators.

An electronic version of the CHEERS tool was developed to align with increasing use of electronic public health outreach and resources provided through website in addition to the desire of educators to access the tool in an online context. While reliability and validity of the CHEERS tool have been demonstrated, it is not known if administration using an online-based interface is consistent with paper and pencil versions. The aim of this study was to assess the reliability and utility of an electronic version of the CHEERS tool.

This study was conducted in Alberta, Canada throughout provincial health zones. Ethical approval to conduct the study was obtained from the Mount Royal University Human Research Ethics Board (no. 100016). Participants were fully informed about the purpose and procedures of the study and provided written consent for paper first participation or digital consent for electronic first participation. No minors were involved in this study.

Participants

Licensed ECEC centres, identified as Day Care Programs (Schedule 1 of the Childcare Licensing Regulation), are facility-based centres that serve infants, toddlers and pre-school-aged children in Alberta, Canada. They typically provide care throughout the day, from the morning to early evening. To be eligible for the study, ECEC educators had to be employed full time in licensed ECEC centres that provided care for a minimum of 15 preschool aged (3–5 years) children with the classification of day care program. Inclusion criteria included access to an electronic device (smartphone/tablet/laptop) and fluent in English. Exclusion criteria were exclusive employment in family day home or after school care programs or employment in unlicensed ECEC centres.

Creating healthy eating and activity environments (CHEERS) audit tool

The CHEERS audit tool includes 59 items divided into four subscales: food served (23 items), healthy eating environment (18 items), healthy eating program planning (6 items), and physical activity environment (12 items). Response options for each item range from always to never which are then recorded from 1 to 7. Scores are averaged for each subscale and summed for the CHEERS total score (range 4–28). The questions can be accessed at https://cheerskids.ca/survey.

Study Design

This study is a randomized crossover trial design comparing paper and electronic survey formats. A convenience sample of ECEC educators were recruited from ECEC centres throughout Alberta. Flyer advertisements were distributed provincially through email, inviting ECEC educators to contact the project coordinator through email. A $20 gift card was provided to respondents as an incentive. Interested individuals were contacted via email or telephone and given detailed instructions about the study to ensure sample validity and data integrity (20). Educators were randomly assigned by random generated number to complete the CHEERS survey either in paper version or electronic survey platform. The two study groups were reversed according to the randomized crossover design and the alternate mode was completed within two to three weeks.

Study methods

Participants assigned to paper first administration received the CHEERS survey in a sealed envelope with instructions on how to complete the survey as well as a prepaid pre-addressed envelope to use to return the paper survey to the research team. An email reminder was sent to the participant ten days after mailing to ensure participants received the survey and to encourage return of the survey. In three cases, the paper surveys had not arrived and an additional package had to be resent. Upon return, surveys were coded and data entered into a Microsoft Excel 2016 spreadsheet. In the case of missing data, the input was left blank. If a page of survey responses were missed the survey was considered spoiled and the participant considered lost to follow up. Once surveys were received and verified, participants received a personalized email link to the electronic CHEERS survey, which could be completed at participants’ convenience. A reminder email was sent one week after the first email to those individuals who had not yet completed the survey with the link to the online-based questionnaire. Participants assigned to the electronic first administration received a personalized the link to the electronic version of CHEERS which contained instructions on how to complete the survey and was followed by a reminder email one week later. Once the electronic system recorded the completion of the survey, the paper survey package was mailed to participants. An email reminder was sent to participants ten days later to check for survey receipt and encourage return. In two cases, paper surveys had to be resent and in three cases participants indicated surveys were mailed back however these were never received by the research team.

Online Interface

The electronic questionnaire system for the CHEERS administration was built in Qualtrics® online-based survey platform. This survey system is compatible across computer and mobile smartphone platforms (e.g. Apple Safari, Google Chrome, Internet Explorer, Mozilla Firefox). The survey is optimized to the mobile experience to adjust for screen size and formatting. The online version of the CHEERS questionnaire was developed to replicate the paper-based questionnaire experience as closely as possible. The online questionnaire presents three questions per page with a progress tracking bar at the top of the survey. The survey platform also records the duration (time to complete the survey). A sample screenshot of the online-based interface for the survey tool is presented in Fig. 1. Participants were provided unique email link once identity and eligibility verified.

Sample Size

Sample size needed for the reliability analysis was calculated using the sample size calculator (21) using the parameters of required number of participants when β = 0.20, α = 0.05, the number of repetitions per participant as two, with a potential dropout rate of 30%. We chose an ICC = 0.70 as minimally acceptable and an ICC = 0.90 as desired based on the recommended minimum reliability of 0.70 when the scale is used in research (22). This calculation returned a required sample size of 23 participants per group (paper 1st or online 1st ) or 46 participants estimated to be sufficient for testing reliability (21–23).

Statistical Analysis

The results of the paper questionnaire were manually inputted into Microsoft Excel 2016, while the data for the electronic collection was captured by Qualtrics software program and derived directly from the system database. Statistical analysis was performed using SPSS statistical package version 26 (SPSS Inc, Chicago, IL), with alpha level of p < 0.05. Descriptive statistics were used to report means and variation between trials for the overall CHEERS score and subscales. Group comparisons were performed using independent t-test and Pearson chi-square tests for numerical and categorical demographic variables, respectively, to assess heterogeneity between ECECs.

To assess the magnitude of the association between electronic and paper data, we calculated an intraclass correlation coefficient (ICC) for each administration and administration time points along with their 95% confidence interval. The ICC takes into account the selection of raters as well as the correlation and agreement between raters (24–26). The intraclass correlation coefficient estimates and their 95% confident intervals for test-retest reliability were based on a single rater/measurement, absolute-agreement, 2-way mixed-effects model (24,27). In this study, interpretations of ICC results followed Koo and Li’s categorizations: poor (less than 0.5), moderate (0.5–0.75), good (0.75–0.9), and excellent (greater than 0.90) [21]. To further assess response stability, the standard error of measurement (SEM) was calculated using the following formula: SEM = SD × √(1 – ICC) (28,29). The SEM provides an index of the “trial-to-trial noise in the data” which provides an absolute index of reliability in the original measurement unit without the influence of variance among participants (28). Smaller SEM values demonstrate lower response variation from test to retest. Absolute reliability provides additional information about a change in score and what change would be required to provide a measure of true change beyond expected measurement error and individual variabilities. This can be evaluated using the minimal detectable change (MDC₉₅) which provides a value of reliability in the original units of the measure reflecting the amount of change required to demonstrate a difference not due to chance at the 95% confidence level (30). MDC₉₅ was calculated using the following formula: MDC₉₅ = SEM × 1.96 × √2 (31). This change can be expressed as relative change (MDC95%) and was calculated in this study using the following formula: MDC₉₅% = MDC₉₅/mean × 100 (citation). An MDC₉₅% less than 30 has been interpreted in the literature as acceptable random measurement error and under 10 as excellent (32,33). Utility was evaluated by documenting the length of time taken to complete the CHEERS tool as recorded in the survey platform housed in Qualtrics (34).

Participants

A total of 80 educators aged between 26 and 55 + years were recruited in the study (Fig. 2). Eight educators did not meet inclusion criteria or did not respond after initial interest and were excluded from randomization and the final 72 randomly assigned to a survey platform. Eleven participants did not continue after paper first survey (group A) and twelve after the online-based survey (group B). Two surveys in group B were excluded due to missing multiple survey questions on double sided page. Missing cases were excluded when the questionnaire scores were compared.

Participant demographics

Demographic characteristics of the child care centres are presented in Table 1. ECEC centres were representative of both not-for-profit and profit-based programs. Half of participating ECEC centres were from large urban centres with the remaining a mix of medium and small centres. No statistically significant differences among demographic characteristics of child care type (χ²(1) = 1.647, p = 0.199), city size (χ²(2) = 3.163, p = 0.206), or number of preschoolers enrolled (t(47) = 0.452, p = 0.145) between the two groups were noted.

Table 1

ECEC centre demographic information
	Group A Paper 1st (n = 25)		Group B Online 1st (n = 24)
	n	%	n	%
Childcare Type
Not-for-profit	15	60	10	40
For-profit	10	40	14	60
City size
Large urban population centre	12	48	10	40
Medium population centre	5	20	10	40
Small population centre	8	32	5	40
Preschoolers enrolled (M, SD)	29.3	23.5	32.0	17.3

Demographic characteristics of the child care educators are presented in Table 2. All respondents were female with representation throughout educator training levels. Educator demographics were similar between the groups in terms of age (χ²(4) = 3.667, p = 0.453), level of education (χ²(2) = .091, p = 0.956), or years of experience (t(47) = 0.278, p = 0.951).

Table 2

ECEC educator demographic information
	Group A Paper 1^st (n = 25)		Group B Online 1^st (n = 24)
	n	%	n	%
Gender
Female	25	100	24	100
Male	-	-	-	-
Educator Age
26-35 yrs	2	8	7	28
36-45 yrs	8	32	7	28
46-55 yrs	8	32	4	16
55+ yrs	4	16	3	12
Prefer not to answer	3	12	4	16
Education Background
Certificate	6	24	7	28
Diploma	14	56	14	56
University degree	6	20	4	16
Years of ECEC Experience (M, SD)	10.4	8.2	9.7	8.2

Reliability

Overall test-retest reliability of form (paper versus online) and sequence (paper first versus online first) was assessed using the ICC two-way mixed-effects model and absolute-agreement. The ICC values for overall CHEERS score and subscales were good to excellent (0.77–0.92) regardless of form (paper vs online) or administration sequence (Table 2).

Table 2

Reliability for the CHEERS tool between form (online vs paper) and sequence (online first or paper first).
Group A - Paper 1st
	Paper n = 25 Mean (SD)	Online n = 25 Mean (SD)	ICC	95% CI
CHEERS overall	22.2 (2.8)	22.9 (3.1)	0.91	0.79–0.96
FS	6.0 (0.5)	6.0 (0.7)	0.91	0.80–0.96
HEE	6.2 (0.6)	6.1 (0.8)	0.81	0.57–0.92
HPP	4.5 (1.1)	5.0 (1.2)	0.77	0.44–0.90
PA	5.5 (0.9)	5.8 (0.9)	0.92	0.74–0.97
Group B - Online 1st
	Online n = 24 Mean (SD)	Paper n = 24 Mean (SD)	ICC	95% CI
CHEERS overall	22.8 (2.7)	23.2 (2.6)	0.93	0.84–0.97
FS	5.9 (0.5)	6.2 (0.5)	0.83	0.25–0.94
HEE	6.2 (0.6)	6.2 (0.5)	0.93	0.83–0.97
HPP	4.9 (1.3)	5.0 (1.3)	0.93	0.84–0.97
PA	5.9 (0.8)	5.8 (0.8)	0.89	0.75–0.95
FS – food served average score; HEE – healthy eating environment average score; HPP healthy program planning; PA – physical activity.

Evaluation between time 1 and time 2 shows good to excellent reliability (Table 3). The ICC value between administration for the overall CHEERS score was 0.92 (95% CI: 0.84–0.96). ICC values for the subdomains ranged 0.85–0.90.

Table 3

Reliability for the CHEERS tool between Allocation Time 1 and Time 2
	Time 1 n = 49 Mean (SD)	Time 2 n = 49 Mean (SD)	ICC	95% CI
CHEERS overall	22.5 (2.7)	23.1 (2.8)	0.92	0.84–0.96
FS	5.9 (0.5)	6.1 (0.6)	0.87	0.72–0.93
HEE	6.2 (0.6)	6.2 (0.7)	0.85	0.74–0.92
HPP	4.7 (1.2)	5.0 (1.2)	0.86	0.72–0.92
PA	5.7 (0.9)	5.8 (0.8)	0.90	0.83–0.95
FS – food served average score; HEE – healthy eating environment average score; HPP healthy program planning; PA – physical activity.

Agreement

The SEM, MDC₉₅ and MDC₉₅% of the CHEERS scores and subdomain tests are shown in Table 4. The SEM between administration of surveys (time 1 versus time 2) were relatively small for the CHEERS overall score (0.79) and subdomains (0.21–0.45) demonstrating an acceptable level of agreement between time points.

The MDC₉₅ of the overall CHEERS score, food served, healthy eating environment, program planning, and physical activity were 2.19, 0.57, 0.69, 1.26, and 0.72. The MDC₉₅% in the overall CHEERS score, food served, healthy eating environment, program planning, and physical activity were 9.6%, 9.5%, 11.1%, 25.8%, and 12.6%, respectively.

Table 4

Reliability for the CHEERS tool between Allocation Time 1 and Time
	SEM	MDC₉₅	MDC₉₅% (%)
CHEERS overall	0.79	2.19	9.6
FS	0.21	0.57	9.5
HEE	0.25	0.69	11.1
HPP	0.45	1.26	25.8
PA	0.26	0.72	12.6
FS – food served average score; HEE – healthy eating environment average score; HPP healthy program planning; PA – physical activity; SEM – standard error of measurement; MDC₉₅ – minimal detectable change at 95% confidence interval; MDC₉₅% - percent minimal detectable change at 95% confidence interval.

Utility measure

The mean time taken to complete the CHEERS tool for all educators, regardless of sequence, in the online environment was 21.5 minutes (SD = 11.4 min), 95% CI [17.9–24.4] where completion times ranged from 9.6 to 48.8 minutes. The mean time taken to complete the survey in an online-first sequence (Group B) was 23.5 min (SD = 12.6 min), 95% CI [18.1–28.8] where completion times ranged from 9.9 to 48.8 minutes. The mean time taken to complete the survey in an online-second sequence (Group A) was 18.9 min (SD = 9.9 min), 95% CI [14.8–23.0] and times ranged from 9.6 to 48.1 min.

The results of this study demonstrate that the online version of the CHEERS tool presents good reliability, agreement, and utility to its paper version. Previous research demonstrates that the paper-based, educator-administered CHEERS survey is a reliable and valid audit tool for evaluating child care centre eating and activity environments (15,16). With billions of internet users worldwide, electronic health (eHealth) and mobile health (mHealth) innovations offer new avenues for health promotion (35). Online methods of health assessment and interventions have the advantages of convenience, ease of operation, adaptability, and accessibility. However, evidence of online reliability, validity, and effectiveness are an important component of building comprehensive evidence for the eHealth initiative (36).

The test-retest reliability of the online-based CHEERS overall score was strongly aligned with the paper-based administration (ICC = 0.91). These results align well with previously reported paper-based reliability data for the overall score (ICC = 0.81) (16). Psychometric properties of “reliability and agreement are estimates that vary based on interactions between a tool, its user, and the context of the assessment” (37). Considering that the current study was done with a new cohort of early childhood educators who had not previously participated and similar intra-rater results were found contributes to the overall psychometric evidence demonstrating reliability of the CHEERS audit tool.

The outcomes of this study are in accordance with outcomes in similar paper to online validation studies of child related health tools. For example, the NutriSTEP® questionnaire screens nutrition risk in preschoolers through a survey completed by primary caregiver/parents. An ICC of 0.91 for CHEERS overall score is in line with the ICC of 0.91 found in the NutriSTEP® adaptation from paper-based to online-based administration (38). There was similar alignment of ICC scores for the MiniPAQLQ, a quality of life questionnaire completed by caregiver’s of children with asthma, ICC of 0.89 (39). This is further supported by the results of a meta-analysis suggesting that computer and paper-based questionnaire administrations result in equivalent scores when comparing electronic and paper-based administrations (40).

The standard error of measurement (SEM) is a measure of stability, reflecting the precision of individual scores expressed in the same units as the original scores (22,28). While ICC reflects reliability among people, SEM quantifies precision within individuals and permits the calculation of the minimal detectable difference that can reliably meaningful difference to measure change that can be used in future research to determine if an intervention has a significant impact on creating change. The current study has provided evidence of stability as demonstrated by the SEM, MDC₉₅, and MDC₉₅% from paper-based to online-based administration.

Tool reliability and validity are important aspects of health measurement scales. However, a measure must also have utility and ease of use for respondents thus an important component of psychometric tool assessment is the usability, functionality, and availability of the tool (41,42). The literature clearly indicates a preference for digital over paper questionnaire completion (38,39,43). eHealth technologies can provide improved efficiency and accessibility for health promotion services for stakeholders. This study adds to the reliability and utility of the CHEERS tool through the addition of a user-friendly electronic version.

Potential limitations exist with this research. First, there is a possibility that participants remembered their responses from the first administration. The recommended interval between administrations appropriate for reliability studies is 2–3 weeks. Sufficiently short to ensure minimal opportunity for real behaviour change while sufficiently long enough to minimize recall and bias. However, some individuals may have remembered their responses and which may have allowed them to report similar answers in the second administration. A second limitation is the dropout numbers. Thirty-six participants were assigned to arms of the crossover study, however, approximately ten were lost to follow-up resulting in a 27% dropout rate which is higher than the 20% expected rate. This may have been due to characteristics of the survey (time demand) but may also be explained by high job turnover in the early childhood educator workforce (44,45) or losses resulting from lack of postal delivery (lost mail). However, it is reasonable to assume the missing data would not have meaningfully impact the study outcome, as reliability measures were similar within the literature.

This study demonstrated the online-based CHEERS tool produced equivalent results to the paper-based version. Health interventions are increasingly being made available via multiple platforms (computer or mobile versions) and demonstrating that these digital options reliable improves the reach and immediacy of support for community-based clients through novel technology products. eHealth administration of the CHEERS tool can be used in public health initiatives in community settings and the outcomes can be interpreted in accordance with findings of the paper version.

CHEERS: creating healthy eating and activity environments survey; CI: confidence interval; eHealth: electronic health; ECEC: early learning and childcare; FS: food served; HEE: healthy eating environment; HPP: healthy eating program planning; ICC: intraclass correlation coefficient; MDC₉₅: minimal detectable change at 95% confidence interval; MDC₉₅%: percent minimal detectable change at 95% confidence interval; PA: physical activity environment; SEM: standard error of measurement;

Ethics approval and consent to participate

This study was approved by the Mount Royal University Human Research Ethics Board (100016). ECEC educators and directors were the participants in this study. ECEC educators and directors were fully informed about the purpose and procedures of the study and provided written or online consent for participation. No minors were involved in this study.

Consent for publication

Not applicable

Availability of data and material

The raw datasets are not publicly available as permission for sharing this dataset or making it publicly available was not requested as part of the application to the Research Ethics Board and hence, participant consent for inclusion of their data in such a dataset was not requested.

Competing interests

The author has no competing interests to declare.

Funding

The CHEERS study was funded by the Mount Royal University Research Grant Fund and the Faculty of Health, Community, and Education Innovation Fund. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Authors' contributions

LL conceptualized the project idea; research design; primary data collection; executed reliability and validation analyses; drafted the manuscript, edited, read, and approved the final manuscript.

Acknowledgements

Appreciation to Alberta Health Services, Population and Public Nutrition for their support of this project. Thanks to all the directors and educators in childcare centres throughout Alberta and Mount Royal University student research assistants.

Tammelin, R, Yang X, Leskinen E, Kankaanpaa A, Hirvensalo M, Tammelin T, et al. Tracking of physical activity from early childhood through youth into adulthood. Medicine and Science in Sports and Exercise. 2014;46(5):955–62.
Jones RA, Hinkley T, Okely AD, Salmon J. Tracking Physical Activity and Sedentary Behavior in Childhood: A Systematic Review. American Journal of Preventive Medicine. 2013 Jun 1;44(6):651–8.
Lioret S, Campbell KJ, McNaughton SA, Cameron AJ, Salmon J, Abbott G, et al. Lifestyle patterns begin in early childhood, persist and are socioeconomically patterned, confirming the importance of early life interventions. Nutrients. 2020 Mar 9;12(3):E724.
Luque V, Escribano J, Closa-Monasterolo R, Zaragoza-Jordana M, Ferré N, Grote V, et al. Unhealthy Dietary Patterns Established in Infancy Track to Mid-Childhood: The EU Childhood Obesity Project. The Journal of Nutrition. 2018 May 1;148(5):752–9.
EDEN Mother–Child Cohort Study Group. Dietary Patterns Track from Infancy to Preschool Age: Cross-Sectional and Longitudinal Perspectives. The Journal of Nutrition. 2015 Apr 1;145(4):775–82.
Feinstein L, Sabates R, Sorhaindo A, Rogers I, Herrick D, Northstone K, et al. Dietary patterns related to attainment in school: the importance of early eating patterns. Journal of Epidemiology & Community Health. 2008 Jan;62(8):734–9.
Pabayo R, Spence JC, Casey L, Storey K. Food consumption patterns: In preschool children. Canadian Journal of Dietetic Practice and Research. 2012;
Kuzik N, Clark D, Ogden N, Harber V, Carson V. Physical activity and sedentary behaviour of toddlers and preschoolers in child care centres in Alberta, Canada. Can J Public Health [Internet]. 2015;106(4). Available from: http://journal.cpha.ca/index.php/cjph/article/view/4794
Temple VA, Naylor PJ, Rhodes RE, Higgins JW. Physical activity of children in family child care. Applied Physiology, Nutrition, and Metabolism. 2009;34(4):794–8.
Chaput JP, Colley RC, Aubert S, Carson V, Janssen I, Roberts KC, et al. Proportion of preschool-aged children meeting the Canadian 24-Hour Movement Guidelines and associations with adiposity: results from the Canadian Health Measures Survey. BMC Public Health. 2017 Nov 20;17(5):829.
Findlay LC. Early learning and child care for children aged 0 to 5 years: A provincial/territorial portrait [Internet]. Ottawa: Statistics Canada; 2019 Jul [cited 2021 Oct 8]. Report No.: Report No. 099. Cat No. 11-626-X. Available from: https://www150.statcan.gc.ca/n1/pub/11-626-x/11-626-x2019013-eng.htm
Mazarello Paes V, Ong KK, Lakshman R. Factors influencing obesogenic dietary intake in young children (0–6 years): systematic review of qualitative evidence. BMJ Open. 2015 Sep 1;5(9):e007396.
Larson N, Ward DS, Neelon SB, Story M. What Role Can Child-Care Settings Play in Obesity Prevention? A Review of the Evidence and Call for Research Efforts. Journal of the American Dietetic Association. 2011 Sep;111(9):1343–62.
van de Kolk I, Verjans-Janssen SRB, Gubbels JS, Kremers SPJ, Gerards SMPL. Systematic review of interventions in the childcare setting with direct parental involvement: effectiveness on child weight status and energy balance-related behaviours. International Journal of Behavioral Nutrition and Physical Activity. 2019 Nov 21;16(1):110.
Lafave L, Tyminski S, Riege T, Hoy D, Dexter B. Content Validity for a Child Care Self-assessment Tool: Creating Healthy Eating Environments Scale (CHEERS). Canadian Journal of Dietetic Practice and Research. 2016;77:89–92.
Lafave LMZ. Creating a healthy eating and active environment survey (CHEERS) for childcare: an inter-rater, intra-rater reliability and validity study. BMC Public Health. 2019 Oct 1;19(1):1–9.
Evans JR, Link to external site this link will open in a new window, Anil M. The value of online surveys: a look back and a look ahead. Internet Research. 2018;28(4):854–87.
Zeleke AA, Naziyok T, Fritz F, Christianson L, Röhrig R. Data Quality and Cost-effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Systematic Review. Journal of Medical Internet Research. 2021 Jan 22;23(1):e21382.
Benjamin Neelon SE, Briley ME. Position of the American Dietetic Association: Benchmarks for Nutrition in Child Care. Journal of the American Dietetic Association. 2011 Apr;111(4):607–15.
Pozzar R, Hammer MJ, Underhill-Blazey M, Wright AA, Tulsky JA, Hong F, et al. Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire. Journal of Medical Internet Research. 2020 Oct 7;22(10):e23021.
Arifin WN. A Web-based Sample Size Calculator for Reliability Studies. Education in medicine journal. 2018 Sep 28;10(3).
Streiner DL, Norman GR, Cairney J. Health Measurement Scales: a practical guide to their development and use. USA: Oxford University Press; 2015.
Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Statistics in Medicine. 1998 Jan;17(1):101–10.
Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine. 2016;15(2):155–63.
Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychological bulletin. 1979;86(2):420–8.
Trevethan R. Intraclass correlation coefficients: clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology. 2017;17(2):127–43.
Qin S, Nelson L, McLeod L, Eremenco S, Coons SJ. Assessing test–retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula. Qual Life Res. 2019 Apr;28(4):1029–33.
Weir JP. JP: Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Con Res.:231–40.
de Vet HCW, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. Journal of Clinical Epidemiology. 2006 Oct 1;59(10):1033–9.
de Vet HC, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM. Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health Qual Life Outcomes. 2006 Aug 22;4:54.
Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek ALM. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res. 2001 Aug 1;10(7):571–8.
Huang SL, Hsieh CL, Wu RM, Tai CH, Lin CH, Lu WS. Minimal Detectable Change of the Timed “Up & Go” Test and the Dynamic Gait Index in People With Parkinson Disease. Physical Therapy. 2011 Jan;91(1):114–21.
Kather F, Hadzic M, Hehle T, Eichler S, Klein J, Völler H, et al. Test-retest reliability of the Mini Nutritional Assessment–Short Form (MNA-SF) in older patients undergoing cardiac rehabilitation. J Geriatr Cardiol. 2020 Sep 28;17(9):574–9.
Anderson J, Manias E, Kusljic S, Finch S. Testing the validity, reliability and utility of the Self-Administration of Medication (SAM) tool in patients undergoing rehabilitation. Research in Social and Administrative Pharmacy. 2014 Jan 1;10(1):204–16.
Grady A, Yoong S, Sutherland R, Lee H, Nathan N, Wolfenden L. Improving the public health impact of eHealth and mHealth interventions. Australian and New Zealand Journal of Public Health. 2018 Apr;42(2):118–9.
Enam A, Torres-Bonilla J, Eriksson H. Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. J Med Internet Res. 2018 Nov 23;20(11):e10971.
Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hroóbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies. 2011;48(6):661–71.
Carducci B, Reesor M, Haresign H, Rysdale L, Keller H, Beyers J, et al. NutriSTEP® is Reliable for Internet and Onscreen Use. Canadian Journal of Dietetic Practice & Research. 2015;76(1):9–14.
Minard JP, Thomas NJ, Olajos-Clow JG, Wasilewski NV, Jenkins B, Taite AK, et al. Assessing the burden of childhood asthma: validation of electronic versions of the Mini Pediatric and Pediatric Asthma Caregiver’s Quality of Life Questionnaires. Qual Life Res. 2016 Jan;25(1):63–9.
Gwaltney CJ, Shields AL, Shiffman S. Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review. Value in Health. 2008 Mar 1;11(2):322–33.
Wakefield BJ, Turvey CL, Nazi KM, Holman JE, Hogan TP, Shimada SL, et al. Psychometric Properties of Patient-Facing eHealth Evaluation Measures: Systematic Review and Analysis. J Med Internet Res. 2017 Oct 11;19(10):e346.
Keszei AP, Novak M, Streiner DL. Introduction to health measurement scales. Journal of Psychosomatic Research. 2010 Apr 1;68(4):319–23.
van Kollenburg RA, de Bruin DM, Wijkstra H. Validation of the Electronic Version of the International Index of Erectile Function (IIEF-5 and IIEF-15): A Crossover Study. Journal of Medical Internet Research. 2019 Jul 2;21(7):e13490.
Cumming T, Logan H, Wong S. A critique of the discursive landscape: Challenging the invisibility of early childhood educators’ well-being. Contemporary Issues in Early Childhood. 2020 Jun 1;21(2):96–110.
Nislin MA, Sajaniemi NK, Sims M, Suhonen E, Maldonado Montero EF, Hirvonen A, et al. Pedagogical work, stress regulation and work-related well-being among early childhood professionals in integrated special day-care groups. European Journal of Special Needs Education. 2016 Jan 2;31(1):27–43.

No competing interests reported.

CHEERS is reliable for online use: a randomized crossover paper versus online test-retest reliability study.

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods

Participants

Creating healthy eating and activity environments (CHEERS) audit tool

Study Design

Study methods

Online Interface

Sample Size

Statistical Analysis

Results

Participants

Participant demographics

Reliability

Agreement

Utility measure

Discussion

Conclusions

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and material

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Additional Declarations

Status:

Journal Publication

Version 1