Trial design, setting and sample
The study was a single-blinded parallel group randomised controlled trial undertaken with childcare services within New South Wales (NSW), Australia. Details of the trial protocol have been published elsewhere (32). In summary, the sample of 54 services was drawn from a pool of 252 long day care services in NSW that both provided food to children in care and were current clients of a single specific childcare management software (CCMS) provider. The intervention involved access to a web-based menu planning and decision support tool.
Eligible services had to meet the following requirements: i) be open for ≥8 hours each weekday; ii) prepare and provide at least one main meal and two snacks to children onsite each weekday; iii) have autonomy to make menu planning decisions within the service and; iv) have a staff member responsible for menu planning (menu planners) with sufficient English to engage with the intervention. Excluded services were those that outsourced menu planning, did not cater for children aged 3-6 years, catered exclusively for special needs children, or were run by the NSW Department of Education and Communities. Twenty-seven services were randomised to intervention and 27 services to the control group.
The trial was prospectively registered with the Australian New Zealand Clinical Trials Registry (ACTRN12616000974404) and approved by the Hunter New England (approval no: 16/02/17/4.05) and the University of Newcastle (approval H-2016-0111) Human Research Ethics Committees.
Economic study
A prospective, trial-based economic evaluation of the intervention versus usual practice was conducted from both health care sector and modified societal perspectives over a one-year time horizon, consistent with the length of the trial (12 months). All costs were calculated and reported in $AUD, 2017/18. The modified societal perspective was constrained to those who would be impacted financially by the intervention, health care providers and childcare services.
Usual practice (control)
Relevant international, national and state guidelines recommend that childcare services implement evidence-based practices to improve the provision of healthy food. (33-35) Childcare services in Australia that provide food to children are required by national accreditation standards to serve foods consistent with the Australian Dietary Guidelines (ADG). (36) In NSW, the Caring for Children (35) resource outlines best practice dietary guidelines for the childcare sector, which are consistent with the ADG. Assessment and compliance officers in NSW regulate service accreditation and use the Caring for Children guidelines to determine if services meet accreditation standards in relation to dietary guidelines for the sector. Services in NSW are also required by law to list all food served to children whilst in care on menus and to make these menus publicly available. In this context, usual practice for childcare services across NSW comprises support from a health promotion officer employed by the local health district to implement the NSW state-wide obesity prevention program for early childhood, Munch and Move (37). Support is typically provided upon request from the childcare service, via telephone or face-to-face contact, with referral to supporting resources. For menu planning, support may also be in the form of a menu review and provision of feedback on compliance with dietary guidelines. Childcare services that were randomly allocated to the control group did not receive access to the directed implementation support strategies described below.
Menu planning support (intervention)
The 12-month intervention targeted menu planners and nominated supervisors within each childcare service. The intervention comprised access to a web-based menu planning tool, titled feedAustralia, online resources, online reminders and feedback, as well as training and support for menu planners and nominated supervisors to use the program. The intervention was informed by previous research where more traditional modes of implementation support, such as printed resources, were trialled with limited success and uncertain cost-effectiveness (15) (38). In this trial greater effect, sustainability and cost-effectiveness were anticipated by using a technology platform and online resources. Details of the components in this intervention are detailed in the original effectiveness study (39). In brief, the intervention comprised: A web-based menu planning program with decision support including automated menu planning, audit and feedback; online resources and reminders; communication strategies and managerial support; training and support to use the program and a portable computer tablet.
Identification, measurement and valuation of trial outcomes
The aim of the implementation intervention was to increase menu compliance with recommended dietary guidelines for childcare services in the state (Caring for Children resource (35)). These guidelines require services to provide daily serves for each of the following food groups: (1) vegetables and legumes/beans (two serves); (2) fruit (one serve); (3) whole grain cereals, foods and breads (two serves); (4) lean meat and poultry, fish, eggs, tofu, seeds and legumes (3/4 serve); (5) milk, yoghurt, cheese and alternatives (one serve); and (6) no ‘discretionary’ foods that are high in energy and low in nutrients (zero serves). Compliance was defined as the provision of the recommended number of serves for that food group per child per day over a 1-week period.
The primary trial outcome was defined as the mean number of food groups compliant with dietary guidelines: Compliance of the menu with nutrition guidelines was assessed using best practice assessment methods, based on calculation of the serves of each food group provided per child, per day. The assessment was conducted by a blinded dietitian who randomly selected one week of each services’ current menu cycle for detailed menu review. At baseline, one week for the current menu cycle was randomly selected during the recruitment phone call for those services that consented. For follow-up, one week for the current menu cycle was again randomly selected (approximately 12 months later). The dietitian obtained all recipes, quantities of food served and number of children attending each day to enable detailed calculation of serves of food groups. The mean number of compliant food groups per service (a score out of six) was compared between intervention and control groups at 12 months follow-up.
Secondary outcomes were:
- Compliance with guidelines for all food groups: The proportion of services compliant for all of the six food groups was compared between the intervention and control group as assessed via one-week menu review at baseline and 12 months follow-up.
- Individual food group compliance with dietary guidelines: The proportion of services compliant with dietary guidelines for each of the six individual food groups was compared between the intervention and control group as assessed via one-week menu review at baseline and 12 months follow-up.
- Mean servings of individual food groups: The mean number of serves for each of the six food groups provided on the menu was compared between the intervention and control groups as assessed via one-week menu review at baseline and 12 months follow-up.
Identification, measurement and valuation of resource use
Micro-costing was used to calculate the incremental costs of the intervention compared to usual practice. Specific cost components, assumptions and sources of unit costs are provided in Table 1. Resource use associated with the execution of the intervention was prospectively collected using a customised cost data capture template, designed by health economists from the Hunter Medical Research Institute and compiled by the health promotion officers delivering the intervention. Categories of cost included; labour, materials, joint costs where costs are shared across multiple programs, and miscellaneous. Resource use was tagged according to stakeholder expense (public health, childcare service, other).
<<Table 1 Micro costing assumptions and sources of unit costs >>
Resource use data associated with the labour time spent undertaking menu planning and reviewing was collected for both intervention and usual practice services at baseline and follow-up. These data were collected using survey instruments employed in the trial and completed by nominated supervisors and menu planners within each of the childcare services via telephone. Baseline and follow up surveys included questions on how much money the services usually spent on buying food and drinks for children per month.
Labour costs, including opportunity costs incurred during the intervention uptake by childcare service staff (cooks, educators, supervisors and directors), were captured in the form of staff time (hours) and valued using the midpoint from relevant Australian wage rate ranges (40). All other resource use categories were valued using market rates. All costs were reported in 2017/18 Australian dollars ($AUD).
Economic analysis
The costing analysis was undertaken from both health sector and modified societal perspectives. All analyses were carried out using Microsoft Excel software Office 365. The direct health sector cost to support uptake of the web-based menu planning intervention was calculated. Costs to the childcare services in each study arm to undertake menu planning and reviewing were also calculated. The incremental cost of the intervention was calculated as the net difference in the cost to undertake menu planning and review (essential elements to planning healthier menus that meet guidelines) plus the direct cost of the intervention.
Owing to the complex set of outcomes included in the effectiveness study, the evaluation included both CCA and CEA. The cost-consequence analysis reports the incremental cost of executing the intervention alongside the primary and secondary outcomes by way of a score-card. Cost-effectiveness analysis was also conducted to assess the productive efficiency of the intervention compared to usual practice. However, the absence of an explicit willingness to pay threshold for trial-specific cost-effectiveness results can make interpretation of the incremental cost-effectiveness ratio (ICER) difficult. A recent method of generating a threshold to aid decision making was published by Hyewon and Levine (41). In this method, the cost and outcomes associated with usual practice, combined in an average cost-effectiveness ratio (ACER), are assumed to represent an implicit willingness-to-pay threshold, having already been implemented by the health care system or society. A relative value index (RVI) is calculated by dividing the usual practice ACER by the ICER calculated for the new intervention. The decision rule follows that if the RVI is greater than 1 the intervention is offering additional outcomes at an “acceptable” cost and should be implemented. That is, the incremental cost per unit increase in compliance score with the web-based intervention is lower than the cost per level of compliance attainable with usual practice.
Cost-utility analysis, an alternate method of economic evaluation where intervention effects are measured in terms of impact on length of life and impact on quality of life (utility) summarised as quality-adjusted life years (QALYs), was not selected in this study for the reason that the underlying trial was an implementation trial. This trial was appropriately focussed on the measurement of compliance as the implementation outcome, as opposed to final health outcomes.
Handling of bias and missing data
To avoid bias in the economic analysis (42), any baseline differences in cost between the groups were adjusted by calculating trimmed or truncated means. Similarly, it is important not to ignore missing data. Inappropriate handling of missing data can lead to misleading inferences in economic evaluations (43). While cost-effectiveness analyses conducted alongside trials are an important source of information for decision makers, trials rarely succeed in collecting all the required information (44). Guidance for handling missing data in trial-based cost-effectiveness analyses and for treating missing cost data specifically recommends multiple imputation (45) (46, 47). In this evaluation the treatment of missing data was handled using a combination of methods: multiple imputation using linear regression models and quantile modelling.
Uncertainty and sensitivity analyses
To account for uncertainty due to sampling variation, nonparametric bootstrapping analysis with 2000 iterations was used. The bootstrapped ICERs were graphically mapped on a cost-effectiveness plane and used to derive a cost–effectiveness acceptability curve (CEAC) indicating the probability of the intervention being cost-effective at various levels of society's willingness to pay per unit change in outcome (27). The willingness to pay threshold was informed by the ACER calculated for usual practice (48).
Sensitivity analysis is used to illustrate and assess the level of confidence that may be associated with the conclusion of an economic evaluation (49). In this study, we undertook several one-way sensitivity analyses. First, on the basis that labour time in menu planning and review was the largest driver of cost, we examined the robustness of results to changes in the value of the wages rates for childcare service staff involved in menu planning and review. In the base case analysis, the midpoint of rate ranges were used. In the sensitivity analysis, the upper and lower rates were used (40). In the second sensitivity test, we conducted complete case analysis, including only those variables with no missing data for both intervention and usual practice services. Third, we adjusted for missing data making the less conservative assumption that the proportional change in childcare service cost for the known observations would apply to the missing data for both intervention and usual practice services.