We investigated the accuracy of three different automated scoring algorithms for determining the rest-interval of older adults who wore a WWA for ≥14 days of observation during the Sleep and Cognition Study [19, 20], a cross-sectional study which examined age and cognitive differences in objectively measured sleep. All participants provided written and informed consent (H14-01301).
Participants
We recruited 154 participants from Vancouver, British Columba. Participants were included if they met the following criteria: 1) men and women 50+ years of age living in the Metro Vancouver area; 2) scored >24/30 on the Mini-Mental State Examination (MMSE) [21]; and 3) able to read, write, and speak English with acceptable visual and auditory acuity. We excluded individuals: 1) diagnosed with dementia of any type; 2) diagnosed with another neurodegenerative or neurological condition (e.g., Parkinson’s disease or Multiple Sclerosis) which affects cognitive function and/or sleep quality; 3) taking medications which may negatively affect cognition; 4) planning to participate or currently enrolled in a clinical drug trial; or 5) unable to speak as judged by an inability to communicate by phone.
Measures
Instrumentation and Software: We used the MotionWatch8© (MW8) WWA to collect activity (i.e., counts) and light data (lux).[22] Data were collected in 60 second epochs. Activity counts are an arbitrary unit of measurement, which is calculated for each epoch as the sum of the peak acceleration relative to a minimum acceleration threshold of 0.1g, in a range of 0.1-8g, sampled at a frequency of 6Hz. The MW8 also has a lux range from 0 to 64000lux and samples the light exposure at a frequency of 1Hz. The per epoch lux value recorded represents the average lux over the specified epoch length when sampled once per second. In the standardized protocol, we used the event marker time stamp button to create event marker recordings in the data to help define the rest-interval. The MW8 is the updated version of the Actiwatch7, an actigraph with evidence of validity against polysomnography in healthy adults (Mean age: 30 ± 6 years; 45% female; [23]), and also adults with chronic insomnia (Mean age: 41 ± 12 years; 78% female; [24]). The MW8 has evidence of validity among: 1) 54 adults with suspected sleep disorders including obstructive sleep apnea, insomnia, hypersomnia, and Ehlers Danlos syndrome (Mean Age: 53 ± 16 years; 61% female); and 2) 19 healthy adults (Mean Age: 28 ± 5 years; 53% female; [25]).
After the rest-interval had been determined using one of the methods we define below (see Rest-Interval Scoring Methods), sleep estimates were determined using the MotionWare software. The sleep/wake algorithm used by the MotionWare software is available from the manufacturer (CamNtech; [email protected]).
Consensus Sleep Diary (CSD): The CSD is a self-report sleep diary designed and used predominantly in insomnia research that has been shown to be effective in clinical and research settings [26]. We elected to use the CSD-core which has 9-questions and is effective for both “good” and “poor” sleepers. Question’s 2 and 7 were used to confirm the rest-interval defined by event marker recordings in the standardized protocol scoring method and were used to set the rest-interval in the CSD scoring method.
Procedure
Each participant was asked to wear the MW8 for 14 consecutive days [27]. During the 14-day period, the watch was set to record uniaxial acceleration as well as light exposure. Participants were asked to press the event marker button on the watch when they got into bed and upon their final awakening in order to set the GU and LO times. Each participant was also given the CSD and asked to fill it out each morning after they woke up. 151 (98%) of the participants had actigraphy data recorded from the specified continuous 14-day period. 21 participants were excluded because either question 2 or 7 from the CSD were not complete for any single day within the 14-day study window. Of the remaining 130 participants (84.4%), 1 had irregular sleep patterns where they slept past 12:00, violating one of the programs simplifying assumptions. The remaining 129 (83.7%) participant’s data were used for testing and analysis.
Rest-Interval Scoring Methods
We tested and compared five rest-interval scoring methods by looking at the sleep parameters generated from those rest-intervals using the sleep/wake scoring algorithm (Fig. 1). We highlight each of these scoring methods below.
CSD Only Method: The CSD scoring method used participants answers to questions 2 and 7 of the CSD as the LO and GU time respectively to establish the rest-interval.
Standardized Protocol: A trained research assistant was responsible for scoring the rest-interval in the standardized protocol method. The four major sleep indices considered in order of priority were event markers, Q2 and Q7 from CSD, cessation/onset of light and cessation/onset of activity. If the event marker and CSD were within 30 minutes of one another, an appropriate rest-interval had been established. In cases where the indices agreement was less clear and the event marker was not within the 30 minutes of the CSD, the LO and GU times were set at based on the cessation/onset of light and motion. Since the precise definition of cessation/onset was not defined, whether to check light, motion or both and the number of consecutive counts of cessation/osnset was to be determined at the lab staff’s discretion. When following the protocol, the data was looked at visually using the MotionWare software graphics (Fig. 2). Lab staff were asked to set the limits of the display to be 1000 and 35 for activity counts and lux respectively before proceeding with scoring. A rough interval was then selected in order to allow for accurate cursor placement. The LO and GU placements were made by adjusting a cursor displayed over top a visualization of the actigraphy data indicating the exact time (to the minute) of the cursor placement.
Automatic Scoring Algorithm: An algorithm was created and then tested using three different threshold settings, constituting three separate scoring methods, which were then used for comparison with the CSD and standardized protocol methods. This algorithm will be referred to as the Rest-Interval Scoring Algorithm or RISA (Fig. 3).
The RISA algorithm works as follows:
- The specified period of interest (14 days in our study population) is partitioned into 24-hour segments.
- Within each 24-hour segment, the window of 6 hours which contains the smallest sum of epochs with an activity count below 20 and a lux count of 0 is designated as the Potential Sleep Window (PSW).
- A 6-hour window containing the potential LO time is found by segmenting data 3 hours before and 3 hours after the start time (ST) of the PSW.
- A 7-hour GU window is found by segmenting data 1 hour before and 6 hours after the end time (ET) of the PSW.
- The potential LO and potential GU windows are then fed into two separate functions (the LO and GU functions), described below, which return the LO and GU times.
The RISA algorithm was determined heuristically (i.e. through trial and error methods). Actigraph data whose rest-intervals had been set by the protocol method were visually inspected using the MotionWare software from an arbitrary number of participants across 4 independent studies to discern patterns in how the rest-intervals were set following the protocol method. Parameter values within the RISA and its subsequent components were initially set arbitrarily and were adjusted manually through iteration by running the algorithm on a training data set which consisted of baseline wrist-worn actigraphy recordings from the Buying Time study [28]. We elected to avoid optimization of the parameter values to prevent overfitting and allow for generalization.
The LO function abstracted away in Fig. 3 and shown in detail in Supplementary Material S1, keeps track of four counters. Each counter tracks consecutive epochs of zero lux, activity count less than 20, activity count of zero, or activity counts of zero activity and lux simultaneously. The GU function (Supplementary Material S2), keeps track of three counters. The counters track consecutive epochs of lux greater than zero, activity greater than zero, activity greater than 20, or activity and lux greater than zero simultaneously. Once a counter exceeds its respective threshold value, the start index is marked as the LO (or GU depending on the function) time for that day.
The threshold values within the LO function of the RISA were given 3 different sets of values; one for biasing light (BL), one for biasing motion (BM), and one which balanced both light and motion (LM). The RISA is thus the parent algorithm and is the underlying logical structure for LM, BM and BL. The set of child algorithms developed from the RISA are the LM, BM and BL. The GU function’s threshold values were held constant. Each set of threshold values in the LO function constituted a separate method which was then compared to the CSD and protocol scoring methods. The BL, BM and LM threshold values are given in Supplementary Material S3
Statistical Analysis
We calculated means and standard deviations for sleep duration, fragmentation, efficiency, and latency using each of the methods for estimating the rest-interval length. We estimated between method differences in estimates of sleep quality using analysis of variance (ANOVA). Significant differences were then explored using pairwise comparisons (i.e., estimated marginal means). Further, we performed bi-variate correlation analyses to examine consistency in sleep quality estimates between methods. We subsequently conducted Bland Altman plot analyses in order to investigate differences in estimated sleep quality between the standardized protocol method and our automated methods of determining the rest-interval length.