Initially a steering group was established to conduct and assess the process of developing the tool for measuring children’s SMU. Further, parents representing the target group and a panel of Danish media experts were recruited as key informants and in initial validation of the questionnaire.
Scientific steering group
A steering group was formed to initiate and guide the development of a questionnaire to asses children’s screen media environment and its plausible proximal correlates. Members of the Steering group are the authors of this paper and are all part of the academic staff at the Center for Research in Childhood Health and Research unit for Exercise Epidemiology at the University of Southern Denmark. The steering group was responsible for initial item generation, the selection of parents and media experts, design and distribution of the questionnaire, analyses of responses from parents and experts, and drafting the final questionnaire.
Key informants – parents
A convenience sample consisting of 10 parents of 6-8-year-old children (this age-range was chosen for convenience and because we wanted to address the youngest) were recruited for the face- and content validation interviews of the first draft of the questionnaire. Inclusion criteria for this convenience sample were being parents of children between 6-8-years of age (pre-school or first grade) with a regular use of screen media and we wanted parents of both boys and girls. An email invitation letter was sent to parents from public school in the local area of the city of Odense (Denmark) containing written participant information.
Key informants – screen media experts
Ten Danish media experts were recruited to evaluate a draft of the SCREENS-Q. Criteria for being an “expert in the field of SMU” were: having published peer-reviewed scientific papers on the topic, or having authored books on media use, or being involved with descriptive national reporting of media use in Denmark. We were aware of recruiting experts of both genders, and publicly were advocates, opposed, or neutral towards children’s heavy use of screen media. The final panel of Danish media experts were representing areas of psychology, media, communication, journalism and medicine (see list of media experts in acknowledgement).
Steps of development
The developing process of the SCREENS-Q was accomplished through an iterative process with several intertwining steps (see Figure 1). Based on solid methodological literature (19-21), the initial process comprised the following steps: 1. Definition and elaboration of the construct to be measured, 2. Selection and formulation of items, 3. Pilot testing for face and content validity (parents and screen media experts), 4. Field testing in a sample of parents with children 7 years of age for test-retest reliability, and 5. Another field test in a larger sample of the target group from Odense Child Cohort (OCC) for assessment of construct validity and item analysis for final scale evaluation.
Steps 1, 2 and 3 were primarily qualitative evaluations. They were conducted as an iterative process in close collaboration with parents of 6-8-year-old children, the scientific steering group, and Danish screen media experts. Step 4 and 5 were primarily a quantitative evaluation of the test-retest reliability and analysis of item correlation and response distributions.
Defining the construct and initial generation of items (steps 1 and 2)
With the SCREENS-Q we aimed to measure children’s SMU (time and content) and specific screen-media behavior, the screen media home environment including important putative proximal correlates of children’s SMU. Several methods were used to identify relevant factors of these constructs. For the proximal correlates the scientific steering group initially established a theoretical model based on a socio-ecological model, to provide a foundation to define and comprehensively understand how various factors that may determine children’s media use are interrelated (see Figure 2). The socio-ecological model worked as a unifying framework for identifying and investigating potential correlates of children’s SMU. Subsequently, a literature search identified and supplemented constructs from the socio-ecological model (22, 23). Based on this model we also included relevant questionnaire items from former or ongoing studies (13, 16, 17, 24-27).
With the SCREENS-Q we aimed to assess possible direct and indirect causal factors that may influence children’s SMU. The questionnaire is multidimensional and based on a formative model (21, 28) meaning that it is intended to cover and measure all indicators that might possibly contribute to the construct “children’s SMU”. Potential causal factors may have different impacts but in a formative perspective we aimed also to identity factors with little impact. Therefore, in the initial phase we attempted to obtain a comprehensive analysis of the construct (20, 21, 28) to generate a broad list of domains and items, that were not necessarily correlated. Reduction of redundant items was carried out in later steps during pilot and field testing (20, 21, 28).
The amount of questions and items within each domain is not necessarily an expression of importance or weighing of the specific domain, but rather a question of meaningful wording and/or how accurately we wanted to measure the specific domain. This first version of SCREENS-Q was developed to use in a large ongoing birth cohort Odense Child Cohort (OCC). Therefore, relevant demographic, social and health behavior questions are obtained from measures and questionnaires in OCC (i.e. family structure, religious and ethnic origin, TV in the bedroom, other health related variables, attendance of institutions, socioeconomic status)
Pilot testing: Face and content validity (step 3)
A first draft of SCREENS-Q was developed based on the socio-ecological model, and face- and content validation was tested in a convenience sample of key informants. Ten parents of children aged 6-8 years filled out the questionnaire, while researchers were present for eventual questions about understanding and interpretation of wording of the questionnaire. Right after completing the questionnaire a semi-structured interview was conducted on relevance, importance, and if some important domains or areas of children’s SMU were missing. The key informant interviews were recorded and transcribed. Every item in the questionnaire was analyzed and revised based on the interviews, in relation to wording, understanding, interpretation, relevance and coverage for SMU in the sample of parents. Relevant changes were adapted after careful consideration in the steering group.
Fifteen Danish media experts unaffiliated with the study were contacted by telephone, informed about the project and asked if they were willing to evaluate the SCREENS-Q questionnaire. Ten of the 15 experts agreed to participate. An updated draft was sent to the ten media experts for another evaluation of face and content validity. The experts received an email with a brief description of the aim of our project, the purpose of the questionnaire, and a link to the online questionnaire in SurveyXact. They were asked not to fill it out, but to comment on every item and/or domain in the questionnaire. They were also asked to comment on wording, understanding and relevance for each item. Finally, they were asked whether the domains in the questionnaire adequately covered all significant areas of children’s use of screen media. Based on the responses and subsequent discussion in the steering group the questionnaire was further refined, and some items were modified, deleted, or added to the questionnaire.
The experience from these first steps were discussed in the scientific steering group and a final draft for field testing in the target group now comprised a questionnaire covering 6 domains, 19 questions summing up to 96 items about children’s SMU and potential correlates (see Table 1 for Domains, questions and items included in the SCREENS-Q). Step 1-3 was conducted as an iterative process from February to July 2017.
Field testing in the target group (step 4 and 5)
Step 4: Examination of test-retest reliability
Another convenience sample was recruited from schools (1st grade and 2nd grade) in the municipalities of Odense and Kerteminde, Denmark. Inclusion criteria were: 1) Being parents to children at 7-9 years of age, and 2) the child must have access to- and use minimum two of the following screen media devices in the household: Tablet, smartphone, TV, gaming console or computer. In total 35 parents agreed to participate in this field testing for test-retest reliability. The questionnaire was sent to the parents, and responses collected electronically. The participants were asked to fill out the SCREENS-Q twice, separated by a two-week interval. Step 4 was conducted in November 2017 and December 2017.
Step 5: Construct validity and item analysis.
After evaluating test-retest reliability in the convenience sample, the SCREENS-Q was implemented in OCC, an ongoing closed birth cohort initiated in 2010-2012 (24). The evaluation of construct validity was done with two items measuring screen time (item 9 and 13 from n=243). Furthermore, item analysis (based on descriptive analysis of data and qualitative evaluation of response patterns and feedback), and feasibilityas evaluated on a subsample of parents from the cohort (n=243). Items would be deemed redundant if they had too little variation. Item responses were analyzed to investigate whether any answer categories were unexpectedly unused/not answered and therefore seemed redundant or irrelevant. Participating parents were asked to fill out the SCREENS-Q on a tablet while attending the 7-year old examination at the hospital. If the child did not attend the planned 7-year old examination the questionnaire was sent to the parents by email. Step 5 was conducted on data from ultimo November 2017 to primo March 2018.
Data management
The questionnaire was distributed and answered online. In the pilot testing (step 3) SurveyXact was used for management and initial response analysis. For the field testing, a professional data entry organization (Open Patient data Explorative Network) entered the data in the survey/data management software REDCap. A series of range and logical checks were undertaken to clean the data.
Statistical methods
To determine test–retest reliability selected relevant items were compared between the first and second administrations of SCREENS-Q during field testing (n=35). For categorical/binominal variables (questions 4, 5 and 11) levels of agreement were determined using Kappa coefficients which were defined as poor/slight (κ = 0.00–0.20), fair (κ = 0.21–0.40), moderate (κ = 0.41–0.60), substantial (κ = 0.61–0.80) and almost perfect (κ = 0.81–1.00) (29). Reliability for questions on an ordinal scale (item 3, 6, 17 and 11) was assessed using weighted kappa and/or intra-class correlation (ICC) as these estimates are identical if the weights in kappa are quadratic (28). To avoid excluding items with a low Kappa value despite showing a high percent agreement (due to a high percent of responses in one category, creating instability in the Kappa statistic) it was decided that items with a κ > 0.60 and/or percent agreement ≥ 60% were considered to have acceptable reliability (30, 31).
Test-retest reliability of continuous variables (item 9, 13 and 19) was evaluated by calculating ICC and standard error of measurement. An ICC value of 0.75 or higher was considered to represent a good level of agreement. ICC values of 0.50–0.74 were considered to represent moderate reliability and those below 0.50 represented poor reliability. Bland-Altman plots were created, and 95 percent limits of agreement calculated to investigate agreement between the first and second administration of the SCREENS-Q for continuous variables.
As SCREENS-Q is a multidimensional tool based on a formative model, item analyses were primarily done by qualitative evaluation of distributions and usefulness. Factor analysis and definition of internal consistency does not apply, as items are not assumed to be internally correlated in a formative model (21). This applies to all items in the questionnaire except from questions 9 and 13 that each can be summarized to provide a total screen time use variable. Construct validity is about consistency – not accuracy (19, 28). Thus construct validity of these questions was assessed using pairwise non-parametric correlations (Spearman’s) and 95% CI calculated by bootstrap estimations with 1,000 reps (32).
All analyses were conducted in Stata/IC 15.