Delphi research is a structured process of coordinating expert opinions anonymously to gather information about a research question until a consensus is reached among experts23–25. In this study, a consensus process was conducted among national and international experts on the importance of physical literacy assessment indicators for junior high school students through three rounds of Delphi’s online questionnaire star survey. The results of the systematic evaluation were provided to all the experts who participated in the questionnaire. At each step, the participating experts were anonymous and independent to avoid the possibility of individuals dominating the consensus.
The working group of this project is composed of one professor, two associate professors and three postgraduate students at the institute, and the main work is to construct the measurement system and preliminarily compile the expert consultation questionnaire based on the results of the literature. To select the experts and obtain the experts' email addresses, to collect the expert consultation questionnaire, to statistically analyse and collate the questionnaire results, and to provide good feedback.
The selection of expert teachers will be determined based on the clarity of the research purpose and research tasks of the physical literacy indicator system study. The selection criteria were as follows: first, experts and scholars who have been engaged in related work for more than ten years; second, who are more familiar with the study of physical literacy, who have the ability to conduct academic research or who have engaged in related work; and third, who are able to enthusiastically participate in and actively support this study. Since this study focuses on constructing a Chinese assessment system, many invited experts are Chinese. Generally, the wider the geographical distribution of the experts is, the greater the proportion of high professional titles, the greater the proportion of experts engaged in their specialities, the greater the representativeness and authority of the experts, and the greater the credibility of the results. We searched domestic and foreign experts (including corresponding authors and first authors) who had published articles related to physical literacy based on major domestic and foreign databases such as the Web of Science (WOS) core database, EBSCO database, and China Journal Network (CNKI) and invited them to participate in the questionnaire survey to form the expert panel of this study, which must ensure that the experts are representative and authoritative. The determination of the number of experts is not based on statistical ability but is generally a pragmatic choice based on the size of the research project and the breadth of the area involved, and 10–50 people are generally recommended. We sent a letter of introduction and information sheet to 20 potential Delphi expert panel experts, and 16 experts responded and were recognised. The 16 experts and scholars related to the field of physical education mainly consisted of 15 Chinese experts and 1 foreign expert.
Table 1
Summary of basic information on the selected experts
serial number
|
qualifications
|
title
|
field of research
|
A1 Specialist
|
PhD
|
lecture on
|
Physical Education Curriculum and Pedagogy
|
A2 Specialist
|
PhD
|
lecture on
|
Sports Industry Management
|
A3 Specialist
|
PhD
|
lecture on
|
school sports
|
A4 Specialist
|
PhD
|
lecture on
|
sports management
|
A5 Specialist
|
PhD
|
Associate Professor
|
Outdoor Education
|
A6 Specialist
|
PhD
|
Associate Professor
|
physical education
|
A7 Specialist
|
Master
|
lecture on
|
Functional assessment of sports training
|
A8 Specialist
|
PhD
|
lecture on
|
Sports and Health
|
A9 Specialist
|
PhD
|
Associate Professor
|
Humanities and Sociology of Sport
|
A10 Specialist
|
PhD
|
lecture on
|
Youth sports
|
A11 Specialist
|
PhD
|
Associate Professor
|
school sports
|
A12 Specialist
|
PhD
|
Associate Professor
|
Physical Education and Training
|
A13 Specialist
|
PhD
|
lecture on
|
school sports
|
A14 Specialist
|
PhD
|
tutors
|
school sports
|
A15 Specialist
|
PhD
|
lecture on
|
Physical exercise and physical and mental health
|
A16 Specialist
|
PhD
|
tutors
|
sports management
|
Informed consent
Participation specialists will be contacted by email and provided with information about the study, and participants will be asked to leave their information, including name and email address, which emphasises voluntary participation, unpaid participation and the right to withdraw from the study at any time. Completion of the initial registration via the link embedded in the email will imply informed consent to participate in the study.
Delphi Round one
Purpose: To initiate the Delphi research process, experts’ initial views on indicators for assessing physical literacy among junior high school students were collected, and an extensive list of indicator candidates was compiled. Steps: (1) Preparation: Identify the research team and clarify roles and responsibilities. A preliminary list of indicators with questions broad enough to stimulate experts’ thinking and share their views was designed. The composition of the expert team was confirmed to ensure the diversity and representation of the industry. (2) Distribute questionnaires: The questionnaires were distributed in digital or paper form to the experts to evaluate the validity of the indicators. (3) Collect data: Collect feedback from all experts by a set deadline. (4) Analyse and collate: Analyse all the feedback in the questionnaire, identify the key indicators according to the frequency of comments and the degree of importance attached by the experts, and summarise them to obtain a list of indicators. (5) Feedback of results: The aggregated results are fed back to the experts, thus making the process transparent and preparing for the next round of surveys.
Delphi Round two
Purpose: To refine and review the list of indicators obtained in the first round, assess the relative importance of each indicator, and form a preliminary evaluation system. Step: (1) Questionnaire design: A more structured questionnaire was designed based on the results of the first round, which included a Likert scale (e.g., a rating from 1 to 5) for assessing the importance of the indicators. (2) Distribution of the questionnaire: The quantitative questionnaire was distributed to all the participating experts. (3) Data collection: Feedback from the second round of questionnaires was collected to ensure that each expert had sufficient time to evaluate each indicator in detail. (4) Data analysis: The mean score and standard deviation for each indicator were calculated to determine the level of consensus among the experts. The indicators with high scores (widely agreed to be important) and low scores (more controversial or considered unimportant) were identified. (5) Collation of results: Prepare a streamlined list of indicators with average importance ratings for each indicator.
Delphi Round three
Purpose: To further improve the consensus among experts, we aimed to determine the indicators screened in the second round and to develop a system of physical literacy assessment indicators for junior high school students. Step: (1) Questionnaire design: Create a questionnaire that includes the results of the second round of ratings so that experts can see their previous ratings as well as the average ratings of the overall expert group. (2) Distribute the questionnaire: Distribute the questionnaire and ask the
experts to assess the indicators again, either by revising their previous ratings or by commenting on the ratings of others. (3) Data collection: Feedback from the third round of questionnaires was collected. (4) Final analysis and decision-making: Analyse the data and establish the final set of consensus ratings. (5) Formation of the assessment index system and report: Based on the results of the three rounds of the survey, the final index system for assessing the physical literacy of junior high school students was formed. A detailed report describing the construction process, the formation of an expert consensus, and the specific content of the indicator system was written. The weighting coefficient of each indicator is calculated.
Statistical analysis
Analysis of the degree of expert motivation and the degree of authority. The degree of expert motivation, i.e., The recovery rate of the expert consultation form (recovery rate = the number of experts participating in the judging of the entries/the number of all experts) can reflect the degree of expert interest in the study. In general, the higher the coefficient of expert motivation is, which indicates that the experts care about the study, the greater the credibility of the results of the study.
The degree of authority of the expert mainly includes the degree of familiarity, the level of academic power and the basis of judgement on the indicators. The expert’s degree of familiarity refers to the expert’s knowledge of the physical literacy assessment indicator system. The experts’ familiarity with the indicators (q1) consists of five grades—very familiar, familiar, general, not too familiar and unfamiliar—which are assigned values of [1.0, 0.8, 0.5, 0.2, 0.0], respectively. The expert’s academic level right (q2) is customised based on the expert’s title. Generally, the higher the title of an expert is, the greater the corresponding academic level, and the more valuable the opinions he expresses. The academic level weights are designated as doctoral supervisor, master’s supervisor or professor, other senior position, deputy senior position, and others. The values are assigned in order of [1.0, 0.9, 0.7, 0.5, 0.3]. The basis of the experts’ judgement of the indicators (q3) was mainly rated at three levels—large, medium and small—in four aspects: theoretical analysis, practical experience, peer understanding and expert intuition.
The formula for the degree of authority of the expert: q
Analysis of the degree of concentration of expert opinion. The physical literacy indicators at all levels of junior high school students in China were listed, and the degree of importance of each indicator was judged. To facilitate quantitative analysis, a 5-point Likert scale was used for the assessment, in which the options ranged from "very important" to "very unimportant", and the indicators were scored by experts as 5, 4, 3, 2, or 1. The recovered data were processed, and the indicators were filtered by the degree of concentration and coordination of expert opinions. Distribution of questionnaires. In the first round, the preliminary construction of the evaluation index system was modified through the scores and suggestions of the experts; the results of the first round of survey were fed back to the experts before the second round of expert opinion survey, and then the second round of expert opinion survey was carried out. The indicators were screened through the three analytical parameters, namely, the mean value (M), standard deviation (SD), and coefficient of variation (CV), of the indicators of all levels, as well as the expert’s suggestions for modification. According to the two rounds of the experts’ questionnaire survey, the opinions returned from the experts were summarised and collated, and the experts’ assessment results were subjected to a consistency test, i.e., P < 0.01 or P < 0.05. After the opinions of the 16 experts converged, the physical literacy assessment index system of junior high school students in China was finally determined.
Concentration Analysis: By counting the mean values of the first-, second- and third-level indicators, we can determine their importance. To avoid missing any of the key indicators, we excluded from this study all indicators with a mean value of less than 4. Analysis of the degree of concordance: The analysis of the coefficient of variation allows us to better assess the degree of dispersion of the data, thus reflecting more accurately the concordance of the indicators by the experts. After expert assessment, we can derive the coefficient of variation for each indicator. According to previous studies, a coefficient of variation lower than 0.2 indicates relative agreement among experts, while a coefficient of variation greater than or greater than 0.2 indicates disagreement among experts. Indicators with a coefficient of variation above 0.2 were excluded from this study to ensure accuracy. The coefficient of variation, in turn, is closely related to the mean and standard deviation, so the formula is as follows:
Average of the ith indicator: " \({\text{M}}_{\text{i}}=\frac{1}{\text{m}}\sum _{\text{k}=1}^{\text{m}}{\text{X}}_{\text{k}\text{i}}\) (1)"
The standard deviation of the ith indicator: " \({\text{S}}_{\text{i}}=\sqrt{\frac{1}{\text{m}}\sum _{\text{k}=1}^{\text{m}}{\left({\text{X}}_{\text{k}\text{i}}-{\text{M}}_{\text{i}}\right)}^{2}}\)(2)"
Coefficient of variation for the ith indicator:"\({\text{C}\text{V}}_{\text{i}}=\frac{{\text{S}}_{\text{i}}}{{\text{M}}_{\text{i}}}\)(3)"
All the above data were analysed with Excel and SPSS 22.0 software.
Determining weights using AHP
This study adopts the hierarchical analysis method, which is an assignment method in which experts assign weight coefficients to the factors at each level of the measured indicator system according to the degree of importance. Combined with the requirements of the calculation method of this study, the use of hierarchical analysis can be carried out in the following four steps: (1) establish an ordered hierarchical structure that can describe the characteristics of the system in accordance with the requirements of the indicator construction and derive the corresponding scale; (2) judge the relative importance of the indicators, use the 1–9 scale method, and compare the relative importance of the indicators of the two indicators in each level; (3) use Yaahp software to calculate the relative degree of each level of indicators in the physical literacy assessment index system and then carry out statistical processing to calculate the weight coefficients of the physical literacy assessment indicators of junior high school students in China; and (4) process the judgement matrix to obtain the relevant elements of the relative importance of the indicators of the upper level and finally carry out a consistency test.
Ethical approval
In this study, we adhered to ethical principles and obtained approval from the Ethics Committee of Jinan University. Written and verbal consent was given by the members of the panel, whose identities remained anonymous. All rounds of test answers were coded according to a numerical code.