This study aimed to address four research questions related to measuring online learning environment in medical education. These questions are:
- Which items are relevant to measure the online learning environment in medical education?
- What is the content validity evidence of the Digi-MEE instrument?
- What is the response process validity evidence of the Digi-MEE instrument?
- What is the factorial structure and internal consistency of the Digi-MEE instrument?
Yusoff, Arifin, and Hadie [20] divided the development and validation of the Digi-MEE instrument into two main phases. In the first phase, experts provided feedback on content and terminology related to the design of the instrument's items and domains. In the second phase, the instrument was validated to establish its content validity, response process validity, construct validity, and internal consistency.
The study received approval from the Human Research Ethics Committee of Universiti Sains Malaysia (USM/JEPeM/21050350) on October 11, 2021, and November 3, 2022. Ethical approvals were also obtained from the review boards of the data collection sites for different phases of validation. Specifically:
- Response Process Validation: Letter No. IEC/80-21, dated April 23, 2021
- Construct Validation - Exploratory Factor Analysis: Letter No. ABWA MC/DME/609/2022, dated September 2, 2022
- Confirmatory Factor Analysis: Data Collection Approval dated September 16, 2022.
The study was conducted between July 2022 and December 2022 and involved medical educationists and medical students in different parts of the study, as explained in relevant sections. Figure 1 represents the flow chart for the development and validation process of Digi-MEE.
Development of Digi-MEE items
The domains of online learning environment in medical education were generated based on the scoping review findings by Naeem et al [21]. The online learning environment in medical education has the following nine domains [21]:
- Cognitive enhancement promotes information processing, knowledge application, analysis, and evaluation by learners in learning environments.
- Content curation is the process of searching, reviewing, organizing, and presenting content about a specific subject from multiple sources in a context that is relevant to a particular audience.
- Cybergogical practices refer to learning activities in online learning environment for the enhancement of cognitive, emotional, and social learning of the students.
- Learner characteristics is a concept that revolves around how the student’s learning experience is influenced by personal, social, cognitive, and academic elements.
- Digital capabilities are referred to as the set of skills, knowledge and understanding required to facilitate someone to live, learn and work in a digital world.
- Platform Usability refers to the capacity of a platform to provide a condition for its users to perform online tasks effectively, efficiently, and safely while enjoying the experience.
- A learning facilitator is an active and direct learning helper whose roles are to create an atmosphere for study, to help learners get to their goals, and to encourage social interaction among professors and students.
- Social representations are referred to be “a system of values, ideas and practices” concerted through interactions between individuals, groups, institutions, and the media.
- Institutional support refers to the organizational active engagements in the form of policies, regulations, and support system, that motivate stakeholders to use online learning environment effectively.
Based on the nine domains, items were developed by the researchers (NKN, SNH, IM, MSBY) through literature review and modified Delphi study that involved 18 experts (i.e., medical educationists, medical teachers, and instructional designers). English language was used to construct the items representing the online learning environment domains.
Validation of Digi-MEE domains and items.
Content Validation
The first version of Digi-MEE (Digi-MEE 1.0) was reviewed by a panel of content experts to evaluate the relevance of the Digi-MEE items to the nine online learning environment domains. A minimum of six experts can be selected based on their qualifications and related experience in online learning environment in medical education [22].
A total of 10 experts were invited to perform the content validation task, and they were provided with a content validation form and asked to rate each item's relevance to the measured domain on a 4-point scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant) and provide written comments on any items that required modifications or removal [23,24]. Content validity index (CVI) was calculated based on I-CVI and S-CVI values.
I-CVI has been defined as “the proportion of content experts giving an item a relevance rating of 3 or 4, while S-CVI is the average of the I-CVI scores for all items on a scale” (i.e., an online learning environment domain) [22] .The acceptable I-CVI value has been set at a minimum of 0.78, while the acceptable S-CVI value should be at a minimum of 0.80 [23]. Based on the I-CVI values, items with I-CVI of 1.0 were accepted, items with I-CVI at least 0.79 but less than 1 were re-discussed, and items with I-CVI less than 0.78 were rejected[24].
Response Process Validation
The second version of Digi-MEE (Digi-MEE 2.0) was reviewed by a panel of medical students who are the potential users of the Inventory and who have had at least one year of experience of online learning in a private medical school in Pakistan.
A total of 21 respondents were invited to perform the response process validation task, and they were given a response process validation form and asked to rate each item's clarity in the measured domain on a 4-point scale (1 = not clear, 2 = somewhat clear, 3 = quite clear, 4 = very clear) and give written comments on any item needing revision or removal [23,24]. Face validity index (FVI) was calculated based on Item face validity index (I-FVI) and scale face validity index (S-FVI values).
I-FVI is the proportion of content experts giving an item a clarity rating of 3 or 4, while S-FVI is the average of the I-FVI scores for all items on a scale (i.e., an online learning environment domain). The acceptable I-FVI and S-FVI value has been set at a minimum of 0.78, and 0.80 respectively [25]. Based on the I-FVI values, items with I-FVI of 1.0 were accepted, items with I-FVI at least 0.79 but less than 1 were revised, and items with I-FVI less than 0.78 were rejected [24].
For the qualitative analysis, open comments given by the medical students related to each item were analyzed for any revision in sentence structure, format, spelling errors as well as any addition, deletion or shifting of any item to another component as suggested. The outcome of this study was the development of third version of Digi-MEE instrument (Digi-MEE 3.0) with 9 components and 46 items.
Assessment of factorial structure and internal consistency
Following the content and response process validation, the third version of Digi-MEE (Dig-MEE 3.0) was administered to 230 undergraduate medical students via a cross-sectional survey. The sample size was determined according to the recommendation by Osborne and Costello indicated that the sample size calculation for factorial analysis should be in the ratio of 5:1 for the number of items or greater than 100 [26].Participating medical students were asked to rate their experience of their school’s online learning platform using the Digi-MEE v.3.0 in a Likert scale from 1-4 (scale:1= strongly disagree ,2= disagree,3= agree, and 4= strongly disagree) after taking informed consent.
All the data was analyzed by Exploratory Factor Analysis (EFA) through Statistical Package for Social Sciences version 26 (SPSS 26) to determine the number of domains of online learning environment and related items for Digi-MEE. First, data suitability for factorial analysis was determined by the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) and Bartlett's Test of Sphericity. The KMO value > 0.7 indicates a good level of factor distinction based on an adequate sample, whereas the significant Bartlett’s Test (p-value <0.05) demonstrates that the factor analysis is appropriate [27]. The factors were extracted through principal component axis method to check for total and cumulative variance for the nine extracted components and factor loading of each item. The initial component matrix was then rotated via Varimax with Kaiser normalizations after 28 iterations. This was based on prior assumption that correlation coefficient between the factors of more than 0.3 warrants oblique rotation method is used otherwise the orthogonal rotation axis should be used [27]. For each component, Eigenvalues >1and item factor loading values > 0.40 signify convergent validity [28]. Items with factor loading <0.4 were removed. The items were shifted in respective components where they loaded maximally, after being reviewed by two independent researchers who considered the theoretical meaning of each component and whether the item fitted conceptually with the other items that are already assigned to that component.
After EFA, confirmatory factor analysis (CFA) was carried out on the data obtained from cross-sectional survey on 450 medical students for their ratings of their school’s online learning environment using Digi-MEE 4.0. Sample size was determined based on criteria of ten participants per item for adequate sample for CFA [29].
Data was through SPSS version 26.0 and Analysis of Moment Structure (AMOS) version 26.0. Model chi-square goodness of fit (set at 0.05) demonstrated model fit. Approximate fit indexes were applied to check the available data for measurement model fitness. For absolute fit indexes, Goodness of Fit Index (GFI) (model fit >0.9) and root mean square error of approximation (RMSEA) (model fit=RMSEA<0.08 and Root mean squared residual-RMR<0.05) were used. For incremental fit measurement, Comparative fit index (CFI) (model fit >0.9), Normed fit index (NFI) (model fit >0.9), and incremental fit index (IFI) (model fit >0.9), and Tucker Lewis fit index (TFI))(model fit >0.9) were used. Parsimonious fit was measured through Chi Square/Degree of Freedom (Chisq/df) [19,20].
Convergent validity was determined with size of factor loading, as well as composite reliability (CR). The item factor loading values should be reasonably high (which are 0.5 or more) to signify convergent validity for each construct. Composite Reliability (CR) was calculated on Microsoft Excel using formulae given by Fornell and Larckers [30]. Values of CR of more than 0.6 indicate good construct reliability and adequate convergent validity [31].
After CFA, same data was used to perform reliability statistics to assess the internal consistency of final version of Digi-MEE (Digi-MEE 5.2). Cronbach’s alpha coefficient was determined as the measured parameter to determine internal consistency. Cronbach’s alpha values between 0.7 and 0.9 are considered to represent high internal consistency and values between 0.6 and 0.7 are considered as satisfactory [32]. Similarly Composite reliability was also calculated with >0.6 having high internal consistency [31].