- Study design and ethical approval.
The research is quasi-experimental in nature, used to measure faculty experiences with item banking before and after the software introduction (intervention). Ethical approval was obtained from the institutional review committee, with reference no. ‘(Riphah/IIMC/ IRC/22/205). It was conducted in a single Institute of Riphah International University. The total duration of the study was 12 months (October 2022 to September 2023).
- Instrument development and validation.
The overall development of the questionnaires occurred in two stages: 1) Questionnaire development and validation process. 2) reliability. The instruments used in the study were: (a) to examine the challenges faced by the faculty members in current item banking practices. (b) to determine the faculty satisfaction with the use of ETMS “Schoolhouse technologies software.” The questionnaires used in the study to collect the data, drawn upon established research, and focused on relevant concerns related to faculty perceptions and experiences in item banking practices (Ying, Yang, 2008) (Yusuf, Lim, 2017) (Meesanga, 2021) (Masila, 2014). Based on thorough literature review, the first draft of the questionnaires was developed containing 26 items and 13 items, respectively. To ensure content validity of the instruments, a judgmental sampling method was employed to invite the expert panels based on their expertise and credibility, followed by the pre-cognitive interviews. The panel of 10 experts who were expert medical educationists with specific experience of assessment rated the items in terms of two attributes (its relevance and clarity) on a 4-point ordinal scale (Dalwali, 2023). Ratings of 1 and 2 represent not a relevant item, while a rating of 3 and 4 represents relevant content. For clarity, a 3-point Likert scale was used, the average was calculated for each item. An average above 2.5 (80%) is a clear item (Khan, 2021) Table-1 below added to show the scale for scoring method.
The ratings given by the panel experts were used to calculate the CVI two indexes which were the item level (I-CVI) and scale-level (S-CVI). Considering the recommendations in the literature the acceptable cut of score for CVI for 10 panels is ≥ 0.7 (Yusoff, 2019). Valuable comments of experts were considered. Items having CVI of ≥ 0.70 were included, items with I-CVI ≤ 0.78 were removed (Shi, 2022). Therefore, out of 26 items initially, 6 items were eliminated with the remaining 20 items in challenges measuring questionnaires were sent for the response process. Similarly, in the questionnaire measuring satisfaction of the faculty with the software, out of 13 items, 1 item was removed with the remaining 12 items sent for the interviews. An important requirement of instrument validation is the Response process validity of an instrument, which is determined by cognitive interviews (Artino, 2014 ). Through the process, the interpretation and understanding of the experts related to both survey items were determined. Face to face interviews were held using a verbal probing method (Artino, 2014) Participants were asked to read aloud and clear all the items one by one, reiterate the statements in one’s own words and clarify any ambiguity. This step in the validation process helps to identify and rectify any issues with the questions, making them clearer and more understandable to ensure that the instrument is valid. The data collected through cognitive interviews were analyzed according to predefined criteria shown in table below. Through the process, 4 items were excluded based on repetition of the concepts, hence, a total of 16 items were finalized in the questionnaire related to challenges and the same 12 items were finalized with minor revisions in the satisfaction survey.
- Reliability:
Once the items were finalized for both questionnaires, a pilot test was conducted to ensure the readability. Pilot testing was done for the reproducibility and credibility of questions. The table below represents Cronbach's alpha (α) reliability coefficient. For the Challenges questionnaire, which consists of 16 items, the calculated Cronbach's alpha reliability coefficient is 0.630(Keith, 2017). This coefficient suggests moderate internal consistency among the items, indicating a reasonable degree of reliability in measuring challenges related to item banking practices. On the other hand, the Satisfaction questionnaire, comprising 12 items, demonstrates a higher level of internal consistency, with a Cronbach's alpha reliability coefficient of 0.946.
- Sampling and Data collection:
Purposive sampling technique was employed to target the faculty members involved in item banking practices at the medical college of Riphah International University. Participation was voluntary and anonymous. The questionnaires were distributed through google forms for its feasibility and anonymity of the participants. The procedure of data collection occurred in two phases: Pre-Intervention Phase in which cross sectional survey was done to collect data on the challenges associated with current item banking. This data served as the baseline measurement for the study. A significant proportion of the population responded i.e., 45 out of 72, resulting in a response rate of 62.5%. After the survey, intervention was done in the form of introducing ETMS “Schoolhouse.” For intervention, various workshops were conducted to train the nominated faculty members from basic and clinical departments in the software and were given 3 months to use the software in their assessment processes. Post-Intervention Phase, following the intervention (i.e., the workshops and the use of the software for three months), data were collected through a satisfaction survey administered to the group of participants who underwent the intervention. The total population included in the satisfaction survey was 25. The data was analyzed by SPSS software.