The main focus of the PATS tool was that it should be an indigenous tool, addressing the domains of Professionalism identified in the framework of ABIM in the local context. The purpose was that students could improve their professionalism as they graduated from medical college.
Key findings
This four-phase study identified domains of professionalism similar to those of the American Board of Internal Medicine. Characteristics identified are fairly similar to those found in other instruments for assessing professionalism, indicating that these are universal characteristics of medical professionalism which are useful in teaching and assessing for learning in the local context.
The construct of respect was observed to be most important by the participants in the local context. They were of the view that it should reflect in the overall approach of a clerkship student towards patients. This element is of utmost priority in south-Asian culture. Duty should be seen in students’ activities, such as completing assignments on time. Effective communication skills are very important and are needed when dealing with peers, patients and seniors. The elements of Honor and Integrity should be displayed by the medical students as future doctors in their interactions and especially in high-pressure situations. In regard to excellence, it was agreed that the students need to maintain a proper dress code. They should not misbehave as they are representing a respectable profession. The construct of accountability was perceived as the least important.
Relevance to similar tools in the literature
The domains of professionalism have been identified in the literature. However, the study done by Nath did not specify domains.15 In this study the items were developed in a more methodical manner. The study by Nath, one of the initial studies carried out on professionalism, is a survey. That study compared the perceptions of professionalism among different professional groups. The survey had 29 circumstantial statements describing behaviors of professionals on components such as respect, accountability, honesty and integrity, empathy self-directed learning, skillful communication etc. In the present study, a focus group was conducted by the author to develop the items on the ABIM domain of professionalism, then they were validated through modified Delphi rounds. The developed tool was cognitively pre-tested and then it was piloted on a large sample of 1511 respondents. The study had three scales to rate the behaviors, but the present study has five scales to rate behavior.
Comparing the present study with other such studies, the objective was clear from the start: to develop a tool that would be able to assess professionalism in the local context. A tool developed by Al-Eraky in the Arabian context, though in a Muslim context, was different in the perspective that the tool was for the assessment of professionalism of medical students and interns.16The developed tool in the present study is not intended to be used for the assessment of professionalism in interns or postgraduate trainees.
One of the methods for assessment of professional behavior is situational judgment tests which are personnel selection instruments that present junior doctors with clinical scenarios and possible responses to the situation. These have two types of instructions related to behavioral tendency and knowledge and the respondents are asked to evaluate the effectiveness of possible responses in a given situation. This method is a type of test, whereas the present instrument can be used by the faculty, patients or peers to assess the behavior of the student.7
The present study is in accordance with a study carried out for the development of a tool to assess professionalism in the Taiwanese context. That study was also carried out on 7th year senior medical students in Taiwan by Tsai and a 34-item tool was developed.17 The current study developed a 27-item tool. The reference panel was not specified in the study by Tsai; however, the current study involved a panel of 9 participants for the focus group discussion. The methodology for piloting is not clearly given in Tsai’s study, and the number of students on which it was piloted on was 133. In contrast, in the present study, the tool was first of all evaluated through pre-cognitive testing and then modifications were made and then it was pilot tested on 228 students.
The Penn State Professionalism Questionnaire has 36 items and was developed with four parallel forms (Medical Students; Residents; Basic faculty and clinical faculty).19 The present tool targets the clerkship undergraduate medical students for feedback purposes.
Content validity of the tool
The content validity of the tool was calculated during and after the three modified Delphi techniques were performed. Content validity index (ICV-I), Content validity scale (S-CVI) and Content validity universal agreement (I-CVI/Avg) were determined and I-CVI/Avg was 0.94. This aspect of the study is in accordance with a tool development study to evaluate the communication skills aspect of professionalism by Zamanzadeh Vahid et al in 2015.19 The S-CVI in that study was 0.93.18 The calculated values of the final tool of the present study were I-CVI/Avg (0.94) and I-CVI/UA (0.63) which is within the acceptable range.
For the clinicians and researchers, establishing the content validity of a tool is important.19 If it is not suitable then examining the behavior and attitude might not be reflective of the target population and the socio-cultural background.
Construct validity of the tool
In the present study there were six domains around which two to eight scales were developed. Most of the studies that have been conducted for the development of a professionalism tool have not included a confirmatory factor analysis. 21
Adequacy of a theoretical model is judged in terms of how well it fits the observed data statistics which are: x2/d, CFI, TLI, RMSEA and SRMR. 22These indices are used to determine the assessment of fitness indices and the researcher's theoretical justification for fitness to practice. Statistically, it is not appropriate to reject the tool. The RMSEA was 0.05, which suggests a good fit as the cut off value is suggested at 0.06 values are good, indicating it is one of those indexes which are least affected by the sample size. The x2/d is less than 5, suggesting a good model fit. The basic rule of thumb is that the fitness indices CFI and TLI should be 0.90. 23 These are incremental fitness indices that compare the proposed model with a null model. The absolute fitness indices suggest an adequate fit for the developed instrument for the assessment of professionalism. The author reports indices of 0.76 and 0.73 for CFI and TLI in the present study. Weak correlations among observed variables might be a cause of this. The results for RMSEA and CFI in the present study are inconsistent as the two indices disagree. This might be because they evaluate by design the magnitude of the model’s fit function from different perspectives so the values can differ. Moreover, the cut off values for theses indices are arbitrary.24
Upon comparing the CFA results of the present study with a study by Maat et al, a good model fit can be seen. However, that study was in the learning environment of high school students and had a sample size of 1887 which might be the reason for high CFI and TLI in that study.25
Reliability
The alpha reliabilities of most factors were good, except for altruism which was low but acceptable.26 One of the main reasons might have been in the understanding of the concept of altruism. The causes could have been in the translation of the concept of altruism identified during the focus group discussion.
Practical Implications
The methodological micro-scenario statement can help regulate the professional aspect of the behavior and attitude of the students towards better patient care through feedback at the undergraduate level, which improves the professionalism of the students. It can be modified and used to assess the professionalism of the students in basic sciences in the earlier years of medical training.
Study limitations
Limitations that needs mentioning is the sample size was small for the piloting of the tool and due to time constraints, longitudinal follow-up was not possible. As the data was collected in one institution generalizability of the tool is not well established.
Recommendation
The tool should be further piloted with increased sample size for context validation of the developed tool. To ensure generalizability it needs to be tested in other medical institutes of Pakistan. Outcome studies can be conducted to examine the effect of the developed tool on the professionalism of the students after its intervention.