Development of PATS: A tool to assess professionalism

Background: Professionalism has become a specic requirement for the medical graduates because of dissatisfaction regarding their preparedness to adequately deal with the challenges of the medical world. It should be assessed as a competency in the nal year of medical students through a valid and reliable instrument. The purpose of the study was to develop and validate an instrument to assess the development of professionalism in nal year medical students in the local context to assess the readiness off medical graduates. Method: This was a mixed method study with sequential qualitative and quantitative components.Micro scenarios for the preliminary instrument were developed after focus group. These were validated through 3 round modied Delphi technique by emails. A total of 24 participants responded for the rst round, 15 for the 2nd and 3 rd rounds. After rst round relevance of the item was obtained. Content validity ratio was calculated after second round. Cut off value for items was appropriated to be 0.6, 0.73 for modication and 0.8 for item retention. Second round analysis for revision of the opinions were send to participants for third round and content validity index average and universal agreement were performed. Cognitive pretesting was done. Cronbach’s Alpha for reliability and Conrmatory factor analysis was carried out after piloting. Results: Focus group resulted in 46 micro scenarios through manual qualitative analysis. Modied Delphi technique was used for content validation. First round of Delphi resulted in 35 items. After second round Content validity ratio was calculated. Twelve items were retained, 18 revised and 5 were removed. Content validity index (I-CVI) and content validity scale (S-CVI) was 0.94 and 0.64 respectively. Pre-cognitive testing resulted in modication of 4 items. Conrmatory factor analysis was 4.1. Cronbach’s Alpha was 0.96. Conclusion: The nal instrument developed is a 27 item 5-point Likert scale. It has good content validity, reliability and acceptable construct validity. The strength of the instrument was in its process of development, which was focus group discussion, the iterative Delphi rounds, cognitive pre-testing and piloting. It can be used to assess the development of professionalism in the students of nal year medical students.


Background
Professionalism for a graduating student has been de ned as "commitment to carrying out professional responsibilities, adherence to ethical principles and sensitivity to diverse patient population" by the Accreditation Council for Graduate Medical Education, USA. 1,2 Thus the competency implies commitment to the patients to fellow professionals bridging the gap between science and society. 3 The lapses in professionalism of medical students pose a challenge to medical profession. 4 The six domains of professionalism that the graduating doctor must show are: respect, accountability, commitment to excellence, altruism, integrity and compassion. 5 An important aspect of the undergraduate medical training is that it exposes the students to the demands of real-life practice through a foundation of clinical competence, communication skills, and ethical understanding, upon which are built the domains of professionalism. 6 While the teachers can foster the development of professionalism, a student's personal as well as environmental factors also play a role in the attainment of professionalism as a competency. 4 Evidence from the literature suggests that professionalism should be deliberately assessed as assessments motivates the individuals to learn. Tools assessing professionalism should bring together clinical competence, communication skills in medical practice in the local context. 6 In this context, it is important for the local institutions to de ne professionalism and develop tools to assess them according to their own culture and circumstances. 7,8,9 This study aims to develop a tool which is reliable, valid and applicable for assessment of professionalism in the undergraduate local medical context.

Method
This mixed method study with sequential qualitative and quantitative components was conducted in which aimed to get an insight in to the professional competence of a graduating medical student to measure the continuum of performance. The questions were validated by medical educationists.
Phase 2: Focus group Discussion and preliminary tool development Nine participants who were certi ed medical educationist were selected. They had experience in curriculum and in development of assessment of undergraduate medical students. The recorded data was transcribed and analyzed manually by open coding to de ne concepts and categories. A preliminary tool with 5-6 item statements was developed. Phase 3: Three round modi ed Delphi technique 12 was employed for consensus on item statements. Delphi Round 1: Forty clinicians supervising nal year medical students from Rawalpindi region were selected. A description of the project and a request to participate in the internet-based validation process through Google forms software was send.
Data collection method: The preliminary tool had 46 statements on a ve-point Likert scale for relevance. Twenty-four experts responded in the rst round. Relevance was rated on scale of 5 with 1 being "Not relevant" to 5 being "Highly relevant". Participants were given a period of 3 weeks for their responses.
Data analysis: Percentage responses and median score for each item were calculated as ordinal data and comments given by the panelists noted. A modi ed tool was prepared on Google forms software in light of the feedback.
Delphi Round 2: The tool was recirculated to the participants, who were asked to assess how each of the mentioned elements is essential in the professional behavior of the medical student's local context. The "Content Validity Ratio"(CVR) was calculated for retention/revision or removal of the items by the formula: 13 CVR = (Ne-N/2)/ (N/2) Delphi Third round: The expert panelists were sent a summarized rating of the items of the tool from the previous round.
They were asked to revise their judgment or to specify the reason if they wanted to abstain. A 4-point ordinal rating was employed to avoid a neutral and uncertain midpoint.
Calculation of "Content Validity Index" I-CVI for the individual tool items was calculated. "Content Validity Scale" S-CVI was also calculated for the constructed tool. S-CVI/UA was calculated by the number of items rated relevant by all experts combined and then divided by the total number of rating. 13 Phase 4: Cognitive pretesting and Piloting of the prepared tool Phase 4 a: Cognitive pretesting A 27-item tool was nalized for Pre-cognitive testing. Ten faculty members were randomly selected through convenience sampling. Cognitive interviews were done after informed consent in a quiet and closed room.

Data collection method:
Interviews were conducted with concurrent verbal probing. The participants were asked to: Read out the tool items.
Repeat the questions in own words.
Clarify any speci c term in the items of the tool found unclear.

Data analysis:
The data was analyzed manually by using pre-determined coding criteria to analyze the responses of the respondents. Most of the responses paralleled with the ndings of the phase 3. However, 4 items were modi ed. Cronbach's alpha: The internal consistency of the nal tool with its subscale was calculated by SPSS 22 employing Cronbach's alpha for reliability.

Results
The focus group resulted in the development of micro scenario for the tool. It was validated through modi ed Delphi technique, the rst round resulted in 35 item statements. The content validity ratio was calculated for second round for every item. Items with CVR of 0.6 were deleted. Those items with score of 0.73 were modi ed and resend to the experts for revision/retention and those items having 0.8 score and above were retained without any change. Twelve items were retained, 18 revised and 5 were removed. After 3rd Delphi round, Content validity scale Average (S-CVI/Ave) and Content validity scale/Universal agreement (S-CVI/UA) was calculated shown in Table 1. Table 2 shows overall t indices of the tool. Squared distribution with degree of freedom (x2/df) was less than < 5, Cronbachs alpha was as shown in Results of the data after piloting of the tool for construct validity and reliability of the tool are given in Table 3. The table shows high internal consistency of 0.96 despite the fact that the tool had multifaceted elements. The inter-item correlation was high it ranged from 0.71to 0.933. The gure 2 shows the nalized measurement model of the tool obtained from M-plus software after considering statistical t index tests.

Discussion
The main focus of the developed tool called PATS was that it should be an indigenous tool, addressing the domains of Professionalism identi ed in the framework of ABIM in the local context. With the purpose that students can improve their professionalism as they graduate from medical college.

Key ndings
This four-phase study identi ed domains of professionalism similar to those of the American Board of Internal Medicine. Characteristics identi ed are fairly similar to those as in other instrument for assessing professionalism. Indicating that these are universal characteristic of medical professionalism which are useful in teaching and assessing for learning in the local context.
The construct of respect was observed to be most important by the participants in the local context. They were of the view that it should re ect in the overall approach of a clerkship student towards patients. This element is of utmost priority in south-Asian culture and especially Pakistan with different ethnic population but a common religion which is Islam which can guide and in uence the behavior of doctors.
Duty should be seen in students' activities such as completing assignments on time. Effective communication skills are very important and are needed when dealing with peers patients and seniors. The element of Honor and Integrity should be displayed by the medical students as future doctors in their interactions and especially under high -pressure situations during which it may be di cult to exhibit. In regard to excellence it was agreed that the students need to maintain a proper dress code. They should not misbehave as they are representing a respectable profession. Whereas, the construct of accountability was perceived as the least important.

Relevance to similar tools in the literature
The domains of professionalism have been identi ed in the literature. However, the study done by Nath did not specify domains. 14 Whereas for the present study the items were developed in a more methodical manner. The study by Nath, one of the initial studies carried out on professionalism, is a survey. That study compared the perceptions of professionalism among different professional groups. The survey had 29 circumstantial statements describing behaviors of professionals on the components such as respectful, accountability, honesty and integrity, empathy self-directed learning skillful communication etc. In the present study, a focus group was conducted by the author to develop the items on the ABIM domain of professionalism, then they were validated through modi ed Delphi rounds. The developed tool was cognitively pre-tested and then it was piloted on a large sample of 1511 respondents. The study had three scales to rate the behaviors, but the present study has ve scales to rate.
Comparing the present study with other such studies the objective was clear from the start which was to develop a tool that would be able to assess professionalism in the local context. A tool developed by Al-Eraky in the Arabian context though in a Muslim context was different in the perspective that the tool is for the assessment of professionalism of medical students and interns. 15 The developed tool in the present study is not intended to be used for the assessment of professionalism in interns or postgraduate trainees.
The present study is in accordance with a study carried out for the development of a tool to assess professionalism in the Taiwanese context. That study was also carried out on the 7th year, senior medical students in Taiwan by Tsai and a 34-item tool was developed. 16 The current study developed a 27 item tool. For the tool development, the reference panel is not speci ed in that study, however, the current study involved a panel of 9 participants for the focus group discussion. The methodology for piloting is not clearly given in that study, and the number of students on which it was piloted on was 133. In contrast, in the present study, the tool was rst of all evaluated through pre-cognitive testing and then modi cations were made and then it was pilot tested on 228 students.
The Penn state professionalism Questionnaire has 36 items and was developed with four parallel forms (Medical Students; Residents; Basic faculty and clinical faculty). 17 While the present tool is targeting the clerkship undergraduate medical students for feedback purposes.

Content validity of the tool
The content validity of the tool was calculated during and after the three modi ed Delphi technique. Content validity index (ICV-I), Content validity scale (S-CVI) and also Content validity universal agreement was determined which was 0.94. This aspect of the study is in accordance with a tool development study to evaluate the communication skill aspect of professionalism by Zamanzadeh Vahid et al in 2015. 18 The S-CVI in that study was 0.93.18 The calculated values of the nal tool of the present study were of I-CVI/Avg as was 0.94 and I-CVI/UA was 0.63 which is within the acceptable range.
For the clinicians and researchers establishing the content validity of a tool is important. 19 Because if it is not suitable then examining the behavior and attitude might not be re ective of the target population and the socio-cultural background.

Construct validity of the tool
In the present study there were six domains around which two to eight scales were developed. Most of the studies that have been conducted for the development of a professionalism tool con rmatory factor analysis has not been performed. 20 Adequacy of a theoretical model is judged in terms of how well it ts the observed data statistics which are: x2/d, CFI, TLI, RMSEA & SRMR. 21 These indices are used to determine the assessment of t indices and the researcher's theoretical justi cation for tness to practice. To justify the results, the t index needs to be discussed. The basic rule of thumb is that the t index CFI and TLI should be 0.90. 22 The author reports indices of 0.76 and 0.73 in the present study. Statistically, it is not appropriate to reject the tool. The relative chi-squared value is 0.5 which is at an acceptable range, the reason that the RMSEA values are good. Although the x2/d is less than 5, suggesting a good model t. The low CFI and TLI are explained by the less sample size which was 228. Upon comparing the CFA results of the present study with a study by Maat et al, a good model t can be seen. But that study was on the learning environment of high school students and had a sample size of 1887 which might be the reason for high CFI and TLI in that study. 23

Reliability
The alpha reliabilities of most factors were good, except for altruism which was low but acceptable. 24 One of the main reasons might have been in the understanding of the concept of altruism. The causes could have been in the translation of the concept of altruism identi ed during the focus group discussion.

Practical Implications
The methodological micro-scenario statement can help regulate the professional aspect of the behavior and attitude of the students towards better patient care through feedback at the undergraduate level improving the Professionalism of the students. It can be modi ed and used to assess the professionalism of the students in basic sciences in the earlier years of medical training.

Study limitations
Limitations that needs mentioning include a small sample size for the piloting of the tool and due to time constraints, longitudinal follow-up was not possible. As the data was collected in one institution generalizability of the tool is not well established.

Recommendation
The tool should be further piloted with increased sample size for context validation of the developed tool.
To ensure generalizability it needs to be tested in other medical institutes of Pakistan. Outcome studies can be conducted to examine the effect of the developed tool on the professionalism of the students after its intervention.

Conclusion
The nal tool developed is a 27 item, 5 point Likert scale. Strength of the tool was in its process of development, which was on the focus group, the iterative Delphi rounds, cognitive pre-testing and piloting. The tool has good face and content validity. The calculated CFA provides evidence of reliability and construct validity. It is hoped that through the application of this tool in the clinical setting professionalism of the students can be assessed for the betterment of the patient care.