A 360-degree evaluation of the professionalism and communication skills of technologists working in the radiology department of a public sector tertiary care hospital in Karachi, Pakistan

doi:10.21203/rs.3.rs-4355018/v1

Download PDF

Research Article

A 360-degree evaluation of the professionalism and communication skills of technologists working in the radiology department of a public sector tertiary care hospital in Karachi, Pakistan

https://doi.org/10.21203/rs.3.rs-4355018/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Radiologic technologists (RTs) are at the forefront of radiology departments and require optimal skills to demonstrate professionalism and effective provider–patient communication. Hence, the objective of this study was to determine the reliability of 360-degree evaluation as a feedback tool to improve competencies in this crucial workforce.

Methods: A planned 360-degree evaluation was conducted using a 10-item scale to evaluate communication skills and professionalism among radiological technologists working at Dow University Hospital, Karachi. In total, 311 evaluations of fifteen radiological technologists were conducted while performing various diagnostic radiology procedures at the radiology department of Dow University Hospital. The data were analysed using SPSS version 26 to compute the evaluation scores and the reliability of 360-degree evaluation.

Results: A total of 311 RT-patient interactions were recorded over 48 days, with scores from all three raters. The evaluation tool was found to have good internal consistency for patients, faculty, and RT, with Cronbach’s alpha values of 0.89, 087, and 0.74, respectively. The study found an intraclass correlation of 0.66 (95% CI; 0.58–0.72), showing moderate reliability of the 360-degree assessment across different evaluators.

Conclusion: We conclude that 360-degree evaluation is a valid and reliable tool for determining the professionalism and communication skills of radiologic technologists and should be incorporated into training programs for formative and summative assessments. However, large-scale multicenter studies are crucial for generalizing these findings and incorporating 360 evaluations in radiological technologist training programs for formative and summative assessments.

Radiological technologist

professionalism

communication skills

evaluation

Radiologic technologists (RTs) are at the forefront of diagnostic radiology departments and are constantly engaged in interactions with patients. They must excel in the professionalism and communication skills required for optimum service delivery and patient care (2). Their day-to-day responsibilities include verifying the patient’s identity and the procedure to be performed, taking a history, giving instructions regarding radiologic procedures, ensuring that the patient understands the instructions, addressing the patient’s concerns and queries, explaining postprocedural instructions, and coordinating patient care (3). However, the literature reports that some areas related to core competencies, such as professionalism and interpersonal communication, are still compromised among radiological technologists and need improvement. (4.5) Hence, there is increasing emphasis on improving professionalism and interpersonal communication in this essential healthcare cadre. (6–8) The 360-degree evaluation has emerged as a potential source for assessing performance and has been widely used for assessing core competencies among healthcare professionals from different cadres and with different sets of clinical skills. It allows simultaneous evaluation of performance by different evaluators, which provides an opportunity to obtain comprehensive feedback from evaluators with different roles in healthcare, from peers, and from self-evaluations. (9, 10). However, evidence is scarce regarding the reliability of 360-degree evaluation for evaluating radiological technologists. To the best of our knowledge, there is no published evidence regarding the reliability of 360-degree evaluation among radiological technologists. Determining the readability of this comprehensive evaluation method will help in the use of this method as part of a performance appraisal to identify areas of improvement to ensure continuous quality improvement in radiology services in related healthcare settings. Hence, this study aimed to determine the reliability of a 360-degree evaluation to evaluate professionalism and communication skills during RT-patient interactions.

This cross-sectional study was conducted at the Dow Institute of Radiology, Dow University of Health Sciences, from January 1, 2021, to December 31, 2022. All consenting adult patients (above 18 years of age) at the radiology department for fluoroscopy, CT scan, MRI scan, VIR or mammography were included in the study. The sample size for this study was calculated using a confidence level of 95%, a precision of 7%, and an anticipated intraclass correlation coefficient of 0.4 for interobserver reliability across three evaluators; we obtained a sample size of 311 observations. (11) In total, fifteen radiological technologists were recruited using a nonprobability convenient sampling technique. Each radiological technologist was evaluated by three different evaluators approximately 21 times to complete 311,360-degree evaluations. A total of 311 patients who underwent any radiology investigation as well as five faculty members from the Department of Radiology volunteered for this study and completed 311 evaluations as needed. The selected RTs were evaluated according to their conduct while interacting with patients, including introducing one’s self, addressing patients respectfully, explaining procedures, addressing patients’ queries and concerns, making patients feel comfortable and respecting their privacy. The 360-degree evaluation was performed using a structured assessment tool adapted from a previous study while considering the local sociocultural context and study objectives. Each individual item in the assessment tool was scored on a scale ranging from one to five. The assessment tool comprised 10 main domains or items where the initial four items were related to communication skills, while six subsequent items were related to professionalism. (12) The 360-degree RT assessment was conducted at three different levels or by three different evaluators simultaneously. A total of 311 RT—patient interactions were observed, and each interaction was assessed by a faculty member from the Department of Radiology, the patient who participated in that interaction and RT himself or herself. The selected RTs were briefed about the 360-degree evaluation by the principal investigator, and informed consent was obtained. Patients were recruited by applying a convenient sampling technique. In addition, before any RT-patient interactions were assessed, each participating patient was provided with a detailed orientation about the study objectives as well as the assessment processes by the principal investigator. Written informed consent was also obtained from the patients who agreed to volunteer for the study. The assessment proforma was translated into the Urdu language to enhance the understanding of the study participants, particularly the patients. For each assessment, a randomly assigned faculty member also evaluated each RT-patient interaction and completed the 360-degree assessment on a separate form. Finally, the RTs themselves evaluated their own performance using a third copy of the same assessment or evaluation form. Each assessment form had a unique coding number to maintain the confidentiality of the study participants. None of the findings about individual performance or assessment outcomes were shared with the relevant authorities at work, and the collected data were used only for research purposes. Ethical approval for this study was obtained from the Institutional Review Board (IRB) of Dow University of Health Sciences. After all the observations were completed, all the data were entered into Excel and later transferred to SPSS. The data were analysed using SPSS version 26. Descriptive statistics were calculated for the basic demographic characteristics of the study participants. Cronbach’s alpha was calculated to determine the reliability of the assessment tool or scale for each category of evaluators. The total score for the 10 items on the scale was calculated for each evaluation, and the mean score was then calculated for each type of evaluator. Median scores were compared across the categories of the evaluators by using mixed analysis of variance with random subject effects. The Kruskal‒Wallis test was applied to test the differences in mean scores according to rater type. Intraclass correlation was calculated between evaluators to measure agreement. (13, 14) A P value < 0.05 was considered to indicate statistical significance.

In total, 15 RTs participated in the study and volunteered for 360-degree evaluation. A total of 54% (n = 8) of all RTs were male, while 46% (n = 07) were female. 86.6% (n = 13) of the RTs were between 18 and 40 years of age, while 13.4% were 41 years of age or older; 33.3% of the RTs had a bachelor’s degree in radiology technique (BSRT), while 20% had a bachelor’s degree or a degree in a subject other than radiology. A total of 46.7% of the RTs in the study had an education level less than graduation. However, 93.3% (n = 14) of the RTs included in the study had work experience as a technologist in the same field for more than five years. The study evaluated 311 RT-patient interactions by performing a 360-degree assessment over a period of 48 days. (Table 1)

Table:1 Sociodemographic characteristics of the technologists working in the radiology department who participated in the 360-degree assessment of performance (n = 15)

Variable

Frequency

Percentage

Age (n = 15)

18–40 years

41 years and above

86.6

13.4

Sex

Male

Female

54.0

46.0

Education

BSRT

Graduate

Diploma

Intermediate

MSC

33.3

20.0

13.3

26.7

0.7

Work Experience

5 years or less

Greater than 5 years

6.7

93.3

Frequency of various procedures observed by evaluators (n = 311)

CT Scan

MRI Scan

Mammogram

Fluoroscopy

VIR

144

28.9

46.3

16.4

2.6

5.8

Each assessment was independently and simultaneously rated by three evaluators or examiners, i.e., the patient, faculty and the performer, i.e., the RT him or herself. On average, approximately 20 interactions were assessed for each technologist who participated in this study while performing different imaging procedures or radiology examinations. Among all 311 360-degree assessments, MRI was the most frequently observed procedure (46.3%, n = 144), followed by CT and mammography (28.9%, n = 90 and 16.3%, respectively) (n = 51). The study calculated the median scores of performance per the evaluator group.

The study revealed statistically significant differences in the median performance scores of the three types of evaluators (p value < 0.005). The median scores were 43 (IQR; 10), 40 (IQR; 8) and 44 (IQR; 6) for the patient, faculty and RT evaluations, respectively. The RT self-evaluation scores were greater than the median patient and faculty evaluation scores. Similar patterns were observed when scores were calculated for the initial four items on the assessment scale, which were related to the communication skills domain. However, in the domain of professionalism, the highest scores were given by the evaluator category, which was patients, followed by RTs and faculty. The study revealed good reliability of the assessment tool for each type of evaluator group, i.e., patient, faculty, and RT, with Cronbach’s alpha values o 0.89, 0.87, and 0.74, respectively. The study also found a moderate intraclass correlation of 0.66 (95% confidence interval; 0.58–0.72), which indicates a high level of agreement in the evaluation scores or responses across all three types of evaluators assessing the performance of RTs, and this finding was statistically significant (p value < 0.001). (Table: 2).

Table:2 Comparison of performance scores, inter-observer agreement and reliability of the 360-degree assessment among technologists working in the radiology department
	Intra-observer Agreement
Variable	Patient (n = 311)	Faculty (n = 311)	Technologists (n = 311)
Overall Scores	40.8 (8.7)	38.9(7.5)	42.0(6.7)	< 0.001
Scores for Professionalism	16(7.0)	14(5.0)	17(4.0)	<0.001
Scores for Communication skills	28(5.0)	26(6)	27(5)	<0.001
Cronbach’s Alpha for internal consistency	0.89	0.87	0.74	-
Intra-class Correlation Coefficient	0.66 (0.58–0.72)

The study also calculated the intraclass correlation coefficient for each item on the evaluation scale and found a moderate correlation between the responses of all three evaluators, with an intraclass correlation ranging from 0.44 to 0.67. The lowest scores were observed for the item related to communication skills, i.e., the technologist introducing himself/herself (Table 3).

Table:3 Item-wise inter-rater correlation in each domain across three evaluators involved in 360-degree evaluation of radio-technologists (n = 311)
Scale-item	Patient (n = 311)	Faculty (n = 311)	Technologists (n = 311)	Intra-class Correlation Coefficient (95% CI)
Scale-item	Median Score (IQR)			Intra-class Correlation Coefficient (95% CI)
Communication Skills
Technologist introduce himself/herself	3(3)	2(1)	3(2)	0.51 (0.40–0.60)
Clearly explain the risk and benefits of procedure	4(2)	3(1)	4(2)	0.58(0.47–0.66)
Technologist address patient`s queries	4(2)	4(2)	4(1)	0.50(0.40–0.59)
The information communicated was easy to understand	5(1)	4(1)	5(1)	0.44 (0.31–0.53)
Professionalism
The technologist address patient respectively	5(1)	4(1)	5(1)	0.58(0.49–0.65)
Technologist showed concern for patient`s` comfort	5(1)	4(1)	5(1)	0.51(0.10–0.59)
Technologist was sensitive to differences in gender/ language/ disability/ age	5(1)	4(1)	5(1)	0.53(0.43–0.61)
Patient`s privacy respected by the technologist	5(1)	5(1)	5(1)	0.46(0.35–0.56)
Technologist was knowledgeable about the procedure	5(1)	4(1)	5(1)	0.51(0.41–0.60)
Overall assessment of technologist.	5(1)	4(1)	5(1)	0.60(0.52-067)

This study provides valuable evidence from a public sector teaching hospital in Pakistan regarding the reliability of 360-degree evaluation and its potential use in healthcare settings. This study revealed good reliability of the assessment scale across all three groups of evaluators included in this study, which is consistent with previous evidence. (12, 15) The study compared the overall average of the evaluation scores on the selected scale, as rated by three different types, of evaluators. However, the highest scores were observed for self-evaluations as rated by RTs themselves. On the other hand, the lowest scores were assigned to the group of evaluators comprising faculty members from the same department. This discrepancy in the scores across the evaluator groups can have many explanations, including subjective interpretation of the items on a Likert scale and differences across the evaluators regarding knowledge and understanding of interpersonal communication as well as professionalism. Moreover, faculty might have evaluated RTs more stringently than patients because they were officially in a position to provide feedback to the RTs and because of their natural tendency as educators to critically appraise their performance as part of medical education processes. On the other hand, the patients might have given relatively better scores due to possible fear of any negative consequences for their treatment and investigation outcomes as a result of poorly rating the RTs. A slightly varying pattern was observed when average scores of evaluations were compared across the evaluators for the scale items assessing professionalism only where the highest average scores were reported by the patients, followed by RTs and faculty. The study also revealed moderate interrater reliability with a moderate intraclass correlation coefficient for the evaluation scores, indicating low variability and significant agreement in the evaluation scores obtained by the three different evaluators. This finding is in line with previous evidence from Pakistan supporting the reliability of 360-degree evaluations conducted by multiple raters or evaluators. A study was conducted at Aga Khan University, Karachi, Pakistan, in a cohort of Internal Medicine residents from all four training years between November 2009 and March 2010. Every resident (n = 49) was evaluated for interpersonal and communication skills by eight raters, including physicians, nurses, and unit staff. Each study participant was also supposed to provide a self-evaluation. The study calculated the reliability of those evaluations across the different evaluator groups. The study found a correlation coefficient of 0.39 with fair consistency of ratings across different evaluators. (15) Another 360-degree evaluation of sixteen emergency medicine residents in Turkey involved seven distinct categories of evaluators, including residents, faculty members, nurses, ancillary staff, unit clerks, paramedics and patients. The study included a total of 1088 evaluations and revealed high interrater reliability, i.e., > 0.7 for 360-degree evaluations across different types of evaluators. The evaluation scores given by nurses were significantly lower than those given by other evaluators, while the highest scores were given by the paramedics (16). However, the interrater reliability may widely vary across studies depending on the design and implementation of the evaluation tool and the type of evaluators. (11)

The current study also computed interrater reliability for individual items in the evaluation scale. The study found moderate reliability for all ten items in the scale, with intraclass correlations ranging from 0.44 to 0.67. However, the faculty reported the lowest average scores for two items related to communication skills. One of those items was related to introducing oneself to the patient, and the other was related to effectively explaining the risk and benefits of the procedure to the patients. Poor scores indicate a lack of satisfaction among faculty members regarding performance in communication skills. Interpersonal communication skills are believed to be the most sensitive indicators of performance in healthcare settings because formal education and technical knowledge are not required to assess flaws or deficiencies in interpersonal communication. In addition, interpersonal communication is a major source of showing healthcare providers’ empathy and responsiveness toward the patient; hence, any compromise in the provider’s interpersonal communication can be identified easily. A retrospective analysis of patient complaints evaluated all radiology-related complaints received by the Salazar institution's Office of Patient Advocacy from April 1999 to December 2010. The qualitative analysis of patients’ complaints revealed that a considerable proportion of complaints were related to a lack of patient-centred care and poor interpersonal skills of radiology staff members, identifying these as major areas of improvement (17). Similarly, a previous study conducted at the radiology department of Fayoum University Hospital (FUH), Egypt, in Oct-Dec 2016 assessed the quality of healthcare and patient satisfaction with the intent of highlighting areas for improvement. The study revealed that a lack of explanation of the radiological procedure to the patient was a major factor resulting in patient dissatisfaction with radiological services. (18) Another study conducted in a public and private hospital in Southeast Nigeria revealed that the professionalism of radiographers influences patients’ satisfaction and dissatisfaction with radiology services. (19) In the current study, the lowest rating for introducing oneself to the patient was also consistent with the evaluation by the technologist themselves, as well as the evaluation by the patients. However, poor ratings for the abovementioned items in our study can be explained by the increased workload due to high patient turnover, resulting in a shortage of time for adequate communication with patients.

This study has several strengths. It evaluated the performance of RTs for a variety of procedures with approximately 20 observations for each performer. Hence, it provided a holistic view of the performance while implementing various procedures involving different machines and imaging protocols. This variety in observation while assessing a performer has resulted in high consistency in the assessment. Moreover, all the assessments were conducted using a previously tested scale, which facilitated the comparison of the study findings with previous evidence.

Nevertheless, this study has certain intrinsic limitations. First, the participants who were observed, i.e., RTs, were aware of the situation. Hence, the knowledge that they are being observed might have resulted in deliberate change or improvement in their usual practices or performance. Hence, the Hawthorn effect is an inherent limitation of the 360-degree evaluation approach, which might have affected the results of the present study as well. This limits the validity of 360-degree evaluations, especially for behavioural outcomes. Second, despite prior assurance, the scores of evaluations conducted by patients might be affected by their fear of compromising treatment or service quality and outcomes in the long run if they give low scores to the service provider, i.e., RT, in this study. Third, it was a single-center study, and the participants included were RTs only; hence, generalizability to other cadres of healthcare providers and technologists in other specialties requires caution. Furthermore, the use of a five-point Likert scale for this evaluation might have affected the study results due to differences in the subjective interpretation of the scale items as well as a natural tendency to avoid extreme options on a Likert scale.

The 360-degree evaluation is a reliable method of performance evaluation and has high interrater reliability for evaluating communication skills and professionalism among radiological technologists when evaluated simultaneously by three different types of evaluators. It supports the application of 360-degree assessment as a feedback tool for continuous quality improvement in healthcare settings. However, large-scale multicenter studies are crucial to generalize these study findings and to incorporate 360-evalauation in radiological technologist training programs for formative and summative assessments.

Radiologic Technologist

Ethics approval and consent to participate
Consent for publication
Availability of data and materials
Competing interests
Funding
Authors' contributions
Acknowledgements

Ethics approval and consent to participate

The Ethics Committee of Dow University of Health Sciences approved the study (Ref Number: IRB-2461/DUHS/Approval/2022/981).

Informed consent was sought from all study participants.

Consent for publication

Not applicable

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests

Funding

No funding was received for this research

Authors' contributions

NR was involved in the conception and design of the research, ethics committee approval, data analysis and final write-up of research article.

NN conceived and planned the research, analyzed data, contributed to writing discussion and proofread the article.

RM, J, KK and H planned the research, collected data, contributed to writing introduction and results of the article

Acknowledgements

I would like to acknowledge the Radiologic technologists, faculty, and patients who participated in the study.

Rawson JV, Kitts AB, Carlos RC. Patient- and Family-Centered Care: Why Radiology? J Am Coll Radiol. 2016 Dec; 13(12):1541-1542. doi: https://doi.org/10.1016/j.jacr.2016.09.041.
Watson L, Odle TG. The technologist’s role in patient safety and quality in medical imaging. Radiologic Technology. 2013 May 1; 84(5):536-41.
Itri JN. Patient-centered Radiology. Radiographics. 2015 Oct; 35(6):1835-46.doi: https://doi.org/10.1148/rg.2015150110.
Sarah J, Nelly L. Radiological technologists’ self-assessment of professional competence in Japan. A survey study. Japan Journal of Research. 2020 Dec; 17; 1(4):1-11.
Lim CS, Lee YS, Lee YD, Kim HS, Jin GH, Choi SY, Hur Y. The job competency of radiological technologists in Korea based on specialists opinion and questionnaire survey. J Educ Eval Health Prof. 2017 May 11; 14:9. doi: https://doi.org/10.3352/jeehp.2017.14.9.
Haynes KW. The importance of professional values from radiologic technologists’ perspective. Radiol Technol. 2020 Jul 1; 91(6):525-32.
Bwanga O. Teaching Professionalism to Radiography Students in the Diagnostic Imaging Department. South Asian Res J App Med Sci. 2019 July-August; 1(1):12-15.
Kruskal, Jonathan & Eisenberg, Ronald & Ahmed, Muneeb & Siewert, Bettina. Ongoing Professional Practice Evaluation of Radiologists: Strategies and Tools for Simplifying a Complex Process. Radiographics. 2018; 38:1593-1608. doi: https://doi.org/10.1148/rg.2018180163.
Hosain S. 360 Degree Feedback as a Technique of Performance Appraisal: Does it Really Work? Asian bus. rev. 2016 Feb 14; 6(1).
Donnon, Tyrone PhD; Al Ansari, Ahmed MBBCh, MRCSI, PhD; Al Alawi, Samah MD; Violato, Claudio PhD. The Reliability, Validity, and Feasibility of Multisource Feedback Physician Assessment: A Systematic Review. Acad Med 89(3):511-516, March 2014. doi: https://doi.org/10.1097/ACM.0000000000000147.
Chandler N, Henderson G, Park B, Byerley J, Brown WD, Steiner MJ. Use of a 360-degree evaluation in the outpatient setting: the usefulness of nurse, faculty, patient/family, and resident self-evaluation. J Grad Med Educ. 2010 Sep; 2(3):430-4. doi: https://doi.org/10.4300/JGME-D-10-00013.1.
Wood J, Collins J, Burnside ES, Albanese MA, Propeck PA, Kelcz F, Spilde JM, Schmaltz LM. Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol. 2004 Aug;11(8):931-9.doi: https://doi.org/10.1016/j.acra.2004.04.016.
Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: Measures of agreement. Perspect Clin Res. 2017.Oct-Dec; 8(4):187-191. doi: https://doi.org/10.4103/picr.PICR_123_17.
Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016 Jun; 15(2):155-63. doi: https://doi.org/10.1016/j.jcm.2016.02.012.
Tariq M, Boulet J, Motiwala A, Sajjad N, Ali SK. A 360-Degree Evaluation of the Communication and Interpersonal Skills of Medicine Resident Physicians in Pakistan. Educ Health. Sep–Dec 2014; 27(3): 269-276, doi: https://doi.org/10.4103/1357-
Oktay C, Senol Y, Rinnert S, Cete Y. Utility of 360-degree assessment of residents in a Turkish academic emergency medicine residency program. Turk J Emerg Med. 2017 Mar 1; 17(1):12-5. doi: https://doi.org/10.1016/j.tjem.2016.09.007.
Salazar G, Quencer K, Aran S, Abujudeh H. Patient satisfaction in radiology: qualitative analysis of written complaints generated over a 10-year period in an academic medical center. J Am Coll Radiol. 2013; 10:513–517. doi: https://doi.org/10.1016/j.jacr.2013.03.013
Wafaa Yousif Abdel Wahed, Shaimaa Elsayed Mabrook. Assessment of patient satisfaction at Radiological Department of Fayoum University Hospitals. IJMDC. 2017; 1(3): 126–131. doi: https://doi.org/10.24911/IJMDC.1.3.2
Ochonma OG, Eze CU, Eze SB, Okaro AO. Patients’ reaction to the ethical conduct of radiographers and staff services as predictors of radiological experience satisfaction: a cross-sectional study. BMC Med Ethics. 2015 Dec; 16(68):1-9. doi: https://doi.org/10.1186/s12910-015-0062-4.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

A 360-degree evaluation of the professionalism and communication skills of technologists working in the radiology department of a public sector tertiary care hospital in Karachi, Pakistan

Status:

Version 1

Abstract

Introduction

Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

Consent for publication

Not applicable

Availability of data and materials

Competing interests

The authors declare that they have no competing interests

Funding

No funding was received for this research

Authors' contributions

NR was involved in the conception and design of the research, ethics committee approval, data analysis and final write-up of research article.

NN conceived and planned the research, analyzed data, contributed to writing discussion and proofread the article.

RM, J, KK and H planned the research, collected data, contributed to writing introduction and results of the article

Acknowledgements

I would like to acknowledge the Radiologic technologists, faculty, and patients who participated in the study.

References

Additional Declarations

Status:

Version 1