Reliability and Validity of PARENTS Tool for Assessment of Pediatric Residents’ Non- Technical Skills in a Tertiary Care Teaching Hospital of Pakistan

doi:10.21203/rs.3.rs-2094043/v1

Download PDF

Research Article

Reliability and Validity of PARENTS Tool for Assessment of Pediatric Residents’ Non- Technical Skills in a Tertiary Care Teaching Hospital of Pakistan

https://doi.org/10.21203/rs.3.rs-2094043/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Non-Technical Skills (NTS) of post graduate residents (PGR) are as important as their clinical skills. Other than direct observation of supervisors, the feedback of caregivers provides an objective assessment of medical residents’ ability to effectively deal with patients. Our research objective was to test the validity and reliability of an instrument named “PARENTS Tool” in a local environment and its use to evaluate NTS of residents at the hospital.

Methods: This multi-phase mixed-method study used an Instrument Design Model called PARENTS Tool to assess NTS of residents. The PARENTS tool was translated into Urdu for our local environment and administered in the emergency department of Children’s Hospital and The University of Child Health Sciences.

Results: The authors collected data from 471 parents and coded them for analysis in AMOS and SPSS. Both tests met acceptance criteria to establishes the validity and reliability of the Urdu version of PARENTS tool. Results showed that parents were satisfied with the knowledge and skill of residents but less satisfied with the patience or empathy that the residents showed towards the children under their care.

Conclusion:PARENTS has shown to be valid and reliable tool for measuring the NTS of residents from the patient’s perspective. Medical administrators can use PARENTS as a performance measurement tool for monitoring the soft side of patient care.

Assessment

Feedback

Non-Teaching Skills (NTS)

PARENTS

Paediatric Residents Performance

In medicine, the treatment/medication provides the physical cure while counselling helps to address the psychological aspects of the malady. Patients who have been counselled properly are more likely to follow the treatment plan and hence show better outcomes. A recent study by Sajjad et al. (2020) in a TB program in Karachi showed improvement in treatment compliance when doctors properly counselled their patients.¹

Particularly in case of paediatric patients, skills are required for listening to the patients and their caregivers, explaining the treatment being given, answering their questions, and giving post-discharge instructions. Effective communication is based on trust between patients and physicians – and this requires developing patience, empathy, and clarity on part of the treating doctor.^2,3 In a public sector healthcare facility setting, where patient load is high, doctors can lose patience with the parents and vice versa which can adversely impact outcomes.

An important part of Post Graduate Resident (PGR) training is to inculcate values and skills needed for effective communication with caregivers and patients. A supervisor can directly observe the soft skills of the PGRs, but this has its own limitations.⁴ It is important to learn how PGRs deal with parents or patients when they are not being observed by a supervisor.

Parents are considered as parallel stakeholders along with patients, health care providers, and medical institutes in Paediatric care facilities, hence they can play a key role in assessing the Non-Teaching Skills (NTS) of PGRs.⁵ Parents feedback can encourage self-improvement of residents’ behaviour with parents and patients. Residency Programme in Paediatric practice (both medical and surgery) can benefit from the feedback of parents of patients as a part of multisource feedback (MSF).⁶ Important sources of feedback on residents’ performance can be from peers, supervisors/seniors, paramedics, paraprofessionals, and parents of patients.⁷

One of the difficulties in getting feedback from parents that it is not feasible to conduct long interviews – and even if qualitative interviews are conducted it would then be a complex task to convert the narrative into measurable scales for rating the interaction of PGRs.⁸

The paediatric emergency department is a good place to develop and practice non-technical skills. However, evaluating NTS is especially challenging for supervisors in the Emergency Departments (EDs) due to several factors including high patient loads, time constraints, and administrative commitments. The use of objective, easily understood and comprehensive tool for getting feedback from parents is needed to get useful feedback from parents. Such a tool can allow parents to assess residents NTS based on their direct interaction with PGRs at discharge.

The current study was done to identify and validate an assessment tool and use it for getting feedback directly from parents of paediatric patients. This tool would provide assessment of cognitive, social, and personal skills that work together to contribute to safe, efficient, and high-quality health care.

This multi-phase mixed-method study used an Instrument Design Model. We reviewed literature to identify an instrument that can be used for assessing NTS of residents and selected the PARENTS tool. PARENTS tool is a 20-item questionnaire including close ended and open-ended questions. The PARENTS tool was developed in Canada in English language. For our local environment we followed a translation protocol to translate PARENTS into Urdu. The translated instrument was pilot tested after getting ethical approval from the hospitals’ Institutional Review Board (IRB). The study was conducted at the Paediatric Emergency Department of the Children’s Hospital and University of Child Health Sciences Lahore in both medical and surgical sections from March 2021 to April 2021. The two-week time window for data collection was set to ensure minimum variability in the data collection environment. The parents of the patients admitted to the paediatric emergency department were interviewed with written consent (voluntary basis) by two enumerators and asked to fill the translated PARENTS tool-based questionnaire in Urdu. The collected data was entered and analysed in computer software SPSS (Statistical Package for Social Sciences) version 22.0 and AMOS. For validity we used AMOS to conduct Confirmatory Factor Analysis and for reliability we used SPSS for Cronbach’s Alpha test. Descriptive statistical analysis was performed on all items except open-ended items (Item 11 and 12) to determine the mean, standard deviation, kurtosis, and skewness.

A total 471 respondents participated in the study – they reported on 80 PGRs at the Paediatric Emergency Department of the Children’s Hospital and University of Child Health Sciences Lahore. The age of patients ranged from neonates to 148 months (school-going). Interestingly, there were significantly more male patients (59.4%) than female patients (40.6%). 47.1% patients were interviewed from medical emergency and 52.9% from surgical emergency.

The initial and final diagnoses of patients were recorded which showed significant skew towards severe disease (Mild= 27.6%, Moderate= 32.9%, Severe= 39.5%). We found that the majority of PGR (90.7%) failed to introduce themselves properly to the parents. (Table 1). Eighty-three percent of PGRs had not identified themselves as residents during their interaction with the parents. (Table 2)

Parent’s feedback coded from Item 11 (What can the resident do to improve his/her interactions with caregivers and their children?) suggested that the PGRs could improve their interaction by talking politely, giving more time, showing more patience, and providing more details on the treatment of their child. (Table 3) Parents' feedback on Item 12 (Please use the space below to provide additional comments on the resident’s skills when interacting with you and your child?) was coded in terms of their general satisfaction with the behaviour of the PGR. (Table 4)

Cronbach’s alpha analysis for 18 items (excluding item 11 and item 12) was done in SPSS we got a value of 0.884 which is considered highly reliable. A descriptive analysis of the computed PARENTS scale had a mean of 38.397 and median of 37.00 which shows values centred on Satisfactory for an overall rating by parents of paediatric patients. Skewness (.481) and Kurtosis (0.53) are within range for a normal distribution.⁹ (Table 5)

We used IBM AMOS tool for conducting the CFA analysis. AMOS showed that the model met all the criterion for acceptable fit with data. (Figure 1)

To assess for validity, we used Bi-variate Pearson Correlation between the 18 items of the PARENTS scale. All the items’ correlations were significant at the 0.01 p-value with high Pearson's correlation scores which met the conditions of internal validity of scale. (Table 6)

The mean value of items in the PARENTS scale provided us with important insights into the NTS of PGRs. Classification of satisfaction based on feedback from parents for the items falls into three bands – More than Satisfactory, Satisfactory, and Less than Satisfactory. Five items (3, 4, 13, 14 & 15) were rated as more than satisfactory, six items (2, 5, 7, 8, 16, and 17) as satisfactory and seven items (1, 6, 9, 10, 18, 19 & 20) as less than satisfactory.

The development of NTS is an important part of the post graduate clinical training of PGRs. In developing NTS, feedback plays an important part. The primary source of feedback is through the direct observation of the residents by their supervisors. In recent years medical educationist have pointed to the importance of incorporating patient’s voice as an important input in enabling the effective delivery of health care.¹⁰ The key challenge in getting the feedback (of parents in case of paediatric patients) was to develop a tool that is comprehensive, objective and easily understood by respondents. In our current study we identified the tool developed by in a Canadian hospital; the PARENTS tool has 20 items of which 18 are close-ended and 2 are open-ended questions.¹¹

Since its development, the PARENTS tool was shown to be effective in mostly Western settings. The goal of the current study was to localize, validate and use the PARENTS tool in Pakistan. For this, we first translated the instrument in URDU following standard translation protocols used in research.^12,13 We then administered the instrument to the parents of children in an emergency unit who were being discharged after spending 24 hours in the facility. Once the parents knew that the survey results would be kept anonymous and hence not affect their interaction with the treating PGR, they freely consented to participate. We collected 471 responses using two trained enumerators over a two-week period.

Our study showed excellent reliability scores for the PARENTS scale – Cronbach’s Alpha value of 0.884 (Field, 2013).¹⁴ The sample size of 471 has a confidence level of 97%. The reliability of the translated PARENTS tool is better than the reported value of 0.7 by Moreau et al.¹¹

Validity of the “PARENTS” Tool was established using confirmatory factor analysis (CFA). CFA confirms a single-factor model (CMIN/DF 3.99, RMSEA 0.958, CFI 0.932) which is consistent with the EFA results reported by Moreau et al.¹¹ Internal Validity test using Pearson Correlation shows all items with regression coefficients significant at p < 0.01 level.

Our results showed that the Urdu version of PARENTS can effectively be used in the Pakistani environment for measuring the NTS of PGRs. This is the first study that has tested the PARENTS reliability and validity in a non-western environment – the previous study in Thailand only used the items in PARENTS for qualitative analysis.¹⁵ Like the Thailand and earlier studies we found that selection of items in PARENTS scale provide a useful feedback mechanism for accessing the NTS of PGRs in an efficient and objective manner.^16,17

The five items (3, 4, 13, 14 & 15) are related to the skills of the PGR to assess the condition of the child and to be able to explain it to the parent effectively – it also covers basics like hygiene and display of proper identification. In all these aspects the PRGs were rated as more than satisfactory. This is a good reflection on the effectiveness of training on basic practices in engaging with patients.

In the next band, we have six items (2, 5, 7, 8, 16, and 17) and these are related to counselling of the patient. These items relate to skills of the PGR in being to explain the diagnosis to the parent, the proposed treatment plan, what to expect in terms of possible side effects, and follow-ups required. In this type of communication, the PGR must be willing to listen to the concerns of the parent. Readiness for discharge from hospital is an area of study that has in recent years started receiving attention from researchers – this is of particular importance for chronic or serious patients who need to be given confidence that their treatment can be safely continued at home.¹⁸ A parent can often be worried about what to do once they leave the hospital – how to know if things are not going well, who to contact if there is a problem, etc. In this interaction, it is important to hear them out and offer the needed advice to settle their anxieties or fears. In this band, we found that the PGR’s skills were rated as satisfactory.

Finally, we have seven items (1, 6, 9, 10, 18, 19 & 20) in the band where the PGRs got less than satisfactory ratings from parents. These are skills that involve patience and empathy from the PGR. They are needed to gain the trust/confidence of the parent and the child. They help in getting the acceptance of the parent for the PGR to examine the child without any inhibition or resistance. By building good rapport the PGR can help to reduce the feeling of anxiety in the parent – this is especially important in an emergency context where a patient is often brought in a serious condition. Studies have shown the importance of rapport building by doctors in gaining the trust of patients to help in improving treatment outcomes.¹⁹ These set of skills are not taught in medical education – those who have empathy by nature can develop it further during their clinical training. But those who lack this soft skill are unlikely to develop these without the intervention of supervisors. It is in this area that we noticed a shortcoming in the NTS of the PGRs. Parent’s ratings were on the unsatisfactory side – during the interview, the respondents recommended that PGRs should show more patience when dealing with caregivers and the patients. Not showing concerns for the feelings of the caregiver or child and lacking the flexibility to address their specific needs were an item of particular concern. Supervisors often instruct PGRs in this area during their teaching rounds – but the feedback that we have received shows that empathy and counselling skills still need a lot more work.

The translated PARENTS tool in Urdu has shown to be valid, reliable, and easy to administer in the local environment. Furthermore, items analysis from the PARENTS tool has shown to be helpful in highlighting areas of strength and weakness in NTS. The availability of PARENTS Urdu version is therefore a useful addition in helping supervisors to assess the NTS of PGRs.

This study has provided medical educationists with a valuable tool to be incorporated in assessment strategies for soft skills. We have gained good insight in the specific areas where curriculum planners need to focus on building the soft skill of PGRs as a part of professionalism. Supervisors can use the feedback from PARENTS to design and assess training approaches that can help in achieving desirable levels of NTS for PGRs to enable them to become capable clinicians.

Normally no formal training is given to medical professionals for developing non-technical skills and making them realize its importance in dealing with difficult and emergency critical situations. Urdu version of PARENTS tool can be a part of under-grad as well as post-graduate medical curricula during their clinical rotations. This will be the best time when feedback on its basis given to student/trainee can be impactful in bringing desired changes in attitudes and behaviours. It can also be used as a useful metric as part of the annual performance report for confirmation/promotion of paediatric health care workers.

Strengths

The study was conducted at the largest paediatric tertiary care center (1200 beds) in Pakistan with a diverse patient population. The sample size was sufficient to evaluate the reliability and validity of the tool. Only parents whose child had spent sufficient (18 + hours) time in hospital were interviewed to ensure that reliable feedback on behaviour of treating resident could be obtained.

Limitations

Although the collection of data from a single source (CHICH) is limitation in this research; achieving the required sample size for conducting statistical analysis to establish reliability and validity off sets this limitation.

PARENTS has shown to be valid and reliable tool for measuring the NTS of residents from the patient’s perspective. The items in the PARENTS scale provide specific feedback on important aspects such as resident knowledge, empathy in dealing with patients and the willingness/ability to patiently listen to and respond to patients concerns. Supervisors can use the translated PARENTS tool as an important part of their assessment tool set for providing useful feedback to their under-training residents. Medical administrators can use PARENTS as a performance measurement tool for monitoring the soft side of patient care.

PARENTS tool is a step towards measuring the satisfaction of patients, specifically it provides an objective measure of satisfaction. Qualitative research can be conducted to identify the reasons for satisfaction or lack of satisfaction expressed by patients. This research can help in the developing of effective training strategies for medical educationists.

Ethics approval and consent to participate: The current study protocols were approved by -The Children's Hospital & The institute of Child Health, Lahore- Pakistan (202-114-CHICH).

All methods were performed in accordance with the relevant guidelines and regulations (Declaration of Helsinki).

Informed consent was taken before the interviews.

Consent for publication: Not applicable

Availability of data and materials: The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Competing interests: The authors declare that they have no competing interests.

Funding: None

Authors' contributions: NT provided the theme of the study and done major data analysis and manuscript writing. RAK helped in manuscript writing and proof reading. MUA helped in data collection, analysis and manuscript writing. WT helped in data collection and analysis. MBM helped in manuscript writing. All authors read and approved the final manuscript.

Acknowledgement: My great appreciation for Prof. Dr. Khalid Ahmad Khan who helped a lot in organisation of data and in-depth statistical analysis of content. I also feel grateful to Dr. Ahsan Sethi, Prof. Muhammad Saleem, Dr. Tayyaba Azhar, Dr. Mahwish, Team Medical and Surgical emergencies The University of Child Health Sciences and The Children Hospital Lahore for all facilitation.

Sajjad SS, Sajid N, Fatimi A, Maqbool N, Baig-Ansari N, Amanullah F. The impact of structured counselling on patient knowledge at a private TB program in Karachi. Pakistan Journal of Medical Sciences. 2020 Jan;36(1):S49.
Bagnasco A, Tubino B, Piccotti E, Rosa F, Aleo G, Di Pietro P, Sasso L, Passalacqua D, Gambino L. Identifying and correcting communication failures among health professionals working in the Emergency Department. International emergency nursing. 2013 Jul 1;21(3):168-72.
Brown J. How clinical communication has become a core part of medical education in the UK. Medical education. 2008 Mar;42(3):271-8.
Urquhart LM, Rees CE, Ker JS. Making sense of feedback experiences: a multi‐school study of medical students' narratives. Medical education. 2014 Feb;48(2):189-203.
Ahuja AS, Williams R. Involving patients and their carers in educating and training practitioners. Current Opinion in Psychiatry. 2005 Jul 1;18(4):374-80.
Moreau KA, Pound CM, Eady K. Pediatric caregiver involvement in the assessment of physicians. BMC Medical Education. 2015 Dec;15(1):1-5.
Ende J. Feedback in clinical medical education. Jama. 1983 Aug 12;250(6):777-81.
DOCTOR FO, CHISHOLM A, ASKHAM J. What do you think of your doctor?.
Leech N, Barrett K, Morgan GA. SPSS for intermediate statistics: Use and interpretation. Routledge; 2013 Jul 11.
Towle A, Farrell C, Gaines ME, Godolphin W, John G, Kline C, Lown B, Morris P, Symons J, Thistlethwaite J. The patient's voice in health and social care professional education: the Vancouver statement. International Journal of Health Governance. 2016 Mar 7.
Moreau KA, Eady K, Tang K, Jabbour M, Frank JR, Campbell M, Hamstra SJ. The development of the PARENTS: a tool for parents to assess residents’ non-technical skills in pediatric emergency departments. BMC medical education. 2017 Dec;17(1):1-0.
Birbili M. Translating from one language to another. Social research update. 2000 Mar 15;31(1):1-7.
Worthington RL, Whittaker TA. Scale development research: A content analysis and recommendations for best practices. The counseling psychologist. 2006 Nov;34(6):806-38.
Field A. Discovering statistics using IBM SPSS statistics. sage; 2013 Feb 20.
Pakakasama S, Donkers J, Puranitee P. Parent feedback: perception of parents and pediatric residents in Asian culture.
Bosse, H.M., Nikendei, C., Hoffmann, K., Kraus, B., Huwendiek, S., Hoffmann, G.F., Jünger, J. and Schultz, J.H., 2007. Communication training using" standardized parents" for paediatricians--structured competence-based training within the scope of continuing medical education. Zeitschrift fur Arztliche Fortbildung und Qualitatssicherung, 101(10), pp.661-666.
Moreau KA, Eady K, Frank JR, Hamstra SJ, Karwowska A, Murnaghan A, Pound CM, Tse S, Jabbour M. A qualitative exploration of which resident skills parents in pediatric emergency departments can assess. Medical Teacher. 2016 Nov 1;38(11):1118-24.
Galvin EC, Wills T, Coffey A. Readiness for hospital discharge: A concept analysis. Journal of advanced nursing. 2017 Nov;73(11):2547-57.
Dang BN, Westbrook RA, Njue SM, Giordano TP. Building trust and rapport early in the new doctor-patient relationship: a longitudinal qualitative study. BMC medical education. 2017 Dec;17(1):1-0.

Table 1 - Item 1 response analysis

Item 1 – PGR Introduction	Cases	Percent %
Don’t remember	10	2.1
No	292	62.0
Incomplete	125	26.5
Yes	44	9.3
Total	471

Table 2 - Item 2 response analysis

Item 2 – PGR Identification	Cases	Percent %
Don’t remember	3	0.6
No	391	83.0
Incomplete	73	15.5
Yes	4	0.8
Total	471

Table 3 – Item 11 Suggestions on open-ended feedback

Suggestion Code	Cases	Percent %
No Suggestion	249	52.9
Talk politely	98	20.8
More Patience	46	9.8
More time	67	14.2
More Details	11	2.3
Total	471

Table 4 - Item 12 Overall satisfaction on open-ended feedback

Overall satisfaction	Cases	Percent %
Excellent	87	18.5
Good	249	52.9
Satisfactory	135	28.7
Total	471

Table 5 - PARENTS Descriptive Statistics

Measure	Value
Count	471
Minimum	23.00
Maximum	59.0000
Mean	38.3970
Std. Deviation	6.45982
Skewness	0.481
Kurtosis	0.53

Table 6 - Pearson correlation of items with PARENTS score

Item No	Pearson Correlation	Sig. (2-tailed)
1	.293**	0
2	.167**	0
3	.641**	0
4	.727**	0
5	.639**	0
6	.632**	0
7	.689**	0
8	.780**	0
9	.734**	0
10	.746**	0
13	.367**	0
14	.482**	0
15	.693**	0
16	.693**	0
17	.638**	0
18	.400**	0
19	.544**	0
20	.718**	0

**All correlations significant at the 0.01 level (2-tailed)

No competing interests reported.

Appendeces.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Reliability and Validity of PARENTS Tool for Assessment of Pediatric Residents’ Non- Technical Skills in a Tertiary Care Teaching Hospital of Pakistan

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Results

Discussion

Conclusions

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1