Downing’s (18) model is an extension of Messick’s validity framework (19), which argues that assessments are not valid in and of themselves; rather they are the result of researchers’ evidence gathered in support of a specific interpretation. As such, we position this study as an argument for the validity evidence collected on the adaptation of Avey et al.’s (16) measure of psychological ownership, a construct that originates in organizational psychology to describe the feeling of responsibility individuals have for the tasks, processes, or people within their sphere of influence and control. We chose to adapt this instrument because, like employees who feel that various projects belong to them, physicians share similar feelings about their patients and their care (20). Further, we understand that while medical students are not expected to take ownership of their patients’ care, they are expected to develop these feelings while in training and are granted access to patients in a limited manner to learn how to do so.
According to Downing (18), researchers arguing for validity must provide scientifically sound evidence within five components: content, response process, internal structure, relationship to other variables, and consequences. In using these components, he frames the validation process as an argument in which each new study adds validity evidence and demonstrates further evidence for the instrument’s strengths and weaknesses within the types of validity evidence it is best designed to capture. We used Downing’s model to frame our work because it is widely accepted in medical education (21-24).
We were granted permission to use Avey et al.’s (16) original instrument by MindGarden in 2014, which contained 16 items and five subscales. When these five subscales are combined, they are thought to coalesce in a measure of ownership. The subscales are: a) Territoriality - when individuals feel they must mark their place or objects, believing they have exclusive rights to them; b) Accountability - the expectation that one may be asked to justify one’s beliefs, feelings and actions; c) Self-Efficacy – the idea that people’s beliefs facilitate or constrain success as they attempt to implement action or complete a specific task; d) Belongingness - the psychological need individuals have for feeling they have a home or place; and e) Self-Identification - when individuals internalize the organizational identity as an extension of one’s self. Previous studies had noted how these various subscales rank in clerkships as students develop ownership over their patients and their care (8).
Avey et al.’s (16) original instrument prompted respondents to comment on 16 items using a 6-point Likert scale ranging from “Strongly Disagree (1) to Strongly Agree (6).” The validity evidence for the original version of the instrument included moderate to good coefficient alphas for the all subscales: Self-Efficacy (α = .90), Accountability (α = .81), Belongingness (α = .92), Self-Identification (α = .73), and Territoriality (α = .84), and good relationship to other meaningful constructs such as Transformational Leadership, Organizational Citizenship Behavior, Organizational Commitment, Workplace Deviance, Intentions to Stay, and Job Satisfaction (16).
Our team, which has expertise in the conceptual and theoretical underpinnings of patient care ownership, medical education, and survey validation, modified the items to fit within a medical education setting. After adapting the items from business to medical education, the newly modified instrument was piloted in 2016 at [redacted] with third- and fourth-year students to assess feasibility and relevancy for a medical school population (7). Further language modifications were made to be more inclusive of healthcare teams. For example, if the original item was worded as “I feel I belong in this organization,” it was originally written, the item was changed so that it read “I feel I belong on this healthcare team.”
Each revised item was then considered for various interpretations (to ensure clarity) and whether these interpretations would ensure the same construct is being asked about in the original item before being sent to clerkship directors for feedback. The team reviewed the original instruments used to create Avey et al.’s (16) survey and added additional items to four of the five subscales to include a more complete understanding of psychological ownership in healthcare settings. For example, for the Accountability subscale in the original Avey et al. study, we added the following item: “I consistently hold myself accountable for my patients’ care.” Additionally, for the Self-Identification subscale, we added “I have difficulty using my knowledge in patient care on this healthcare team” (reversed item).
The items in the Territoriality subscale remained at 4 items, while items in the other scales all increased from the original 3 items: Accountability (5 items), Self-Efficacy (6 items), Belongingness (5 items), and Self-Identification (7 items). All changes were incorporated into the instrument and the survey was renamed the Patient Ownership Survey.
To collect validity evidence on the Patient Ownership Survey’s relationship to other variables, we also included two other measures in our validation process; the Teamwork Assessment Scale (TSA) (25) and the 1-item Maslach Burnout Inventory (MBI)-Human Services Survey (26). The TSA was previously validated in a medical student population (25) and measures individuals’ level of teamwork within a specific setting. We included this survey because in our pilot study, we found that a sense of belongingness, which is one of Avey et al.’s (16) original subscales had a strong correlation with students’ willingness to take ownership of their patients (7). The TSA is comprised of three sub-scales measuring team adjustment behaviors, team coordination and cooperation, and information exchange. The other measure included with our Patient Ownership Survey was the Maslach Burnout Inventory (MBI)-Human Services Survey, which is a one-item shortened version of the full Burnout scale (26). The scores on this instrument range from 1-5, with higher scores indicating a lower level of burnout. We included these other two measures because we hypothesized that teamwork would be positively associated with aspects of psychological ownership and negatively associated with burnout, as others have suggested (15, 27).
The adapted Patient Ownership Survey included 27 items, which was an 11 item increase from the 16 items included in Avey et al.’s (16) original instrument; it was not anticipated that all new items might function equally well within the scale and some might be dropped. The survey also included questions to collect demographic information (i.e., age, gender, ethnicity/race) and descriptions of the medical education setting of respondents (i.e., clerkship, year in medical school, campus, clinical setting). Demographic information was used to collect validity evidence on relationships with other variables. Based on our previous research, variables like type of clerkship and clinical settings can influence the level of patient care ownership (7, 8). This data was also used to collect validity evidence.
The adapted instrument was distributed to third-and fourth-year students through the [redacted medical school’s] secure online evaluation system. Participants included third- and fourth-year medical students enrolled in the 2018-2019 school year who were rotating through their clerkships. Third year clerkships included: Internal Medicine, Family Medicine, Obstetrics and Gynecology (OB/GYN), Surgery, Pediatrics, Neurology, and Psychiatry. Fourth-year rotations included Emergency Medicine, Ambulatory Medicine, and various electives.
To collect consequential validity evidence on how patient ownership may develop over time, and to recruit as many students as possible into the study with at least two data points, we sent the survey out several times. During the academic year 2018-19 the survey was sent approximately three times three months apart to third year and only once in the spring to fourth year students.
Participation in this study was voluntary and all students provided full and informed consent prior to participating. To document response process, trained researchers kept a decision journal on the challenges and questions they encountered in the writing of items, administration of the survey, and interpretation of the results. These entries were used to continually adjust the items throughout the validation process, and track where we experienced challenges in the development of the new instrument. The study was approved by [redacted] University’s Institutional Review Board (Protocol #920339).
All statistical analysis were performed using SAS 9.4 and statistical significance was assessed using an alpha level of 0.05. Descriptive statistics on all variables were determined overall, and by measurement time where appropriate.
In using Downing’s (18) multi-component approach to assessing validity, we prioritized internal structure validity in the analysis. To this end, the team assessed the instrument’s internal structure using confirmatory factor analysis (CFA) by making use of each student’s first administration of the survey. Fit statistics including the chi-square test, root mean square error of approximation (RMSEA)<0.08, Bentler’s Comparative Fit Index (CFI)>0.95, Bentler and Bonnett’s Non-Normed Index (NNI)>0.90, and Bentler and Bonnet’s Normed Fit Index (NFI)>0.90. If fit statistics indicated an adequate fit to the factor structure, parameter estimates and whether each was statistically different from 0, variances of exogenous variables being different than 0, and covariances among the exogenous variables being different than 0 were examined. The analysis plan included use of full information maximum likelihood for parameter estimation, the five factors identified in Avey et al.’s (16) original ownership instrument were used in the CFA: Territoriality (4 items), Accountability (5 items), Self-efficacy (6 items), Belongingness (5 items), and Self-Identification (6 items).
As the CFA did not confirm the initial five factor models, an exploratory factor analysis was performed using a principal components extraction method with varimax rotation and Kaiser normalization. The number of factors was determined using eigenvalues>1, a scree plot, and parallel analysis. Items that loaded on two or more factors or did not load within a minimum factor loading of 0.40 were excluded.
Cronbach’s alpha was determined for the overall and sub-scales in the revised ownership scale. Pearson correlations of the overall and sub-scales revised factors were calculated.
To examine relationships between the revised sub-scales and various variables, two-sample t-tests, Pearson correlations, and one-way ANOVA between different racial/ethnic groups with a Tukey-Kramer multiple comparison procedure were examined. Other demographics included age, gender, year in medical school, clinical settings, including identification of the campus, type of clinical setting, and number of students on rotation. Assessing the number of students on the team is important because it has potential to affect feelings of patient care ownership and ownership behavior (7). Given the literature on patient care ownership and how it develops, the research team expected to find correlations between the adapted Patient Care Ownership Survey and Teamwork Assessment Scale (25), and the Burnout Inventory Survey (26).
To assess consequential validity, mixed models were used to examine whether changes over time were seen for both the original and exploratory sub-scales. The subject was considered the random effect and survey administration time (1, 2, or 3) was considered a fixed effect.