A mixed sequential exploratory study was used to develop a valid and reliable questionnaire to determine Cognitive Appraisals of Health Situation Scale in TSCI (CAHSSTSCI).
The present study was conducted in two qualitative and quantitative phases for tool development(15).
Qualitative phase
- Study design in the qualitative phase
This phase is related to development of the tool. Indeed, an inductive-deductive approach was conducted to generate appropriate initial items. First, a systematic scoping literature review was performed. Then a qualitative study with the content analysis approach was done to explain the concept of cognitive appraisal from the persons with SCI following an accident or traumatic incident (16). The items were extracted from both studies.
- The review process
A systematic scoping literature review was performed using key words and search strategy was designed by a specialist in library sciences and medical research. Some of key words derived from MeSH, Emtree terms, and free text words included "Spinal Cord Injuries", "Trauma", "cognitive", "Appraisal", "health status" through databases, PubMed, Web of Science, Scopus, and Embase from 1990 to 16 May 2020, without any language limitation.
- The participants
Participants were selected using a purposive sampling (identification and selection of information-rich cases related to the phenomenon) (17) from NSCIR from Trauma research center and Brain and Spinal Cord injury Research (BASIR) center in Tehran. Participants were people with TSCI who all agreed to participate in this study. All participants had TSCI for at least 6 months.
Maximum variation in participant demographics was prioritized during the sampling (gender, different socio-economic levels, and different clinical situation)(18).
- The process of interview and data collection
After explaining the purpose of the study to the participants, the researcher reassured them about confidentiality of their personal information. Ultimately, written consent for data collection was obtained. A deep face to face, semi-structured individual interview with open-ended questions was conducted. In the interview, participants were encouraged to share their experiences about the TSCI and related health problems. Some questions are in Appendix1, part A.
The interview was continued until data saturation when no new code of data was added to the study. In general, 12 people participated from July 2020 to March 2021 in this study. The interviews were done by the first author who has qualitative research experience and experience working with people with TSCI. Also, field notes were made during and after the interview. The average duration of interviews was 60 to 90 minutes.
- Data analysis
Data analysis was performed with qualitative conventional content analysis approach which was designed by Graneheim and Lundman (19). MAXQDA v.2018 was used for data management.
- Rigor
The codded data were assessed by people with SCI, peers, and experts and their suggestions such as correct sentences clearly and understandably were considered. Credibility, transferability, conformability, and dependability of the qualitative data were evaluated by the method proposed by Lincoln and Guba’s criteria (20).
Quantitative phase: psychometric properties of CAHSSTSCI
The psychometric properties of the questionnaire were assessed as follows:
Quantitative and qualitative face validity, quantitative and qualitative content validity, constructs validity, and the reliability of the tool.
- Face validity assessment
As a pre-test, it was performed both qualitatively and quantitatively. To determine the qualitative face validity, 10 people with SCI caused by traumatic accidents and 15 health care providers in this field were interviewed by voice and video online. Potential problems arose regarding the understanding of the items or words, the existence of ambiguity, and misinterpretations of items. The meanings of the sentences were examined. At this stage, some items were corrected. The following formula was used for calculating face validity:
Item impact score = Frequency of response (percentage) × Importance of item (21).
Content Validity: Content validation is usually used to examine the components of a measuring instrument. In this stage, the preliminary questionnaire was evaluated by 15 health care providers who previously published articles about coping, cognitive appraisals and related subjects: 4 psychologists, 1 psychiatrists, 2 PhD in health education, 3 PhD in disaster health, 3 PhD in nursing who were specialist in long term care and psychology nursing, and 2 Neuro-trauma epidemiologists. In the present research, the opinion of the experts was used to provide quantitative credit (content validity ratio (CVR) and content validity index (CVI) calculation) and qualitative validity (grammar, use of proper words, etc.) of the questionnaire.
To calculate the CVR score, the items were rated based on the 3 options: "necessary", "useful but not necessary", and "not necessary." Comments or suggestions about the content of the items were also collected. The content validity ratio was evaluated based on the 2014 Lawshe(22) table and the minimum acceptable score of CVR per 15 experts was 0.6. Finally, out of the items selected for CVR, 14 were removed and 36 were left, which were examined for content validity index (CVI) in the following table.
Furthermore, the CVI was assessed by 15 experts who scored items of the questionnaire based on their "simplicity", "relevance" and "clarity" using the 4-level Likert scale (scores 0 to 3 for “not at all ”to “completely”) based on Waltz & Bausell’s content validity index(23). CVI was calculated with the following formula:
(CVI= (Number of raters choosing points 3 and 4/ Total number of raters))
Items with a CVI more than 0.79, between 0.70 and 0.79, and less than 0.70 were considered suitable, needing modification, and unacceptable, in order (23). The scale’s CVR (S-CVR) and the scale’s CVI (S-CVI) were acquired through calculating of mean of items’ CVR and CVI.
Comprehensiveness of the tool: The ability of the tool to cover all areas related to the subject under study constitutes the comprehensiveness of the tool. Four options were used: “completely comprehensive”, "comprehensive", "somewhat comprehensive", "incomplete". To calculate this index, the total number of "completely comprehensive" specialists who have recognized the comprehensiveness of the tool as comprehensive and completely comprehensive was divided by the total number of specialists. If the comprehensiveness score of the confirmation questionnaire / overall comprehensiveness of the instrument was equal or greater than 0.7.
Construct validity
In this research, construct validity of CAHSSTSCI was evaluated through exploratory factor analysis (EFA) using principal component analysis (PCA) (24).
- Convergent validity
To evaluate the convergent validity, the correlation between new scale (CAHSSTSCI) and the Appraisals of DisAbility Primary and Secondary Scale (Kennedy et al.)(10) was assessed.
- Sampling
Plichta et al. cited that the required number of participants for EFA is between 3 and 10 persons per item (25). In this study, we tried to select the maximum number of samples, about 8 people were considered for 36 question, so a total of 305 persons with TSCI were selected of NSCIR-IR and brain and SCI center (BASIR) affiliated Tehran University of Medical Sciences (TUMS) in this study. For the sampling, the researcher used of respondents who were available (convenience method). The inclusion criteria were: persons with SCI due to an accident or traumatic incident that occurred more than 6 months ago, willingness to participate, knowing Persian language, no amputation in the limbs, no coma in the last six months, and no severe mental problems.
- Data analysis
Two criteria of Kaiser-Meyer-Alekin (KMO) and Bartlett Sphericity Test were used to confirm if there is enough correlation between the variables for EFA. The sample size sufficiency for EFA can be calculated KMO index is more than 0.8 and the p-value of the Bartlett Sphericity test is less than 0.05(26). After that, the items were tested about propriety to enter factor analysis by calculating commonalities. And then, items with commonalities of more than 0.4 were selected for the analysis. According to the Kaiser (1960) criterion that is the acceptance of factors having an Eigen value of more than one and also drawing the Scree plot, the factors of the tool were extracted. However, for more accuracy, parallel analysis was used for determining the exact number of factors.(27).
- Reliability
For the reliability the internal consistency assessment and Stability assessment were done:
- The internal consistency assessment
The internal consistency of CAHSTSCI was calculated by the Cronbach’s alpha coefficient, and the values higher than 0.7 were considered acceptable (28). To assess internal consistency, 58 eligible persons with SCI filled the questionnaire.
- Stability assessment
Stability of CAHSTSCI was assessed through the test–retest method. This questionnaire was completed twice within an interval of 10 days by 58 eligible persons with SCI. For the scores of two tests, Pearson correlation and intra-class correlation coefficients (ICC) were used.
The Pearson correlation coefficient more than 0.7 [23] and ICC higher than 0.4 were considered as the acceptable levels for stability [30]. The SPSS version 22 was used to calculate all statistical analyses.
A summary of steps in the current method were presented in Appendix 1 part B.