Test–retest reliability and convergent validity of the Test of Nonverbal Intelligence-Fourth Edition in patients with schizophrenia.

doi:10.21203/rs.3.rs-27553/v1

Download PDF

Research article

Test–retest reliability and convergent validity of the Test of Nonverbal Intelligence-Fourth Edition in patients with schizophrenia.

https://doi.org/10.21203/rs.3.rs-27553/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 13 Jan, 2021

Read the published version in BMC Psychiatry →

You are reading this older preprint version

Read the latest preprint version →

Background Fluid intelligence deficits affect executive functioning and social behaviors in patients with schizophrenia. To help clinicians manage fluid intelligence deficits, a psychometric sound measure is needed. The purposes of this study were to examine the test–retest reliability and convergent validity of the Test of Nonverbal Intelligence-Fourth Edition (TONI-4) assessing fluid intelligence in patients with schizophrenia.

Methods A total of 103 patients with stable condition were assessed with the TONI-4 twice with a 4-week interval to examine the test–retest reliability. We further used the Montreal Cognitive Assessment (MoCA) and the Tablet-Based Symbol Digit Modalities Test (T-SDMT) to examine the convergent validity of the TONI-4.

Results The intra-class correlation coefficient was 0.73 for the TONI-4. The percentages of standard error of measurement and minimal detectable change for the TONI-4 were 5.1% and 14.2%, respectively. The practice effect of the TONI-4 was small (Cohen’s d = -0.03). Convergent validity showed small to moderate significant correlations between the TONI-4 and the MoCA as well as the T-SDMT (r = 0.35, p = .011 with the T-SDMT and r = 0.61, p < .001 with the MoCA). The results demonstrated that the TONI-4 had good test–retest reliability, limited random measurement error, and a trivial practice effect. The convergent validity of the TONI-4 was good.

Conclusions These findings indicate that the TONI-4 has potential to be a reliable and valid assessment of fluid intelligence in patient with schizophrenia.

Psychiatry

cognition

fluid intelligence

intelligence

psychometrics

schizophrenia

Fluid intelligence can be defined as the ability to think logically and solve problems in novel situations [1, 2]. Conceptually, fluid intelligence has been linked to executive functioning and complex social behavior [3, 4]. Fluid intelligence is a critical cognitive ability affecting a wide variety of daily activities [5, 6]. Fluid intelligence deficits have been considered one of the core features of patients with schizophrenia, which accounts for much cognitive impairment in this patient group [3]. The deficits of fluid intelligence in patients with schizophrenia are associated with difficulties in daily independent functioning [7–9]. Moreover, poor fluid intelligence has been shown to be a risk factor for the onset of schizophrenia [10–14]. To help clinicians manage patients’ fluid intelligence, clinicians and researchers have to administer reliable and valid assessments of such deficits.

Three assessments are commonly used to assess fluid intelligence. They are the Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2) [15, 16], the Raven Advanced Progressive Matrices Test (RAPM) [17, 18], and the Test of Nonverbal Intelligence–Fourth Edition (TONI-4) [19]. However, some items of the CTONI-2 have cultural bias. For example, one item includes pictures related to American football or faces of Caucasians. In contrast, both the RAPM and the TONI-4 are administered with geometric patterns, which are not culturally dependent. Comparing the administration time, the CTONI-2 and the RAMP usually take an average of 40–60 minutes to finish, while the TONI-4 can be finished within 15 minutes. For time-pressed clinicians, the TONI-4 has great potential for assessing fluid intelligence in patients with schizophrenia.

Some supportive evidence on the psychometric properties has been found for the TONI-4 (e.g., sufficient test–retest reliability and construct validity) in healthy groups [19, 20]. However, the TONI-4 has not yet been validated in patients with schizophrenia. Because psychometric properties are generally sample dependent [21, 22], psychometric studies are needed to confirm whether the TONI-4 is reliable and valid in patients with schizophrenia. Particularly, sufficient psychometric properties (e.g., test–retest reliability, practice effect, random measurement error, and validity) are required for a measure to ensure its clinical utility for repeated assessments in patients with schizophrenia.

The current study aimed to examine the test–retest reliability, practice effect, random measurement error, and convergent validity of the TONI-4 in patients with schizophrenia. The results of the study should help clinicians and researchers determine the utility of the TONI-4 when applied to patients with schizophrenia.

Participants

We recruited participants via convenience sampling from a psychiatric hospital in southern Taiwan. Patients were included in this study if they met the following criteria: (1) diagnosis of schizophrenia according to the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) [23]. DSM-5 criteria for schizophrenia were assessed and validated by board-certified psychiatrists and supported by clinical observations and interviews during hospitalization, past medical records, and information provided by main caregivers, (2) age ≥ 20 years, and (3) stable use and dosage of antipsychotic medication for at least one month prior to recruitment. The exclusion criteria were (1) diagnosis of other neurological or psychiatric diseases affecting cognition (e.g., stroke or depression), (2) another severe medical condition or psychiatric disorder that required treatment during the study, or (3) an unstable severity of symptoms [specifically, a change in score of more than 2 on the Clinical Global Impressions Scale–Severity (CGI-S)] [24].

This study was approved by the Institutional Review Board of the local hospital. All participants signed consent forms before participating in this study.

Procedure

This study was comprised of three assessments with 2-week intervals between adjacent assessments (i.e., early, middle and late assessments). At the early and late assessments, the participants completed alternate forms of the TONI-4 (i.e., Form A at the early assessment and Form B at the late assessment) in a four-week interval. At the middle assessment, we administered the Tablet-Based Symbol Digit Modalities Test (T-SDMT) [25], and Montreal Cognitive Assessment (MoCA) [26]. All assessments were administered by a trained occupational therapist. In addition, the CGI-S was administered in each session to ensure that the participants' symptoms did not change during the study period. We collected the patients’ demographic characteristics from chart review.

Measures

TONI-4

The TONI-4 is designed to assess fluid intelligence for individuals aged 6 years to 89 years and 11 months. The TONI-4 has alternate forms (i.e., Form A and Form B) to reduce the practice effect [19]. Items for Form A can be found on one side of the picture book, and items for Form B are found on the reverse side. The two forms are not interchangeable. Each item is composed of a sequence of abstract figures with a figure missing from the sequence. Each sequence includes one or more attributes, such as shape, position, direction, rotation, contiguity, shading, size, and movement. Items ascend in level of difficulty as more attributes are added. When three of the five consecutive items are incorrectly answered, the test is terminated. The items are scored dichotomously: correct answers earn one point and incorrect answers earn zero points. The rater writes the score on the answer sheet. The TONI-4 is norm referenced and yields an index, which is a standardized score (quotient) with a mean of 100 and a standard deviation of 15. Higher index scores indicate better fluid intelligence [19].

T-SDMT

The T-SDMT was developed from the SDMT to assess processing speed [25]. This test includes 9 different symbols, each associated with a number (1–9), presented to the examinee on a tablet computer screen (i.e., an iPad). All trials are conducted with the tablet in landscape orientation, held in place by a case that is adjusted to a 30-degree tilting angle. To respond to each item, the participant is required first to look at the symbol in the center of the screen, then to search for the corresponding number in the table at the top of the screen, and finally to choose the corresponding number on a 3-by-3 grid at the bottom of the screen. The tablet computer automatically records the number of correct answers during the test. A higher number of correct answers indicates better performance of processing speed. The T-SDMT has acceptable psychometric properties in patients with schizophrenia [25].

MoCA

The MoCA briefly measures overall cognitive functioning, including orientation, memory, visuospatial skills, executive functioning, language, and attention [26]. The total scores range between 0 and 30, and higher scores indicate better cognitive functioning. The total score (including the addition of one point for examinees with 12 or fewer years of education) is used for analysis. The MoCA has demonstrated high sensitivity as a cognitive screening test for severe mental illness [27].

CGI-S

The CGI-S assesses symptom severity on a 7-point scale (1–7) [24]. One point on the CGI-S represents that a patient is not ill, and 7 points represents most severely ill. We used the CGI-S to examine whether the symptom severity of the participants was stable during the study period.

Statistical Analyses

Test–retest reliability

Test–retest reliability was estimated using the intra-class correlation coefficient (ICC) between the early and late assessments, on the basis of a two-way random-effects model with absolute agreement [28]. The following criteria were used to interpret ICC values: an ICC value ≥ 0.80 indicated excellent test–retest reliability; 0.60–0.79, good; 0.40–0.59, moderate; and < 0.40, poor [29].

The standard error of measurement (SEM) is an index of random measurement error that can be used to present the precision of individual scores [30]. The SEM% was calculated by dividing the SEM by the mean of the early assessment score and then multiplying the result by 100% (SEM%). An SEM% of less than 10% is considered to indicate limited random measurement error for a measure [31].

We also calculated the minimal detectable change (MDC) and MDC percentage (MDC%) to examine the change between adjacent assessments that could be considered as a real change (beyond the score change caused by random measurement error) at the 95% confidence level. The MDC% was calculated by dividing the MDC by the mean of the early assessment score and then multiplying the result by 100% [32].

In addition, the agreement between test–retest measurements was analyzed by Bland–Altman plots with 95% limits of agreement (LOA) [33]. In these plots, the differences (d) between each pair of assessments were presented against the average value for each pair of assessments. To examine whether heteroscedasticity existed, Pearson’s correlation coefficient (r) was used to calculate the correlation between the absolute value of the difference of two assessments and the mean score of two assessments [34]. When Pearson’s r was ≥ 0.3 or ≤ -0.3, it meant that the absolute value of the difference was related to the mean score of two assessments, and that there was heteroscedasticity [35]. In other words, the higher the assessment score, the greater (r ≥ 0.3) or smaller (r ≤ -0.3) the difference between the two assessments.

Effect size (Cohen’s d) was used to estimate the magnitudes of practice effects due to repeated assessments of the TONI-4. An effect size ≥ 0.80 was considered as a large practice effect; 0.50–0.79, medium; 0.20–0.49, small; and < 0.20, trivial [36].

Convergent validity

Convergent validity was examined by correlating the scores of the TONI-4 at the early assessment with those of the MoCA and the T-SDMT using Pearson’s r. We hypothesized that we would find moderate correlations between the scores of the TONI-4 and the MoCA (i.e., fluid intelligence and cognition) [37], and that small correlations would be found between the scores of the TONI-4 and the T-SDMT (i.e., fluid intelligence and processing speed) [3, 38].

We recruited 106 patients with schizophrenia who were eligible for the study. Of these, 103 participants completed all assessments. Half of the participants were male (50.5%), and the mean age was 47.4 years. The demographic and clinical characteristics of the participants are shown in Table 1. The early and late assessments scores of the TONI-4, on average, were very similar (92.4 and 91.9), indicating that the participants had slight impairment of fluid intelligence.

Test-retest reliability

Table 2 shows the results of the test–retest reliability analyses. The ICC of the TONI-4 was 0.73 (95% confidence interval: 0.62 to 0.81).

The SEM (SEM%) and MDC (MDC%) of the TONI-4 scale were 4.7 (5.1%) and 13.1 (14.2%) points, respectively. The results were smaller than our preset criterion.

In Figure 1, the LOAs ranged from -14.6 to 13.6 points. Pearson’s r between the absolute value of the difference of the early and late assessments and the mean score of the early and late assessments was 0.31.

Analysis of the practice effect revealed that the effect size of score change in the TONI-4 was small (Cohen’s d = -0.03) between the early and late assessments.

Convergent validity

The index scores of the TONI-4 were moderately correlated with the scores of the MoCA (r = 0.61, p<.001), whereas a small correlation was found between the scores of the TONI-4 and the T-SDMT (r = 0.35, p =.011).

Test–retest reliability

A measure with sufficient test-retest reliability ensures users to obtain reproducible scores. Good test–retest reliability (ICC = 0.73) was found for the repeated assessments of the TONI-4. In comparison with previous studies, the test–retest reliability of our study was slightly lower than those found for healthy controls (r = 0.82–0.93) [19], and was consistent with those of other cognitive assessments examining patients with schizophrenia [39, 40]. There are three possible reasons for the slightly lower ICC of the TONI-4. First, the test–retest reliability was estimated by Pearson correlation coefficients in the previous studies, which tends to overestimate reliability [28]. Second, alternate forms (i.e., Forms A and B) were used in this study, which may have resulted in more variation compared to using the same form in previous studies [19]. Third, the heterogeneity of our sample appeared limited. In particular, the variances of the TONI-4 in this study (SDs = 9.1 and 9.9) were smaller than those of a previous study (SDs = 13–15) [19], which may have underestimated the ICC values in this study [41]. In summary, our findings indicate that the TONI-4 appears to be reliable in repeatedly assessing fluid intelligence of patients with schizophrenia.

We found that the SEM% (5.1%) was far below our preset criterion (10%), which suggests limited random measurement error. Our findings are consistent with those in previous studies examining healthy groups, where the SEM% were 4.0–5.5% [19]. These findings support that random measurement error is similar in patients with schizophrenia and in healthy adults. Therefore, the scores of the TONI-4 tend to be stable in patients with schizophrenia.

In addition, MDC can be viewed as the threshold for a statistically significant change for individual patients in clinical and research settings [42]. Conceptually, a change exceeding the MDC of the first assessment can be interpreted as a real improvement with the corresponding certainty (e.g., 95%). Thus, a fixed MDC value can be used to interpret the change scores for patients with different levels of fluid intelligence. However, we found that the association between the absolute value of the difference of the early and late assessments and the mean score of the early and late assessments (Pearson’s r = 0.31) was above 0.30, implying the existence of heteroscedasticity [35]. That is, the absolute difference and the mean of the early and late assessments increased simultaneously. Accordingly, a fixed value of MDC is not appropriate for different levels of fluid intelligence.

In such assessments with heteroscedasticity, the MDC% is more suitable than the MDC for interpreting a true change for a patient [43]. That is, as seen in this study, the MDC value can be adjusted based on the MDC% and the patient’s early assessment score. Specifically, the MDC% (14.2%) of the TONI-4 can be multiplied by the patient’s early assessment score to achieve an adjusted MDC value. For example, a patient with a score of 92 points at the early assessment requires an improvement of more than 13.1 points (92 × 0.142) to indicate a true change. These adjustments can help clinicians and researchers interpret changes in score on the TONI-4 of an individual patient after intervention, and develop further treatment plans accordingly.

We found that the scores between the early and late assessments had almost no change (effect size = -0.03). Our findings are consistent with those in a previous study, where the change scores within one-to-two-week intervals were small (effect size = 0.00–0.07) [19]. These findings indicate that the scores of the TONI-4 do not systematically increase given that the early assessment (or practice) has already been completed. The trivial practice effect may have been due to the use of alternate forms (i.e., Forms A and B) [44, 45]. However, using alternate forms may lead to underestimation of the practice effect as compared to using a single form. In this study, all participants were administered the forms in a fixed order (i.e., Form A first and Form B second). The fixed order design was used because previous findings had indicated that test–retest reliability is not affected by the order effect [20]. Thus, clinicians could use alternate forms of the TONI-4 in their routine repeated assessments to effectively minimize practice effects.

Convergent validity

Good convergent validity was demonstrated for the TONI-4. We found that the TONI-4 was moderately correlated with the MoCA (r = 0.61) and significantly correlated with the T-SDMT (r = 0.35), supporting our hypotheses. These results support the validity of the TONI-4 for assessing fluid intelligence in patients with schizophrenia.

This study had two merits. First, the sample size (103 participants) was relatively large. A large sample size tends to provide robust estimates, which improves the generalizability of our findings [46]. Second, we used alternate forms of the TONI-4. Due to this study design, the practice effects of the TONI-4 were well controlled and thus confirmed its utility in repeated assessments.

Implications

The fluid intelligence encompasses the ability to think logically and solve problems in novel situations, which is a critical cognitive ability affecting clients’ performance on a wide variety of daily activities. Knowledge and evidence of the test–retest reliability and convergent validity of the TONI-4 help clinicians select a measure for assessing fluid intelligence in patients with schizophrenia.

The TONI-4 appears reliable in repeatedly assessing the fluid intelligence in patients with schizophrenia.
Due to heteroscedasticity of the TONI-4, an adjusted MDC, the patient’s early assessment score multiplying by the MDC% (14.2%), is suggested to be used to determine whether the change of score of a patient is outside the range of random measure error.
The good convergent validity of the TONI-4 provides a preliminary basis to support its utility for assessing fluid intelligence in patients with schizophrenia.

Study limitations

Two limitations of this study should be noted. First, the study sample was a convenience sample recruited from a psychiatric center in southern Taiwan. In addition, our participants, on average, had slightly impaired fluid intelligence (the mean score of the TONI-4 was 92.4 points at the early assessment). The above sampling limitations might have affected the generalizability of our findings. Second, we used alternate forms to examine the test–retest reliability of the TONI-4. Thus, our results on test–retest reliability might not be generalizable to single-form assessment of the TONI-4. Using alternate forms may lead to underestimation of the test–retest reliability as compared to using a single form.

We found good test–retest reliability and good convergent validity of the TONI-4 in patients with schizophrenia. These findings provide preliminary evidence supporting the utility of the TONI-4 in patients with schizophrenia.

CTONI-2: Comprehensive Test of Nonverbal Intelligence–Second Edition; RAPM: Raven Advanced Progressive Matrices Test; TONI-4: Test of Nonverbal Intelligence–Fourth Edition; DSM-5: Diagnostic and Statistical Manual of Mental Disorders, 5th edition; CGI-S: Clinical Global Impressions Scale–Severity; T-SDMT: Tablet-Based Symbol Digit Modalities Test; MoCA: Montreal Cognitive Assessment; ICC: Intra-class correlation coefficient; SEM: Standard error of measurement; MDC: Minimal detectable change; LOA: Limits of agreement; SD: standard deviation

Ethics approval and consent to participate

Ethical approval for this study was granted by the Institutional Review Board (IRB) of the Kaohsiung Municipal Kai-Syuan Psychiatric Hospital. All subjects were informed about the study and signed consent forms before participating in this study.

Consent for publication

Not Applicable

Availability of data and materials

The (anonymized) datasets analyzed during the current study are available from the first author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This study was supported by the Kaohsiung Municipal Kai-Syuan Psychiatric Hospital (grant number KSPH-2016-03). The funder had no role in study design and collection, analysis, and interpretation of data, decision to publish and in writing the manuscript.

Author contributions

KWC and CLH designed the study; LJC, and CYC collected the data; KWC and CLH analyzed the data and wrote the first draft; KWC, YCL and TYY revised the manuscript; all authors gave insightful comments and improved the manuscript. All authors approved the final version of the manuscript.

Acknowledgements

The authors would like to thank all the participants in the study.

Cattell RB. Theory of fluid and crystallized intelligence: A critical experiment. J Educ Psychol. 1963;54(1):1-22.
Horn JL, Cattell RB. Refinement and test of the theory of fluid and crystallized general intelligences. J Educ Psychol. 1966;57(5):253-270.
Roca M, Manes F, Cetkovich M, Bruno D, Ibanez A, Torralva T, Duncan J. The relationship between executive functions and fluid intelligence in schizophrenia. Front Behav Neurosci. 2014;8:46.
Huepe D, Roca M, Salas N, Canales-Johnson A, Rivera-Rei AA, Zamorano L, Concepcion A, Manes F, Ibanez A. Fluid intelligence and psychosocial outcome: from logical problem solving to social adaptation. PLoS One. 2011;6(9):e24858.
Gray JR, Thompson PM. Neurobiology of intelligence: science and ethics. Nat Rev Neurosci. 2004;5(6):471-482.
Jaeggi SM, Buschkuehl M, Jonides J, Perrig WJ. Improving fluid intelligence with training on working memory. Proc Natl Acad Sci U S A. 2008;105(19):6829-6833.
Tucker-Drob EM. Global and domain-specific changes in cognition throughout adulthood. Dev Psychol. 2011;47(2):331-343.
Sternberg RJ, Wagner RK. Practical intelligence: Nature and origins of competence in the everyday world. Yew York, NY: CUP Archive; 1986.
Kievit RA, Davis SW, Griffiths J, Correia MM, Cam C, Henson RN. A watershed model of individual differences in fluid intelligence. Neuropsychologia. 2016;91:186-198.
Blair C. How similar are fluid cognition and general intelligence? A developmental neuroscience perspective on fluid cognition as an aspect of human cognitive ability. Behav Brain Sci. 2006;29(2):109-125.
Snitz BE, Macdonald AW, 3rd, Carter CS. Cognitive deficits in unaffected first-degree relatives of schizophrenia patients: a meta-analytic review of putative endophenotypes. Schizophr Bull. 2006;32(1):179-194.
Caspi A. Cognitive performance in schizophrenia patients assessed before and following the first psychotic episode. Schizophr Res. 2003;65(2-3):87-94.
Zanello A, Perrig L, Huguelet P. Cognitive functions related to interpersonal problem-solving skills in schizophrenic patients compared with healthy subjects. Psychiatry Res. 2006;142(1):67-78.
Johnson MK, McMahon RP, Robinson BM, Harvey AN, Hahn B, Leonard CJ, Luck SJ, Gold JM. The relationship between working memory capacity and broad measures of cognitive ability in healthy adults and people with schizophrenia. Neuropsychology. 2013;27(2):220-229.
Hammill DD, Pearson N. Comprehensive test of nonverbal intelligence. In: Handbook of Nonverbal Assessment. edn. Boston, MA: Springer; 2017: 167-184.
Hammill DD, Pearson NA, Wiederholt JL. Comprehensive test of nonverbal intelligence (CTONI). Austin, TX: Pro-ed; 1997.
Raven JC. Manual for Raven's progressive matrices and vocabulary scales. London: HK Lewis; 1983.
Raven JC. Guide to the standard progressive matrices: Sets A, B, C, D and E. London: Lewis & Co; 1960.
Brown L, Sherbenou RJ, Johnsen SK. Test of nonverbal intelligence: TONI-4. Austin, TX: Pro-ed; 2010.
Ritter N, Kilinc E, Navruz B, Bae Y. Test Review: Test of Nonverbal Intelligence-4 (TONI-4). J Psychoeduc Assess. 2011;29(5):484-488.
Hobart J, Cano S. Improving the evaluation of therapeutic interventions in multiple sclerosis: the role of new psychometric methods. Health Technol Assess. 2009;13(12):iii, ix-x, 1-177.
Nunnally JC, Bernstein IH. Psychometric Theory. New York, NY: McGraw-Hil; 1994.
American Psychiatric Association. Desk reference to the diagnostic criteria from DSM-5®. Arlington, VA: American Psychiatric Publishing; 2014.
Haro J, Kamath S, Ochoa S, Novick D, Rele K, Fargas A, Rodriguez M, Rele R, Orta J, Kharbeng A. The Clinical Global Impression–Schizophrenia scale: A simple instrument to measure the diversity of symptoms present in schizophrenia. Acta Psychiatr Scand. 2003;107(416):16-23.
Tung LC, Yu WH, Lin GH, Yu TY, Wu CT, Tsai CY, Chou W, Chen MH, Hsieh CL. Development of a Tablet-based symbol digit modalities test for reliably assessing information processing speed in patients with stroke. Disabil Rehabil. 2016;38(19):1952-1960.
Nasreddine ZS, Phillips NA, Bedirian V, Charbonneau S, Whitehead V, Collin I, Cummings JL, Chertkow H. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53(4):695-699.
Musso MW, Cohen AS, Auster TL, McGovern JE. Investigation of the Montreal Cognitive Assessment (MoCA) as a cognitive screener in severe mental illness. Psychiatry Res. 2014;220(1-2):664-668.
Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep. 1966;19(1):3-11.
Bushnell CD, Johnston DC, Goldstein LB. Retrospective assessment of initial stroke severity: comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke. 2001;32(3):656-660.
Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res. 2005;19(1):231-240.
Flansbjer UB, Holmback AM, Downham D, Lexell J. What change in isokinetic knee muscle strength can be detected in men and women with hemiparesis after stroke? Clin Rehabil. 2005;19(5):514-522.
Huang SL, Hsieh CL, Wu RM, Tai CH, Lin CH, Lu WS. Minimal detectable change of the timed" up & go" test and the dynamic gait index in people with Parkinson disease. Phys Ther. 2011;91(1):114-121.
Bland JM, Altman DG. Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement. Lancet. 1986;1(8476):307-310.
Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135-160.
Atkinson G, Nevill AM. Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Med. 1998;26(4):217-238.
Cohen J. Statistical power analysis for the behavioral sciences, 2 edn. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
Cochrane M, Petch I, Pickering AD. Aspects of cognitive functioning in schizotypy and schizophrenia: evidence for a continuum model. Psychiatry Res. 2012;196(2-3):230-234.
Shelton JT, Elliott EM, Matthews RA, Hill BD, Gouvier WD. The relationships of working memory, secondary memory, and general fluid intelligence: working memory is special. J Exp Psychol Learn Mem Cogn. 2010;36(3):813-820.
Hahn E, Vollath A, Ta TT, Hahn C, Kuehl LK, Dettling M, Neuhaus AH. Assessing long-term test-retest reliability of the CPT-IP in schizophrenia. PLoS One. 2014;9(1):e84780.
Pietrzak RH, Snyder PJ, Jackson CE, Olver J, Norman T, Piskulic D, Maruff P. Stability of cognitive impairment in chronic schizophrenia over brief and intermediate re-test intervals. Human Psychopharmacology. 2009;24(2):113-121.
Lexell JE, Downham DY. How to assess the reliability of measurements in rehabilitation. Am J Phys Med Rehabil. 2005;84(9):719-723.
Jette AM, Tao W, Norweg A, Haley S. Interpreting rehabilitation outcome measurements. J Rehabil Med. 2007;39(8):585-590.
Flansbjer UB, Holmback AM, Downham D, Patten C, Lexell J. Reliability of gait performance tests in men and women with hemiparesis after stroke. J Rehabil Med. 2005;37(2):75-82.
Brown L, Sherbenou RJ, Johnsen SK. Test of Nonverbal Intelligence: A language-free measure of cognitive ability. Austin, TX: Pro-ed; 1990.
Martin JD, Blair GE, Bledsoe JR. Measures of concurrent validity and alternate-form reliability of the Test of Nonverbal Intelligence. Psychol Rep. 1990;66(2):503-508.
Stockwell DRB, Peterson AT. Effects of sample size on accuracy of species distribution models. Ecol Modell. 2002;148(1):1-13.

Table 1: Demographic characteristics of the participants (n=103).

Characteristic	Value
Gender, n (%) Male Female	52 (50.5%) 51 (49.5%)
Age, years, mean (SD) *	47.4 (10.3)
Age of onset, years, mean (SD) *	25.5 (8.3)
Education level, n* (%) Illiterate Elementary Middle school High school or vocational school University or graduate school	0 6 (6.0%) 39 (39.0%) 41 (41.0%) 14 (14.0%)
Marital status, n (%) Married Unmarried Other	28 (27.2%) 71 (68.9%) 4 (3.9%)
CGI-S (SD)	3 (0.8)

*There were some missing data in the variables of marital status and severity.

Note: CGI-S = Clinical Global Impressions Scale–Severity

Table 2. The Mean, SD, ICC, Cohen’s d, SEM and MDC of the TONI-4 (n=103).

Early assessment

Mean (SD)

Late

assessment

Mean (SD)

Cohen’s d

ICC

(95% CI)

SEM

(SEM%)

MDC

(MDC%)

TONI-4

92.4 (9.1)

91.9 (9.9)

-0.03

0.73

(0.62–0.81)

4.7

(5.1)

13.1

(14.2)

Note: SD = standard deviation; ICC = intra-class correlation coefficient; SEM = standard error of measurement; SEM% = percentage of standard error of measurement; MDC = minimal detectable change; MDC% = percentage of minimal detectable change

Download PDF

Journal Publication

published 13 Jan, 2021

Read the published version in BMC Psychiatry →

Editorial decision: Major revision
08 Jun, 2020
Editor assigned by journal
12 May, 2020
Submission checks completed at journal
11 May, 2020
Editor invited by journal
11 May, 2020

You are reading this older preprint version

Read the latest preprint version →

Test–retest reliability and convergent validity of the Test of Nonverbal Intelligence-Fourth Edition in patients with schizophrenia.

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Status:

Journal Publication

Version 1