The two test batteries were administered among 274 children living with HIV and 330 children without HIV with a mean age of 9.48 (SD = 1.31), of which roughly half were male. Table 2 summarizes the demographics of participants in the two groups. The second assessment of the BENCI among 38 Children not living with HIV consists of 21 females, with a mean age of 9.18 (SD = 1.21).
Table 2
Socio Demographic Information
Variables | HIV Negative N (%) | HIV Positive N (%) |
Gender | Male | 163 (49.40) | 148 (54.00) |
Female | 166 (50.30) | 125 (45.60) |
Missing | 1 (0.30) | 1 (0.40) |
Age in months (Mean ± SD) | 117.2 ± 16.24 | 119.40 ± 14.63 |
Age in Years (Mean ± SD) | 9.41 ± 1.37 | 9.56 ± 1.24 |
Nutrition | Weight in kg (Mean ± SD) | 34.98 ± 7.12 | 32.27 ± 5.85 |
Height in cm (Mean ± SD) | 136.34 ± 8.00 | 133.02 ± 8.11 |
Scale Attenuation Effects
Eight of the scores on the BENCI subtests exhibited ceiling and floor effects that tend to suppress correlations and reliabilities. Specifically, on Verbal Comprehension Figures, 30% (N = 181) of the sample scored the highest possible score of 8 hits, while on Verbal Comprehension Images hits, 51% (N = 308) of the sample scored the highest possible score of 8 hits. Other subtests with ceiling effects included Continuous Performance hits, Go No Go hits, Working Memory hits and Spatial Stroop. Verbal Memory Recognition and Visual Memory Recognition had a bit of ceiling effects. At the same time, floor effects were evident on the Planning Time of First Option and Spatial Stroop errors scores. Semantic Fluency hits, Phonetic Fluency hits, Verbal Memory Delayed hits and Planning time total had a bit of floor effects. The floor and ceiling effects highlighted that these subtests psychometric functioning could be improved by adding easier and more difficult items, respectively, in any future revisions of the BENCI. The remaining the BENCI subtests showed no such attenuation effects.
Internal Consistency of the BENCI
We computed the Cronbach’s Alphas (KR-20s) for seven of the subtests with dichotomous item scores. The internal consistency of the BENCI subtests varied from poor to excellent reliability. As shown in Table 3, the Language Comprehension tests, Verbal Comprehension Images and Figures, had the fewest items (N = 8) and Cronbach Alpha 0.49 < α < 0.68 which was the lowest among the other BENCI subtests. Low Cronbach Alphas tend to suppress correlations, but most of the BENCI subtests had high Alphas. The Abstract reasoning, Planning, Go No Go, Spatial Stroop and Processing Speed tests correlated well with themselves (0.75 < α < 0.97 or alpha range from .75 to .97) hence showing that there was little random measurement error.
Table 3
BENCI Items Internal Consistency
BENCI Subtests | No. of Items | Skewness | Overall Cronbach's Alpha | HIV Negative Cronbach's Alpha | HIV Positive Cronbach's Alpha |
Verbal comprehension images Time | 8 | 3.113 | .689 | .519 | .682 |
Verbal comprehension images Hits | 8 | -1.302 | 0.592 | 0.386 | 0.602 |
Verbal comprehension figures Time | 8 | 1.926 | 0.613 | 0.56 | 0.609 |
Verbal comprehension figures Hits | 8 | − .737 | 0.496 | 0.349 | 0.571 |
Abstract reasoning Hits | 25 | .019 | 0.832 | 0.813 | 0.781 |
Abstract reasoning Time | 25 | .851 | 0.890 | 0.904 | 0.875 |
Go No Go Total Hits | 101 | − .745 | 0.870 | 0.824 | 0.895 |
Go No Go Total Time | 101 | 1.136 | 0.879 | 0.872 | 0.864 |
Planning Total time | 12 | .895 | 0.753 | 0.760 | 0.744 |
Spatial Stroop Hits | 90 | − .933 | 0.973 | 0.966 | 0.975 |
Spatial Stroop Time | 90 | -1.701 | 0.950 | 0.924 | 0.959 |
Processing speed Reaction Time | 50 | 1.598 | 0.832 | 0.832 | 0.822 |
Possibly due to the ceiling effects being less severe because of lower mean scores, we found Verbal Comprehension Figures and Images tests to show higher internal consistencies among children living with HIV (0.57 < α < 0.68) than among children not living with HIV (0.35 < α < 0.56), whose scores were more affected by the ceiling effect. In the Abstract reasoning, Planning, Go No Go, Spatial Stroop and Processing Speed sub-tests the items had acceptable and excellent (0.76 < α < 0.97, or alpha range from .76 to .97) internal consistency showing that the tests were suited for both children living with HIV and those not living with HIV as shown in Table 4. The Alphas in the latter tests were higher in the lower-scoring sample of children living with HIV than in children not living with HIV due to less severe attenuation effects in the former group.
Table 4
Reliability test-retest of the BENCI battery
Test (N = 38) | First Visit Mean (sd) | Second Visit Mean (sd) | ICC | CI 95% | Pearson correlation |
Visuomotor Coordination (TT) | 73772.32 (34587.11) | 54539.74 (27326.12) | .66** | .35 − .82 | .51** |
Alternate Visuomotor Coordination (TT) | 75473.65 (32581.90) | 50385.71 (23478.43) | .74** | .49 − .87 | .62** |
Sustained Attention CPT (CA) | 49.06 (12.06) | 51.70 (6.26) | .13 | − .74 − .57 | .08 |
Sustained Attention CPT (RT) | 626.96 (191.41) | 618.57 (196.13) | .81** | .62 − .90 | .68** |
Immediate Verbal Memory (CA) | 5.19 (2.60) | 6.26 (3.38) | .58* | .12 − .77 | .39* |
Delayed Verbal Memory (CA) | 5.30 (3.01) | 5.58 (3.00) | .71** | .44 − .85 | .55** |
Verbal Recognition Memory (CA) | 18.89 (3.49) | 20.00 (3.08) | .41 | − .15 − .70 | .26 |
Immediate Visual Memory (CA) | 5.76 (2.60) | 6.30 (3.29) | .75** | .51 − .87 | .61** |
Delayed Visual Memory (CA) | 5.35 (3.22) | 6.47 (3.29) | .55* | .13 − .77 | .38* |
Visual Recognition Memory (CA) | 44.30 (6.08) | 45.47 (4.11) | .52* | .07 − .75 | .38* |
Comprehension of Images (CA) | 7.53 (0.97) | 7.78 (0.42) | .49* | − .01 − .74 | .45* |
Working Memory (CA) | 11.58 (5.69) | 13.76 (4.92) | .71** | .43 − .85 | .55** |
Reasoning (CA) | 13.89 (4.11) | 15.74 (4.39) | − .34 | -1.63 − .32 | − .15 |
Semantic Fluency (CA) | 8.00 (3.01) | 6.84 (3.58) | .64* | .30 − .81 | .48* |
Phonetic Fluency (CA) | 4.89 (2.48) | 5.68 (2.83) | .48* | − .00 − .73 | .32 |
Go/No-Go (CA) | 0.87 (0.14) | 0.84 (0.16) | .43* | -11 − .71 | .27 |
Go/No-Go (RT) | 0.64 (0.08) | 0.66 (0.11) | .14 | − .71 − .56 | .08 |
Selective Attention (RT) | 575.21 (148.28) | 573.16 (153.80) | .66** | .32 − .83 | .49* |
Planning FO (RT) | 5047.03 (5865.45) | 2600.13 (3168.40) | .43* | − .11 − .71 | .32* |
Table 5: BENCI – Kilifi Toolkit Convergent Validity | | |
BENCI Tests | Kilifi Toolkit Tests | | |
| People Search test | Digit Span test | Contingency Naming test | Self-Ordered Pointing Test | Verbal List Learning (VLL) test – Total CA | Nonverbal Selective Reminding Memory Test (NVSRT) | Tower Test | Ravens Progressive Matrices test | VLL Immediate Memory Span | VLL Level of Learning |
Domains | Visual Sustained and Selective Attention | Auditory Sustained and Selective Attention | EF: Flexibility - Attention and attention shift | EF: Working Memory | Memory | Non-Verbal Memory | EF: Inhibition - Planning | EF: Reasoning | Memory | Memory |
Sustained Attention CPT (CA) | Sustained Attention | -0.103 | 0.053 | -0.157 | 0.093 | .303** | 0.089 | 0.056 | .288** | 0.056 | .266** |
Sustained Attention CPT (RT) | Sustained Attention | 0.123 | -0.024 | 0.130 | -0.043 | -0.110 | 0.123 | 0.066 | -0.151 | 0.066 | -0.062 |
Working Memory (CA) | EF: Working Memory | .194* | -0.124 | -0.047 | 0.004 | .276** | 0.085 | -0.143 | 0.049 | -0.143 | .297** |
Verbal memory (CA) | Memory | -0.006 | 0.005 | -0.030 | 0.014 | .414** | -0.181 | -0.165 | .346** | -0.165 | .372** |
Verbal Memory Delayed (CA) | Memory | -0.038 | 0.043 | -0.089 | 0.116 | 0.193 | -0.102 | -0.171 | 0.162 | -0.171 | .212* |
Verbal Memory Recognition (CA) | Memory | -0.010 | -0.025 | -0.001 | 0.068 | -0.076 | 0.012 | − .186* | -0.073 | − .186* | -0.085 |
Planning Total Time | EF: Inhibition - Planning | -0.083 | -0.176 | .279** | -0.011 | -0.030 | − .405** | − .209* | -0.010 | − .209* | 0.004 |
Planning Time FO | EF: Inhibition - Planning | -0.169 | -0.128 | .219* | -0.070 | -0.060 | − .310** | -0.113 | 0.047 | -0.113 | -0.028 |
Reasoning (CA) | EF: Reasoning | − .367** | 0.119 | 0.042 | 0.000 | .424** | − .279** | -0.087 | .206* | -0.087 | .380** |
Visual Memory Immediate (CA) | Memory | 0.041 | 0.012 | 0.010 | -0.081 | .322** | -0.112 | 0.060 | 0.119 | 0.060 | .234* |
Visual Memory Delayed (CA) | Memory | 0.038 | -0.033 | -0.007 | 0.064 | .252* | -0.088 | -0.179 | .261* | -0.179 | 0.220 |
Visual Memory Recognition (CA) | Memory | -0.109 | 0.215 | -0.057 | 0.092 | 0.129 | -0.022 | 0.008 | 0.078 | 0.008 | 0.047 |
Spatial Stroop Flexibility | EF: Flexibility | -0.142 | 0.086 | 0.027 | 0.084 | .414** | − .202* | -0.037 | .327** | -0.037 | .361** |
TT – Total Time; RT – Reaction Time; CA – Correct Answers; **. Correlation is significant at the 0.01 level (2-tailed); *. Correlation is significant at the 0.05 level (2-tailed). |
Tests Retest Reliability of the BENCI
Table 4 presents the Intraclass Correlation (ICC) of the test and retest scores of the BENCI and the Pearson correlations between the repeated measurements among the 38 children not living with HIV. The Intraclass correlation for specific tests ranged from − .34 to .81. The coefficients were rather high in Sustained Attention RT, Immediate Visual Memory and Alternate Visuo-motor Coordination (ICC range from .74 to .81, r = .68 − .62). Moderate correlations were found in Immediate Verbal Memory, Delayed Visual Memory and Visual Recognition Memory (ICC range from .52 to .58, r = .39 − .38). Lack of reliability were found in Go/No-Go (RT), Sustained Attention CA and Reasoning (ICC range from .14 to − .34, r = .08 - − .15).
The test-retest reliability results showed that most of the tests were consistent on the two occasions (2 months in between t1 and t2). With clear significant gains in performance as expected by increasing test familiarity and maturation for fifteen out of nineteen subtests, except for Sustained Attention CPT, Verbal Recognition Memory (CA), Reasoning (CA) and Go/No-Go (RT) that showed no clear improvements in mean performance.
Convergent Validity
Table 5 presents the correlations between corresponding BENCI and Kilifi toolkit tests. The attention, memory, inhibition/planning, reasoning and flexibility tests in the BENCI and Kilifi were expected to correlate. However, some of these tests did not correlate as expected due attenuation effects while others correlated as expected despite the attenuation effects.
In domains of reasoning, several inhibition and a few memories related tests in the BENCI were positively correlated with tests in Kilifi toolkit, pointing to strong convergent validity across these domains. In the domain of working memory, the BENCI’s Working Memory was expected to correlate with Kilifi’s Self-Ordered Pointing Test (SOPT). However, the BENCI Working Memory test did not have a significant correlation with Kilifi’s working memory test, Self-Ordered Pointing Test (SOPT). This could be because the BENCI Working Memory test showed ceiling effects and might have been too easy for most test takers.
In the domain of memory, Kilifi’s Verbal List Learning Test and Nonverbal Selective Reminding Memory test were expected to correlate with the BENCI’s Verbal Memory and Visual Memory tests. However, none of the BENCI’s memory tests had a significant correlation with Kilifi’s Nonverbal Selective Reminding Memory Test (NVSRT). Moreover, the BENCI’s Verbal Memory Recognition and Visual Memory Recognition tests had no significant correlation to any of the Kilifi’s memory test. This outcome could be because the BENCI’s Verbal Memory Recognition and Visual Memory Recognition tests had a bit of ceiling effects while Kilifi’s NVSRT had floor effects. However, the BENCI’s Verbal Memory Immediate hits had a significant correlation with Kilifi’s Verbal List Learning’s (VLL) Immediate Memory Span (r = .37), Level of Learning (r = .40) and Total correct answers (r = .41). In addition, the BENCI’s Verbal Memory Delayed Trial was also significantly correlated with Kilifi’s Verbal List Learning’s Immediate Memory Span (r = .21). Moreover, the BENCI’s Visual Memory Immediate hits had a significant correlation with Kilifi’s Verbal List Learning’s (VLL) Immediate Memory Span (r = .23), Level of Learning (r = .34) and Total correct answers (r = .32). In addition, BENCI’s Visual Memory Delayed Trial was also significantly correlated with Kilifi’s Verbal List Learning’s (VLL) Level of Learning (r = .23) and Total correct answers (r = .25). The significance was found despite the BENCI’s Verbal Memory Delayed having a bit of floor effects. The rest of the memory tests in the BENCI and Kilifi had no ceiling or floor effects. The correlation between Kilifi’s Verbal List Learning’s (VLL) Level of Learning and Total correct answers and the BENCI’s Reasoning test was not expected. As expected, the BENCI Abstract Reasoning Test significantly correlated with Kilifi’s Raven’s Progressive Matrix (RPM) (r = .21). Both reasoning tests had no attenuation effects.
In the attention domain, Kilifi’s People Search test and FNRT test were expected to correlate with the BENCI’s Continuous Performance test and Spatial Stroop Attention test. Among the attention tests, the BENCI sustained attention test, Continuous Performance hits and reaction time test, did not have a significant correlation with Kilifi’s visual sustained and selective attention - People Search test (r = − .10; r = .12), as well as auditory sustained and selective attention test - Forward Digit Span total score (r = − .14; r = .07). People Search test had floor effects while Continuous Performance hits had ceiling effects. Moreover, the BENCI tests that contain an attention component, Reasoning (r = − .37) and Working Memory (r = .19) were also significantly correlated to Kilifi’s People Search. Kilifi’s People Search and its correlation with the BENCI’s Reasoning and Working Memory tests was unexpected as these BENCI tests are not primarily meant to measure attention.
In the flexibility domain, the BENCI’s Spatial Stroop was expected to correlate with Kilifi’s Contingency Naming test (CNT). However, the Spatial Stroop test, had no significant correlation with Contingency Naming test (CNT) (r = .03). Spatial Stroop test showed ceiling effects while CNT had no attenuation effects.
In the inhibition domain, Kilifi’s Tower Test was expected to correlate with the BENCI’s planning test. This is indeed the case, as the BENCI Planning Total Time test had a significant association with Kilifi’s Tower test (r = − .21). However, BENCI’s Planning Time of First Option test had no significant association with Kilifi’s Tower test (r = − .11). This results should be interpreted cautiously because the BENCI’s Planning Total Time test had a bit of floor effects while the Planning Time of First Option had floor effects indicating that items were relatively difficult for our test takers.
Overall, in reasoning domain, much convergence between the BENCI and Kilifi Toolkit was supported, whereas in memory and inhibition there was only partial convergence. Subtests in the flexibility, attention and working memory domains showed little convergent validity with the Kilifi mostly because of attenuation effects.
The BENCI Functionality in Age and HIV Groups
As can be seen in Table 6, children not living with HIV had better mean performance on all BENCI tests compared to those living with HIV. However, the mean group difference was significant in all subtests except Continuous Performance Test hits and reaction time, Go No Go hits, Verbal Memory Recognition hits, and Planning total time.
Table 6
Mean Group Differences in BENCI Subtests Responses
| N | Mean | Std. Deviation | Std. Error Mean | Mean Difference | Significance (2-tailed) |
Verbal Comprehension Figures Hits | HIV negative | 317 | 7.030 | .990 | .056 | .498 | .000 |
HIV positive | 258 | 6.530 | 1.171 | .073 |
Verbal Comprehension Images Hits | HIV negative | 318 | 7.540 | .743 | .042 | .760 | .000 |
HIV positive | 259 | 6.780 | 1.220 | .076 |
Continuous performance Hits | HIV negative | 322 | 47.329 | 12.756 | .712 | 1.928 | .097 |
HIV positive | 264 | 45.401 | 14.891 | .916 |
Continuous performance RT Median | HIV negative | 318 | 585.997 | 149.796 | 8.400 | 20.093 | .061 |
HIV positive | 260 | 565.904 | 107.263 | 6.652 |
Go No Go Hits | HIV negative | 315 | 42.333 | 8.007 | .451 | .287 | .709 |
HIV positive | 258 | 42.047 | 9.968 | .621 |
Go No Go Mean RT | HIV negative | 315 | .825 | .007 | .000 | − .002 | .023 |
HIV positive | 252 | .827 | .009 | .001 |
Processing Speed Median Reaction Time | HIV negative | 313 | 584.931 | 146.541 | 8.283 | 20.6036 | .048 |
HIV positive | 267 | 564.328 | 102.179 | 6.253 |
Phonetic Fluency Hits | HIV negative | 319 | 5.100 | 2.729 | .153 | 1.658 | .000 |
HIV positive | 271 | 3.440 | 2.546 | .155 |
Semantic Fluency Hits | HIV negative | 321 | 7.310 | 3.295 | .184 | 1.909 | .000 |
HIV positive | 271 | 5.410 | 3.440 | .209 |
Working Memory Hits | HIV negative | 319 | 10.510 | 6.060 | .339 | 1.955 | .000 |
HIV positive | 270 | 8.560 | 6.314 | .384 |
Verbal Memory Hits | HIV negative | 264 | 6.240 | 2.249 | .138 | .747 | .000 |
HIV positive | 206 | 5.500 | 2.040 | .142 |
Verbal Memory Hits Delayed | HIV negative | 317 | 4.670 | 3.192 | .179 | 1.125 | .000 |
HIV positive | 270 | 3.540 | 2.706 | .165 |
Verbal Memory Hits Recognition | HIV negative | 317 | 18.310 | 3.533 | .198 | .517 | .089 |
HIV positive | 270 | 17.790 | 3.810 | .232 |
Planning Total Time | HIV negative | 303 | 18178.770 | 12791.017 | 734.825 | 148.873 | .891 |
HIV positive | 259 | 18029.900 | 12776.036 | 793.864 |
Planning Time of First Option | HIV negative | 304 | 3404.050 | 3027.831 | 173.658 | -567.889 | .036 |
HIV positive | 259 | 3971.940 | 3316.305 | 206.065 |
Abstract Reasoning Hits | HIV negative | 319 | 14.870 | 4.835 | .271 | 4.261 | .000 |
HIV positive | 269 | 10.610 | 4.775 | .291 |
Visual Motor Total time | HIV negative | 320 | 70080.740 | 28797.814 | 1609.847 | -18559.339 | .000 |
HIV positive | 254 | 88640.080 | 35451.218 | 2224.407 |
Alternative Visual-Motor Total Time | HIV negative | 315 | 74940.830 | 36611.539 | 2062.827 | -22239.897 | .000 |
HIV positive | 255 | 97180.730 | 45435.071 | 2845.254 |
Visual Memory Immediate Hits | HIV negative | 278 | 6.140 | 2.295 | .138 | 1.277 | .000 |
HIV positive | 207 | 4.860 | 2.030 | .141 |
Visual Memory Delayed Hits | HIV negative | 273 | 6.150 | 2.459 | .149 | 1.289 | .000 |
HIV positive | 201 | 4.860 | 1.990 | .140 |
Visual Memory Recognition Hits | HIV negative | 314 | 44.580 | 5.122 | .289 | 1.527 | .003 |
HIV positive | 264 | 43.050 | 6.975 | .429 |
Spatial Stroop Hits | HIV negative | 328 | 66.500 | 21.956 | 1.212 | 7.270 | .000 |
HIV positive | 270 | 59.230 | 23.769 | 1.447 |
Spatial Stroop Omission Errors | HIV negative | 328 | 9.050 | 10.469 | .578 | -5.962 | .000 |
HIV positive | 270 | 15.010 | 16.667 | 1.014 |
Spatial Stroop Commission Errors | HIV negative | 328 | 11.710 | 15.242 | .842 | -2.719 | .032 |
HIV positive | 270 | 14.430 | 15.571 | .948 |
Spatial Stroop Mean Time | HIV negative | 328 | 979.504 | 223.457 | 12.338 | -68.133 | .001 |
HIV positive | 270 | 1047.638 | 249.922 | 15.210 |
We checked whether the performance within the BENCI subtests aligned with developmental models’ expectation of growth in cognitive performance as children aged and report Pearson correlations between age in years and the BENCI subtest performance for the children living with HIV- and those not living with HIV separately in Table 7. Among the children living with HIV, there were significant associations in the expected direction between age and Verbal Comprehension Images hits, Verbal Memory hits, Verbal Memory Recognition hits, planning total time, Planning Time of First Option, Abstract Reasoning hits, Visual Memory Immediate hits, Visual Memory Recognition hits and Spatial Stroop omission errors. Amon children not living with HIV, there was a significant association between age and Continuous Performance reaction time, Processing Speed reaction time, Verbal Memory hits, Abstract Reasoning hits and Visual Memory Delayed hits. The lack of significant correlations between some cognitive indicators and age could be because of attenuation effects, but might also relate to sampling issues (e.g., older participants appearing in the sample because of delayed development and the repeating of grades in school).
Table 7
Age Correlations in BENCI Subtests Responses
| HIV Positive | HIV Negative |
Pearson Correlation | Sig. (2-tailed) | N | Pearson Correlation | Sig. (2-tailed) | N |
Age in years | | | 252 | | | 292 |
Verbal comprehension Images Hits | .188** | .004 | 239 | .038 | .520 | 283 |
Continuous Performance Hits | − .050 | .435 | 244 | − .004 | .945 | 288 |
Continuous Performance RT Median | − .054 | .409 | 240 | − .168** | .004 | 285 |
Go No Go Total Hits | .074 | .253 | 239 | − .023 | .698 | 281 |
Go No Go Mean RT | − .113 | .085 | 233 | .048 | .422 | 281 |
Processing Speed Median Reaction Time | − .123 | .053 | 247 | − .156** | .009 | 279 |
Phonetic Fluency Hits | − .063 | .321 | 250 | .034 | .571 | 284 |
Semantic Fluency Hits | .025 | .689 | 250 | .006 | .921 | 286 |
Working Memory Hits | − .046 | .468 | 249 | − .100 | .094 | 284 |
Verbal Memory Hits | .156* | .032 | 190 | .255** | .000 | 234 |
Verbal Memory Hits Delayed | .048 | .450 | 249 | − .006 | .921 | 282 |
Verbal Memory Hits recognition | .151* | .017 | 249 | .034 | .564 | 282 |
Planning Total Time | .128* | .049 | 238 | − .015 | .803 | 267 |
Planning Time of First Option | .237** | .000 | 240 | .006 | .922 | 269 |
Abstract Reasoning Hits | .156* | .014 | 248 | .210** | .000 | 284 |
Visual Motor Total Time | − .126 | .055 | 233 | − .083 | .161 | 285 |
Alternative Visual Motor Total Time | .105 | .108 | 237 | − .006 | .914 | 281 |
Visual Memory Immediate Hits | .160* | .027 | 191 | .082 | .196 | 248 |
Visual Memory Delayed Hits | .096 | .195 | 185 | .230** | .000 | 243 |
Visual Memory Recognition Hits | .150* | .019 | 245 | .037 | .541 | 279 |
Spatial Stroop Hits | .106 | .095 | 249 | .034 | .561 | 292 |
Spatial Stroop Omission Errors | − .152* | .017 | 249 | − .082 | .163 | 292 |
Spatial Stroop Commission Errors | − .013 | .838 | 249 | .003 | .964 | 292 |
Spatial Stroop Mean Time | − .017 | .788 | 249 | − .072 | .218 | 292 |
**. Correlation is significant at the 0.01 level (2-tailed). |
*. Correlation is significant at the 0.05 level (2-tailed). |
Validity and Reliability of the English Version of Computerized Battery for Neuropsychological Evaluation of Children (BENCI) |
Confirmatory Factor Analyses
We tested the construct validity of Executive Functioning as proposed by Diamond for normal development (26). According to his model, the subtests that measure inhibition, flexibility, reasoning, memory and fluency together constitute executive functioning (26). These are tests that evaluate ability to make decisions, exercise self-control, pay attention, be creative, problem solve and plan all towards having good health, success in life and development. These are considered core functions in the brain hence the name executive functions. We fitted a confirmatory factor analysis model previously fitted successfully in the Arabic version of the BENCI (19) and sought to adjust the model slightly to improve fit if necessary.
A second-order model with Executive Functioning as a second-order latent factor and five first-order latent factors (i.e., Fluency, Reasoning, Memory, Inhibition and Flexibility) measured by the specific the BENCI subtests (Fig. 2.1) was specified and tested with the pooled sample including missingness handled by Full Information Maximum Likelihood. The model fit indexes suggested a good fitting model (χ2 (100, N = 604) = 245.55, p < 001, RMSEA = .049, CFI = .908, TLI = .875). However, this model had several issues. First, the Fluency factor was estimated to have a negative residual variance that we fixed at zero. Second, in this revised model, the Verbal memory factor also yielded an estimate negative residual variance that we treated similarly by fixing it at zero. Third, in the third model, the residual variance of the Alternate Visuo-motor total time also needed to be fixed to zero. Next, we considered modification indices and found that the model could be improved if we included a covariance between the residuals of Reasoning and Flexibility and between the residuals of Semantic Fluency correct answers and Verbal Memory Recognition correct answers. This further modified model showed an acceptable fit (χ2 (101, N = 604) = 205.73, p < .001, RMSEA = .041, CFI = .934, TLI = .911). Figure 2.2 presents the standardized factor loadings. An inspection of this model showed that not all indicators of Inhibition (Go No Go RT = λ − .46 ; Go No Go CA = λ .74) had significant loadings on their respective factor, indicating that these specific tests did not measure Inhibition as intended (Fig. 2.2). It also showed that the latent factor of Inhibition did not load on the Executive Functioning factor. Therefore, we removed the Inhibition factor together with its indicators and tested a second-order factor with only four factors. This model fitted well (χ2 (51, N = 604) = 135.57, p < .001, RMSEA = .052, CFI = .944, TLI = .914). Figure 2.3 presents the factor loadings of this model. Therefore, the five components of Executive Functioning as validated before did not all show up in the Kenyan sample, while Executive functioning comprised of fluency, reasoning, verbal memory and flexibility was found to fit well in the Kenyan sample. The final model with four factors each measuring executive functioning supports the construct validity for the BENCI battery, despite Heywood cases on Alternative Visuo-motor subtest.
AMOS treats missing data using full information maximum likelihood, which is considered a robust method for treating missing data. However, we checked whether model fit would be affected when using a dataset with no missing data. On running the model with no missing data, the model fit was excellent (χ2 (51, N = 327) = 64.07, p > .05, RMSEA = .028, CFI = .968, TLI = .958). This shows that the BENCI does have good construct validity though some changes in some test items and instructions are needed in future revisions of some subtests.
The BENCI Measurement Invariance
We set out to test whether the BENCI behaves the same way across the HIV positive (N = 274) and HIV negative groups (N = 330) using measurement invariance testing with multigroup confirmatory factor analysis. We used the factor model that was identified as having an excellent fit using the pooled sample as the basis and modified it to have only the four correlated first-order factors (i.e., Fluency, Reasoning, Memory, and Flexibility, each of them had their observed indicators) but no second-order factor (which is not reuired for testing measurement invariance). The model fit was excellent (χ2 (47, n = 604) = 107.76, p < 001, RMSEA = .046, CFI = .960, TLI = .933) as shown in Fig. 3.0.
We first tested for configural invariance where all factor loading, item intercepts and residual parameters were freely estimated. The model fit indexes suggested a well-fitting model (χ2 (94, N = 604) = 175.09, p < .001, RMSEA = .038, CFI = .941, TLI = .902). The factor loadings of all the indicators in both groups were significant.
We then specified a model for metric invariance where all the factor loadings were restrained to be the same across the two groups and all the other parameters were freely estimated. This model had a good fit (χ2 (102, N = 604) = 198.35, p < .001, RMSEA = .040, CFI = .930, TLI = .893). On comparing the configural to the metric invariance model, we found that there was no statistically significant difference between the chi square values, suggesting that the metric invariance was supported (Δχ2 = 23.26, df = 8, p = .003). This meant that the factor loadings were invariant and the indicator items across groups have the same associations with the latent constructs. Differences in other fit indexes also showed that the metric invariance was tenable (ΔCFI from configural to metric model < 0.01).
A scalar invariance model was then specified where the item intercepts and factor loadings were restrained to be the same across groups, while the latent mean of the latent factors in the HIV positive group was released (with an aim to check latent mean differences in flexibility, fluency, verbal memory and reasoning). This model had a poorer fit compared to the metric invariance model (χ2 (110, N = 604) = 245.12, p < .001, RMSEA = .045, CFI = .901, TLI = .860). On comparing this scalar invariance model to the metric invariance model, there was a worsening fit due to constraints on the intercepts; this was due to a statistically significant difference between the chi square values of the scalar invariance and metric invariance model (Δχ2 = 46.77, p < .001). The CFI difference also showed that the scalar invariance was not holding across all subtests (ΔCFI = 0.029). This indicates that some intercepts were not invariant and that these subtests are uniformly biased.
Using modification indices, we then specified a partial scalar invariance model where we constrained one intercept for each indicator at a time and tested whether this restraint resulted in a significant chi-square difference. For items for Verbal Comprehension (figures) CA and Visual Memory Delayed CA, the tests showed significant chi-square difference hence we freely estimated these two intercepts across groups while holding the rest of the intercepts and factor loadings to be the same across groups. This partially invariant model fitted well (χ2 (108, N = 604) = 218.38, p < .001, RMSEA = .041, CFI = .920, TLI = .884). The fit for the partial scalar invariance was better than the strict scalar invariance, and the difference between the chi square values between this model and the metric invariance model shows that partial scalar invariance fits reasonably well (Δ χ2 = 20.03, df = 6, p > .001). The CFI difference also showed that the partial scalar invariance was tenable (ΔCFI 0.010).
To summarize the series of measurement invariance tests, we conclude that metric invariance is achieved indicating that factor loadings of the BENCI are comparable across the HIV positive and HIV negative samples, and we can compare the association of the BENCI with other invariant constructs across the two groups, but not the mean comparisons of Verbal Comprehension (figures) CA and Visual Memory Delayed CA. They are not well calibrated. A partially scalar invariant model fitted the data reasonably well meaning you could compare mean difference for most of the subtests with caution for Verbal Comprehension Figures CA and Visual Memory Delayed CA.