Discriminative Validity
When applying the suggested cutoff score of ≥ 15 for a classification of ASD versus non-spectrum to all the samples, the AUC was 0.605, with a sensitivity of 0.677 and specificity of 0.593. When adjusting the cutoff score of the SCQ (e.g., cutoff ≥ 11 to ≥ 20), the AUC continued to reveal poor accuracy (range 0.581–0.634) with a better sensitivity when using a lower cutoff score (e.g., cutoff ≥ 11, sensitivity = 0.857, specificity = 0.315; Table 1) and a greater specificity when using a higher cutoff score (e.g., cutoff ≥ 20, sensitivity = 0.406, specificity = 0.759; Table 1).
Table 1. Prediction of ASD status based on Social Communication Questionnaire (SCQ)
SCQ cutoff
|
AUC
|
Sensitivity
|
Specificity
|
PPV
|
NPV
|
All (N = 187)
|
|
|
|
|
|
≥ 11
|
0.586
|
0.857
|
0.315
|
0.755
|
0.472
|
≥ 15
|
0.605
|
0.677
|
0.593
|
0.804
|
0.427
|
≥ 20
|
0.583
|
0.406
|
0.759
|
0.806
|
0.312
|
Gender
|
|
|
|
|
|
Males (N = 128)
|
|
|
|
|
|
≥ 11
|
0.539
|
0.838
|
0.241
|
0.753
|
0.350
|
≥ 15
|
0.631
|
0.675
|
0.586
|
0.818
|
0.395
|
≥ 20
|
0.597
|
0.400
|
0.793
|
0.842
|
0.324
|
Females (N = 59)
|
|
|
|
|
|
≥ 11
|
0.612
|
0.867
|
0.357
|
0.591
|
0.714
|
≥ 15
|
0.686
|
0.800
|
0.571
|
0.667
|
0.727
|
≥ 20
|
0.624
|
0.533
|
0.714
|
0.667
|
0.588
|
Age
|
|
|
|
|
|
Age 4–7 years (N = 84)
|
|
|
|
|
|
≥ 11
|
0.627
|
0.873
|
0.381
|
0.809
|
0.500
|
≥ 15
|
0.587
|
0.698
|
0.476
|
0.800
|
0.345
|
≥ 20
|
0.508
|
0.444
|
0.571
|
0.757
|
0.255
|
Age 8–11 years (N = 58)
|
|
|
|
|
|
≥ 11
|
0.553
|
0.833
|
0.272
|
0.652
|
0.500
|
≥ 15
|
0.638
|
0.639
|
0.636
|
0.742
|
0.519
|
≥ 20
|
0.649
|
0.389
|
0.909
|
0.875
|
0.476
|
Age 12 years and above (N = 45)
|
|
|
|
|
|
≥ 11
|
0.563
|
0.853
|
0.273
|
0.784
|
0.375
|
≥ 15
|
0.702
|
0.676
|
0.727
|
0.885
|
0.421
|
≥ 20
|
0.586
|
0.353
|
0.818
|
0.857
|
0.290
|
Note: AUC: the area under the receiver operating characteristic (ROC) curve; Sensitivity: number of true positive/(true positive + false negative); Specificity: number of true negative/(true negative + false positive); PPV: positive predictive value (number of true positive/[true positive + false positive]); NPV: negative predictive value (number of true negative/[true negative + false negative]).
When examining by sex, using the suggested cutoff score of ≥ 15, the SCQ achieved acceptable sensitivity among females with poor accuracy and specificity (N = 51; AUC = 0.686, sensitivity = 0.800, specificity = 0.571; Table 1), whereas AUC, sensitivity, and specificity remained poor in males (N = 136; AUC = 0.631, sensitivity = 0.675, specificity = 0.586). When adjusting the cutoff score of the SCQ (e.g., cutoff ≥ 11 to ≥ 20), the AUC among males (range 0.539–0.631) and females (AUC range 0.579–0.691) continued to reveal poor accuracy (Table 1).
To examine the discriminative validity of the SCQ with respect to age, the sample was separated into three age groups: age 4 to 7 years (N = 84), age 8 to 11 (N = 58), and age 12 and above (age range from 12.07–28.21; N = 45) based on previous literature (Corsello et al. 2007). In the youngest group, the AUC, sensitivity, and specificity were low when using the suggested cutoff score (cutoff ≥ 15; AUC = 0.587, sensitivity = 0.698, specificity = 0.476; Table 1), but the sensitivity improved when adjusting the cutoff to ≥ 11 (cutoff ≥ 11; AUC = 0.627, sensitivity = 0.873, specificity = 0.381; Table 1). For the older age groups, the 8 to 11 group had similar accuracy, sensitivity, and specificity to the whole group (cutoff ≥ 15; AUC = 0.638, sensitivity = 0.639, specificity = 0.636; Table 1), and age 12 and above yielded a fair AUC with better specificity (cutoff ≥ 15; AUC = 0.702, sensitivity = 0.676, specificity = 0.727; Table 1). All of the results were lower than reported in the original studies [1, 2].
Please insert Table 1 here.
Comparison Between Groups
False Negatives (FN) vs. True Negatives (TN). Compared to the FN, TN had more individuals who received ADHD, anxiety disorders, and depressive disorders as their final diagnoses, but fewer individuals comorbid with two or more diagnoses (Table 2). Regarding autism symptomatology, FN exhibited more restricted and repetitive behaviors than TN based on clinician observation (ADOS RRB CSS; Table 3, Fig. 1A), but caregivers’ reports (BASC Withdrawal and Atypicality) did not reveal significant differences among the two groups (Table 3, Fig. 1B). The adaptive skills assessed by the caregiver-reported BASC and VABS revealed similar results among FN and TN, suggesting similar adaptive skills among those misclassified and non-spectrum individuals (Table 3, Fig. 1C & 1D). Moreover, FN reportedly demonstrated better adaptability (i.e., the ability to adapt readily to changes in the environment) than TN on the BASC (Table 3, Fig. 1C). No significant differences on cognitive functioning, internalizing problems, or externalizing issues between groups (Table 3).
Table 2. Demographics for accurately and inaccurately classified cases
Mean ± SD
|
True Positives (TP)
|
False Negatives (FN)
|
False Positives (FP)
|
True Negatives (TN)
|
|
Comparison
|
|
Chi-Square/ F value
|
Post hoc analysis (Bonferroni or Fisher’s exact test)
|
Gender, N (%)
|
(N=90)
|
(N=43)
|
(N=22)
|
(N=32)
|
|
|
|
Male
|
65 (72.22)
|
29 (67.44)
|
14 (63.63)
|
20 (62.50)
|
|
1.37
|
|
Female
|
25 (27.78)
|
14 (32.56)
|
8 (36.37)
|
12 (37.50)
|
|
|
Age (Mean±SD)
|
|
|
|
|
|
|
|
Child
|
9.49±4.64
|
9.33±3.85
|
8.91±3.23
|
9.24±2.91
|
|
0.13
|
|
Comorbidity, N (%)
|
|
|
|
|
|
|
|
ADHD
|
41 (45.56)
|
8 (18.60)
|
11 (50.00)
|
21 (65.63)
|
|
18.78 c
|
TN & FP > FN
|
Behavioral Disorders
|
1 (1.11)
|
1 (2.33)
|
6 (27.27)
|
1 (3.13)
|
|
17.88 c
|
FP > TP, FN, & TN
|
Anxiety Disorders
|
25 (27.78)
|
9 (20.93)
|
9 (40.91)
|
18 (56.25)
|
|
12.45 b
|
TN > TP & FN
|
Depressive Disorders
|
5 (6.10)
|
1 (2.33)
|
1 (4.55)
|
6 (18.75)
|
|
8.78 a
|
TN > TP, FN, & FP
|
Language Disorders
|
41 (45.56)
|
14 (32.56)
|
4 (18.18)
|
6 (18.75)
|
|
11.00 a
|
TP > FP & TN
|
Learning Disorders
|
1 (1.11)
|
1 (2.33)
|
1 (4.55)
|
3 (9.38)
|
|
5.43
|
|
Intellectual Disability
|
24 (26.67)
|
6 (13.95)
|
1 (4.55)
|
1 (3.13)
|
|
12.96 b
|
TP > FP & TN
|
Any Comorbidity
|
77 (85.56)
|
33 (76.74)
|
16 (72.73)
|
16 (50.00)
|
|
16.48 d
|
TP, FN, & FP > TN
|
Note: p value a: p < 0.05; b: p < 0.01; c: p < 0.001; d: p < 0.0001
Table 3
Comparisons for accurately and inaccurately classified cases
Mean ± SD
|
True Positives (TP)
|
False Negatives (FN)
|
False Positives (FP)
|
True Negatives (TN)
|
Comparison
|
Chi-Square/F value
|
Post hoc analysis
(Bonferroni or Fisher’s exact test)
|
Cognitive Functioning (Mean ± SD)
|
(N = 90)
|
(N = 43)
|
(N = 22)
|
(N = 32)
|
|
|
Full-scale IQ
|
88.57 ± 23.67
|
89.97 ± 27.40
|
92.94 ± 12.74
|
103.33 ± 16.53
|
2.83 a
|
TN > TP
|
Verbal IQ
|
86.61 ± 27.47
|
94.29 ± 23.19
|
83.06 ± 14.59
|
105.85 ± 16.56
|
4.29 b
|
TN > TP
|
Nonverbal IQ
|
86.68 ± 27.28
|
89.87 ± 25.85
|
92.56 ± 16.03
|
101.93 ± 16.78
|
2.62 a
|
TN > TP
|
Symptoms Severity (Mean ± SD)
|
|
|
|
|
|
|
ADOS
|
|
|
|
|
|
|
ADOS Comparison Score
|
7.38 ± 2.54
|
6.84 ± 2.47
|
3.12 ± 2.91
|
4.22 ± 2.76
|
14.24 d
|
TP > TN & FP,
FN > FP
|
Social Affect CSS*
|
6.79 ± 2.27
|
6.29 ± 1.27
|
3.67 ± 2.12
|
4.24 ± 2.46
|
9.64 d
|
TP > TN & FP
FN > FP
|
Restricted & Repetitive Behavior CSS*
|
7.98 ± 2.43
|
6.97 ± 2.01
|
3.00 ± 2.45
|
4.12 ± 1.73
|
12.28 d
|
TP & FN > TN & FP
|
BASC
|
|
|
|
|
|
|
Externalizing Problem
|
61.04 ± 12.54
|
59.77 ± 12.95
|
72.00 ± 18.56
|
68.55 ± 12.43
|
5.44 b
|
FP > TP & FN
|
Internalizing Problems
|
54.03 ± 10.12
|
56.86 ± 13.71
|
63.50 ± 13.51
|
60.70 ± 15.04
|
3.87 a
|
FP > TP
|
Behavioral Symptoms Index
|
71.51 ± 11.87
|
65.43 ± 11.93
|
75.50 ± 10.72
|
70.32 ± 13.55
|
3.40 a
|
FP > FN
|
Adaptive Skills**
|
29.21 ± 8.49
|
37.91 ± 7.99
|
30.80 ± 5.03
|
34.83 ± 7.33
|
10.68 d
|
FN > FP & TP,
TN > TP
|
Hyperactivity
|
67.19 ± 12.61
|
64.14 ± 11.81
|
71.40 ± 12.63
|
69.09 ± 13.68
|
1.59
|
|
Aggression
|
57.63 ± 13.85
|
56.74 ± 13.23
|
71.61 ± 20.10
|
66.70 ± 13.55
|
6.93 c
|
FP > TP & FN
|
Conduct Problems
|
54.59 ± 13.06
|
54.54 ± 14.24
|
68.56 ± 17.89
|
61.79 ± 11.18
|
5.56 b
|
FP > TP & FN
|
Anxiety
|
54.07 ± 13.27
|
55.71 ± 14.69
|
62.65 ± 16.07
|
55.87 ± 13.55
|
3.17 a
|
FP > TP
|
Depression
|
58.33 ± 11.89
|
59.91 ± 13.78
|
66.20 ± 13.41
|
65.74 ± 14.40
|
3.06
|
|
Somatization
|
49.89 ± 9.41
|
51.57 ± 12.32
|
54.60 ± 13.49
|
55.04 ± 16.50
|
1.52
|
|
Atypicality
|
78.34 ± 17.84
|
64.86 ± 15.40
|
77.00 ± 13.97
|
64.91 ± 13.05
|
7.89 d
|
TP > TN & FN,
FP > FN
|
Withdrawal
|
73.38 ± 12.95
|
62.51 ± 10.75
|
69.75 ± 14.45
|
58.83 ± 14.83
|
10.08 d
|
TP > TN & FN,
FP > TN
|
Attention
|
65.97 ± 8.90
|
63.00 ± 8.34
|
66.00 ± 6.18
|
65.57 ± 7.93
|
1.10
|
|
Adaptability**
|
36.67 ± 8.39
|
42.49 ± 10.26
|
32.70 ± 5.80
|
34.96 ± 7.08
|
7.16 c
|
FN > TP, FP, & TN
|
Social Skills**
|
32.58 ± 10.01
|
40.00 ± 7.26
|
34.55 ± 8.60
|
37.00 ± 5.82
|
6.08 c
|
FN > TP
|
Leadership**
|
31.78 ± 6.95
|
38.77 ± 6.21
|
34.00 ± 7.22
|
37.11 ± 6.81
|
7.23 c
|
FN & TN > TP
|
Activities of Daily Living**
|
30.66 ± 10.72
|
38.00 ± 9.24
|
32.35 ± 6.74
|
37.57 ± 13.19
|
5.20 b
|
FN & TN > TP
|
Functional Communication**
|
29.03 ± 10.85
|
38.06 ± 9.90
|
32.85 ± 6.43
|
37.04 ± 9.37
|
8.08 d
|
FN & TN > TP
|
VABS
|
|
|
|
|
|
|
Communication
|
65.84 ± 18.13
|
75.90 ± 19.08
|
75.94 ± 12.52
|
81.83 ± 10.44
|
7.93 d
|
TN & FN > TP
|
Daily Living Skills
|
67.86 ± 14.39
|
76.50 ± 14.46
|
78.00 ± 8.62
|
80.90 ± 10.51
|
9.07 d
|
TN, FP, & FN > TP
|
Social Skills
|
61.54 ± 13.02
|
73.24 ± 11.73
|
68.59 ± 10.79
|
76.72 ± 8.66
|
15.84 d
|
TN & FN > TP
|
Motor Skills
|
77.76 ± 12.04
|
81.57 ± 15.68
|
76.71 ± 20.84
|
89.38 ± 18.28
|
1.06
|
|
Adaptive Behavior Composite
|
64.22 ± 12.73
|
73.93 ± 12.46
|
82.53 ± 7.32
|
77.59 ± 7.95
|
12.72 d
|
TN, FP, & FN > TP
|
a: p < 0.05; b: p < 0.01; c: p < 0.001; d: p < 0.0001
* The Calibrated Severity Score (CSS) is a standardized version of ADOS-2 raw total scores aimed to minimize the impact of factors such as age, language, and cognitive ability.
** Scores are reversed for the Adaptive Scales; lower scores represent poorer skills/performances, and higher scores reflect better skills/performances.
False Negatives (FN) vs. True Positives (TP). FN demonstrated similar comorbidity, cognitive functioning, and autism symptomatology with TP based on direct neuropsychological measures and clinicians’ observation on the ADOS-2 (Tables 2 & 3, Fig. 1A). Caregivers also reported similar internalizing and externalizing problems on the BASC among FN and TP (Table 3). Compared to TP, FN displayed less atypical behaviors and social withdrawal based on caregivers’ observation on the BASC (Table 3, Fig. 1B). FN also exhibited better adaptive skills then TP on the BASC and VABS (Table 3, Fig. 1C & 1D).
False Positives (FP) vs. True Positives (TP). Compared to TP, FP had more individuals who were diagnosed with behavioral disorders, but fewer individuals received diagnosis of intellectual disability (Table 2). Consistent with their final diagnoses, TP displayed higher ADOS CSS than FP on SA, RRB, and total comparison score of ADOS (Table 3, Fig. 1A). However, caregivers of FP reported greater externalizing and internalizing problems than TP (Table 3). Specifically, caregivers of FP reported clinically significant scores on domains of hyperactivity, aggression, and conduct problems, and their scores on aggression and conduct problems were significantly higher than TP (Table 3, Fig. 1B). Caregivers of FP also reported greater internalizing problems, especially anxiety, which were significantly higher than TP group (Table 3, Fig. 1B). In general, caregivers of FP reported similar adaptive skills with TP across domains on the BASC and the VABS (Table 3, Fig. 1C & 1D), though FP reportedly displayed better daily living skills than TP on the VABS, resulted in greater overall Adaptive Behavioral Composite than TP (Table 3, Fig. 1D).
False Positives (FP) vs. False Negatives (FN). Compared to FN, FP had more individuals who received ADHD and behavioral disorders as their final diagnoses among this sample (Table 2). Consistent with their final diagnoses by the licensed psychologists, FN displayed higher ADOS CSS than FP on SA, RRB, and total comparison score of ADOS (Table 3, Fig. 1A). When examining internalizing and externalizing symptoms on the BASC, caregivers of FP reported greater behavioral issues than FN, including aggression and conduct problems (Table 3, Fig. 1B). The FP group also exhibited more atypical behaviors than FN (Table 3, Fig. 1B). In terms of adaptive skills, caregivers of FN and FP generally reported similar adaptive skills on the BASC and VABS (Table 3, Fig. 1C & 1D), with an exception that FN reportedly demonstrated better adaptability than FP based on the BASC (Table 3, Fig. 1C).
Please insert Tables 2 & 3 here.