Confirmatory Factor Structure of the Kaufman Assessment Battery for Children Second Edition: Consistency With Cattell-Horn-Carroll Theory

Transcription

1 Confirmatory Factor Structure of the Kaufman Assessment Battery for Children Second Edition: Consistency With Cattell-Horn-Carroll Theory Matthew R. Reynolds, Timothy Z. Keith, Jodene Goldenring Fine, Melissa E. Fisher, and Justin A. Low University of Texas at Austin The Kaufman Assessment Battery for Children-Second Edition (KABC-II) is a departure from the original KABC in that it allows for interpretation via two theoretical models of intelligence. This study had two purposes: to determine whether the KABC-II measures the same constructs across ages and to investigate whether those constructs are consistent with Cattell-Horn-Carroll (CHC) theory. Multiple-sample analyses were used to test for equality of the variancecovariance matrices across the 3- to 18-year-old sample. Higher-order confirmatory factor analyses were used to compare the KABC-II model with rival CHC models for children ages 6 to 18. Results show that the KABC-II measures the same constructs across all ages. The KABC-II factor structure for school-age children is aligned closely with five broad abilities from CHC theory, although some inconsistencies were found. Models without time bonuses fit better than those with time bonuses. The results provide support for the construct validity of the KABC-II. Additional research is needed to more completely understand the measurement of fluid reasoning and the role of time bonuses on some tasks. Keywords: intelligence, assessment, Kaufman Assessment Battery for Children, confirmatory factor analysis, invariance The first edition of the Kaufman Assessment Battery for Children (KABC; Kaufman & Kaufman, 1983) was a ground-breaking measure. Unlike most tests at the time, it was based on cognitive processing and neuropsychological theories. The KABC was designed to assess simultaneous and sequen- Matthew R. Reynolds, MA, is a doctoral student in School Psychology and Quantitative Methods at the University of Texas at Austin. His research interest areas are in intelligence and pervasive developmental disorders. His methodological interests include general latent variable modeling with a specific interest in multiple-group mean and covariance structure analysis. He earned his MA from East Carolina University and worked as a school psychologist before attending the University of Texas. The PhD School Psychology Program at the University of Texas at Austin is accredited by the APA. 511 School Psychology Quarterly Copyright 2007 by the American Psychological Association 2007, Vol. 22, No. 4, /07/$12.00 DOI: /

2 512 Reynolds et al. tial mental processing modeled after some of the early work of Luria and Das. The technical characteristics of the KABC set a new standard for subsequent measures. Numerous validity studies were conducted throughout the test development phase, and the manuals provided in-depth coverage of development, norming, reliability, and validity. Questions dogged the KABC, however, as to what exactly the test measured. Designed to measure simultaneous and sequential mental processing, evidence suggested that these tests instead measured nonverbal reasoning and verbal memory skills, whereas the achievement scale measured a mixture of verbal reasoning and reading achievement (e.g., Keith & Novak, 1987). In modern parlance, the test may have measured visual processing (Gv), short-term memory (Gsm), crystallized intelligence (Gc), and reading achievement (see Kamphaus & Reynolds, 1984; Keith & Dunbar, 1984; and Flanagan & McGrew, 1997, for a cross-section of opinions and research concerning the constructs measured by the KABC). The recent revision of the KABC, the KABC-II (Kaufman & Kaufman, 2004) allows for interpretation from two theoretical models: parts of Luria s theory and Cattell-Horn-Carroll (CHC) theory. Interpretation from Luria s model involves sequential and simultaneous processing, as well as learning and planning abilities. The focus is on mental processing, and the model includes all of the subtests except for those explicitly designed to measure acquired knowledge. The CHC interpretative approach includes all of the same subtests as the Luria approach, as well as the subtests that explicitly measure acquired Timothy Z. Keith, PhD, is Professor of Educational Psychology in the School Psychology Program at the University of Texas at Austin. His research focuses on the nature and measurement of intelligence and on understanding the influences on school learning. Tim also has methodological interests in multiple regression analysis, confirmatory factor analysis, and structural equation modeling. Tim earned his PhD from Duke University in Jodene Goldenring Fine, PhD, is a post-doctoral research fellow at Michigan State University in the Department of Psychology. Her research interests include neuroimaging to identify the neural pathways of developmental disability and neuropsychological tests and measurement. Dr. Fine received her PhD in School Psychology at the University of Texas at Austin. Melissa E. Fisher, BA, is a doctoral student in the School Psychology Program at the University of Texas at Austin. Her current research interests include the prevention and treatment of depressive disorders in children and early adolescents. Justin A. Low is a graduate student in the Department of Educational Psychology at the University of Texas at Austin. His current research interests include intelligence and the assessment of intelligence. Mr. Low is working towards his doctorate in school psychology. None of the authors have any financial interest in the KABC-II or in any of the other tests mentioned in this article. We are grateful to Alan Kaufman, Mark Daniel, and the American Guidance Service for providing the data used in this research. We are responsible for any errors and for the opinions expressed in this article. Correspondence concerning this article should be addressed to Matthew R. Reynolds, University of Texas at Austin, Department of Educational Psychology, 1 University Station D5800, Austin, TX matthew.reynolds@mail.utexas.edu

3 KABC-II Structure 513 knowledge. In general, the CHC model is recommended over the Luria-based model, unless the examiner expects that scores from acquired knowledge would compromise the general ability (Fluid-Crystallized Index; FCI) score (Kaufman & Kaufman, 2004, p. 4). We chose the CHC model as the theoretical model to investigate in this study. CHC theory, derived from a psychometric orientation, serves as an umbrella term under which Cattell and Horn s Gf-Gc theory and Carroll s three-stratum theory are combined (McGrew, 2005). CHC theory posits three strata: stratum III, general ability (g); stratum II, 8 to 10 broad abilities; and stratum I, over 60 narrow abilities. The eight broad CHC abilities typically classified as intellectual in nature include fluid reasoning (Gf), crystallized intelligence (Gc), visual processing (Gv), auditory processing (Ga), short-term memory (Gsm), long-term retrieval (Glr), processing speed (Gs), and sometimes quantitative reasoning (RQ) or math knowledge (Gq; see for review McGrew, 2005). The KABC-II is designed to provide estimates of g as well as measures of Gc, Gf, Gv, Gsm, and Glr broad abilities. The theory and the nature of these abilities are described in detail elsewhere (e.g., Carroll, 1993; McGrew, 2005). KABC-II TEST DEVELOPMENT The structure of the KABC-II was developed via confirmatory factor analyses used in exploratory fashion (Kaufman & Kaufman, 2004). In general, the CHC models presented in the manual suggest that the various factor structures provide good fit to the standardization data. At the individual subtest level, the goal was to construct measures that integrated CHC stratum I abilities because a pure measure of a stratum I ability would lack the complexity needed to assess higher level functioning (Kaufman & Kaufman, 2004, p. 16). The manual presents alternative CHC stratum I abilities that may be measured by specific subtests. The manual does not, however, indicate whether all of these alternative interpretations of stratum I abilities were tested empirically. In sum, the manual provides a thorough description of the test development process. Nevertheless, as with the release of any new measure, independent evaluation is necessary to assess the psychometric properties of the KABC-II and to understand the nature of the constructs being measured by the test. Despite the laudable reliance on theory and research in developing the KABC-II, questions remain concerning the test and the constructs that it measures.

4 514 Reynolds et al. REMAINING QUESTIONS Age Invariance The KABC-II is designed to measure processing and cognitive abilities across the 3- to 18-year age span. The subtest structure of the instrument changes across these ages; research presented in the manual suggests that different, and fewer, constructs may be measured for younger children. Thus, for ages 7 through 18, the authors contend that the KABC-II measures memory, visual processing, long-term retrieval, fluid reasoning, and crystallized intelligence (Gsm, Gv, Glr, Gf, and Gc); for ages 4 through 6, however, the test measures Gsm, Gv, Glr, and Gc (Kaufman & Kaufman, 2004, p. 104). This finding reflects a classic conundrum in intelligence research: is intelligence less complex for younger children or is it simply more difficult to measure separate abilities for younger children (cf. Carroll, 1993, pp )? Therefore, it is unclear whether the KABC-II actually measures different constructs across the ages or whether changes in factor structure are a result of some constructs being difficult to measure for young children. What Does the KABC-II Measure? Rival Hypotheses Questions remain about the nature of the constructs measured by the KABC-II. Although the manual presents justification for the placement of subtests on each scale, alternative interpretations are possible, particularly because many of these tests are designed to measure multiple first-stratum abilities (Kaufman & Kaufman, 2004). Alternative interpretations of contemporary CHC theory suggest that some of the KABC-II subtests might measure different abilities than those that are suggested in the manual. Does Gestalt Closure Measure Gc? Gestalt Closure is considered a measure of the CHC narrow ability closure speed (CS), which is subsumed by Gv (Kaufman & Kaufman, 2004, p. 66). CS has been defined as the ability to combine visual parts quickly into a meaningful whole without knowing the pattern in advance (Carroll, 1993; McGrew, 2005). Given that definition, it is plausible that Gestalt Closure measures CS. In addition to CS, Gestalt Closure might also measure the CHC narrow ability General Information (K0), which has been defined as a range of general knowledge, and is subsumed by Gc or

5 KABC-II Structure 515 crystallized intelligence (Flanagan & Ortiz, 2001). Carroll (1993) indicated that successful performance on CS tasks requires that the object stimulus be represented in a person s long-term knowledge so that it is available for retrieval. A necessary condition, then, is that the stimuli be of common experience to all examinees. Gestalt Closure appears to meet this condition, but though necessary, common experience is not sufficient because it is not given that the objects are stored in long-term knowledge and available for immediate retrieval. Although everyone is exposed to the object, some individuals acquire this knowledge more easily than others, which by definition, is related to definitions of Gc (McGrew, 2005). For this reason, it is important to determine whether Gestalt Closure measures Gc. Do Hand Movements Measure Gf? Hand Movements is considered a measure of Gsm (Kaufman & Kaufman, 2004, p. 62). Unlike other Gsm tasks on the KABC-II, Hand Movements seems to be a cognitively complex task requiring the use of several cognitive processes, not all of which are related to short-term memory. Successful performance on this subtest requires strategy, the ability to manipulate and sequence information, and the ability to synthesize information for retrieval, all of which are more suggestive of working memory and fluid reasoning than basic short-term memory ability. Some researchers from experimental psychology argue that working memory capacity is related more to Gf than to Gsm (Kane, Hambrick, & Conway, 2005; Kyllonen & Christal, 1990; Sü, Oberauer, Wittmann, Wilhelm, & Schulze, 2002). Tasks designed to measure working memory and Gf are often cognitively complex and require the utilization of many different cognitive processes. Given the nature of the Hand Movements tasks, if working memory has a stronger relation with Gf than it does with Gsm, Hand Movements may require fluid reasoning ability in addition to short-term memory. Does Pattern Reasoning Measure Gv? Pattern Reasoning was designed to measure the CHC stratum I abilities of induction and visualization (Kaufman & Kaufman, 2004, p. 69). This design appears valid because induction tasks often share variance with spatial visualization tasks (Carroll, 1993). In the analyses of the factor structure, however, Pattern Reasoning was only allowed to load on the Gf factor. If indeed Pattern Completion requires visualization, then it should also load on a Gv factor.

6 516 Reynolds et al. Does Story Completion Also Measure Gc or Gv? Story Completion requires the examinee to select appropriate pictures to complete a story. According to the manual, Story Completion measures induction, visualization, sequential reasoning, and general information (Kaufman & Kaufman, 2004, p. 69). Given the panoply of skills required for success on the Story Completion subtest, it seems important to determine whether Story Completion taps Gc and Gv abilities in addition to Gf. Do Rover and Block Counting Measure Gf? To the surprise of the KABC-II authors, who developed Rover to measure Gf, Rover measured Gv, not Gf. The authors concluded that Rover is primarily a measure of Gv because visualization is required, but it is also a measure of Gf... (Kaufman & Kaufman, 2004, p. 65). More research is needed to help explain the discrepancy between the theoretical hypothesis of Rover being a measure of Gf and the empirical finding of Rover being a measure of Gv. The manual indicates that Block Counting measures math achievement as well as visualization (Kaufman & Kaufman, 2004, p. 63). It is also plausible that Block Counting requires the narrow ability quantitative reasoning (RQ) and not necessarily math achievement. That is, Block Counting might require Gf-like reasoning abilities that are important for successful performance on mathematical tasks, rather than acquired math knowledge. Are Gf and Gv Distinct Factors? The authors of the KABC-II found that on the core battery, ages 7 to 18, Gf and Gv were distinct but similar constructs. For younger ages, 5- and 6-year-olds, the Gf and Gv factors were not separable (Kaufman & Kaufman, 2004). The Gf-Gv distinction needs to be explored. What Is the Effect of Bonus Points for Timed Tests? Three subtests (Story Completion, Triangles, and Pattern Reasoning) allow for points related to time. Carroll (1993) argued that reasoning tasks should be administered without a time limit so that reasoning, not speed of processing, is measured. The manual notes that the effects of subtests with

7 KABC-II Structure 517 and without time points were explored via factor analysis, but few details of these analyses are reported (Kaufman & Kaufman, 2004, p. 84). Does the addition of this time component improve measurement, or does it introduce an unintended speed component? Purpose This research had two purposes. The first purpose was to determine if the KABC-II measured the same constructs across its 3- to 18-year age span. The second purpose was to understand the nature of the constructs measured in the 6- to 18-year-old age range. We focused on the factor structure of ages 6 to 18 because space limitations preclude a thorough examination of separate factor structures for younger children. We applied CHC theory to our research for several reasons. First, space limitations and a limited number of subtests prevent the application of other theories such as the PASS (Naglieri & Das, 2005) or Vernon s hierarchical model of intelligence (Vernon, 1950). Second, CHC theory is generally recommended as the primary theoretical approach to interpretation of the KABC-II (Kaufman & Kaufman, 2004) and thus the evaluation of the KABC-II structure as it relates to CHC theory has important implications for the interpretation of the test from this orientation. Third, on the KABC-II, the CHC theory interpretation includes all subtests, whereas the Luria-Das interpretation does not include the Knowledge/Gc tests. This research is designed to add to knowledge concerning the construct validity of the KABC-II. This particular aspect of construct validation is referred to as structural validation because it involves testing alternative factor structures as rival hypotheses to determine how the subtests are related to the theoretical structure of the test (see Benson, 1998). One analytical method useful for this type of construct validation is confirmatory factor analysis (CFA; Keith, 2005). We applied CFA to compare the factor structure of the KABC-II to rival factor structures derived from CHC theory. Factor analysis is useful for construct validation because it shows the extent to which the latent factors explain performance on various subtests. The standardized loadings from factor analysis are similar to standardized regression coefficients. The loadings of the subtests on the factors represent the link between theoretical and observable worlds. CFA also produces model fit statistics that suggest plausible versus unreasonable models, as well as fit statistics that help researchers determine which factor structure provides a better fit to the data, and thus allows for tests of competing factor models. It is worth noting that the KABC-II authors (Kaufman &

8 518 Reynolds et al. Kaufman, 2004) used confirmatory factor analyses as part of the development process of the KABC-II. Not all of the analyses were explicated in detail, however, and independent evaluation of new measures is always worthwhile. It should be noted that the primary purpose of this research was not to evaluate the placement of KABC-II subtests onto scales or the division of subtests into core versus supplemental measures. Although the research has implications for those questions, our primary purpose was to determine what constructs are measured by the KABC-II and whether the test is consistent with CHC theory. METHOD Participants The sample for the investigation was the KABC-II standardization sample (ages 3 18) used by AGS Publishing in their reported CFA studies. As described in the KABC-II manual, the entire sample was stratified according to age, maternal education, ethnic group, geographic region, parental education within ethnic group, educational placement, and educational status (for 18-year-olds). Important sample characteristics are shown in Table 1. Table 1. Demographic Characteristics of the Normative Sample for the KABC-II Variable N Total sample 3,025 Sex Male 1,504 Female 1,521 Age per year per year Race/ethnicity White 1,861 Hispanic 545 African American 465 Other 154 Maternal education 11th grade 429 High school grad, GED years of college year degree 714

9 KABC-II Structure 519 Measure Kaufman Assessment Battery for Children, Second Edition (KABC-II) The KABC-II is an individually administered measure of cognitive abilities designed for use with children and adolescents ages 3 to 18. The expanded battery takes about 90 minutes to administer to children ages 5 and 6, and about 100 minutes to administer to children ages 7 to 18. Analyses Age Invariance The first purpose of this research was to determine whether the KABC-II measures the same constructs across its 3- to 18-year age span. To do so, we tested the equality of the variance/covariance matrices across the ages. Because any factor structure is contained within the covariance matrix, this process tested the equality of measurement of constructs across ages without specifying the nature of those constructs (Keith, 2005). This process also allowed us to test the equivalence in construct measurement across ages even when the structure of the KABC-II (i.e., the subtests administered) changes across ages. Amos 5.0 (Arbuckle, 2003) was used for these and all CFA analyses. The first step was to test the equivalence of the variances and covariances across ages. Model fit was evaluated to determine whether these constraints were reasonable. Across-age equivalence was further evaluated by freeing the constraints at each age level. Change in 2 ( 2 ) was used to test the statistical significance of model changes. Thus for step two, the variances and covariances for age 3 were no longer constrained to be equal to those of the other age levels, and 2 was examined to determine whether this change led to a statistically significant improvement in model fit. If model fit improved significantly, we would conclude that the KABC-II measures a different set of constructs at age 3 than at other age levels. Although the subtests that make up the KABC-II are consistent from ages 7 through 18, the structure of the test changes at younger ages; new subtests are added and others are deleted. Variances and covariances were specified as equivalent for the ages where the subtest was used. Thus, the covariance between Atlantis and Triangles was specified as invariant across all ages because the two tests appear at all ages. In contrast, the covariance between Atlantis and Conceptual Thinking was only constrained to be

10 520 Reynolds et al. equal for ages 3 through 6 because the Conceptual Thinking subtest was not standardized beyond age 6 (cf. McArdle, 1994; Wothke, 2000). The sample sizes varied across ages and, to a lesser extent, across subtests (from 99 participants for the Gestalt Closure test at age 18 to 250 for many subtests for the younger age levels). To make sure each age level was weighted equally in the analysis, and to control for excessive power, the sample size for each matrix was set to 100 for a total sample size of Full-information maximum likelihood methods were used to deal with variations in sample sizes within ages. Maximum likelihood methods are recommended by methodologists as one of the preferred methods of handling incomplete data (cf. Arbuckle, 1996; Schafer & Graham, 2002). Rival Hypotheses Higher-order confirmatory analyses were used to test rival models against a CHC baseline model that included all of the available KABC-II subtests. A calibration/validation approach was used. To perform these analyses, the standardization sample for ages 6 to 18 was divided in half at random. Two variance/covariance matrices (a calibration matrix and a validation matrix) were estimated from these two subsamples; these matrices were used as input for CFAs. All analyses were performed initially with the calibration matrix. For purposes of cross-validation, all model modifications supported in the calibration analyses were incorporated into a single model that was analyzed using the validation matrix. Last, using the output obtained from the analysis with the validation matrix, we calculated the proportion of subtest variability explained by the general factor and the broad abilities. Several indices of model fit are presented in the tables and text. Because different fit indices are attuned to different aspects of fit, varying fit indices are presented in the tables. To assess the fit of single models, we used the rules of thumb provided by Hu and Bentler (1998, 1999). Specifically, Hu and Bentler (1998, 1999) recommended using joint criteria when determining whether to retain or reject a model as being plausible: the Bentler s Comparative Fit Index (CFI) of.96 and the Standardized Root Mean-Square Residual (SRMR) of.10, or the Steiger and Lind Root Mean-Square Error of Approximation (RMSEA) of.06 and the SRMR of.10. To compare rival nested models (that is, constrained versions of the other models), we used change in chi-squared ( 2 ; Keith, 2006; Loehlin, 2004). To compare non-nested models, we used the Akaike Information Criteria (AIC); smaller AIC values are preferred (Loehlin, 2004). These fit indices were used to compare competing models because they tend to be more sensitive to differences in competing models than are the other indices reported (Loehlin, 2004).

11 KABC-II Structure 521 RESULTS Does the KABC-II Measure the Same Constructs Across the 3 18 Age Span? Table 2 shows the results of the tests of invariance across the 3- to 18-year age levels of the KABC-II. Although not shown in the table, the initial invariance model (the all-constrained all-groups model), specifying that the covariances and variances of the KABC-II subtests were the same across the entire age range, showed an excellent fit to the KABC-II standardization data (CFI 1.00, SRMR.064, RMSEA.000). These results suggest that the KABC-II indeed measures the same constructs across ages. The SRMR suggests an average difference in correlations of only.064 between the actual matrices and a single, averaged matrix across all ages. Subsequent models freed the variances and covariances from a single age range. A comparison of fit ( 2 ) for each of these models with the initial model tested whether allowing each age to vary resulted in a better fit than did the model in which the age was not allowed to vary. A finding of a statistically significant decrease in 2 would suggest that the KABC-II measures different constructs for the age tested than for the other age levels. As shown in the table, none of the 2 values was statistically significant, further supporting the conclusion that the KABC-II measures the same constructs across all age levels of the test. The table also Table 2. Test of Invariance of Measurement of the KABC-II for Ages 3 to 18 Model 2 df 2 df p All constrained all groups 1, ,918 Age 3 free 1, , Age 4 free 1, , Age 5 free 1, , Age 6 free 1, , Age 7 free 1, , Age 8 free 1, , Age 9 free 1, , Age 10 free 1, , Age 11 free 1, , Age 12 free 1, , Age 13 free 1, , Age 14 free 1, , Age 15 free 1, , Age 16 free 1, , Age 17 free 1, , Age 18 free 1, , Ages 3-6 free , Ages 7-12 free , Ages free ,

12 522 Reynolds et al. shows the results of analyses in which groups of ages were allowed to vary. These age groupings were chosen because they represent substantive differences in structure according to the KABC-II manual (Kaufman & Kaufman, 2004, pp ). For these comparisons as well, the changes in 2 were not statistically significant, meaning that we could not reject the hypothesis that the KABC-II measures the same constructs for these ages as for others. The analyses thus strongly support the contention that the KABC-II measures the same constructs across ages. Rival Hypotheses Calibration and validation variance/covariance matrices from the KABC-II standardization were used to test rival hypotheses. 1 Because some subtests were not administered to all participants, we used the full information maximum likelihood method in Amos to create these matrices. Sample sizes for covariances ranged from 859 to 1,188 in the calibration sample and from 835 to 1,187 in the validation sample. For analysis purposes, sample sizes were set to 1,000 for each matrix. The matrices are available from the first author, by request. The KABC-II baseline model shown in Figure 1 was used to compare rival models. All of the subtests were included in the model. The baseline model is a second-order factor model that is very similar (except for the two additional delayed subtests) to the model presented in the KABC-II manual for the 7- to 12- and 13- to 18-year-old age groups (Kaufman & Kaufman, 2004, p. 106). The model includes 16 subtests, five first order CHC factors, and one second order or g factor. This model is similar to the scoring structure of the test where the subtests are summed into the five broad scales (Gc, Gv, Gf, Glr, Gsm) and the overall composite (Fluid- Crystallized Index) score. The fit indices shown in Figure 1 and in Table 3 suggest that this initial model, a model consistent with the theoretical and scoring structure of the KABC-II, is a plausible model. Said differently, the fit indices suggest that the KABC-II model could have indeed produced the KABC-II standardization data. All first- and second-order factor loadings are reasonable, supporting the grouping of subtests and the hierarchical nature of the tests. The possible exception is the loading greater than one of the first-order Gf factor on the second-order g factor. Loadings of Gf on g approaching or even exceeding unity are not uncommon, however, suggesting the near- 1 The ages 6 to 18 were chosen because in the standardization sample these age groups were administered the same subtests, and the constructs measured by these subtests are similar across the age levels.

13 KABC-II Structure 523 Figure 1. Baseline KABC-II Model for ages 6 to 18 used for model comparisons. CFI Comparative Fit Index; RMSEA Root Mean Square of Approximation; SRMR Standardized Root Mean Square Residual; AIC Akaike Information Criterion. equivalence of g and Gf (Gustafsson, 1994; Keith, 2005). The Gv factor had the second highest loading on g, followed by Gc. Gestalt Closure The first rival hypothesis was that Gestalt Closure measures Gc. This and subsequent hypotheses were tested by comparing competing models

14 524 Reynolds et al. with different patterns of constraints. To test this hypothesis, Gestalt Closure was allowed to load on both Gv and Gc first-order factors. This change resulted in a statistically significant improvement in model fit as shown by the 2 in Table 3. This improvement suggests that Gestalt Closure also measures crystallized intelligence. To test whether Gestalt Closure measures Gc only or both Gc and Gv, the loading of Gestalt Closure on Gv was constrained to be zero, but the loading of Gestalt Closure on Gc was freed. As shown in Table 3, this constraint led to a statistically significant increase in 2, indicating that this model provided a worse fit for the data. The model with Gestalt Closure loaded on Gc only, however, had a lower AIC than the model with Gestalt Closure loading on Gv only, suggesting that Gestalt Closure may be a better measure of Gc than of Gv. In addition, the factor loadings in the model with Gestalt Closure loading freely on both Gc and Gv showed a loading of Gestalt Closure on Gc of.34, compared to a loading on Gv of.24. To test if these loadings were significantly different from each other, the unstandardized loadings were constrained to be equal. This constrained model did not fit statistically significantly worse than the model allowing the test to load freely on both factors ( 2 [97] , 2 [1] 0.15, p.70). Gestalt Closure thus appears to measure both Gc and Gv abilities. 2 Hand Movements The second rival hypothesis was that Hand Movements measures Gf. To test this hypothesis, the path from Hand Movements to Gf was unconstrained so that Hand Movements loaded freely on both factors. This change led to a statistically significant improvement in model fit (see Table 3). Next, to determine if Hand Movements measures Gsm (in addition to Gf), the loading of Hand Movements on Gsm was constrained to be zero. This constraint led to degradation in the model fit, as shown by the 2 value in Table 3. The AIC for the model with Hand Movements on Gf only was lower than the model with Hand Movements on Gsm only, suggesting 2 Again, it is possible to test such hypotheses by comparing the fit of competing models with different patterns of constraints versus free loadings. As noted by an anonymous reviewer, the plethora of fit indices available continues to be a source of debate in the SEM literature (cf., Marsh, Hau, & Grayson, 2005). AIC and 2 were used to compare competing models in this research because these fit indices are sensitive enough to detect differences in competing models (cf. Hoyle & Panter, 1995; Loehlin, 2004). Furthermore, 2 allows a statistical test of model differences. The other fit statistics shown change little from model to model (cf. Cheung & Rensvold, 2002).

15 KABC-II Structure 525 that if the test measures one ability, that ability may be closer to Gf and than to Gsm. When the unstandardized factor loadings of Hand Movements on Gf (.37) and Gsm (.25) were constrained to be equal, the model did fit worse ( 2 [97] , 2 [1] 4.61, p.032) meaning that the loadings are significantly different from each other. Hand Movements appears to measure both Gf and Gsm, but it appears to measure Gf more strongly than Gsm. Pattern Reasoning Pattern Reasoning was tested to determine if performance on this task requires visualization ability in addition to fluid reasoning. The fit of the model improved significantly when Pattern Reasoning was freed to load on Gv in addition to Gf (see Table 3; Model 8). The factor loadings were not statistically significantly different from each other ( 2 [97] , 2 [1] 0.43, p.51) when the unstandardized factor loadings were constrained Table 3. Tests of Rival Hypotheses of the Structure of the KABC-II for Ages 6 to 18 Model 2 df df 2 p CFI RMSEA SRMR AIC 1. Baseline model Gestalt on Gv and Gc 3. Gestalt on Gc a Hand movements on Gsm and Gf 5. Hand movements b on Gf 6. Rover on Gv and Gf 7. Block counting on Gv and Gf 8. Pattern reasoning on Gv and Gf 9. Story completion on Gc and Gf 10. Story completion on Gv and Gf Note. CFI Comparative Fit Index; RMSEA Root Mean Square of Approximation; SRMR Standardized Root Mean Square Residual; AIC Akaike Information Criterion. Models are compared to Model 1 unless otherwise noted. a Model 3 compared with Model 2. b Model 5 compared with Model 4.

16 526 Reynolds et al. to be equal. Thus, as the KABC-II manual suggests, Pattern Reasoning does indeed measure both visualization and fluid reasoning skills. Story Completion Story Completion was evaluated to determine if it measures Gc or Gv in addition to Gf, as indicated in the manual (Kaufman & Kaufman, 2004, p. 69). As shown in Table 3 (Models 9 and 10), tests of these hypotheses led to statistically nonsignificant improvements in model fit. These findings suggest that Story Completion is best interpreted as a measure of Gf and not as a measure of Gc or Gv. Rover and Block Counting Rover and Block Counting were tested to determine if they measured Gf in addition to Gv. First, Rover was freed to load on Gf in addition to Gv. Table 3 (Models 6 and 7) shows that the fit of the model did not improve when compared to the baseline model as indicated by a nonsignificant decrease in 2. In a separate test, Block Counting was freed to load on Gf in addition to Gv. This modification also led to a nonsignificant improvement in the fit of the model. These findings suggest that Block Counting and Rover are best interpreted as measures of Gv and not Gf. Does Rover Measure Gf? Because Rover was developed to be a measure of Gf, additional analyses were performed in an attempt to better understand the nature what is being measured by Rover. First, a covariance matrix was estimated using the 6- and 7-year-old age groups only. These two age groups were selected because both groups were administered an additional fluid reasoning subtest, Conceptual Thinking, during the standardization process. The inclusion of this additional subtest allowed us to create an initial Gf factor with three subtest indicators instead of two, which should provide a more robust estimate of Gf. Three models were tested using the covariance matrix estimated for 6- and 7-year-olds. The fit statistics for these models are shown in Table 4. The first model was similar to the original KABC-II model in which Rover loaded only on Gv. In the second model, the constraint from Rover to Gf was removed so that Rover was free to load on both Gf and Gv. In the third

17 KABC-II Structure 527 Table 4. Tests of Rover as a Measure of Gf and the Separability of Gf and Gv Factors Model 2 df df 2 p CFI RMSEA SRMR AIC Rover: Ages Rover on Gv Rover on Gv & Gf Rover on Gf Rover: Gf Gv a Standardized models for Gf-Gv Separability: Ages Factor correlations free 6. Gf-Gf factor r 1; all constrained Gf-Gv tests: Ages Factor correlations free 8. Gv-Gf r 1; all constrained Second-order models 9. One Gf-Gv factor: Ages 6-18 b 10. Baseline One Gf-Gv factor 6-7 c Note. CFI Comparative Fit Index; RMSEA Root Mean Square of Approximation; SRMR Standardized Root Mean Square Residual; AIC Akaike Information Criterion. All 2 model comparisons are with the previous model unless noted otherwise. a Compared to Model 2. b AIC of Model 9 is compared to the AIC of Model 1 in Table 3. c AIC of Model 11 is compared to the AIC of Model 10. model, Rover was free to load on Gf, but the loading of Rover on Gv was constrained to be zero (see Table 4). The results indicated that the second model (shown in Figure 2) led to a statistically significant improvement in the model fit when compared to the first model. The third model, however, led to a statistically significantly worse fit for the data when compared to the second model. The standardized loadings of Rover on Gf and Gv were very similar in the second model:.27 and.28. To test whether the loadings were significantly different from each other, the same model was analyzed with the unstandardized loadings of Rover on Gv and Gf set to be equal. This model resulted in a nonsignificant increase in 2. This finding suggests that for ages 6 and 7, Rover measures Gv and Gf equally well. Furthermore, comparison of the AIC values as shown in Table 4 shows little difference in whether Rover was free to load only on Gf or only on Gv. At ages 6 and 7, Rover appears to measure both Gf and Gv equally. These findings are obviously inconsistent with those presented previously and in Table 3. Taken together, our findings suggest that Rover indeed

18 528 Reynolds et al. appears to measure Gv, but the issue of whether Rover measures Gf is not completely settled. Are Gv and Gf Distinct? Two sets of analyses were conducted to answer the question of whether Gf and Gv were measured as distinct constructs on the KABC-II for ages 6 to 18 and for a sample of 6- and 7-year-olds. A first-order factor model (with no g factor) was used for the first set of analyses. This model was a standardized model, meaning that the factor variances were set to 1 rather than a single factor loading set to 1, allowing us to set the correlations to 1 (Keith, 2006, chap 14). In the baseline model, the first-order broad abilities were allowed to correlate freely, meaning that Gf and Gv were separate constructs. To test whether the two were distinct constructs, we equated the Gf and Gv factors by setting the factor correlation between them to 1, and then constraining the other factor correlations to be equivalent across the Gf and Gv factors (e.g., the correlation between Gv and Gc was constrained to be equal to the correlation between Gf and Gc). This model, a model equivalent to a model in which all Gf and Gv subtests load on a single factor, was nested within the baseline model. The additional constraints led to a statistically significant degradation in the model fit (see Table 4), suggesting that Gf and Gv are indeed better represented as two distinct factors. This method was also used to determine if the Gf and Gv distinction held up in the 6- to 7-year-old age group. These analyses were performed because the inclusion of the Conceptual Thinking measure at these age levels allowed for a Gf factor with three measures as indicators. Setting the correlation between Gv and Gf to one, while constraining the other factor correlations to be equivalent across the Gf and Gv factors, again led to degradation in model fit (see Table 4). This finding suggests that Gf and Gv are better represented as two distinct factors for 6- and 7-year-olds. The second set of analyses tested the Gf and Gv distinction using a second-order factor model. In this set of analyses, the Gf and Gv factors were combined to form a single factor. That is, all Gf and Gv subtests were loaded onto a single Gf-Gv factor. The AIC values were compared to determine if the single Gf-Gv factor model provided a better fit to the data when compared to the model that represented Gf and Gv as distinct factors. As shown in Table 4, the AIC value for the one-factor Gf-Gv model was larger than the AIC value for the baseline model. Hence, the combined Gf-Gv factor model did not explain the data better than did the baseline model, further supporting a Gf-Gv distinction for 6- to 18-year-olds. The same pattern was found using the

19 KABC-II Structure and 7-year-old data (see Table 4). Gf and Gv indeed appear to be distinct constructs on the KABC-II, at least for ages 6 and older. What Is the Effect of Bonus Points for Timed Tests? The final set of analyses used with the calibration sample tested the rival hypothesis that bonus points for timed tests change the model fit. To test this hypothesis, the untimed scores for Triangles, Story Completion, and Pattern Reasoning subtests were substituted for the timed scores for these subtests using the initial baseline model. The fit statistics suggested excellent model fit (CFI.971, RMSEA.047, SRMR.04). The 2 for this model ( with 97 df) was lower than for the same model using the timed tests, but the models are not nested so the 2 s cannot be compared. However, a comparison of the AIC index for this model (387.26) with the AIC index from the KABC-II baseline model (409.43) showed that the untimed model provided a better fit to the data than did the timed model. Thus, it appears that the factor structure of the KABC-II is better explained when a time component is not included in the scoring. This finding, in turn, suggests that the timed version of these tests adds a component to the test that results in degradation of model fit. Final Model The final step in these analyses was to apply all of the changes that were significant in the calibration analyses to a single final model using the validation matrix. The purpose of these analyses was to establish a final model that fit the data well and that had reasonable factor loadings. Two criteria were used to determine whether a factor loading was reasonable: (1) the loading had to be statistically significant and (2) the loading had to be.25 or greater. The first model included the three cross-loadings that significantly improved the model fit in the calibration analyses. Thus, Gestalt Closure was freed to load on Gv and Gc, Hand Movements was freed to load on Gsm and Gf, and Pattern Reasoning was freed to load on Gf and Gv. The model showed an excellent fit to the validation data, as evidenced by the CFI (.987), RMSEA (.033) and SRMR (.026). All of the factor loadings for this model were statistically significant. The loading of Gestalt Closure (.24) on Gv, however, was less than the.25 criterion that we established prior to the analyses (it was also less than.25 with the calibration sample model). Therefore, another

20 530 Reynolds et al. Figure 2. Model used to test whether Rover measures Gf (ages 6 and 7). CFI Comparative Fit Index; RMSEA Root Mean Square of Approximation; SRMR Standardized Root Mean Square Residual; AIC Akaike Information Criterion; TLI Tucker-Lewis Index. model was analyzed with this path constrained to be zero. The final model is presented in Figure 3. Again, the model showed an excellent fit to the data (CFI.984, RMSEA.035, SRMR.029). All of the factor loadings were statistically significant and each was above the.25 criterion. The lowest loading was.32 of Pattern Reasoning on Gv. The major difference between this model and the KABC-II factor structure is that Gestalt Closure measures Gc and not Gv, Hand Movements measures both Gf and Gsm, and Pattern Reasoning measures both Gv and Gf.

21 KABC-II Structure 531 Figure 3. Final validation model, ages 6 to 18. CFI Comparative Fit Index; RMSEA Root Mean Square of Approximation; SRMR Standardized Root Mean Square Residual; AIC Akaike Information Criterion. Most of the subtests had large loadings on their corresponding first-order factors. Moreover, as indicated by the strong second-order loadings, the g factor provides a good structure for the first-order factors. The Gf factor had the largest loading (.97) on the second-order g factor (cf. Keith, 2005), and the Gc, Gv, and Glr factors also had loadings over.80 on the second-order g factor.

22 532 Reynolds et al. Contribution of g and the Broad Abilities Using the final model we also obtained the loadings of the subtests on the second-order general (g) factor. These results are shown in Table 5. The loadings are considered indirect effects of g via the broad abilities on the subtests. Several loadings were greater than.60, and the Gc and Gf tests were generally the best measures of g. Gestalt Closure (.40) had the lowest loading on g. In a related fashion, we estimated how much of the subtest variance was explained by g versus how much of the subtest variance was explained by the broad abilities (also shown in Table 5). Although factor loadings show the effects of the latent variables to the subtests, decomposing the explained variance into different sources may be a useful tool to investigate the relative importance of various latent factors in explaining variation in subtest performance. This decomposition of variance is analogous to a Schmid-Leiman (1957) variance decomposition in exploratory factor analysis. The results are displayed graphically in Figure 4. Clearly, as expected, g accounted for most of the variance in the subtests; on average, g accounted for 34% of the variance in the subtests. The broad abilities explained, on average, 16.3% of the variance in the subtests. The contribution of the variance explained by the broad abilities, however, differed based on the broad ability. In fact, there was very little variance explained in the fluid reasoning subtests (e.g., Story Completion and Pattern Reasoning subtests) that was not explained by g. This finding is not surprising given the strong loading of Gf on g, and given that some researchers Table 5. Loading of KABC-II Subtests on the Second-Order g Factor for Ages 6 to 18, and Variance Explained by g and the Broad Abilities Subtest Loading g Variance explained by g Broad ability Riddles Verbal knowledge Expressive vocabulary Pattern reasoning Story completion Rebus Rebus delayed Triangles Atlantis Word order Hand movements Rover Block counting Atlantis delayed Number recall Gestalt closure

23 KABC-II Structure 533 Figure 4. Proportion of variance in the subtests accounted for by g and by the broad abilities. consider g and fluid reasoning to be the isomorphic (Gustafsson, 1994). Short-term memory, on the other hand, explained more of the variance in the Gsm subtests than did g. DISCUSSION Conclusion The KABC-II is a new and expanded version of the original Kaufman Assessment Battery for Children. The KABC-II is unique in that it allows for interpretation from a psychometrically oriented CHC theory and a Luria-based neuropsychological-process oriented theory. The KABC-II technical manual provides detailed information about test development, the factor structure, and test interpretation (Kaufman & Kaufman, 2004). The analyses in the manual generally support the factor structure of the KABC-II. Nevertheless, independent evaluation of new tests is important for understanding the nature of the constructs those tests measure. This study had two general purposes related to construct validity. The first purpose was to assess whether the KABC-II measures the same constructs across the 3- to 18-year age range. The second purpose was to understand the constructs measured by the test for school-age children and

24 534 Reynolds et al. adolescents. The findings have implications for construct validation, practical implications for interpretation, and theoretical implications for understanding the nature of ability constructs and future test development. The research generally supported the construct validity of the KABC- II. The first important general finding is related to measurement invariance. Establishing measurement invariance is an essential component of test validation. If constructs are not measured similarly across groups, then comparisons of scores across those groups are tenuous. The KABC-II indeed measures the same constructs across its 3- to 18-year age range. The second important general finding is that for school-age children (i.e., ages 6 18) the constructs measured by the KABC-II generally match those specified by CHC theory and operationalized in the scoring structure of the test. There were, however, minor inconsistencies between the test and its factor structure. In particular, these results suggest that three tests Gestalt Closure, Pattern Reasoning and Hand Movements are best conceived as measuring multiple abilities. Gestalt Closure appears to measure crystallized intelligence (Gc) in addition to, or perhaps instead of, visual processing (Gv); Pattern Reasoning appears to measure visual processing in addition to fluid reasoning (Gf), and Hand Movements measures fluid reasoning in addition to short-term memory (Gsm). These findings were supported in both calibration and validation samples. Other subtests appear to measure the single broad ability suggested by the current KABC-II factor structure. Block Counting measures visualspatial ability and not fluid reasoning. Story Completion measures fluid reasoning and not crystallized intelligence or visual-spatial ability, even though crystallized intelligence and visual-spatial ability are recommended as alternative interpretations in the KABC-II manual. Our findings were equivocal concerning the nature of the fluid reasoning measures on the KABC-II. The research showed that the Gf factor on the KABC-II is statistically distinguishable from the Gv factor for children ages 6 and older. When analyzed for ages 6 and older, a fairly narrow, three-test Gf factor emerged. Although this factor was consistent with the KABC-II scoring structure, it was only measured strongly by one test (Story Completion) and did not include one test, Rover, that was designed originally to measure Gf. At ages 6 7, however, Rover did load onto a Gf factor that included three tests. These ambiguous findings could not be resolved in our research because we were constrained by the tests within the KABC-II battery. These findings may suggest inadequate representation of Gf on the KABC-II, or they may reflect the often difficult task of separating Gf tasks from Gv tasks. Moreover, these findings highlight the need for additional study of the Gf factor on the KABC-II; cross-battery analysis with established measures of such abilities would be particularly valuable (cf. Keith, Kranzler, & Flanagan, 2001; Phelps, McGrew, Knopik, & Ford, 2005).