GRE R E S E A R C H. Validity of GRE General Test Scores for Admission to Colleges of Veterinary Medicine. Donald E. Powers.

Transcription

1 GRE R E S E A R C H Validity of GRE General Test Scores for Admission to Colleges of Veterinary Medicine Donald E. Powers May 2001 GRE Board Research Report No R ETS Research Report Princeton, NJ 08541

2 Validity of GRE General Test Scores for Admission to Colleges of Veterinary Medicine Donald E. Powers GRE Board Report No R May 2001 This report presents the findings of a research project funded by and carried out under the auspices of the Graduate Record Examinations Board. Educational Testing Service, Princeton, NJ 08541

3 ******************** Researchers are encouraged to express freely their professional judgment. Therefore, points of view or opinions stated in Graduate Record Examinations Board reports do not necessarily represent official Graduate Record Examinations Board position or policy. ******************** The Graduate Record Examinations Board and Educational Testing Service are dedicated to the principle of equal opportunity, and their programs, services, and employment policies are guided by that principle. EDUCATIONAL TESTING SERVICE, ETS, the ETS logos, GRADUATE RECORD EXAMINATIONS, and GRE are registered trademarks of Educational Testing Service. Educational Testing Service Princeton, NJ Copyright 2001 by Educational Testing Service. All rights reserved.

4 Abstract This paper documents a case study of the validity of the Graduate Record Examinations (GRE ) General Test in a comprehensive sample of veterinary medical colleges. Extensive and complete data available from these schools allowed us to control several artifacts that are typical of one-shot validation studies and thus enabled a relatively definitive assessment of the test s validity. Overall undergraduate grade point average (GPA), undergraduate GPA in the last 45 hours of courses, and GRE verbal, quantitative, and analytical scores were examined individually and in combination for their ability to predict success in veterinary school. For each of 16 veterinary medical colleges, statistical methods were applied to correct for the effects of range restriction in the predictors and unreliability of the criterion, and the results were summarized across all schools. When corrections were made for both range restriction and unreliability, the resulting validity coefficients (median multiple correlations) were.53 for three GRE scores, when used together;.56 for overall undergraduate GPA; and.72 for GRE scores and overall undergraduate GPA considered together. Adding GRE scores to undergraduate GPA increased the amount of variance explained, on average, by about 18% a proportion that can be regarded as being medium to large. Keywords: Graduate Record Examinations, GRE, veterinary medical colleges, test validity, predicting success, graduate admissions

5 Acknowledgements Special thanks go to each of the following people, without whose help or advice this report would not have been possible: Kurt Geisinger for suggesting that veterinary medical colleges would provide a good opportunity for GRE validity studies Beth Winter for helping us gain entry to veterinary schools, for facilitating communication with them, and for advice throughout the course of the study Mary Herron and Curt J. Mann, of the Association of American Veterinary Medical Colleges, for their support and advice Julie Lind and Don Elder for information about the Veterinary Medical College Application Service and for facilitating the transfer of data needed for the study Tony Confer for helping us locate prior validity studies of veterinary medical college admissions Sheila Allen, Linda Blythe, David Bristol, Barbara Coats, Ronnie Elmore, William Fenner, Robert Hansen, Kathyrn Kuehl, Michael Lorenz, Sherry McConnell, Claire Miceli, Kenneth Myers, Charles Newton, Denise Ottinger, Gerald Pijanowski, John Rhoades, Rebecca Russo, James Thompson, Eldon Uhlenhopp, John Van Vleet, Yasmin Williams, and Yvonne Wilson for acting as our contacts and for facilitating data collection and other aspects of the study at their institutions Susan Leung for coordinating data collection Michelle Najarian for managing the database and conducting the analyses; Tom Jirele for advice on analyses; and Fred Cline for producing graphics describing analyses Charlie Lewis for advising us on the data analysis Ruth Yoder for helping to process data, produce data collection instruments, and coordinate numerous other aspects of the project Leona Aiken, Narayan Bhat, Nancy Burton, Brent Bridgeman, Carol Dwyer, and Diane Halpern for providing helpful feedback on an earlier draft of this report The Graduate Record Examinations (GRE ) Board and the GRE Research Committee for their support of this effort

6 Executive Summary Traditionally, it has been difficult to conduct good validity studies for graduate admissions. Graduate-level validity studies have been hampered in several ways for example, by the availability of relatively small samples, by the use of unreliable criteria, and by the influence of selection procedures that restrict the range of test scores and other predictors of success. These factors can serve either to depress validity estimates or to make them appear more variable across graduate institutions than they really are. This study took advantage of a unique opportunity to address several shortcomings typical of many previous Graduate Record Examinations (GRE ) validity studies. This opportunity arose in the context of admissions to colleges of veterinary medicine, most of which require applicants to submit scores from the GRE General Test. From the standpoint of conducting validity studies, several features of veterinary schools made such an opportunity attractive: (a) larger class sizes than are typical of most graduate departments; (b) relatively uniform curricula across institutions; (c) the availability of information about applicants; and (d) an opportunity to assess the reliability of criteria. These features enabled us to statistically correct for several of the factors that can lead to inaccurately low estimates of true validity. They also allowed us to estimate the degree to which apparent between-school variation in validity estimates is due to statistical artifacts, rather than to real differences in the predictability of success from school to school. As anticipated, the highly selective nature of veterinary medical school admissions was clearly apparent from the data. There was ample evidence of factors for example, restriction in the range of test scores and undergraduate grade point averages (GPAs) that can result in serious underestimates of the validity of preadmission measures. Even with the dampening effects of these factors, however, both undergraduate GPA and GRE General Test scores exhibited significant relationships with first-year grades in veterinary medical school. Moreover, when used together, grades and test scores constituted a more powerful predictor combination than did either one used alone. Adding GRE scores to grades increased prediction by an amount that social scientists usually consider as being medium to large.

7 More importantly, when statistical corrections were made to counteract the attenuating effects of selection and criterion unreliability, validity estimates increased significantly. When fully corrected for both range restriction in the predictors and unreliability in the criterion, the combination of undergraduate GPA and GRE General Test scores accounted, on average, for more than half the variation in first-year veterinary school grade averages. By focusing on a particular context veterinary medical school admissions we were fortunate to obtain the cooperation of a majority of U.S. veterinary colleges of medicine, and hence information about a preponderance of first-year veterinary medical students in the academic year. Thus, unlike the results of validity studies based on single institutions, the findings reported here are reasonably representative of a specific universe of interest in this case, veterinary medical schools. Moreover, an analysis of the variation among schools suggested little reason to question the generalizability of our findings across veterinary medical schools: With the exception of one school, validity estimates seemed to apply about equally well to all of the schools that participated in the study.

8 Table of Contents Page Introduction... 1 Factors Affecting the Estimation of Validity Coefficients... 2 Validity Generalization... 4 Veterinary Medical School Admissions... 5 Relevant Research... 6 Research Questions... 6 Method... 7 Sample Selection... 7 Instruments/Data... 9 Data Analysis... 9 Results Discussion Limitations Conclusion References Appendix... 41

9 List of Tables Page Table 1. Participating Universities... 8 Table 2. Table 3. Table 4. Table 5. Correlations of Preadmissions Variables with First-Year Average in Veterinary Colleges of Medicine Means and Standard Deviations of Preadmissions Variables for Applicants and Enrolled Students Correlations of Preadmissions Variables With First-Year Average in Veterinary Colleges of Medicine Correlations of Preadmissions Variables With First-Year Average in Veterinary Colleges of Medicine Table 6. Correlations Among Preadmissions Variables for Applicants and Enrolled Students Table 7. Student Perceptions of the First Year of Veterinary Medical School Table 8. Statistics for Individual Course Grades List of Figures Figure 1. Distributions of validity coefficients before and after corrections (UGPA T as the undergraduate grade indicator) Figure 2. Distributions of validity coefficients before and after corrections (UGPA 45 as the undergraduate grade indicator) Page

10 Introduction The Graduate Record Examinations (GRE ) General Test is a standardized test of verbal, quantitative, and analytical reasoning that is designed, primarily, to facilitate admissions to U.S. graduate schools. Since its inception in 1949, the original measure and several revisions have been widely used and frequently studied (see, for example, Briel, O Neill, & Scheuneman, 1993). In fact, according to Hambleton (1998), there have been some 1,500 studies of the validity of the GRE General Test plus one! The one (Sternberg & Williams, 1997) a case study of psychology graduate students at a prestigious academic institution (Yale University) published in an esteemed scholarly journal (American Psychologist) by two prominent psychologists attracted considerable, though perhaps undue, attention. While understandable, the degree of interest in this single study and its largely negative results flies in the face of current conceptions of test validation, which is now generally viewed as the process of accumulating evidence regarding the meaning and value of test score inferences (Messick, 1989). In short, one study does not a test validation make. In addition to the attention it received, the Sternberg and Williams (1997) study is also notable with regard to the criticism it drew, much of which concerned professional requirements for defensible validity studies. The ensuing, mostly critical commentary (Andre & Hegland, 1998; Cornell, 1998; Darlington, 1998; Kuncel, Campbell, & Ones, 1998; Melchert, 1998; Miller, Barrett, & Doverspike, 1998; Roznowski, 1998; Ruscio, 1998; Thayer & Kalat, 1998) suggested that, like numerous other validity efforts, the study was multiply flawed because it: relied on small samples employed unreliable criteria overgeneralized from a single, atypical department discounted the effects of compensatory selection failed to account for range restriction in the criteria and in the predictors With these criticisms in mind, we undertook a case study of the validity of the GRE General Test in a comprehensive sample of veterinary medical colleges. The extensive and

11 complete data available from these schools allowed us to control several of the artifacts that affect many one-shot validation studies, and thus enabled a relatively definitive assessment of the test s validity. Factors Affecting the Estimation of Validity Coefficients While some of the factors mentioned above (e.g., the use of small samples) can give rise to inconsistent results, others (e.g., range restriction, criterion unreliability, and compensatory selection) can occasion serious underestimates of the validity of admissions measures. The potential influence of each of the latter factors is discussed briefly below. Range restriction. It has long been known (since Pearson, 1903) that sampling from a population can, by curtailing the range of a variable, artificially depress its correlation with other variables. Such restriction typically occurs in academic admissions when students are selected from a larger pool of applicants for example, on the basis of test scores. Generally, the effect is to underestimate the true correlation between the selection device (e.g., test scores) and first-year grades (or some other criterion of success) in the original population that is, the applicant pool. That range restriction can sometimes have a dramatic effect on the magnitude of validity coefficients has been clearly demonstrated (e.g., Linn & Dunbar, 1982). Conversely, very high validity coefficients can be realized when selection is not based on test scores and when, therefore, the range of performance on the test is not restricted. For instance, Huitema and Stein (1993) found that when admissions decisions were made without reference to test performance, GRE General Test scores were reasonably strong predictors of graduate course grades and faculty ratings, with validity coefficients ranging from.55 to.70. Undergraduate grade point averages (GPAs), on the other hand, which were used in selection, did not correlate significantly with these criteria. Criterion unreliability. A second factor, criterion unreliability, can also serve to decrease estimates of validity. Because of the attenuation that results from the use of imperfectly reliable indicators of success, it has often been deemed appropriate to correct for measurement error in the criteria. Although, historically, this practice has not been accepted unconditionally, 2

12 correcting for unreliability in the criterion, if not in the predictors, is now generally regarded as appropriate, provided that both the corrected and the uncorrected results are presented (AERA/APA/NCME, 1999). The chief remaining debate concerns the proper method of computing the reliability estimate that is used to make the corrections (Muchinsky, 1996). For some criteria of success first-year GPA, for example the factors that contribute to imperfect reliability are at least partially understood. For instance, GPAs are often a peculiar amalgam of individual course grades based on different subject matter, different instructors, and different grading standards. Such heterogeneity of component parts is known to contribute to the less-than-perfect reliability of a composite. Combining grades from disparate courses into a single average often obscures important differences in the meaning of individual course grades, thereby depressing the reliability of the composite (Willingham, 1990). Grades in single individual courses, on the other hand, can sometimes be nearly as predictable as composites based on several courses (Ramist, Lewis, & McCamley, 1990). Accordingly, as discussed below, one of our objectives was to examine the relationship of both GRE test scores and undergraduate GPAs to grades in key individual first-year courses. Compensatory selection. When good standing on one selection variable is allowed to offset, or compensate for, lower status on another, compensatory selection is said to occur. Like range restriction and criterion unreliability, this circumstance can also have a dramatic impact on validity estimates. The phenomenon is germane here because compensatory selection is, very likely, the norm for admission to graduate education generally and veterinary medicine specifically. Most schools, we believe, follow recommended practice by eschewing rigid cutoffs with respect to test performance and other admissions criteria. Most likely, perhaps, is the following scenario described by Cornell (1998). Applicants with relatively low GRE scores may be selected because they show evidence of other outstanding traits, such as motivation or maturity. These important qualities may, however, receive less scrutiny for applicants with exemplary test scores. Subsequently, some students with high test scores may fail because they lack motivation or maturity, whereas some low-scoring students may succeed largely because they do possess these qualities. 3

13 Beyond these anecdotes, there is also strong empirical evidence of the effects of compensatory selection. Differential compensatory selection practices, as indexed by the extent to which preadmission measures fail to correlate strongly with one another, can sometimes explain much of the observed variation among validity coefficients in the case of law school admissions, for example, more than 50% (Linn & Hastings, 1984). As Ruscio (1998) has pointed out, compensatory selection often stacks the deck squarely against the predictive validity of the GRE (p. 569). Validity Generalization Besides acting to depress validity estimates, the factors discussed above can also contribute to apparent differences in the results of validity studies across situations (so-called situational specificity ). Interest in the extent to which the validity of test-score inferences is similar, or generalizes, from one situation to another was, in large part, the motivation for the development of validity generalization methods. Since the 1970s, these methods have been adopted in numerous validity studies, most of them concerned with employment testing and some with academic admissions testing. The general conclusion from the bulk of these efforts is that the apparent variability in test score validity across situations or among institutions is often largely artifactual. That is, differences from one situation to another are more likely the result of statistical phenomena than of inherent differences among situations in the predictability of criteria. In fact, it is now commonly accepted that statistical artifacts account for much, if not most, of the apparent variation among validity estimates across situations. Although a variety of factors, including computational errors, have been implicated in between-study differences (Hunter & Schmidt, 1990), the bulk of the variation seems due mainly to differences among situations with respect to the reliability of the criterion, the size of the study samples, and the effects of selection in curtailing the range of test scores. For instance, Linn, Harnisch, and Dunbar (1981) estimated that, for a large sample of law schools, fully 70 percent of the variation in observed validity coefficients could be explained by these three factors. To a lesser degree, the same result has been found in graduate admissions 4

14 (Boldt, 1986). (The greater generalization in the law school context is, most likely, due to the fact that, when compared with graduate departments, law schools are typically more homogeneous in many respects for example, with regard to their curricular offerings and their course requirements.) More recently, Kuncel, Hezlett, and Ones (2000) also concluded that the relatively low validities observed in GRE validity studies, as well as their variability across graduate departments, is largely the result of sampling error and range restriction in test scores. Veterinary Medical School Admissions Currently, there are 27 U.S. schools of veterinary medicine. As of January 1997, a majority of these schools required applicants to submit GRE General Test scores, a small number required scores from the Veterinary College Admission Test (VCAT), and a few others allowed scores from either the GRE, the VCAT, or the Medical College Admission Test (MCAT) (Veterinary Medical College Application Service, 1995). Veterinary schools are therefore a major user of GRE scores. Befitting this use, a number of veterinary medical professionals have studied the effectiveness of admissions procedures in their locales, as discussed in the next section. However, to our knowledge, there has been no large-scale, multi-institution study of the validity of GRE test scores for veterinary medical school admissions. The significant number of validity studies that have been conducted by the GRE program on behalf of graduate departments are relevant, of course (see chapter IX, Validity of the GRE Tests, in Briel, O Neill, & Scheuneman, 1993). But, the extent to which these studies generalize without reservation to colleges of veterinary medicine is unclear. On one hand, both graduate education and veterinary medical education entail challenging post-baccalaureate academic experiences the kind for which the GRE General Test was designed. On the other hand, there are some obvious and meaningful differences between these educational contexts. 5

15 Relevant Research As suggested above, there have been several studies of the effectiveness of veterinary medical admissions (see Appendix for a summary). There is considerable variation among these studies for example, with respect to the samples that were studied, the preadmission variables that were considered, and the criteria of success that were specified. For most of the studies, samples were pooled across several entering classes within a single institution. Some of the efforts examined academic performance in the first year of veterinary school, while others studied performance over a longer period. Various tests were used as predictors, with GRE scores being considered in about half the studies. The studies span a variety of institutions, with Oklahoma State University being represented more often than other institutions, due to several studies by Confer and his colleagues at that institution (Confer, 1990; Confer & Lorenz, in press; Confer, Turnwald, & Wollenburg, 1995). Research Questions The following questions motivated the study described here: What is the predictive power both observed and true (i.e., corrected for factors that tend to lower correlations) of the GRE General Test and of undergraduate grades with respect to measures of success in veterinary medical school? What do GRE scores contribute to prediction above and beyond undergraduate grades? How much variation is there among veterinary medical colleges with respect to the predictive validity of GRE scores and undergraduate grades? How much of the apparent variation can be explained by statistical artifacts, such as sampling error, differences in the reliability of criteria, and differential restriction of range in the predictors and the criteria? Is there evidence of compensatory selection (i.e., allowing an applicant s strength in one area to compensate for a weakness in another) in veterinary school admissions? If so, what effect does this have on estimates of test-score validity? Secondarily, we were also interested in answering the following questions: How well do GRE General Test scores and undergraduate grades predict a more student-oriented criterion of success in veterinary school (versus the more traditional institution-centered criterion of first-year grades)? 6

16 How does the prediction of grades in key individual courses compare with the prediction of overall first-year averages? Method Sample Selection In the fall of 1998, the Association of American Veterinary Medical Colleges (AAVMC) issued, on our behalf, an invitation to each of the 27 U.S. colleges of veterinary medicine to participate in a study of the validity of GRE General Test for admissions to colleges of veterinary medicine. In order to participate, schools were required to provide first-year students GPAs in each semester of the academic year. For students who had withdrawn without a GPA, schools were asked to designate the student s academic standing at the time of withdrawal (as either "dismissed for academic reasons" or "withdrew in good standing"). Schools were also encouraged, but not required, to provide students grades in individual, key courses of their choice, and to collect information about students perceptions of their first-year of veterinary medical education. The latter was to be accomplished by administering a brief questionnaire (described in more detail below) to first-year students. Given these requirements, nearly all of the schools that require or allow applicants to submit GRE scores agreed to cooperate in the study, resulting in a total of 16 participating schools, which are listed in Table 1. All but one of the cooperating schools also participate in the AAVMC-sponsored Veterinary Medical College Application Service (VMCAS). The VMCAS is a central processing service that receives applications from prospective veterinary school students, conducts analyses of undergraduate transcripts, assembles and forwards applicants information to colleges, and, finally, records admissions decisions made by participating veterinary colleges. For our study, students GRE scores, undergraduate GPAs, and demographic characteristics were available from the VMCAS both for applicants and for admitted students for most schools. Non-VMCAS schools provided GRE scores, undergraduate GPAs, and other required information for enrolled students and applicants directly to us. 7

17 Table 1 Participating Universities Universities Iowa State University Kansas State University Louisiana State University North Carolina State University Ohio State University Oklahoma State University Oregon State University Purdue University Texas A & M University Tufts University University of California, Davis University of Florida University of Georgia University of Illinois, Urbana University of Pennsylvania Washington State University The resulting sample of some 1,400 students included more than half of the estimated 2,300 students who entered U.S. veterinary schools in the fall of In each school, female students were in the majority and, overall, they comprised about 70% of the total study sample. With respect to race/ethnicity, the median percentage of Caucasian students across schools was 86%. African American students constituted the next largest group on average, approximately 1% of the remaining sample. The median percentage of each of 12 other identifiable racial/ethnic groups was less than 1%. Information on racial/ethnic identity was unavailable for 8% of the total study sample. Information was also available for a much larger number of applicants to veterinary schools about 5,400 of the approximately 6,600 students who recently submitted some 23,000 applications through the VMCAS (July 1997, Thus, the study sample comprises a majority of U.S. veterinary schools, first-year veterinary medical students in these schools, and students who were interested in pursuing veterinary medical study during the school year. 8

18 Instruments/Data Specific information available from the VMCAS or directly from schools included: GRE General Test scores 1 for most enrolled students and applicants. (Test scores from the Medical College Admissions Test, the Veterinary College Admissions Test, and the GRE Subject Test in biology were also available for some students, but these data proved too sparse to conduct any meaningful analyses). Undergraduate GPAs (average in last 45 hours of courses, designated below as UGPA 45, and overall, designated below as UGPA T ). (All GPAs were based on the latest transcripts available through and including the fall semester of the final undergraduate year.) Schools to which each applicant had applied. Information obtained from participating veterinary schools included: GPAs in each term of the first year of veterinary school grades in key individual courses for some schools student perceptions of their own first-year success and experiences The student perceptions questionnaire mentioned earlier was adapted from several sources, including a recent revision of two Educational Testing Service (ETS ) questionnaires the Student Instructional Report (SIR II) and Student Reactions to College designed to solicit students' opinions about their college experiences (ETS, 1995). Additional questions were based on a survey of the academic experiences of graduate management students (Baydar, Dugan, Grady, & Johnson, 1995). The resulting five-to-10-minute questionnaire was administered to first-year students shortly after the beginning of their second academic term. Data Analysis To answer each of the questions posed at the outset, several analyses were performed. First, to estimate observed validities, ordinary least-squares regression analyses were conducted for each school. Three separate criteria (cumulative first-year average, grades in individual 9

19 courses, and student perceptions of the first year of veterinary school) were regressed, in turn, on each of several predictors either individually or in combination. GRE General Test scores (verbal, quantitative, and analytical) and undergraduate GPAs (both overall and in the last 45 hours of courses) served as the predictors. Zero-order and multiple correlations were computed for each predictor/criterion combination for each school. In cases where one or more regression weights were negative for a combination of predictors, the negatively weighted variable was deleted, and the regressions were re-run for the reduced set of predictors. Second, to estimate the true (as opposed to observed) validity coefficients for GRE scores and undergraduate grades, we corrected for both the effects of criterion unreliability and the influence of range restriction in the predictors. To adjust for the effects of range restriction, we made use of information about applicants and admitted students at each school. For applicants, this information included the covariances among predictors; for admitted students, it included the covariances among predictors and criteria. Using this information, multivariate range restriction formulas (Gulliksen, 1950, chapter 13) were used to correct both zero-order and multiple correlations. Multivariate procedures were preferred over univariate methods, because the former have been shown to yield more accurate, less conservative estimates than the latter (Held & Foley, 1994; Linn, Harnisch, & Dunbar, 1981). Finally, empirical range restriction procedures (Linn & Hastings, 1984) were used as an additional check. Because of both student attrition and school grading practices, range restriction can also occur on the criterion, as students who perform poorly during their first year of veterinary school (and who perhaps also had low GRE test scores) may drop out before receiving grades. The resulting curtailment on the criterion can also act to depress validity estimates. Our approach to dealing with this type of restriction was to examine the relationship between validity coefficients and the percentage of nonpersisters at each school. A negative relationship would suggest the likelihood that validity estimates had been attenuated because of student attrition. 1 More than one set of GRE scores was available for some students. Correlations among highest, most recent, and average GRE scores were in the mid- to high.90s for each school. A decision was made to use the most recent scores available for each student. 10

20 To adjust for the effects of imperfectly reliable criteria, first-year grades were corrected for attenuation using classical formulas first suggested by Spearman (1904). Our initial intention was to compute the correlation of grades in a single course in one term with grades in the same course in a subsequent term, and then to average the resulting correlations over all multi-term courses. This procedure seemed slightly more appropriate than did internal consistency estimates based on correlations among grades from heterogeneous courses within a single semester, or based on correlations between first- and subsequent-term averages, as this method would better control for differences among courses, both within and across semesters. It would also more nearly meet the assumptions that individual courses are equally demanding, that they are graded in an equivalent manner, and that real academic progress (or backsliding) over two semesters is not plausible. Although we were able to collect information about performance in individual courses, the number of multi-term courses did not prove adequate for our purposes. We therefore used one of the alternative procedures, estimating the reliability of first-year GPAs from the correlations between first- and second-term averages (and for some schools, third-term averages as well). The resulting between-term correlations were stepped up according to the Spearman- Brown prophecy formula in order to estimate the reliability of GPAs for the full first year. To confirm that these estimates were reasonable, we also computed correlations among individual course grades at each school, and then stepped up the estimates again using the Spearman-Brown formula. Third, to determine how much, if any, of the observed variation among validity coefficients was due to statistical artifacts, we employed standard meta-analytic/validity generalization procedures (Hunter & Schmidt, 1990; Pearlman, Schmidt, & Hunter, 1980; Schmidt, Law, Hunter, Rothstein, Pearlman, & McDaniel, 1993). The artifacts of interest were small samples, differential restriction of range in the predictors, and differences in criterion reliability. To begin, we calculated the variation among observed validity coefficients that could be explained solely by sampling error. The validity coefficients for each individual school were then corrected both for range restriction and criterion unreliability. Having data from each participating school, we were able to correct the correlations for each individual school: There 11

21 was no need to resort to using assumed distributions of artifacts, as is customary in many metaanalytic studies (Hedges, 1988). Finally, in order to establish the presence of compensatory selection in veterinary school admissions (and, if present, to determine its effect on validity estimates), the following strategy was used. For each school, correlations between predictors undergraduate grades and GRE scores, for example were computed, and correlations for applicants and enrolled students were compared. When between-predictor correlations were lower for enrolled students than for applicants, this was taken as evidence that preadmission measures were operating in a compensatory fashion; the lower the between-predictor correlations among enrolled students, the greater the degree of compensatory selection was assumed to be. To assess the likely influence of compensatory selection under these assumptions, the relation between these correlations and validity coefficients was examined, and the extent to which the size of these correlations accounted for variation among the size of validity estimates was calculated. The data were also inspected for indications that selection was based on preadmission factors other than those for which we had data. For example, we noted the extent to which applicants with both poor test scores and low undergraduate grades had been admitted. Results Results are presented here in terms of each of the questions posed at the outset. 1. What is the observed validity of the GRE General Test, of undergraduate grades, and of the combination of both for predicting success in each of several veterinary medical schools? Table 2 displays the results of the regression analyses for each of the 16 participating veterinary colleges. Zero-order correlations are displayed for individual predictors, and multiple correlations are shown for combinations of predictors. 2 For each predictor and predictor 2 For some of the analyses that involved combinations of predictors, the regression weights computed for one or more variables were negative. Although none of these negative weights was statistically significant in any of these cases, variables with negative weights were deleted from the predictor set, and the multiple correlation was recomputed for the remaining variables. We note also the existence of possible suppressor effects for some schools that is, a situation in which one predictor has essentially no predictive power by itself, but which contributes to prediction by suppressing irrelevant variance in another predictor, thus improving the power of that predictor. 12

22 combination, there is apparent variation among schools. (For one school a clear outlier the computed validities were negative for each of the three GRE General Test scores.) The mean correlations over all schools were slight to moderate for each individual variable, ranging from.21 for GRE verbal scores to.37 for undergraduate GPA in the last 45 hours of courses. Overall undergraduate GPA and undergraduate GPA in the last 45 hours of courses were, on average, about equally predictive of performance in the first year of veterinary school. Table 2 also shows that, as has been well established in numerous previous validity studies, the combination of test scores and previous grades constitutes a better predictive combination than either test scores or previous grades alone. The mean multiple correlation based on undergraduate GPAs and GRE General Test scores was.51 when overall undergraduate GPA was used and.53 when undergraduate GPA in the last 45 hours of courses was available. 2. What are the estimates of the true validity of GRE scores and undergraduate grades after correcting for the effects of criterion unreliability and range restriction in the predictors? For each school, Table 3 displays for both applicants and enrolled students the means and standard deviations for each predictor variable. As is clear from the differences between applicants and enrolled students at each school, some selection is taking place if not on GRE scores and undergraduate GPAs, then on some other factors that are correlated with them. For instance, at each school the average overall undergraduate GPA is higher for enrolled students than for applicants by.5 to 1.1 standard deviation units, and by.4 to 1.0 standard deviation units for the average undergraduate GPA in the last 45 hours of courses. GRE General Test score means are also higher for enrolled students than for applicants at each school by.2 to 1.0 standard deviation units for GRE verbal ability scores, by.1 to 1.0 standard deviation units for GRE quantitative ability scores, and by 0 to 1.0 standard deviation units for GRE analytical ability scores. Figure 1 presents distributions of validity coefficients for these three scores, taken in combination and also combined with overall undergraduate GPA, before and after corrections. The same combinations are shown in Figure 2 when undergraduate GPA in the last 45 hours of courses is substituted for overall undergraduate GPA. In addition to evidence of student selection by institutions, there is also some indication of student self-selection to veterinary colleges. Although we do not know the mean undergraduate GPAs of all graduate school applicants, it seems that the means for veterinary 13

23 school applicants are relatively high, ranging across schools from 3.13 to 3.35 for overall undergraduate GPA and from 3.25 to 3.45 for undergraduate GPA in the last 45 hours of courses. Likewise, the mean GRE scores of these applicants tend to be higher than the mean scores of all GRE General Test takers. During a recent testing year, GRE General Test score means were 474 for GRE verbal ability (SD = 114), 558 for GRE quantitative ability (SD = 139), and 547 for GRE analytical ability (SD = 130). For GRE test takers who majored in the life sciences (from which a disproportionate number of veterinary school applicants probably come), the means were, respectively, 465 (SD = 96), 547 (SD = 117), and 559 (SD = 117; ETS, 1998). Thus, according to these data, veterinary students are a relatively highly selected group. The major import of this selection is, for us, its effect on the variation of predictor variables. As is evident, there is a clear tendency for there to be less variation among enrolled students than among applicants, hence the need to correct for range restriction. There is also a need to correct for the imperfect reliability of the primary criterion, firstyear GPA. Reliability estimates based on between-term correlations ranged from.74 to.98 across schools (median =.92). Estimates based on correlations among individual courses ranged from.81 to.96 (median =.90). Because more data were available, and fewer assumptions needed, for reliability estimates based on term averages than on individual course grades, the between-term estimates were used to correct for attenuation due to unreliability. 14

24 Table 2 Correlations of Preadmissions Variables with First-Year Average in Veterinary Colleges of Medicine 15 School Variable Number of Individually In combination students UGPA T UGPA 45 GRE-V GRE-Q GRE-A V,Q,A UGPA T, V,Q,A UGPA 45, V,Q,A A B a.58 a.63 a C D E b F c G d n.c. n.c. H I n.a. n.a. n.a..27 e J K c.47 c.56 c L M N O n.a c.57 n.a. P Median Weighted mean Note. UGPA T = overall undergraduate grade point average; UGPA 45 = undergraduate grade point average in last 45 hours; GRE-V = GRE General Test verbal ability score; GRE-Q = GRE General Test quantitative ability score; GRE-A = GRE General Test analytical ability score; V,Q,A = combined GRE General Test verbal ability, quantitative ability, and analytical ability score; n.a. = not available; n.c. = not computed. a Does not include GRE-V; b does not include GRE-A; c does not include GRE-Q; d set to.00 because all weights were negative; e only a total of GRE-V, GRE-Q, and GRE-A was available for this school.

25 Table 3 Means and Standard Deviations of Preadmissions Variables for Applicants and Enrolled Students Variable School UGPA T UGPA 45 GRE-V GRE-Q GRE-A A Applicants 3.21 (.40) 3.32 (.42) 504 (94) 600 (95) 609 (105) Enrollees 3.52 (.30) 3.64 (.27) 591 (89) 697 (60) 713 (66) B Applicants 3.17 (.38) 3.27 (.42) 480 (87) 580 (93) 580 (115) Enrollees 3.41 (.29) 3.46 (.34) 523 (72) 655 (72) 615 (119) C Applicants 3.16 (.37) 3.26 (.42) 476 (89) 561 (93) 571 (110) Enrollees 3.43 (.33) 3.58 (.32) 569 (82) 649 (61) 650 (84) D Applicants 3.20 (.37) 3.32 (.40) 487 (88) 580 (98) 586 (105) Enrollees 3.58 (.24) 3.65 (.26) 516 (80) 628 (72) 648 (84) E Applicants 3.18 (.38) 3.30 (.41) 476 (89) 574 (95) 580 (107) Enrollees 3.53 (.24) 3.58 (.22) 500 (77) 612 (78) 641 (82) F Applicants 3.19 (.36) 3.32 (.39) 476 (86) 574 (92) 579 (106) Enrollees 3.42 (.17) 3.50 (.25) 512 (92) 622 (69) 656 (94) G Applicants 3.13 (.38) 3.26 (.41) 464 (85) 560 (92) 566 (111) Enrollees 3.55 (.26) 3.66 (.23) 491 (83) 584 (69) 566 (130) H Applicants 3.23 (.38) 3.34 (.42) 486 (88) 585 (93) 594 (104) Enrollees 3.62 (.28) 3.71 (.36) 530 (89) 650 (74) 664 (78) I Applicants 3.35 (.36) 3.45 (.37) 495 (91) 590 (94) 600 (105) Enrollees 3.57 (.27) 3.73 (.19) n.a. n.a. n.a. J Applicants 3.14 (.38) 3.25 (.41) 463 (83) 553 (94) 569 (107) Enrollees 3.40 (.28) 3.52 (.24) 480 (79) 567 (100) 593 (99) K Applicants 3.20 (.38) 3.31 (.42) 488 (90) 582 (95) 592 (105) Enrollees 3.60 (.25) 3.71 (.22) 568 (82) 636 (88) 669 (69) L Applicants 3.26 (.38) 3.39 (.39) 509 (92) 601 (93) 610 (103 Enrollees 3.45 (.29) 3.56 (.28) 568 (88) 664 (71) 651 (93) M Applicants 3.17 (.39) 3.27 (.42) 477 (86) 582 (94) 586 (106) Enrollees 3.48 (.30) 3.62 (.25) 505 (96) 630 (80) 621 (80) N Applicants 3.20 (.38) 3.31 (.41) 470 (91) 574 (93) 582 (110) Enrollees 3.61 (.25) 3.69 (.21) 528 (95) 647 (78) 657 (87) O Applicants n.a. n.a. n.a. n.a. n.a. Enrollees 3.45 (.31) n.a. 599 (79) 675 (63) 692 (69) P Applicants 3.23 (.37) 3.34 (.40) 490 (91) 585 (92) 595 (104) Enrollees 3.55 (.24) 3.69 (.18) 508 (84) 602 (65) 635 (92) Median Applicants Enrollees Note. For applicants, undergraduate GPA statistics are based on from 629 to 1550 applicants per institution; GRE General Test scores are based on from 885 to 2356 applicants per institution; n.a. = not available. 16

26 Figure 1 Distributions of Validity Coefficients Before and After Corrections (UGPA T as the Undergraduate Grade Indicator) 1.0 Validity Coefficients UGPA Total GRE V,Q,A UGPA Total 0.0 & GRE V.Q,A N = Uncorrected Corrected Figure 1. Distributions of validity coefficients before and after corrections (UGPA T as the undergraduate grade indicator). 17

27 1.0 Validity Coefficients UGPA Last 45 Hrs GRE V,Q,A UGPA Last 45 Hrs 0.0 & GRE V.Q,A N = Uncorrected Corrected Figure 2. Distributions of validity coefficients before and after corrections (UGPA 45 as the undergraduate grade indicator). 18

28 Correcting back to each school s applicant pool for multivariate range restriction resulted in somewhat higher validity estimates for each school and for each predictor combination (Table 4). Median validity estimates increased by.16 to.24 for individual predictors, and by.17 to.21 for combinations of predictors. When corrections were applied for both range restriction and criterion unreliability (Table 5), correlations increased slightly more. The increase in median correlations was.01 to.05 for individual predictors and.02 to.04 for combination of predictors when correlations were corrected for attenuation due to criterion unreliability. The relatively small size of this additional increase was expected because first-year GPA was reasonably reliable, on average, across schools. Figures 1 and 2 display distributions of uncorrected and corrected validity coefficients for overall undergraduate GPA alone, for GRE verbal, quantitative, and analytical scores in combination, and for undergraduate GPA in the last 45 hours of courses. 19

29 Table 4 Correlations (Corrected for Range Restriction Only) of Preadmissions Variables With First-Year Average in Veterinary Colleges of Medicine 20 School Variable Number of Individually In combination students UGPA T UGPA 45 GRE-V GRE-Q GRE-A V,Q,A UGPA T, V,Q,A UGPA 45, V,Q,A A B a.67 a.71 a C D E b F c G n.c. n.c. n.c..00 d n.c. n.c. H I n.a. n.a. n.a..49 e J K c.64 c.76 c L M N O n.a c.73 n.a. P Median Weighted mean Note. Correlations are corrected for range restriction but not for criterion unreliability. UGPA T = overall undergraduate grade point average; UGPA 45 = undergraduate grade point average in last 45 hours; GRE-V = GRE General Test verbal ability score; GRE-Q = GRE General Test quantitative ability score; GRE-A = GRE General Test analytical ability score; V,Q,A = combined GRE General Test verbal ability, quantitative ability, and analytical ability score; n.a. = not available; n.c. = not computed. a Does not include GRE-V; b does not include GRE-A; c does not include GRE-Q; d set to.00 because all weights were negative; e only a total of GRE-V, GRE-Q, and GRE-A was available for this school.

30 Table 5 Correlations (Corrected for both Range Restriction and Criterion Unreliability) of Preadmissions Variables With First-Year Average in Veterinary Colleges of Medicine 21 School Number of Students Variable Individually In combination UGPA T UGPA 45 GRE-V GRE-Q GRE-A V,Q,A UGPA T, V,Q,A UGPA 45, V,Q,A A B a.78 a.82 a C D E b F c G n.c. n.c. n.c..00 d n.c. n.c. H I n.a. n.a. n.a..49 e J K c.67 c.79 c L M N O n.a c.75 n.a. P Median Weighted mean Note. Correlations are corrected for both range restriction and criterion unreliability. UGPA T = overall undergraduate grade point average; UGPA 45 = undergraduate grade point average in last 45 hours; GRE-V = GRE General Test verbal ability score; GRE-Q = GRE General Test quantitative ability score; GRE- A = GRE General Test analytical ability score; V,Q,A = combined GRE General Test verbal ability, quantitative ability, and analytical ability score; n.a. = not available; n.c. = not computed. a Does not include GRE-V; b does not include GRE-A; c does not include GRE-Q; d set to.00 because all weights were negative; e only a total of GRE-V, GRE-Q, and GRE-A was available for this school.

31 To account for the possible role of student attrition in restricting the range of performance on the criterion (and perhaps the predictors), we asked schools to indicate which students had withdrawn, and why that is, whether they had withdrawn in good standing or had been dismissed for poor academic performance. Twelve schools were able to provide this information, indicating in each case that, in total, five or fewer students had withdrawn in either good or poor academic standing. Thus, attrition exerted little, if any, influence on the accuracy of validity estimates. In summary, when correlations were fully corrected for both range restriction in the predictors and unreliability in the criterion, the resulting validity coefficients were moderately large. Median correlations were in the.40s for each component of the GRE General Test and.53 for the combination of all three GRE General Test scores. Undergraduate GPA proved to be an even stronger predictor, with median correlations of.56 for overall undergraduate GPA and.64 for undergraduate GPA in the last 45 hours of courses. Taken together, GRE scores and undergraduate GPA constitute a relatively powerful predictive combination, accounting, on average, for slightly more than half the variance in first-year veterinary school GPAs. By considering GRE scores along with undergraduate GPA, the amount of variance explained increases by about 18% from 35-40% to 53-58%. According to Cohen s guidelines for effect size (1977, chapter 9), this increase in explained variance can be regarded as medium to large. 3. How much of the apparent variation among validity coefficients across schools can be explained by statistical artifacts such as (a) small samples, (b) differential restriction of range in the predictors, and (c) differences in criterion reliability? To partition the variation among validity coefficients into that due to random fluctuation versus that due to systematic differences among schools, meta-analytic formulae (Hunter & Schmidt, 1990, chapter 3) were applied first to the uncorrected validity coefficients given in Table 2 and then to the fully corrected coefficients in Table 5. The observed variation among validity coefficients across schools, and the proportion attributable solely to sampling error, was as follows for individual predictors and combinations: 22