SUPREME COURT OF OHIO

Transcription

1 SUPREME COURT OF OHIO STATE EX REL., PAUL PERREA 412 Wood Avenue Cincinnati, Ohio Relator,. MANDAMUS PROCEEDING -vs- CINCINNATI PUBLIC SCHOOLS. CASE NO Burnet Avenue Cincinnati, Ohio Respondent. MOTION TO STRIKE AFFIDAVIT OF JOANNE JENSEN; IN THE ALTERNATIVE, MOTION TO FILE AFFIDAVIT OF JEROME D'AGOSTINO, PH.D. TED L. WIELLS_(OhioLRarNo ) _ IYIARK^I-STEPANIA 414 Walnut Street, Suite 707 Taft Stettinius & Hollister Cincinnati, Ohio Walnut Street, Suite 1800 Telephone (513) Cincinnati, Ohio Facsimile (513) Telephone: (513) TedLWills@aol.com Facsimile: (513) Attorney for Relator, Paul Perrea stepaniak@taftlaw.com Attorney for Respondent, Cincinnati Public Schools

2 I. INTRODUCTION/MOTION On April 21, 2008, Relator, Paul Perrea, filed a public records mandamus action against Respondent, Cincinnati Public Schools (CPS). (Complaint with Affidavit of Relator (Complaint).) In his complaint, Perrea asked the Court to order CPS to produce copies of certain standardized tests referred to as Semester Exams. (Complaint at ) The Court set a deadline for the parties to submit evidence, November 4, (Entry, October 15, 2008 (10/15/08, Entry).) When CPS submitted its evidence, it included an affidavit of Elizabeth Holtzapple. (Evidence Submitted by Respondent Cincinnati Public Schools, Affidavit of Dr. Elizabeth Holtzapple.) On December 29, 2008, Perrea filed Motion to Strike Portions of Affidavit of Elizabeth Holtzapple. (MTS, Holtzapple.) The reason was that Holtzapple's affidavit contained inadmissible hearsay. (MTS, Holtzapple at 1-2.) On January 8, 2009, CPS filed a memorandum in opposition to Perrea's motion to strike. (Memo Opp, MTS Holtzapple.) With its memo, CPS attached a second affidavit, Affidavit ofjoanne L. Jensen, Employee of WestEd. (Jensen Affidavit.) CPS claimed that the Jensen Affidavit was not hearsay. Relator, Paul Perrea, now files this motion: (1) to strike the Jensen Affidavit; or (2) in the alternative, to file his own supplemental affidavit, Affidavit of Jerome V. D'Agostino, PIi.D. A more complete explanation follows. H. MEMORANDUM A. Jensen Affidavit This Court set a date of November 4, 2008, as a deadline for the parties to file evidence in this case. (10/15/08, Entry.) In this case, CPS filed the Jensen affidavit on January 8, (Memo Opp, MTS Holtzapple.) That was more than two months after the deadline set by the Court. Because of that, the Jensen Affidavit is not timely.

3 Furthermore, the Supreme Court Practice Rules state that "an application for an order or other relief shall be made by filing a motion for the order or relief." (S. Ct. R. XIV, 4(A).) In this case, CPS filed the Jensen Affidavit without filing a motion for an extension of time to file evidence. (Memo Opp, MTS Holtzapple.) Instead of filing a motion, CPS inserted a footnote in its memorandum in opposition to strike the Holtzapple Affidavit. (Memo Opp, MTS Holtzapple, fii.3) In that footnote, CPS asked the Court to consider the Jensen Affidavit. (Memo Opp, MTS Holtzapple, fn.3.) Then, CPS just attached the Jensen Affidavit to its memorandum in opposition to strike the Holtzapple Affidavit. Because CPS did not file a motion for an extension of time to file the Jensen Affidavit, it has violated S. Ct. R. XIV, 4(A). For these reasons, Perrea, moves to strike the Jensen Affidavit. B. D'Agostino Affidavit If the Court does not strike the Jensen Affidavit, Perrea concludes that the Court wants to give CPS every opportunity to present its arguments, even if CPS has not followed the Supreme Court Practice Rules. If that is so, Perrea moves, in the alternative, to submit his own supplemental affidavit, the Affidavit of Jerome V. D'Agostino, Ph.D. (D'Agostino Affidavit, attached hereto as Exhibit A.) If the Court allows the parties to submit supplemental evidence, the D'Agostino Affidavit will be helpful to reach a full understanding of the issues in this case. Dr. D'Agostino is a professor at the Ohio State University in the Quantitative Research, Evaluation, and Measurement section of the College of Education and Human Ecology. (D'Agostino Affidavit at 1.) That field is called "psychometrics." (D'Agostino Affidavit at 1.) Persons who specialize in psychometrics use a process of "test validation" to analyze standardized tests. (D'Agostino Affidavit at 6.) That is -2-

4 important in this case, because CPS does not have any evidence of test validation for its Semester Exams. (Merit Brief of Relator, Paul Perrea at 20.) Dr. D'Agostino explains in his affidavit that the system of "test validation" is a method that has been subjected to scientific testing. (D'Agostino at 7.) The scientific testing involves statistical analysis ofthe "reliability" o fthe standardized tests. (D'AgostinoAffidavitat 7(a).) The testing also involves evaluation of the "content" of the standardized tests. (D'Agostino Affidavit at 7(b).) And, the testing includes a statistical analysis of examinee scores to determine whether the stated goals of the standardized test are being met. (D'Agostino Affidavit at 7(c).) The system of test validation has also been subjected to publication and peer review. (D'Agostino Affidavit at 8.) In addition, test validation has a known or potential rate of error. (D'Agostino Affidavit at 9.) For high stakes tests - such as the CPS Semester Exams, which are used for 25% of students' grades - psychometricians consider a reliability coefficient of.80 or higher to show sufficient validity. (D'Agostino Affidavit at 9.) The process of test validation is also generally accepted in the area of standardized tests. (D'Agostino Affidavit at 10.) Organizations that use test validation include the United States Department of Education through its No Child Left Behind mandates. (D'Agostino Affidavit at 10.) Psychometricians are also concernedwith the concept of "test security" for standardized tests. (D'Agostino Affidavit at 12.) Test security often involves refreshing the test with new test items so that teachers do not "teach to the test questions." (D'Agostino at 12.) If test items are not refreshed, the validity of the test is severely compromised. (D'Agostino at 12.) -3-

5 Based on his training and experience, Dr. D'Agostino is an expert in the areas of "test validation" and "test security." (D'Agostino Affidavit at 6 and 12.) Dr. D'Agostino has also reviewed that materials that the parties have presented to the Court in this case. (D'Agostino Affidavit at 3.) Based on his review, Dr. D'Agostino is able to give two crucial opinions. (D'Agostino Affidavit at 13(a) and (b).) First, Dr. D'Agostino explains that, by not providing test validation studies related to the Semester Exams, CPS cannot say that the Semester Exams are reliable; CPS cannot say that the content o,f the Semester Exams is accurate; and CPS cannot say that the Semester Exams meet their intended purposes. (D'Agostino Affidavit at 13(a).) Second, Dr. D'Agostino explains that, by using the same Semester Exams indefinitely, CPS has created a situation in which test security likely has been breached. (D'Agostino Affidavit at 13(b).) Because of that, Semester Exams in all likelihood will be subject to "test score inflation." (D'Agostino Affidavit at 13(b).) That means that, over time, the validity of the test scores are greatly compromised. (D'Agostino Affidavit at 13(b).) Based on the contents of his affidavit, Dr. D'Agostino's opinions meet the requirements of Ohio R. Civ. P. 702(C)(l)-(3). (D'Agostino Affidavit at 5-12.) And, the opinions of Dr. D'Agostino closely track the argumerits in Perrea's briefs regarding the validity of the Semester Exams. (Merit Brief of Relator Paul Perra at ) For these reasons, Dr. D'Agostino's affidavit would be helpful to the Court in deciding the merits of the case. III. CONCLUSION Based on all these points, Relator, Paul Perrea, respectfully moves: (1) to strike the Jensen Affidavit; or (2) in the alternative, to file the Affidavit of Jerome V. D'Agostino, Ph.D. (Exhibit A.) -4-

6 Respectfully submitted, TED L. WILLS (Ohio Bar No ) 414 Walnut Street, Suite 707 Cincinnati, Ohio Telephone (513) Facsimile (513) Attorney for Relator, Paul Perrea CERTIFICATE OF SERVICE The undersigned hereby certifies that the above document was served by regular United States Mail on Mark Stepaniak, Taft Stettinius & Hollister, 425 Walnut Street, Suite 1800, Cincinnati, Ohio , this 29' day of January, Ted L. Wills ( ) c:/tlw/perrea.paul/ipleadinglprr.stri ke,jensen.motion -5-

7 SUPREME COURT OF OHIO STATE EX REL., PAUL PERREA Relator, MANDAMUS PROCEEDING -vs- CINCINNATI PUBLIC SCHOOLS CASE NO Respondent. AFFIDAVIT OF JEROME V. D'AGOSTINO, PH.D. TED L. WILLS (Ohio Bar No ) 414 Walnut Street, Suite 707 Cincinnati, Ohio Telephone:(513) Facsimile: (513) Attorney for Relator, Paul Perrea MARK J. STEPANIAK (007758) Taft Stettinius & Hollister 425 Walnut.Street, Suite 1800 Cincinnati, Ohio Telephone: (513) Facsimile: (513) Attorney for Respondent, Cincinnati Public Schools

8 STATE OF ARIZONA COUNTY OF PIMA Now comes Affiant, Jerome V. D'Agostino, Ph.D., who being duly cautioned and sworn upon his oath, states as follows: 1. My name is Jerome V. D'Agostino, Ph.D. I am an associate professor in the Quantitative Research, Evaluation, and Measurement section of the College of Education and Human Ecology, The Ohio State University. My office address is 29 West Woodruff Avenue, Ramseyer Hall, Room 301, Columbus, Ohio, I received my doctorate in educational measurement from The University of Chicago in I have produced over 30 publications related to testing, validity, and student achievement. Presently, I serve as the Chair of the technical review committee for the Arizona statewide achievement tests, and I have served on the technical review connnittee for the Arizona teacher and school administrator licensure examinations. For twelve years, I have taught undergraduate and graduate courses on classroom assessment, test development, validity analysis, statistics, and research methods at universities in Illinois, Arizona, and Ohio. As a general matter, this field of study can be referred to as "psychometrics." A copy of my curriculum vitae is attached hereto. 2. I have been contacted by Paul Perrea and his attorney, Ted L. Wills, to give an opinion on certain aspects of the CPS Semester Exams (CPS). 3. In the course of my consultation, I have reviewed the following items: a. Merit Brief of Relator, Paul Perrea; b. Affidavit of Paul Perrea; -2-

9 c. Second Affidavit of Relator, Paul Perrea; d. Respondent Cincinnati Public Schools' Response in Opposition to Merit Brief of Relator; e. Evidence Submitted by Respondent Cincinnati Public Schools. 4. The materials that I have reviewed establish that the system of Semester Exams at CPS consists of 24 exams administered to CPS 9"', 10`h, and 11'h grade students twice yearly after each semester in Mathematics, Science, English, and History. Each exam consists of 45 multiple choice and four constructed response items, except for the English tests which have 40 multiple choice and four more extensive response writing prompts. 'The tests were developed during the school year by CPS and its subcontractor, WestEd. CPS states that the four purposes of the exam are: (1) strengthen CPS grading consistency; (2) align curriculuni district wide, (3) help CPS students perform well on the Ohio Graduation Test, and (4) to prepare students for further education beyond high school. (Holtzapple Aff. 5) The tests were first administered to 9`s grade students in January 2007 (Perrea Aff. 19), and each test accounts for one quarter of a student's grade in the respective subject. (Holtzapple Aff. 6) Teachers are asked to administer the exams to their students, and to score the constructed response questions. There is no bank of items to refresh the exams with new items on subsequent administrations of the exams. (Holtzapple Aff. 7) 5. My education and professional experience have prepared me to be familiar with "standardized tests." As a general matter, a test is considered to be "standardized" if examinees are asked to complete a uniform set of questions, tasks, or other assessment activities, under very similar environmental situations, after receiving identical or nearly identical instructions, and their responses are scored and interpreted using the same processes. The primary purpose of "standardizing" a test is -3-

10 to hold constant any potential differences in testing and scoring that might unduly influence any student's performance. 6. Also because of my education and experience, I am familiar with the "test validation" process. In general, test validation is a process by which psychometricians determine whether a test actually measures what it is intended to measure. Tests that measure what they are intended to measure have high level of validity. Tests that do not measure what they are intended to measure have a low level of validity. For example, a test may be designed to determine if the examinees understand algebraic equations. A test would have a low level of validity if a large percentage of examinees who actually do understand the equations give "incorrect" answers to the test questions. According to the American Educational Research Association, American Psychological Association, & National Council on Measurement in Education Standards f'or Educational and Psychological Testing (Standards), documenting the validity of test scores for a specific purpose is the responsibility of the test sponsor and user. 7. The psychometric system of test validation has been subjected to scientific testing. a. The first step of scientific testing concems the standardized test's "reliability." In this context, "reliability" means that the examinees' performances are consistent and repeatable within the same testing period and over multiple testing periods. Reliability indices usually are reported as statistical correlation coefficients, and as measurement errors. b. The next step of scientific testing concerns the "content" of the standardized test. This process includes evaluation of the items by content experts for accuracy, alignment to standards, and potential bias. c. The next step of testing concerns statistical analysis. The statistical analyses are -4-

11 related to actual examinee scores based on scientific investigations. For example, giventhat the CPS Semester Exams are designed to prepare students for the Ohio Graduation Test, CPS should generate statistical evidence to detemvne whether the Semester Exam scores converge or correlate sufficiently with student's Ohio Graduation Test scores. Also, given that the Semester Exams represent one quarter of a student's semester grade, CPS should generate statistical evidence correlating scores from the Semester Exams with the other three quarters of the students' semester grades. CPS should undertake other validation efforts, depending on the usage of scores. As an example, given that the Semester Exam scores serve to prepare students for postsecondary school, and help align curriculum throughout the district, CPS should generate statistical evidence througli the scientific process to determine whether students are more prepared for postsecondary school and that curriculum is more aligned as a result of the Semester Exam testing system. 8. Test validation is a system that has been subjected to publication and often peer review. Commonly, the results for validation studies are submitted for peer review to scientific journals in measurement and testing, in which experts in the field judge the merit of the scientific studies and soundness of the conclusions. However, peer review and publication are not reserved for journals. Most typically, test sponsors develop technical review committees consisting of members with testing expertise who review test development, administration, scoring, and reporting methods, and offer suggestions on how to improve those processes. Also, the most common form of validation study publication is to compile study results in a test technical report that sponsors usually release to the public annually. Besides the results from validation studies, technical reports often document the processes followed in item and test development, and a detailed description of the scoring process. 9. Furthermore, the test validation process has a known or potential rate of error. Because of -5-

12 that, test validation studies nearly always include the reliability coefficients of scores and measurement error rates. For high-stakes tests --- i.e., tests that are used for purposes such as students' grades --- the standardized test should have a high reliability coefficient. For high-stakes tests such as the CPS Semester Exams, psychometricians would consider a reliability coefficient of.80 or higher to show sufficient validity. 10. The system of test validation is generally accepted in the psychometric community. For example, organizations such as the Educational Testing Service (ETS) and the Fducational Measurement Group of Pearson, as well as the United States Department of Education through its No Child Left Behind mandates, abide by the Standards for test validation evidence. 11. Based on the system of test validation, there is a general rule in the area of standardized tests. That general rule is that an organization cannot claim that a particular standardized test is sufficiently reliable, that its content is accurate, or that it meets its stated goals unless the standardized test has been subjected to a test validation process. 12. From my education and professional experience, I have become familiar with the concept of "test security." The notion of test security is much broader than merely securing the test booklets so examinees are unaware of test items before the exaniination period. "Test security" entails maintaining the integrity of test scores so that valid inferences can continually be drawn from the scores. In standardized achievement testing in schools, securing the tests often involves refreshing the test with new test items so that teachers do not "teach the test questions" rather than the curriculum are academic standards the tests were designed to measure. If teachers begin to "teach the questions," the validity of the scores are severely compromised, because students' scores no longer reflect the degree to which they learned from the curriculum, or the degree to which they attained the -6-

13 academic standards, but instead likely indicate the degree to which they learned the specific facts and principles embodied in specific test items. 13. Based on my review of the materials from this case and based on my education and expertise in this area, I am able to give the following expert opinions: a. By CPS not providing any validation studies of Semester Exams in this case, CPS cannot claim that its Semester Exams are reliable; CPS cannot say that the content of Semester Exams' is accurate; and CPS cannot say that the Semester Exams help meet their four intended purposes. b. By CPS re-using the same Semester Exam questions indefrnitely, and because teachers have had sufficient time to review and memorize specifics about test questions during the time period the test forms are in their possession, it is likely that test security as defined in 12, already has been breached. Likely, teachers either have or likely will begin to teach the isolated facts and principles tested by specific test questions, and it is my judgment based on prior studies that test score inflation will in all likelihood result. If scores become inflated due to the teaching of concepts presently tested on the exams, the validity of scores over time will be greatly compromised. FURTHER AFFINED SAITH NOT.!^' eb6me V. D'Agostino, Ph.D. Subscribed and sworn to me, a Notary Pylalic, this day of!z^

14 JEROME V. D'AGOSTINO Curriculum Vita, January, 2009 THE OHIO STATE UNIVERSITY College of Education & Human Ecology 29 W. Woodruff Ave., Columbus, OH Phone: (614) Fax: (614) CURRENT POSTTION: Associate Professor (2007 to Present), Quantitative Research, Evaluation, & Measurement (QREM) Section, School of Educational Policy and Leadership, College of Education & Human Ecology, The Ohio State University EDUCATION: Ph.D. (1997), Measurement, Evaluation, and Statistical Analysis (MESA) Program, Deparhnent of Education The University of Chicago, Chicago, Illhiois Lany V. Hedges & Kenneth K. Wong, Advisors M.S. (1991), Experimental Psychology, Department of Psychology Colorado State University, Fort Collins, Colorado Ross J. Loomis, Advisor B.S. (1988), Summa Cum Laude, Major: Psychology, Minor: Sociology Northern Arizona University, Flagstaff, Arizona. Lester J. Hunt, Advisor GRANTS AND CONTRACTS: Prospects: Special Analyses. Contract with United States Department of Education, Office of the Undersecretary (ED Contract # EA ), (Graduate research assistant, $100,000) Testing Mthin-School and School-Community Linkages. Grant from the National Science Foundation, Methodology, Measurement, and Statistics (MMS) prograin (NSF Grant # SBR ), (Graduate research assistant, $300,000) University of Chicago School Mathematics Project Grant, (Graduate research assistant, $10,000) Discovering the Effective Components of Family Literacy Programs. Grants from the College of Education and Office of the Vice President for Research and Graduate Studies, University of Arizona, (Principal hivestigator, $5,000) Measuring the Development ofnew Teachers: Implicationsfor Teacher Certification Testing. National Academy of Education/Spencer Postdoctoral Fellowship, (Principal Investigator, $45,000). Theoretical Analysis and Implementation Study of Comprehensive School Reform Programs. Grant from the United States Department of Education, Office of Educational Research and hnprovement (Award #

15 R306S000033), Participated from (did not participate in third grant year, 2003). (Co-principal Investigator, $1,000,000) Analyzing Parent and Child Outcome Data from Family Literacy Programs. Grant from the Arizona Department of Education, Division of Adult Education, (Principal Investigator, $47,000) Evaluation of the Somerton, Arizona Migrant Even Start Program. Grant from the Arizona Department of Education & Somerton School District, 2002 (Principal Investlgator, $20,000) Validity Analysis of the Arizona Instrument to Mea,sure Standards (AIMS). Grant from the Ari zona Departinent of Education, (Principal Investigator, $235,000) AIILLS High School Study. Grant from the University of Arizona, College of Education, Pima Educational Research Collaborative and Flowing Wells School Distriot, (Principal Investigator, $14,915) Development of Science Unit Assessments Aligned to School Curriculum and the Arizona Science Standard. Grant from the University of Arizona, College of Education, Pima Educational Research Collaborative and Tucson Unified School District, (Co-principal Investigator witli Brace Johnson and Christopher Harris, $15,000) Arizona Project WET Evaluation: Examining Impact and Developing Water Assessment Tools for Students. Grant from the University of Arizona, Technology and Research Initiative Fund, Water Sustainability Program, (Principal Investigator, $50,000) Rodel MAC-Ro Evaluation. Contract with the Rodel Charitable Foundation of Arizona, (Principallnvestigator,$35,000) Impact and Implementation of Discover a Watershed: The Colorado. Contract with Project WET Intetnational, (Principal Investigator, $40,000) Arizona Project WET Evaluation: Examining Impact and Developing a Computer-based Tutorial and Assessment System. Grant from the University of Arizona, Technology and Research Initiative Fund, Water Sustainability Program, (Principal Investigator, $50,000) NEWPATH. Nurturing, Through Entrepreneurship, IT World Leaders. Grant from the National Science Foundation, CISE Pathways to Revitalized Undergraduate Computing Education (CPATFI) program (Award No ), ( External Evaluator, $606,822) Curriculumfor Accelerated Services Engineering (CASE). Grant from the National Science Foundation, Course, Curriculum, and Laboratory Improvement (CCLI) program (Award No ), (Coprincipal investigator/evaluator, $149,981) PUBLICATIONS: Book Chapters: D'Agostino, J.V. (in press). Classroom assessment. In E. M. Anderman (Ed.), Psychology ofclassroom learning: An encyclopedia. Farmington Hills, MI: Macmillan Press. Page 2

16 Welsh, M.E., & D'Agostino, J.V. (2008). Fostering consistency between standards-based grades and large scale assessment results. In T.R. Guskey (Ed.), Practical solutionsfor serious problems in standardsbased grading. Thousand Oaks, CA: Corwin Press. D'Agostino, J.V. (2005). Measuring leaming outcomes: Reliability and validity issues. In G.D. Phye, J.R. Levin, and D.H. Robinson (Eds.), Empirical methods for evaluating educational interventions (pp ). San Diego, CA: Elsevier Academic Press. D'Agostino, J.V., & Stoker, G.L. (2002). Teachers' perceptions of Title I Program Improvement and accountability. In M. Wang & K. Wong (Eds.), Efficiency, accountability, and equity issues in Title I schoolwide program implementation (pp ). D'Agostino, J. V. (2000). Achievement testing in American schools: Where are we after 150 years? In T.L. Good (Ed.), 90 Yearbook of the National Societyfor the Study of Education. Chicago: University of Chicggo Press. D'Agostino, J. V., Hedges, L.V., Wong, K.K., & Borman, G.D., & (2000). Title I home-school partnerships: Effects on parenting practices and student achievement. In G.D. Borman, S.C. Stringfield, & R.E. Slavin (Eds.), Title L Compensatory education at the crossroads. Mahwah, NJ: Lawrence Erlbaum Associates. Borman, G.D., Wong, K.K., Hedges, L.V., & D'Agostino; J. V. (2000). Coordinating categorical and regular prograins: Effects on Title I students' educational opportunities and outcomes. In G.D. Borman, S.C. Stringfield, & R.E. Slavin (Eds.), Title L Compensatory education at the crossroads. Mahwah, NJ: Lawrence Erlbaum Associates. D'Agostino, J.V. (1999). Teacher roles and student achievement in high-poverty schools: hnplications for Title I schoolwide improvement. In G. Or6eld and E. H. DeBray (Eds.), Hard workfor good schools: Facts not fads in Title I reform (pp ). Cambridge MA: Harvard University Civil Rights Project. Roggenbuck, J.W., Loomis, R.J., & D'Agos6no, J.V. (1991). The (earning benefits of leisure. In B. L. Driver, P.J. Brown, & G.L. Peterson (Eds.), The benefits of leisure (pp ). State College, PA: Venture Publishing Inc. Journal Articles: D'Agostino, J.V., & Bonner, S.M. (in press). High school exit exam scores and university performance. Educational Assessment. D'Agostino, J.V., & Powers, S.J. (in press). Predicting teacher performance with test scores and GPA: A meta-analysis. American Educational Research Journal. Ko, Y., Malone, D.C., D'Agostino, J.V., Skrepneck, G.H., Armstrong, E.P., Brown, M., Woosley, R.L. (in press). Potential determinants of prescribers' drug-drug interaction knowledge. Research in Social and Administrative Pharmacy. Milman, L.H., Holland, A Kaszniak, A.W., D'Agostino, J.V., Garrett, M., & Rapcsak, S. (2008). Initial Validity and Reliability of the SCCAN: Using Tailored Testing to Assess Adult Cognition and Communication. Journal of Speech, Language, and Hearing Research, 51, Page 3

17 D'Agostino, J.V., Welsh, M.E., Cimetta, A.D., Falco, L.D., Hester VanWinkle, W., Smith, S., & Powers, S.J. (2008). Comparing the rating and matching item-objective alignment inethods. Applied Measurement in Education, 21, D'Agostino, J.V., Schwartz, K.L., Cimetta, A.D., & Welsh, M.E. (2007). Using a partitioned treatment design to examine Project WET impact. Journal ofenvironmental Education, 38(4), D'Agostino, J.V., & Hester VanWinkle, W. (2007). Identifying prepared and competent teachers with professional knowledge tests. Journal of Personnel Evaluation in Education, 20, D'Agostino, J.V., Welsh, M.E., & Corson, N.M. (2007). Instructional sensitivity of a state's standards-based assessment. Educational Assessrnent, 12, D'Agostino, J.V., & Murphy, J.A. (2004). A meta-analysis of Reading Recovery in United States schools. Educational Evaluation and PolicyAnalys is, 26, Jackson, T.R., Draugalis, J.R., Slack, M.K., Zachry, W.M., & D'Agostino, J.V. (2002). Validation of authentic performance assessment: A process suited for Rasch modeling. American Journal of Pharmaceutical Education, 66, D'Agostino, J.V. (2001). Increasing the role of educational psychology theory in program development and evaluation. Educational Psychologist, 36, D'Agostino, J.V. (2001). Recommendations for changes to Title I testing and program improvement systems. Title I Monitor, 6(2), D'Agostino, J. V. (2000). School effects on students' longitudinal reading and math achievements. School Effectiveness and Schoollmprovement, 11, D'Agostino, J.V. (2000). Testing should serve one purpose: A reply to Paris and Colleagues' papers. Issues in Education, 6, D'Agostino, J.V., Borman, G.D, Hedges, L.V., & Wong, K.K. (1998). Longitudinal achievement and Chapter 1 coordination in high-poverty schools: A multilevel analysis of the Prospects data. Journal of Education for Students Placed at Risk, 3, Borman, G.D., D'Agostino, J.V., Wong, K.K., & Hedges, L.V. (1998). The longitudinal achievement of Chapter 1 students: Preliminary evidence from the Prospects study. Journal of Education for Students Placed at Risk, 3, D'Agostino. J.V. (1996). Authentic instruction and academic achievement in compensatory education classrooms. Studies in Educational Evaluation, 22, Borman, G.D., & D'Agostino, J.V. (1996). Title I and student achievement: A meta-analysis of federal evaluation results. Educational Evaluation and PolicyAnalysis, 18, D'Agostino, J.V., & Day, S.K. (1991). Gender-role orientation and preference for an intimate partner. The Psychological Record, 41, Page 4

18 Roggenbuck, J.W., Loomis, R.J., & D'Agostino, J.V. (1990).1'he learning benefits of leisure. Journal of Leisure Research, 22, Hunt, L.J., Janssen, M., D'Agostino, J.V., & Gruber, B. (1989). Haptic identification of rotated letters using the left or right hand. Perceptual and Motor Skills, 68, SELECTED TECHNICAL REPORTS: D'Agostino, J.V. (August, 2008). The Effectiveness of the Superkids on Student Achievement and Teacher Outcomes. Madison, WI: Rowland Reading Foundation. D'Agostino, J.V., & Rooks, D.L. (September, 2007). Alternate Accommodations Study of the AIMS Mathematics Exams: Final report. Phoenix, AZ: Arizona Department of Education. D'Agostino, J. V., & Kosorok, J. (August, 2007) MAC-Ro (Math Achievement Club by Rodel) Initiative evaluation. Phoenix, AZ: Rodel Foundation. D'Agostjno, J.V., & Meador, S. N. (July, 2007). Discover a Watershed: The Colorado and Discover the Colorado River KIDs Activity Booklet: Evaluation final report. Bozeman, MT: Project WET. D'Agostino, J.V., Schwartz, K.L., Cimetta, A.D., Welsh, M.E. (2006) Arizona Project WET evaluation: Examining impact and developing water education assessments. Final Report. Tucson, AZ: University of Arizona, Technology and Research Initiative Fund (TRIF), Water Sustainability Program D'Agostino, J.V., Welsh, M.E., & Cimetta, A.D. (September, 2005). Alignment analysis ofarizona Academic Standards and Assessments. Phoenix, AZ: Arizona Department of Education. D'Agostino, J.V. (November 2003). AIMS validity analysis studies. Report for Interagency Service Agreement ED. Phoenix, AZ: Arizona Deparlment of Education. D'Agostino, J.V., & Tingelhoff, H. (2002) Family Literacy evaluation: Participant characteristics and outcomes. Phoenix, AZ: Arizona Department of Education, Division of Adult Education. D'Agostino, J.V., & Stoker, G.L. (2001) Family Literacy evaluation: Participant characteristics and outcomes. Phoenix, AZ: Arizona Department of Education, Division of Adult Education. D'Agostino, J.V., Haladyna, T., & Hess, R(2000). Test quality report: Arizona Teacher Proficiency Assessment Phoenix, AZ: Arizona Board of Education. D'Agostino, J.V., & Stoker, G.L. (2000) Family Literacy evaluation. Participant characteristics and outcomes. Phoenix, AZ: Arizona Department of Education, Division of Adult Education. D'Agostino, J.V. (1998). Summative evaluation of the Beloit Even Start Family Literacy Program. Beloit, WI: Beloit, WI Depariment of Education. Wong, K. K., Hedges, L. V., Borman, G. D., & D'Agostino, J. V. (1996). Prospects SpecialAnalyses: Final report. Washington, DC: U.S. Department of Education. Bakall, E., & D'Agostino, J.V. (1995) Reading Recovery program impact on siudem achievement: Page 5

19 "What can we learn from the Observation Survey? " Chicago, IL: Chicago Public Schools. D'Agostino, J. V. (1994). Effectiveness of advanced-skill instruction during the Chapter 1 Summer Programs. Chicago, IL: Chicago Public Schools. D'Agostino, J. V. (1993). Planning-stage evaluation offamily programs at the Oriental Institute Museurn. Chicago, IL: Oriental Institute Museum. D'Agostino, J. V. (1992). Evaluation of a multicultural camp-in at the Denver Museum of Natural History. Denver CO: Denver Museum of Natural History. D'Agostino, J. V. (1991). Visitors' beliefs, attitudes, intended behaviors, and information retention about urban wildlife in the Close to Home exhibit. Denver, CO: Denver Museum of Natural History. D'Agostino, J. V., Harvey, M. L., & Loomis, R. J. (1990). Evaluation ofparticipant learning and interaction during a multicultural camp-in at the Denver Museum of Natural History. Denver, CO: Denver Museum of Natural History. D'Agostino, J. V., & Loomis, R. J. (1990). Museum of Natural History. Visitor reactions to Hall oflife exhibits. Denver, CO: Denver Loomis, R. J., Birjulin, A., D'Agostino, J. V., & Smith, J. (1989). Forrnative testing of the Conception Game exhibit. Denver, Co: Denver Museum of Natural History. Loomis, R. J., Biqulin, A., & D'Agostino, J. V. (1988). Front-end visitor survey about the Hall of Life. Denver, CO: Denver Museum of Natural History. CONFERENCE PRESENTATIONS: D'Agostino, J.V., & Ciinetta, A.D. (March, 2008). The alignment of high school exit exams and college performance. Presented at the annual meeting of the American Educational Research Association Conference, New York, NY. Schwartz, K.L., D'Agostino, J.V., & Cimetta, A.D. (November, 2007). Arizona Project WET evaluation: examining impact. Presented at the annual meeting of the North American Association for Environmental Education, Virginia Beach, VA. D'Agostino, J.V., Citnetta, A.D., & Schwartz, K. (April, 2007). Differential treatment evaluation of Project WET. Presented at the annual meeting of the American Educational Research Association Conference, Chicago, IL. D'Agostino, J.V., & Welsh. M.E. (April, 2007). Standards-based progress reports and standards-based assessment score convergence. In T.R. Guskey (Chair), The challenges of standards-based grading and reporting. Symposium conducted at the meeting of the American Educational Research Association, Chicago, IL. D'Agostino, J.V., & Powers, S. (April, 2006) Predicting teaching competence with teacher test scores: A nteta-analysis. Presented at the annual meeting of the American Educational Research Association Conference, San Francisco, CA. Page 6

20 Welsh, M.E., & D'Agostino, J.V. (April, 2006). Examining instructional validity: An interview-based approach. Presented at the amiual meeting of the American Educational Research Association Conference, San Francisco, CA. D'Agostino, J.V., & Hester, W.S. (April, 2005). What can be learned from teacher certification tests? Presented at the annual meeting of the American Educational Research Association, Montreal, Quebec. D'Agostino, J.V., & Welsh, M.D. (October, 2004). Standards-based progress reports and AIMS scores: Convergent or divergent information? Symposium conducted at the meeting of the Arizona Educational Research Organization, Tempe, AZ. D'Agostino, J.V. (April, 2004). Measuring learning outcomes: Reliability and validity issues. In G.D. Phye (Chair), Experimental methods in educational interventions. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA. D'Agostino, J.V., Corson, N., & Falco, L. (April, 2003). Exploring the Validity of a State's Standards-based Assessment. Symposium conducted at the ineeting of the American Educational Research Association, Chicago,IL. D'Agostino, J.V. (April, 2002). Selection effect issues in studying Comprehensive School Reform in Arizona. In D.K. Aladjem (Chair), Methodological Approaches to Sludying Comprehensive School Reform. Symposium conducted at the meeting of the American Educational Research Association, New Orleans, LA. D'Agostino, J.V. (April, 2002). The school size and SES interaction: True effect or spurious relationship? In L.V. Hedges (Chair), Applications of HLMin Studies of School Effects. Symposium conducted at the meeting of the American Educational Research, Association, New Orleans, LA. D'Agostino, J.V. (April, 2001). Does the Title I accountability model motivate teachers in high-poverty schools to improve their instruction? In J.V. D'Agostino (Chair), Research on the pa.st, present, and future of Title I. Symposium conducted at the meeting of the American Educational Research, Association, Seattle, WA. D'Agostino, J.V., & Stoker, G. (April, 2001). Even Start family literacy program approaches and parent and child outcomes. Presented at the annual meeting of the American Educational Research Association, Seattle, WA. D'Agostino, J.V., & Chen, B. (April, 2000). Construct validatingparent involvement models. Presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Crehan, K. D., Hess, R. K., & D'Agostino, J.V. (April 2000). The technology of teacher certiftcation testing: A discussion of issues and recommendations for practice. Presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Johanning, D., D'Agostino, J.V., Steele, D., & Shumow, L. (April, 1999). Student writing, post writing group collaboration, and learning in pre-algebra. Presented at the meeting of the American Educational Research Association, Montreal, Quebec. Murphy, J.A., & D'Agostino, J.V. (April, 1999). A meta-analysis of Reading Recovery. Presented at the meeting of the American Educational Research Association, Montreal, Quebec. Page 7

21 D'Agostino, J.V. (September, 1998). Types of teacher decision-making and student achievement in highpoverty schools: Implications for Title I schoolwide improvement. Invited paper presentation at the Title I: Seizing the Opportunity conference, Washington, DC. D'Agostino, J.V. (April, 1998). The effectiveness of Title I parent programs: A multilevel analysis of Prospects data. Paper presented at the meeting of the American Educational Research Association, San Diego, CA. Jacobs, S. E., Mishkin, M., & D'Agostino, J. V. (April, 1997). Recognition memory for objects in infant and adult rhesus monkeys. Poster session presented at the meeting of the Society for Research in Child Development. Washington, DC. D'Agostino, J. V. (April, 1996). School effects and successful Title I programs. In E. M. Scott (Chair), A systemic analysis oftitle I effectiveness: Final results of the multi-level study ofprospects. Symposium conducted at the meeting of the Ainerican Educational Research, Association, New York, NY. D'Agostino, J. V. (April, 1995). Advanced-skill instruction in Chapter 1 summer programs and student achievement. Paper presented at the meeting of the American Educational Research Association, San Francisco, CA. Borman, G. D., & D'Agostino, J. V. (April, 1995). Chapter I and student achievement: A meta-analysis of thirty years of test results. Paper presented at the meeting of the American Educational Research Association, San Francisco, CA. D'Agostino, J. V. (April, 1994). Improving the identification ofschools for Chapter 1 Program Improvement. Paper presented at the meeting of the American Educational Research Association, New Orleans, LA. D'Agostino, J. V., Wallace, T., & Zhang, G. (April, 1993). Increasingparent involvement in giftedprograms through information that addresses their beliefs. Paper presented at the meeting of the American Educational Research Association, Atlanta, GA. EDITORIAL EXPERIENCE: Editorial Advisory Board Member: Reading Research Quarterly (2008 to Present) Educational Assessment (2008 to Present) Journal of P,sychoeducational Assessment (2005 to present) Elementary School Journal ( ) Journal Reviewer: American Educational Research Journal American Journal of Education Educational Evaluation and Policy Analysis (recognized at AERA as outstanding reviewer, 2005) Educational Measurement: Issues & Practice Journal of Applied Social Psychology Journal of Educational Psychology Journal of Speech, Language, and Hearing Research Page 8

22 The International Journal of Disability, Development and Education Review of'educational Research Socioloio, ofeducation Grant Proposal Reviewer: Reviewer for National Science Foundation ROLE Program, February Reviewed 12 proposals and met in Washington, DC for group discussion. Reviewer for OERI Comprehensive School Reform Program Grants, June Reviewed about 12 proposals and participated in group conference calls. Reviewer for the National Board for Professional Teaching Standards Grant Competition, June, Judged about ten proposals independently, and then participated in a conference call to make fmal decisions. Reviewer for Field-Initiated Studies Grant Proposals, Office of Educational Research & Improvement (OERI), United States Department of Education -- served on two review panels in October & May, Reviewed about 12 grant proposals per meeting. MEMBERSHIPS: American Educational Research Association American Evaluation Association International Congress for School Effectiveness and Improvement National Council on Measurement in Education National Society for the Study of Fducation Sigma Xi, The Scientific Researcli Society (was nominated and elected as a graduate student) REFERENCES: Dr. Lawrence Aleamoni Professor and Department Head Special Education, Rehabilitation, & School Psychology College of Education University of Arizona Tucson, AZ (520) aleamonl@ .arizona.edu Dr. William Mehrens, Professor Emeritus Michigan State University 2351 Sapphire Lane East Lansing, Ml (517) (home address and phone) wmehrens@msu.edu Page 9

23 Dr. Kenneth Wong Walter and Leonore Annenberg Professor in Education Policy Education Department Brown University Barus Hall, Box 1938 Providence, RI (401) Page 10