1 CASAS Technical Assistance for California WIA Title II Funded Agencies Experienced Staff 2009-10
California Accountability for Experienced Users To participate in this Web Conference, at a minimum all participants should: Have completed a California Accountability for New Users workshop Have at least one year of experience as a teacher, administrator, or database manager at a California WIA Title II agency. 2
California Accountability for Experienced Users 3 At the end of this session, participants will be able to: Describe WIA and NRS requirements Identify the 2009-10 data submission timeline and deliverables Identify key policy guidelines Identify the Payment Points structure for WIA Title II 225, 231, and EL Civics funded agencies Interpret TOPSpro reports that assist with state and federal reporting requirements Locate resources and technical assistance
National Reporting System The National Reporting System (NRS) is the accountability system for federally funded adult education programs. The NRS is important to states, programs, teachers, and students because it provides a means of regular evaluation of the learner outcomes for adult education programs. In August 1998 the voluntary nature of the NRS changed when the Adult Education and Family Literacy Act within the Workforce Investment Act (WIA - P.L. 105-220) became law. 4
5 NRS Resources To learn more information about the National Reporting System: http://www.nrsweb.org http://www.nrsonline.org
WIA Title II Data Deliverables Quarterly TOPSpro data submission Updated TOPSpro export to CASAS via Internet/disk Data Integrity Report Quarterly Core Performance Survey Run TOPSpro Core Performance Reports Disseminate surveys to students and enter results on CASAS Web site Separate results for distance learning students California WIA Title II Instructional Questionnaire CASAS sends questionnaires to agencies mid-october Agencies complete and return to CASAS by mail WIA Title II Program Implementation Survey Complete online on CASAS Web site End of Year Data Submission Submit TOPSpro export to CASAS via Internet/disk Data Integrity Report Certification Letter Payment Points Summary Report AE Personnel Wizard 6
7 WIA Title II Data Collection Timeline Data Submission 1st Quarter Data Submission 2nd Qtr Data Submission 3rd Quarter Data Submission Year End Data Submission None Core Performance 08-09 4 th Quarter Sept 30, 2009 1st Quarter Follow-Up Survey 2nd Quarter Follow-Up Survey 3 rd Quarter Follow-Up Survey 4th Quarter Follow-Up Survey Date Due October 31, 2009 January 31, 2010 April 30, 2010 August 15, 2010 September 30, 2010
First Quarter Data Submission 8
WIA Title II Instructional Questionnaire Submission 9
Second Quarter Data Submission 10
Third Quarter Data Submission 11
Fourth Quarter Data Submission 12
13 Follow-Up Quarter Data Submission By September 30, 2010 Required Data and Documents Fourth Quarter Core Performance Follow-up Survey Submit online/electronically http://www.casas.org Mail hard copy Not Applicable (Online Only)
14 Summary of Required Surveys WIA Title II Requirement Core Performance Survey Frequency of Submission Quarterly Submission Method Online WIA Survey Annually Online Class Questionnaire Annually Paper
15 Core Performance Follow-up Survey WIA II agencies will now be able to distribute Core Performance surveys by phone, mail, or e-mail. Must separate Distance Learning students Download surveys for each format at www.casas.org
16 Data Collection Requirements Entry Records For Each WIA Title II Learner: Collect Gender (Field 4) and Date of Birth (Field 5) Assist the learner in selecting primary and secondary goals, labor force status, and Distance Learning, if applicable Learners with a goal of attaining a high school diploma or GED (Field 12) must have a CASAS test at or above 236
FAQ: Appropriate Goal Setting and Program Placement for ASE/GED Learners Only affects Entry Record Field 12, goal setting. No effect on Payment Point attainment No effect on program placement Specific CASAS Scale Score thresholds: Test Benchmark CASAS Scale Scores (Reading or Math) Low ASE 236 245 High ASE 246 and above 17
Data Collection Requirements Update Records Must be completed after a substantial block of instruction or at the end of the instructional period (semester, quarter, term) Must be completed when a learner exits a program Mark the appropriate learner status, progress, instructional level and reason for exiting 18
19 FAQ: Data Collection Requirements Update Records Field 7 Status. Mark Retained in Program if student plans to return during the program year of if unsure about returning. Field 8 Progress. Mark Completed Level if student advances based on pre/post-test results but receives same instruction; Advanced Level if student receives more advanced instruction. Field 13 Reason for Exiting. Mark only if Field 7 Status = Left Program.
California Assessment Policy The CDE requires WIA Title II funded local agencies to test all students enrolled in ABE, ESL, and ASE instructional programs. Pretests are recommended as soon as the student enrolls in the program Post-tests are recommended at the end of each quarter, semester, or term to document continuous learner improvement after approximately 70-100 hours of instruction. An updated version dated June 2009 is available for download on the CDE and CASAS Web sites 20
21 California Assessment Policy Highlights of 2009-10 CDE Assessment Policy Lists appropriate CASAS assessment instruments authorized for use for WIA II accountability reporting. Details policies for appropriate test administration, scoring, and reporting. Describes appropriate use of test results to inform instruction and improve instructional program outcomes. Documents CDE training attendance policy for WIA II agencies Includes Local Assessment Policy guidelines
California Assessment Policy Key changes in 2009-10 CDE Assessment Policy Mandates that all learners must have a valid pretest for placement on the NRS Federal Tables. Addresses the minimum hours requirement between CASAS pre- and post-tests must have at least 40 hours between pretest and the first post-test. (New DIR Item #28) Establishes a California statewide performance goal of at least 50 percent paired test scores for the 2009-10 program year. Includes more specific guidance for distance learning programs. 22
Local Assessment Policy WIA Title II agencies must develop and implement a Local Agency Assessment Policy, and update it annually. Local agencies may develop their own local assessment policy guidelines, but must address, at a minimum, all CDE assessment policy guidelines and those included in the Local Agency Assessment template. Download the template at the California Accountability page, at www.casas.org CDE will review local policy and implementation during program monitoring. 23
Appropriate Pre- and Post-Test Selection Appropriate 81R-82R-81R 81RX-82RX-81RX 82R-82RX-84R Inappropriate 81R-81R-81R 81R-87R 85R-83R Post-test form of equal or higher level, but not administered consecutively 24
25 Valid Paired Test Records Accurate Pretest Test score is within the accurate range or conservative estimate range (high end or score) Appropriate Post-Test Scored within the accurate or conservative estimate range Post-test form of equal or higher level, but not administered consecutively
FAQ: Conservative estimate test scores Conservative estimate scores will continue to qualify as valid pre- and post-tests during the 2009-10 Program Year Agencies can monitor their performance by reviewing Item #27 on the Data Integrity Report. Guidance will be provided indicating acceptable percentages of conservative estimate ( score) pretests. 26
27 Use of Assessment Modalities ABE/ASE: Use Reading or Math ESL: Use Reading or Listening Pre- and post-test pairs must always be from the same test modality.
28 Multiple Modalities Paired Tests What if a student has two sets of pretest and post-test scores, such as one in reading and one in listening? The pretest and post-test pair with the lowest accurate pretest score determines benchmark attainment
ABE/ESL CASAS Pretest 236 or Higher Payment Points Drop Reason #10 Enroll learner in GED or high school diploma instructional program with goal of GED or high school diploma Test in another modality Do nothing and student will not be included in federal reporting for WIA Title II. 29
30 Payment Point Structure Benchmarks Significant gain (on CASAS scores) Two-level advancement Completion of GED or high school diploma SODs Civic Participation: Additional Assessments (COAAPs) Citizenship Preparation: CIT, G & H for Citizenship Test
Significant Gain Benchmark 5 points gain or greater for pretest scores of 210 or below 3 points gain or greater from pretest scores of 211 and above Earned in only one modality Earned only once for each learner per program year 31
32 Significant Gain Benchmark Calculation Significant Gain Calculated by first accurate pretest to highest appropriate posttest First Accurate 200 206 203 Highest 210
33 Significant Gain Benchmark Calculation Significant Gain Pretest = 206 Post-test = 210 4 point gain 206 First Accurate 203 Highest 210
34 Significant Gain Benchmark Calculation Significant Gain Pretest = 200 Post-test = 206 6 point gain Highest First Accurate 200 206 203
Two Level Advancement Benchmark Refers to CASAS levels within the NRS adjusted levels of performance Gain resulting in two-level movement on California Benchmark Levels See 2009-10 California Administration Manual, Section 6 Earned in only one modality Earned only once for each learner per program year 35
Payment Points for Basic Skills (ABE) NRS (National Reporting System) Level Names for BS CASAS Score Ranges for NRS Levels CASAS Level Names for ABE CASAS pretest scaled score range for California Payment Point Minimum scaled score gain in order to earn a significant learning gain Minimum post-test scaled score to achieve a two level gain Beginning Adult Basic Education Literacy Beginning Basic Education 200 and below 201 210 Beginning Literacy/Pre- Beginning Beginning Basic Skills 180 and below 5 or more 191 181 190 points 201 (3 or more 191 200 points for 211 learners using 201 210 POWER) 221 Low Intermediate ABE 211 220 Intermediate Basic Skills 211 220 236 High Intermediate ABE 221 235 Advanced Basic Skills Low Adult Secondary Education High Adult Secondary Education 221 235 246 3 or more 236 245 Adult Secondary 236 245 points N/A 246 and above Advanced Adult Secondary 246 + N/A 36
Payment Points for ESL NRS (National Reporting System) Level Names for ESL CASAS Score Ranges for NRS Levels CASAS Level Names for ESL CASAS pretest scaled score range for California Payment Point Minimum scaled score gain in order to earn a significant learning gain Minimum post-test scaled score to achieve a two level gain Beginning ESL Literacy 180 and below Beginning Literacy/ Pre-Beginning ESL 180 and below 191 Low Beginning ESL High Beginning ESL 181 190 191 200 Low Beginning ESL High Beginning ESL 181 190 5 or more 201 points 191 200 211 Low Intermediate ESL 201 210 Low Intermediate ESL 201 210 221 High Intermediate ESL Low Advanced ESL 211 220 High Intermediate 211 220 ESL 3 or more points 236 221 235 Advanced ESL 221 235 246 37
38 GED and High School Diploma Benchmarks GED Benchmarks GED students must pass all five sections Must mark Passed GED on Update Record Passing the Spanish GED is acceptable for GED benchmark attainment GED students must use their Social Security number (SSN) or GED ID CDE will perform a data match for GED students
High School Diploma Benchmarks HSD Benchmarks High school diploma students must pass the California High School Exit Exam (CAHSEE) Must mark Earned HS diploma on Update Record Maintain certified list of high school diploma students 39
Student Outcome Dataset (SODS) Civic Participation Pass one or more CDE-approved Additional Assessments (COAAPs) Completion of Entry, Update, Pretest, and Post-test 40
Student Outcome Dataset (SODS) Citizenship Preparation Pass Citizenship Interview Test (Score = 206 or above Pass Government and History for Citizenship Test (Score = 206 or above) Also requires presence of Entry, Update, Pretest, Post-test 41
225/231 Funded Agencies 1. Significant Gain 2. Two-Level Advancement 3. GED/HSD The maximum number of Payment Points a student may earn for 225/231 only funded agencies is three (3). 42
Civic Participation Agencies 1. Significant Gain 2. Two-Level Advancement 3. Additional Assessments (up to 3) The maximum number of Payment Points a student may earn for Civic Participation only funded agencies is five (5). 43
Citizenship Preparation Only 1. Significant Gain 2. Two-Level Advancement 3. Government and History for Citizenship (Passing Score 206) 4. Citizenship Interview Test ( 206) The maximum number of Payment Points a student may earn for Citizenship Preparation only funded agencies is four (4). 44
45 Section 231 and EL Civics Funded The maximum number of Payment Points a student may earn from multiple focus areas is five (5), from a combination of benchmarks and SODS. 231 and Civic Participation 231 and Citizenship Preparation Civic Participation and Citizenship Preparation 231, Civic Participation, and Citizenship Preparation
46 Instructional Program Focus Areas Hierarchy for 225/231 only If a student is enrolled in more than one focus area, programs prioritize in the following order: 1. ESL* 2. ABE* 3. ASE ** *Family Literacy (ABE or ESL) *Vocational Literacy (VABE or VESL) **(ASE learners scoring 236 or above on pretest who also enrolled in ESL or ABE will be reported as ASE)
Instructional Program Focus Areas Hierarchy for EL Civics & 225/231 If a student is enrolled in more than one focus area, then the following takes priority: 1. Citizenship Preparation 2. Civic Participation 3. 225/231 47
48 EL Civics Guidelines Entry Record for each EL Civics focus area a student attends must be for an EL Civics designated class EL Civics students must have instructional program of ESL (not ESL- Citizenship) Update Record in either an ESL or EL Civics focus area after 12 or more hours of instruction CASAS pre- and post-tests can be from any focus area, but must be reading or listening
49 FAQ: Incomplete EL Civics Outcome Dataset Description of drop reasons: 15a: Incomplete dataset in both Civic Participation and Citizenship Preparation 15b: Incomplete dataset in Civic Participation only 15c: Incomplete dataset in Citizenship Preparation only Appears if learner passed CIT, G & H, or SOD s but did not have complete dataset of Entry, Update, and valid pre- and post-test Appears if learner has complete dataset (Entry, Update, pre, post) but no passed AA s For agencies that are also 231 funded, appears if learner earned non-elc Payment Points such as Significant Gain or Two Level Advancement
Civic Participation Due Dates Select at least one Civic Objective Additional Assessment Plan (COAAP) October 31, 2009 Enter New COAAPs Option 3 November 30, 2009 Submit All COAAPs April 30, 2010 50
2009-10 EL Civics Technology Plan A deliverable for EL Civics Civic Participation Due on January 31, 2010 Submitted online Requires a planning team www.cde.ca.gov/sp/ae/fg/ Or, www.otan.us/adulted/
52 Data Integrity Report Displays 28 items that are important for State and Federal requirements.
53 Payment Points Reports TOPSpro Payment Points reports display California accountability information for WIA-funded agencies. Each Payment Points report uses the same rules to determine outcomes, but each differs in the amount of detail it displays. Specific Payment Points reports: Payment Points Audit Payment Points Summary Payment Points Totals Payment Points Monitor Payment Points Exceptions
Payment Points Drop Reasons 01 Missing birth date or age outside of 16-110 02 Less than 12 hrs of instruction 03 Concurrently enrolled in HS/K-12 07 No Entry/Update Record 10 Non-ASE learner scored 236 or higher 15a Incomplete EL Civics Outcome dataset 15b Incomplete Civic Participation Outcome dataset 15c Incomplete Citizenship Preparation Outcome dataset 17 EL Civics student in non-esl program 18 Unfunded Payment Point 19 AA passed and highest test score below 180 54
55 EL Civics Drop Reasons 01 Missing birth date or age outside of 16-110 02 Less than 12 hrs of instruction 03 Concurrently enrolled in HS/K-12 15a Incomplete EL Civics Outcome dataset 15b Incomplete Civic Participation Outcome dataset 15c Incomplete Citizenship Preparation Outcome dataset 17 EL Civics student in non-esl program 19 AA passed and highest test score below 180
Top 10 Reasons Agencies miss out on Payment Points 56 1. No post-test 2. Less than 12 hours of instruction 3. Did not use GED ID/SSN for GED match 4. No pretest 5. Incomplete Civic Participation outcome dataset 6. No Update Record 7. No Entry Record 8. Missing birth date/age 9. Non-ASE learner scoring 236 or above 10. Incomplete Citizenship Preparation outcome dataset
Federal Tables Federal Tables display data in format required by U.S. Department of Education, National Reporting System (NRS). TOPSpro uses the Federal Tables to report statewide data to the Federal Government. 57
List of Federal Tables Table Title 1 Participants by Entering Educational Functioning Level, Ethnicity & Gender 2 Participants by Age, Ethnicity and Gender 3 Participants by Program Type and Age 4a Educational Gains and Attendance by Educational Functioning Level 4b Educational Gains and Attendance by Educational Functioning Level 4c Educational Gains/Attendance by EFL Distance Learning *NEW* 5 Core Follow-up Outcome Achievement 5a Core Follow-up Outcome Achievement Distance Learning *NEW* 6 Participant Status and Program Enrollment 7 Adult Education Personnel by Function and Job Status 8 Outcomes for Adults in Family Literacy Programs (Optional) 9 Outcomes for Adults in Workplace Literacy Programs (Optional) 10 Outcomes for Adults in Correctional Education Programs (Optional) 11 Secondary Outcome Measures (Optional) 12 Work-based Project Learners by Age, Ethnicity and Gender (Optional) 58
59 Federal Table 4 Federal Table 4 displays Educational Gains by Educational Functioning Level. Specifically, it reports learners who entered program at one Instructional Level and finished the program year (June 30) at a higher level.
60 Federal Table 4B Federal Table 4B displays the same level advancement information as Table 4, but it only includes learners who completed a valid pre- and post-test.
61 Federal Table 4C Federal Table 4C displays the same level advancement information as Table 4, but it only includes learners who are enrolled in Distance Learning. This table is new for 2007-08.
Educational Functional Levels 62
Educational Functional Levels 63
64 Level Completion Examples Completed a Level Examples for Low Intermediate ESL: 1. Pretest = 205 Post-test = 208. Completed a level? NO. 2. Pretest = 205 Post-test = 212. Completed a level? YES. 3. Pretest = 209 Post-test = 210. Completed a level? YES.
65 Persister Report The Persister Report compares the percentage of all learners completing a level versus the percentage of learners completing a level who completed a pre- and post-test.
66 NRS Performance Report The NRS Performance Report compares your own agency s results to your statewide goals.
Local Performance Goals The CDE strongly encourages WIA Title II agencies to establish local performance goals. These goals can be maintained locally and should be realistic based on previous agency performance. Review previous agency performance on Federal Table 4 and 4B. 67
**This year s California Administration Manual is available for download at www.casas.org **
2009-10 Ordering Guide for WIA II Agencies WIA Title II agencies can order select CASAS materials free of charge using the 2009-10 Ordering Guide. Access the Ordering Guide in your California Administration Manual or download it separately at www.casas.org. 69
70 Technical Assistance CDE Consultants Program assistance Resources Policy Compliance Legal and fiscal Regional meetings CASAS Staff Provide assessment and curriculum support Support California accountability Provide TOPSpro and other technical assistance Regionally assigned Program Specialists Coordinate with CDE consultants
71 Resources Regional Network Meetings Each region in California holds network meetings where participants can address concerns and discuss ways to better meet accountability requirements. TOPSpro Network Meetings WIA II Network Meetings EL Civics Network Meetings Network groups meet monthly, quarterly, or whatever is most appropriate for each specific region. Register for network meetings a www.casas.org.
72 What s New Online Registration California Accountability Page CASAS Forums Download Centers CASAS Web Site www.casas.org
EL Civics Web site 73
CASAS Forums 74
75 Data Portal CASAS Data Portal enables users to view state and local data by agency, region, agency size, and provider type.
76 Resources CASAS Technical Support: 1-800-255-1036 CASAS Fax # 1-858-292-2910 E-mail: capm@casas.org topspro@casas.org elcivics@casas.org
Contact Information for New Personnel Update WIA II Contact Information Include name, phone #, email and title: Primary WIA or EL Civics contact or district administrator FAX on District Letterhead to: Rich Berry @ CDE, Fax #916-327-7089 Rebeca Mellado @ CASAS, Fax #858-292-2910 77
Thanks for your participation! 78