Oregon Online Assessment Reporting System User s Guide

Transcription

1 Oregon Online Assessment Reporting System User s Guide Version 3.0 Updated on August 12, 2008 Created for the Oregon Department of Education by the American Institutes for Research - 1 -

2 TABLE OF CONTENTS WHAT IS THE OREGON ONLINE ASSESSMENT REPORTING SYSTEM?...4 WHICH ASSESSMENTS ARE INCLUDED IN THE SYSTEM?...5 GETTING STARTED...5 WHAT ARE THE SYSTEM REQUIREMENTS?...5 HOW DO I ACCESS THE SYSTEM?...6 Logging In...6 First-time Users: Acknowledgment Page...6 Selecting a District...7 Selecting a Test Grade and Subject...8 HOW DO I NAVIGATE THE SYSTEM?...8 MANAGE MY ROSTERS...10 WHY SHOULD I ADD ROSTERS?...10 HOW DO I MANAGE MY ROSTERS?...10 STUDENTS WITH NO GROUP...14 DASHBOARD...15 SCORE REPORTS...17 WHAT IS THE FIRST REPORT I SEE (ONLINE SCORE REPORT)?...17 District Online Score Report...17 What can I do with my report?...18 How do I navigate within the Online Score Report?...19 What are the achievement standard levels and cut scores?...19 What do the legend symbols mean?...20 WHAT GRAPH REPORTS ARE AVAILABLE?...20 Performance by Resolved Highest Scale Score Graph...20 Performance by Level of Aggregation Graph...22 Performance by Score Reporting Category Graphs...23 HOW OFTEN ARE THE REPORTS UPDATED?...23 *LONGITUDINAL REPORTS (GRAPHS)...24 TRACKING BY SUCCESSIVE CROSS-SECTIONS...24 TRACKING BY INDIVIDUAL STUDENTS...25 TRACKING BY COHORT...26 * Denotes information relevant to new features added to the system for Fall

3 TABLE OF FIGURES FIGURE 1: OREGON ONLINE ASSESSMENT REPORTING SYSTEM...4 FIGURE 2. ACKNOWLEDGMENT PAGE....7 FIGURE 3. DISTRICT SELECTION (FOR MULTI-INSTITUTION USERS ONLY)...7 FIGURE 4: SYSTEM NAVIGATION...8 FIGURE 5: MANAGE MY ROSTERS FIGURE 6: ADD NEW ROSTER FIGURE 7: ACCESS ROSTERS TO EDIT OR DELETE...12 FIGURE 8: EDIT A ROSTER FIGURE 9. STUDENTS WITH NO GROUP...14 FIGURE 10. ENROLLED GRADE FILTER FOR HIGH SCHOOL...15 FIGURE 11. PERCENT OF STUDENTS WHO HAVE TAKEN EACH SUBJECT TEST SO FAR FIGURE 12. PERCENT OF STUDENTS WHO HAVE TAKEN 1, 2 AND 3 OPPORTUNITIES FOR EACH TEST...16 FIGURE 13. PERCENT OF STUDENTS WHO HAVE MET OR EXCEEDED THE STATE STANDARDS IN EACH SUBJECT FIGURE 14. DISTRICT ONLINE SCORE REPORT FIGURE 15. ACHIEVEMENT LEVEL DESCRIPTIONS...19 FIGURE 16: LEGEND SYMBOLS...20 FIGURE 17. PERFORMANCE BY RESOLVED HIGHEST SCALE SCORE GRAPH (INDIVIDUAL STUDENT REPORT VERSION).21 FIGURE 18. PERFORMANCE BY LEVEL OF AGGREGATION GRAPH...22 FIGURE 19: PERFORMANCE BY SCORE REPORTING CATEGORY GRAPHS FIGURE 20. TRACKING BY SUCCESSIVE CROSS-SECTIONS GRAPHS...24 FIGURE 21. TRACKING INDIVIDUAL STUDENTS GRAPH...25 FIGURE 22. TRACKING BY COHORT GRAPH

4 What is the Oregon Online Assessment Reporting System? The Oregon Online Assessment Reporting System provides interactive assessment reports for authorized state-level representatives, district superintendents, principals, and personnel (teachers and administrators). Figure 1: Oregon Online Assessment Reporting System. Specifically, the online system allows authorized users to: view district, institution, personnel, roster, and individual student reports in table and graph formats; *view longitudinal reports (new!) filter score reports by student subgroup (gender, ethnicity, LEP, Title I, etc.); manage online rosters (groups); access integrated information from ODE's SSID and OAKS Online systems, resulting in a centralized assessment reporting system

5 Which assessments are included in the system? The following OAKS tests are in the system: Math 3-8 and High School Reading 3-8 and High School Social Sciences 5, 8, and High School Science 5, 8, and High School To learn more about these tests, visit the Oregon Department of Education Web site at: For support, please contact your local ESD: Getting Started What are the system requirements? You must meet the following system requirements to access and use the Oregon Online Assessment Reporting System: A computer with a high-speed, broadband Internet connection (dial-up is not recommended); One of the following Internet browsers: Internet Explorer 6.0 and above, Mozilla Firefox 1.5 and above, or Safari 2.0 and above; JavaScript, which should be enabled in your browser. Ensure JavaScript is enabled by following the steps below: 1. Open your Internet browser. 2. From the menu bar, select Tools Internet Options. The Internet Options window will open. 3. Select the Security tab at the top of the Internet Options window. 4. From the list of zones at the top of the Security options, select the Internet icon. 5. Select the Custom Level button (near the bottom). 6. In the new window that pops up, scroll down to the item that reads Active Scripting. 7. If it is not enabled, select the Enable option. Note: The above instructions are for Internet Explorer browsers. The process is similar for Mozilla Firefox and Safari

6 How do I access the system? Logging In All district superintendents, district test coordinators, and institution principals should have received login information for the User Management System (UMS). Use your UMS username and password to access the Oregon Online Assessment Reporting System. If you need UMS login information, please check the Institution Lookup Application at If you are unable to log in, please contact your DSA, DTSA, or STC. He or she can verify your level of access. If you do not have reporting level access and you need it, he or she will grant it to you. You may also contact your local ESD Partner for help. (See for ESD contact information.) Only your DSA, DTSA, or STC can grant you access to the reporting system. The AIR Help Desk cannot grant reporting level access to users. If you have changed your login information for UMS, you must use the revised login when accessing the Oregon Online Assessment Reporting System. Also, if you want to change your login information for the Oregon Online Assessment Reporting System, you must change it first in UMS. Once you have your username and password information, follow these steps to access the system. 1. Open your Internet browser. 2. In the address field of your Internet browser, enter this URL: 3. Enter your assigned username and password provided to you via the UMS Web site. 4. Click [Log In]. You will be directed to the Select a Test Grade and Subject page. First-time Users: Acknowledgment Page Before first accessing the Oregon Online Assessment Reporting System, it is important for users to understand what the system is for and how it can be used for reporting and analysis purposes. Thus, when you log in for the first time, you will see a page that contains important information about the system. Please read this text carefully and click the check box acknowledging that you understand and accept the terms and conditions. You must review the information on this page. After you have done so, you must acknowledge that you have read it by clicking the checkbox at the bottom of the screen

7 Figure 2. Acknowledgment Page. Selecting a District Note: This page will appear only for state- and district-level users who have access to reports for multiple districts. You will see a list of districts whose reports you can access. Select the district whose reports you want to see. You will be directed to the Select a Test Grade and Subject page. Figure 3. District Selection (for multi-institution users only). Please note that after you select a district, you cannot change districts from the Home page. You will need to log out and log in again

8 Selecting a Test Grade and Subject These interactive reports are customized for administrators and personnel. 1. To begin, click the icon to expand the school year test listing. 2. Click the link for the Grade and Test reports you want to view. The tests are listed in order by grade level and subject and then by High School. We currently provide reports for the following: Reading Grades 3-8 and High School Math Grades 3-8 and High School Social Sciences Grades 5, 8, and High School Science Grades 5, 8, and High School How do I navigate the system? To understand the different features of the system, you should be familiar with the layout. Refer to the figure below. Figure 4: System Navigation. Tab Section and Breadcrumbs You can navigate through the system by clicking any of the following tabs: Home, Dashboard, Score Reports, Longitudinal Reports*, Manage My Rosters, and Help. Tab Sections: Home Click this tab to return to the home page and select a different test grade and subject. Dashboard Click this tab to view the following data: o The percent of students who have taken the subject tests so far (within the selected grade). Note: This is also in the Online Score Report mini-dashboard o The percent of students who took the subject test who have taken the test one, two, and three times. Note: This is also in the Online Score Report mini-dashboard o The percent of students who met the state standard in each subject for the current year, *as well as at the end of the testing window the previous year and at the same time the previous year. o The percent of students who met the state standard in each content standard category

9 Score Reports Click this tab to view district, institution, roster, and student scores (in table and graph formats). The Online Score Report also includes a mini-dashboard. *Longitudinal Reports Longitudinal reports can help you analyze test scores over time (for example, across school years). You can build three types of longitudinal analyses: o Tracking by Successive Cross-Sections o Tracking by Individual Students o Tracking by Cohort Manage My Rosters Click this tab to view, add, edit, and delete class rosters. Help If you have questions, click the Help tab to access the user guide and interactive help section. This section will guide you through the features of the online reporting system. You will also find contact information for the regional ESD and OAKS Help Desks. Login Information View your login information by looking at the upper right corner of each page. Verify that your name is correct. To log out at any time, click Logout. Breadcrumbs If you are a multi-institution user, you can always confirm which level of aggregation you are viewing by looking at the breadcrumbs below the tab section. Your current level will be in bold green font. Note: these breadcrumbs appear only in the Online Score Report; they will not appear in the longitudinal reports or dashboard sections

10 Manage My Rosters Why should I add rosters? Creating customized rosters (online groups) allows you to obtain detailed score reports for a specific group or cross-section of students. This feature allows for more advanced and expanded reporting options. You may assign students to multiple rosters via the Manage My Rosters section. You will also have the ability to view students not currently assigned to any roster(s), as these students will be grouped together under the name Students With No Group. How do I manage my rosters? You can manage your data in two ways: 1. Add a new roster to your institution. 2. Edit or delete a roster from your institution. Figure 5: Manage My Rosters. Add a new roster to your institution To do this, you must: 1. Select a school year from the Select a School Year drop-down list. (Note: The roster you create will only be viewable within the selected school year.) a. For multiple-district users, you will also need to select a district from the Select a District drop-down list. 2. Choose your institution from the Select an Institution drop-down list. 3. Select Add Roster from the drop-down list under Select task. The page will automatically load with the form to add a new roster. 4. Add the new roster information, including roster ID, roster name, personnel name, subject(s), grade level, and students. (Note: You may not change your roster ID once you have saved it.) 5. Click Save

11 Figure 6: Add New Roster. Edit a roster from your institution To do this, you must: 1. Select a school year from the Select a School Year drop-down list. a. For multiple-district users, you will also need to select a district from the Select a District drop-down list. 2. Choose your institution from the Select an Institution drop-down list. 3. Select View/Edit/Delete a Roster from the Select task drop-down list. A list of the roster(s) in your institution will appear. 4. Click Edit next to the roster name. 5. Edit the information that needs to be changed, including roster name, teacher, subject(s), grade level, and students. (Note: You cannot change the roster ID.) Refer to figures 7 and 8 below. 6. Click Save

12 Delete a roster from your institution To do this, you must: 1. Select a school year from the Select a School Year drop-down list. a. For multiple-district users, you will also need to select a district from the Select a District drop-down list. 2. Choose your institution from the Select an Institution drop-down list. 3. Select View/Edit/Delete a Roster from the Select task drop-down list. A list of the roster(s) in your institution will appear. 4. Click Delete next to the personnel s name. (Refer to Figure 7 below.) 5. A pop-up window will appear, asking for confirmation that you want to delete the roster. Click OK to delete. After you have saved your roster information, click the Score Reports tab to access the updated reports. Figure 7: Access Rosters to Edit or Delete

13 Figure 8: Edit a Roster

14 Students With No Group This group contains students that currently do not belong to any roster. It will also contain students who were added to a roster but do not have personnel assigned to that roster. This group is separate from the group(s) that appear for specific personnel in the system. To move these students to a roster with assigned personnel, follow the instructions above for managing rosters. Figure 9. Students with no Group

15 Dashboard The Dashboard page provides overall data within each Grade or High School regarding how many students have taken the subject tests so far. Three graph reports are presented: The percent (%) of students who have taken the subject tests (Reading, Mathematics, Science, and Social Sciences). The percent (%) of students who have taken the subject tests one, two, or three times. The percent (%) of students who took the test(s) and who met or exceeded the state standards for each subject and score reporting category within each subject. o NEW! You can now view comparative data between the current school year and the previous school year you will see as of today s date, as of this time last year, and the final score last year. This data will enable you to track state, district, and school progress and compare it to the previous year s performance. Note: For High School reports, an Enrolled Grade Filter has been implemented. This allows you to break down performance data by grade level. Figure 10. Enrolled grade filter for high school. Figure 11. Percent of students who have taken each subject test so far

16 Figure 12. Percent of students who have taken 1, 2, and 3 opportunities for each test. Figure 13. Percent of students who have met or exceeded the state standards in each subject

17 Score Reports What is the first report I see (Online Score Report)? The first report you see is the Online Score Report. You can use this report to summarize the results of your institution(s) in the selected test grade and subject. You will view the institution(s) for which you have access. Click the link of the institution to access corresponding personnel, roster and student reports. Each level of the Online Score Report will show the percentage breakdown of students achievement levels and whether they meet state standards. Roster reports display only students who have completed tests. Students in your roster who have not yet tested will not appear in reports until after they have completed a test. All students remain in their rosters, even if you do not see their names in the online score report. District Online Score Report The district report view shows a summary of results for institutions in the selected test grade and subject. Click any name to access the next lower level (district institution, institution personnel, personnel roster, roster student). Figure 14. District Online Score Report

18 What can I do with my report? With this report, you have several options for managing and accessing your data. You can sort data within columns, print roster and student reports, export the data into an Excel spreadsheet, and show and hide specific data columns. Sort data within columns View overall scores and individual score reporting category levels from lowest to highest and highest to lowest. Click on the top header of each column to sort from lowest to highest. Click again to sort from highest to lowest. Print roster and student reports: Click. The print window will pop up. Click OK to print the report. o Note: In order to print an entire report (more than 20 names), you will need to disable pagination. To disable pagination, click the link that says Display all pages at once above the table. The link should then read Enable separate pages. Export data into Excel Click. A popup window will appear. Choose either Open with Excel or Save to Disk and click OK. If you save to disk, you will need to locate the file and open in Excel. o In order to export an entire report (more than 20 names), you will need to disable pagination. To disable pagination, click the link that says Display all pages at once above the table. The link should then read Enable separate pages. o In the Roster-level report, you have two (2) options for exporting data. 1. You may export the report with the same symbols that appear online (default option). 2. You may export using codes. The codes are textbased, which will allow you to sort data by individual columns in the Excel file. o Note: FERPA prohibits the release of any personally identifiable information. A warning will be placed on both the originating report page and the spreadsheets. Show/hide data columns Click. You will be directed to a page with column options. Click checkboxes to add columns to and remove columns from your online score report. This function allows you to view specific data for performance levels for each score reporting category

19 How do I navigate within the Online Score Report? At any time, you may select another test by changing the subject, grade level, data view (table or graph), or view data using Student Subcategory filters. Changing the Subject To change the subject at any time, click the bubble to the left of the subject name. The page will automatically refresh with the new subject report. Changing the Grade Level To change the grade level at any time, click the tab for the grade or High School report you need to view. You will be automatically redirected to a new page with the new grade or High School report. Viewing Data by Student Subgroup To view reports that include an AYP category, select the desired category from the Filter by Student Subgroup drop-down list. Current filters include Enrolled Grade, Ethnicity, Gender, Limited English Proficiency, Special Education, and Title I status, as well as various combination filters (e.g., Ethnicity by LEP). After you select a category, the report will automatically refresh to include the chosen category. What are the achievement standard levels and cut scores? Below each report is a description of the achievement standard levels and the score ranges for each one. Click on individual student names to view graphs and learn more information about his or her score and why he or she was placed within an achievement level range. Figure 15. Achievement Level Descriptions

20 What do the legend symbols mean? At the top of the roster report is a legend that shows symbols that roughly correspond to the performance levels and cut scores. The symbols in the top row correspond to the students overall achievement standard levels in each subject. The symbols in the bottom row correspond to the students achievement level within each score reporting category. Figure 16: Legend Symbols. What graph reports are available? One of the features of this Web site is graph reports that correspond with the online score reports. The following graphs are available: Performance by Scale Score Performance by Level of Aggregation Performance by Score Reporting Categories Performance by Resolved Highest Scale Score Graph This default graph appears for individual students and acts as the Individual Student Report (ISR). You may also view scale score graphs at the district, institution, and personnel levels. Each report includes the following: o Highest scale score to date. o *Scale score results by score reporting category (content standard), as well as Standard Error of Measurement calculations. o State, District, and School average scale score data (for comparison purposes). o The number of attempt(s) the student made in the current test and the scale score for each test. To view the graph for another institution, personnel, or student within the current district, select the name from the drop-down list next to Select. To go down to the next level (e.g., institution personnel), click the link in the yellow box. o Note: This graph is not available at the roster level. Printing the Scale Score Graph o State, District, Institution, and Personnel levels Click the Print icon print dialogue box will pop up. Click OK to print. o Student level You have two options for printing the student graphs. 1. You may print only the selected student s graph. 2. You may print the student graphs for all students in the rostered group.. The

21 To print, click. The icon will expand to show the print options. Select your desired option and then click Print. Figure 17. Performance by Resolved Highest Scale Score Graph (Individual Student Report version). Note: New Graph Feature Score Reporting Categories data now also includes Scale Score and Standard Error of Measurement for each reported category

22 Performance by Level of Aggregation Graph This graph report shows the overall performance by level of aggregation (subject) for the selected institution or personnel. To view the graph for another institution, personnel, or roster within the current district, select the name from the drop-down list next to Select. To go down to the next level, click the link in the yellow box. o Note: This graph is not available at the roster or student levels. Figure 18. Performance by Level of Aggregation Graph

23 Performance by Score Reporting Category Graphs This graph report shows the overall performance for each scoring category within a subject. Individual graphs for each category are shown for the selected institution or personnel. To view the graphs for other institutions, personnel or rosters within the current district, select the name from the drop-down list next to Select. To go down to the next level, click the link in the yellow box. o Note: This graph is not available at the roster or student levels. Figure 19: Performance by Score Reporting Category Graphs. How often are the reports updated? Reports are updated almost immediately after a student has completed a test. To see the most recent test results, add the student(s) who have just completed a test to a roster so you can see their results

24 *Longitudinal Reports (Graphs) Longitudinal reports can help you analyze test scores over time (for example, across school years). You can build three types of longitudinal analyses: Tracking by Successive Cross-Sections Tracking by Individual Students Tracking by Cohort Tracking by Successive Cross-Sections Build this report to answer this question: Are students scoring higher this year than last year? Select a test grade level to compare groups of students across time. Direct comparisons are made between two groups of students that have a similar attribute for the selected test subject and grade level. For example, how did 3rd graders in 2007 do in Math compared to the 3rd graders in 2008? The report shows this analysis in two ways: by the resolved highest scale score and by the percent of students who scored at or above the proficient level. Note: If you filter by a sub-group (e.g., Ethnicity American Indian) and the entity you are plotting the graph for does not have data for that sub-group, then we will not plot the comparative state data. Figure 20. Tracking by Successive Cross-Sections Graphs

25 Tracking by Individual Students Build this report to answer this question: How much is an individual student s test score increasing from year to year? This report allows you to track individual students from one test grade to the next. For example, a student in 2007 who tested Grade 3 Math may be testing Grade 4 Math in For past school years, this report shows the resolved highest scale score for that student. For the current school year, this report shows all available test opportunities for that student. For example, if Jane Doe took the Grade 3 Math test three times in 2007, her resolved highest scale score is displayed and can be compared to all her test opportunities so far for Grade 4 Math. Figure 21. Tracking Individual Students Graph Note: You may select up to 5 students at once to track

26 Tracking by Cohort Build this report from your existing roster(s) to answer this question: How much is my cohort of students average score increasing from year to year? By selecting a current roster, you can track your group s (of the same students) performance over time. For example, you might have 20 students in Roster A who tested in Grade 3 Math last year, and you would like to analyze the performance of the same 20 students in Roster A this year who are taking the Grade 4 Math test. The report shows this analysis in two ways: by the resolved highest scale score and by percent proficient. Figure 22. Tracking by Cohort Graph