Offsite BOE Report: Rhode Island College June 16, 2011 1:00 pm - 5:00 pm Offsite BOE Team Members Dr. Annie M. Bauer, Chair Mrs. Susan N. Graham, Member Dr. Larry D. Powers, Member Offsite BOE Team Observers Dr. Donna M. Gollnick, NCATE Staff
The following report indicates areas of concern on which the Onsite BOE Team will focus during the upcoming visit. In addition, the last section for each standard is a list of evidence that the team plans to validate during the visit to ensure that the standards continue to be met. This validation will occur as the team interviews faculty, administrators, school-based partners, and other members of the professional community. Validation could also occur in the visits to schools and observations on campus. The validation list also includes some specific documentation that the team would like to review during the onsite visit. In some cases, the Offsite team members could not locate a document or open a link and have requested that the Onsite Team review those documents. Standard 1: Candidate Knowledge, Skills, and Professional Dispositions Candidates preparing to work in schools as teachers or other school professionals know and demonstrate the content knowledge, pedagogical content knowledge and skills, pedagogical and professional knowledge and skills, and professional dispositions necessary to help all students learn. Assessments indicate that candidates meet professional, state, and institutional standards. 1.1 Statement about the evidence In that the specialized program association program reports are currently in review, data related to content knowledge, pedagogical knowledge, and impact on students are limited. In addition, due to the significant revisions to the assessment system (in view of AFIs for the last visit), three years of data were not available for all assessments for all programs. Title II reports indicate a 98-100% pass rate for all programs. The Teacher Work Sample provides evidence that candidates can design, implement, and evaluate their instruction, as well as use assessment to demonstrate an impact on student learning. The unit has devoted extensive time and study to the measurement of dispositions, and, when a ceiling effect was demonstrated, developed a finetuned instrument. In addition, culturally competent teaching is being assessed as an aspect of dispositions. 1.2 Progress toward meeting the target level on this standard. The program has identified this as a growth standard. However, the program has dedicated a great deal of time and study to the development and assessment of dispositions. 1.3 Feedback on correcting previous areas for improvement (AFIs) AFIs corrected from last visit: AFI Number & Text Apply to AFI Rationale The history/social studies programs at the undergraduate and graduate levels have not been nationally recognized. Candidates in art education, health education, secondary history, music education, physical education, secondary physics, and technology education programs had pass rates ITP ITP The program was recognized after the last NCATE visit, and the program has again submitted a report for national review. Pass rates for these programs range from 98-100% 2
below 80 percent on the Principles of Learning and Teaching examination. 1.4 Areas of concern related to continuing to meet the standard (1) Assessment data and other evidence were not available for the modern languages, secondary education (ITP), Music (not that much of an issue because of NASM accreditation), and Exceptional Learning Needs (M.Ed.) Rationale: The links to evidence for these programs were not available. 1.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Evidence of assessments and assessment data for modern languages, secondary education (ITP), and Exceptional Learning Needs (M.Ed.) What assessments are used to determine that candidates meet standards? What assessment data exist for these programs? (2) Reviews of programs by the SPAs. What are the findings of the SPA reviews that were submitted earlier this year? (3) Update on state review process. What were the state findings after the rejoinders were submitted? (4) Summaries of assessment data for each rubric, disaggregated by undergraduate and graduate for each program. What data exist? What has the unit learned from these data? Standard 2: Assessment System and Unit Evaluation The unit has an assessment system that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the performance of candidates, the unit, and its programs. 2.1 Statement about the evidence The unit has made significant progress towards developing and establishing an assessment system that collects, analyzes, evaluates, and stores data that can be used for continuous improvement of the unit and its operations. The units assessment system is aligned to the unit s conceptual framework/institutional standards and state and national standards. The unit has meticulously studied and reviewed national studies and trends to ensure that its assessment system is consistent with best practices and accepted theoretical foundations. The unit has documented these studies and practices. Early on in the unit s assessment of instruments, it was determined that all candidates scored high on selected assessments and that these instruments did not reveal nor provide useful data for program improvement. This provided the unit with the impetus to revise and develop reliable and valid instruments that were helpful for program improvement. Since the last visit (2008) the unit has fully revised its assessment system at the initial level and has made many revisions to its assessment system at the advanced level. The unit s assessment system has four transitions points at the initial level that candidates matriculate through. They are: Program Admission, Acceptance to Student Teaching, Program Exit, and Post Graduation. The unit assessment system has four transition points for its advanced program, they 3
are: Program Admission, Formative Point, Summative Transition Point, and Continuous Follow Up. Each transition point has a number of key assessments that candidates must satisfy before they can move to the next transition point or position in the program. The unit assessments system collects, evaluates, analyzes, and stores data on key assessments indicated at each transition point. The revisioning process to revise the assessment system was guided by a written plan that engaged faculty and unit stakeholders in a process that reviewed, studied and analyzed the current assessment system and engaged them in careful deliberations regarding the development and implementation of changes to provide significant information to inform continuous improvement in the unit. An underlying goal of the units work to revise it assessment system is to develop an assessment culture among faculty and candidates. The unit has allocated resources to enhance the capability of its assessment system. In 2004 the unit hired a Data Management Coordinator and Director of Assessment. In 2008 the unit provided resources to hire a part-time Assistant Director of Assessment and currently the unit provides resources to hire personnel to provide training for users of Chalk and Wire and provides release time for faculty members to train on Chalk and Wire. The IR and other documents present a detailed chronology of the revisioning process the unit engaged in to revise the assessment system and the specific changes that were made to it. The revisioning process was guided by a written plan and conducted by the Director of Assessment and the unit s Program Improvement Committee. Assessment System changes include the elimination of some assessments and the development of others. All changes were made upon careful study and review. For example, the unit s exit portfolio was replaced with a Teacher Candidate Work Sample (TCWS); the unit developed a Teacher Candidate Observation and Progress Report (OPR). The OPR was designed to collect data on candidates growth in teaching skills and professionalism. The unit revised its dispositions and the process and system of assessing them while aligning them to the unit s conceptual framework and state and national standards (INTASC). A full description of specific changes in the initial and advanced program is included in the IR. The Director of Assessment and the Program Improvement Committee meticulously followed a planned process for all revisions that engaged faculty, pilot tested new assessments, collected and assessed piloted data and instituted revisions based upon those assessments. The unit has tested and developed a number of different technologies/software systems while working to establish and implement its assessment system. For example, the unit utilized True Outcomes, Surveygizmo, SurveyMonkey, CheckBox and Chalk and Wire. The unit has an established system to document and resolve candidate complaints. According to the IR, the unit follows the university guidelines when resolving candidates complaints. The file to document candidate complaints is kept in the dean office. The unit is using data to make changes in its programs, courses, policies and procedures and has a plan to continue to use its assessment system to generate data for continuous improvement. Per the documentation provided, the unit has established a culture of assessment and has plans to continuously use their data to inform its candidates performance and enhance unit operations. 4
2.2 Progress toward meeting the target level on this standard The unit has made progress toward moving to the target level regarding the development and implementation of its assessment system. The unit has developed a detailed system to assess its assessments and engage faculty in continuous improvement of unit operations and its ability to produce successful professional candidates. The unit has extensive documentation of its efforts to secure buy-in by unit faculty and to create a culture of assessment. In the unit s IR, the unit indicates that its work to establish an assessment system, which involves the unit and unit faculty in the collection and analysis of data to manage and improve candidate performance and unit operations, has created a more cohesive unit focused on improving its operations through assessment. The units assessment system generates unit data and disaggregated program data as well as specialized data sets for individual faculty. The specialized data sets allow faculty to drill down into specific areas that enable them to improve their effectiveness in their work. Faculty members are regularly provided with raw data so they can conduct their own investigations. The unit has an elaborate system where it closely studies its assessments for validity and reliability, and consistently evaluates its candidate assessments to determine their ability to predict candidate success and the extent to which the assessments actually assist candidate in continuously improving. For example, the unit systematically studies its assessments to establish content, construct, and predictive validity. The unit studies its assessments to determine the extent that they are fair and inform practice. The unit has a systematic approach to studying the effects of its assessments on candidates and to determine if assessments are effective in terms of program improvement. The unit also studies the extent that its assessments may have adverse consequences on its candidates. Efforts are made to keep faculty and stakeholders informed about assessments and any changes that may impact them or candidates. Faculty and cooperating teachers are informed at regularly scheduled retreats and meetings. The unit works to inform candidates about program requirements and assessments through webpages, regularly scheduled candidate orientation meetings at each transition point, and regularly updated handbooks and unit documents. 2.3 Feedback on correcting previous areas for improvement (AFIs) AFIs continued from last visit: AFI Number & Text Apply to AFI Rationale Data collection is inconsistent across programs. ITP 2.4 Areas of concern related to continuing to meet the standard Data on key assessments were not available to the team during the offsite meeting. (1) Evidence was not available to show that members of the professional community are involved in the development and implementation of the unit s assessment system. 5
Rationale: The unit s IR describes engagement of the professional community and other stakeholders in the development and implementation of the assessment system; however, the reviewer did not see engagement of persons for public schools in this process. The unit describes extensive engagement and involvement of unit faculty in the development and revision of assessment system, instrument pilot testing, data analysis and faculty input and implementation. 2.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Data collection on key assessments for initial and advanced programs. Where are these data housed? How are candidates scoring on these assessments? (2) Evidence of advanced program assessments. What assessments are used? What data are available? What is the unit learning from these assessments? (3) The unit s use of data from the assessment system to inform unit operations. What unit operations have been assessed? How does the unit use the data on unit operations? What changes have occurred as a result of unit operations assessments? (4) The operation of the assessment system. How does the system work? What are the timelines for the data collection? (5) Involvement of the broader professional community. How have school partners been involved in the development and implementation of the assessment system? Standard 3: Field Experiences and Clinical Practice The unit and its school partners design, implement, and evaluate field experiences and clinical practice so that teacher candidates and other school professionals develop and demonstrate the knowledge, skills, and professional dispositions necessary to help all students learn. 3.1 Statement about the evidence Initial Teacher Preparation There is evidence that the unit provides resources and training for school site personnel with its workshop presentation that is attended by approximately 75% of cooperating teachers. Judging from the PowerPoint, the workshop models interactive and differentiated instruction. Workshops are also offered for differentiation and math instruction; a graduate course on reflective practice is also available. The unit collects an exit survey from cooperating teachers, which is a requirement for compensation, but it is unclear how the information from that survey is used to improve the program nor is there any clear indication that the unit draws on the expertise or experience of practicing teachers to enhance their own instruction. The unit takes pride in its community service requirement and while this is admirable and may address dispositional goals, there is no assurance community service connects with the knowledge and skills of teaching. The on-line Student Teaching Handbook defines the steps to the student teaching experience and the roles of all parties. It appears that practicum field experience and internship placements are handled by two different offices (RICplacements.org and Office of Partnerships and Placement), but it is unclear how these records are maintained to ensure that all candidates have diverse 6
experiences. Evaluative tools are rubric-based and most have been revised in the last year to include indicators to reflect both candidate knowledge and skills and learner outcomes. The IR states that all candidates have field experiences in urban, outer urban ring, suburban and rural settings as well as with P-12 students from diverse socioeconomic and ethnic backgrounds. Course work reflects preparation to meet the learning needs of diverse populations. Both aggregate and school specific population data are available. It is unclear how the unit confirms that all candidates have diverse experiences. Work Sample rubrics reflect the candidate s ability to adapt instruction to meet exceptionalities and learning styles and address both pre, formative, and summative assessment of student learning. Advanced Programs It is unclear how the unit interacts with the school sites were advanced program candidates are involved in field experiences. Nor is it clear how candidates bring their advanced learning back to the unit to share how coursework supports practice and how practice might enhance program design. The unit does not appear to have a structure in place to interact with school and community partners to enrich the advanced level field experiences. Information in the Assessment System Plans and Descriptions shows progress over the more general statements of the document appended to the Conceptual Framework in the IR, but a consistent structure to align advanced degrees with the Conceptual Framework does not appear to exist. The expectations for field experience in advanced programs vary dramatically from over 1,500 hours to no stated hours because the candidates for these advanced degrees are working professionals. While it is common for advanced candidates to complete their field experience at their home school, there doesn t seem to be an intentional design for demonstration of how these candidates knowledge and skills are applicable in diverse settings. Assessments for graduate program employ a rubric scale, but lack indicators by which the ratings can be judged. In some programs these are self assessments and in others they are faculty assessments. The lack of consistency across programs and lack of external validation is a concern. Advanced programs are less consistent in their expectations and measurement of candidate impact in designing an environment, providing instruction, and analyzing the impact on student learning. 3.2 Progress toward meeting the target level on this standard The unit has undertaken a great deal of work to design and implement candidate assessment instruments that provide clear performance indicators for its initial teacher preparation programs. Chalk and Wire should provide assurance that all competencies have been addressed and met during coursework. Evaluative tools are rubric-based and most have been revised in the last year to include indicators. 3.3 Feedback on correcting previous areas for improvement (AFIs) AFIs corrected from last visit: 7
AFI Number & Text Apply to AFI Rationale Assessments for clinical practice do not reflect candidate proficiencies identified in the unit s conceptual framework. ADV 3.4 Areas of concern related to continuing to meet the standard The revision process for the assessment system ensured that candidate proficiencies were linked to the unit s conceptual framework. (1) Not all advanced candidates have field experiences with students from diverse groups. Rationale: No evidence was available to indicate how the unit ensures that advanced candidates have field experience with students from diverse groups, especially when their own classrooms my not reflect diversity. 3.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Collaboration with school-based personnel at partner schools. How does the unit collaboratively seek out and utilize the expertise of school site partners? How are P-12 partners involved in the unit s work? In what activities/projects do the school partners collaborate with the unit? (2) Field experience placement records. How does the unit track placement of candidates in multiple settings? How do the two offices managing field placement communicate to ensure that candidates have worked with students from diverse groups? (3) Disaggregation and analysis of survey data. What has the unit learned from the survey completed by school partners about the training sessions conducted? (4) First two semesters of Chalk and Wire implementation. What data related to field experiences and clinical practice are available in Chalk and Wire? (5) Use of assessment data to evaluate methods courses. What formal and consistent process exists for compiling, analyzing, and using assessment information to inform program decisions about methods courses? (6) Tracking placement of candidates in field experiences with students from diverse groups. How does the unit ensure all candidates have field experiences in diverse settings and with students from diverse groups? (7) Consistency of field experiences and clinical practice in advanced programs? What are the requirements for advanced teaching programs and programs for other school professionals? Are all placements in the schools in which candidates are working? What is the rationale for the disparity in field experience expectations in advanced programs? (8) Assessments for advanced programs. What are the required assessments for advanced candidates? How do programs ensure that assessments rely on more than self and advisor assessment. What protocols assure the validity and consistency of these assessments? (9) Opportunities for advanced candidates to demonstrate their ability to impact learning for all students in diverse settings if field experiences take place at their own school site. How does the unit ensure that advanced candidates work with students from diverse groups? 8
Standard 4: Diversity The unit designs, implements, and evaluates curriculum and provides experiences for candidates to acquire and demonstrate the knowledge, skills, and professional dispositions necessary to help all students learn. Assessments indicate that candidates can demonstrate and apply proficiencies related to diversity. Experiences provided for candidates include working with diverse populations, including higher education and P 12 school faculty, candidates, and students in P 12 schools. 4.1 Statement about the evidence The unit has identified proficiencies related to diversity aligned with the conceptual framework. With an emphasis on culturally relevant teaching, the unit has developed expectations and assessments, coursework, and field experiences that provide groundwork for understanding a wide range of diversity. Candidates are provided formative assessment results related to their performance as a culturally relevant practitioner. The unit has a clear recruitment plan for recruiting diverse faculty, and retention efforts include professional development, supports, and incentives. Twenty percent of the candidates do not identify themselves as white, not Hispanic, which implies that candidates engage in the professional education experiences with individuals from different socioeconomic and racial, ethnic, and language groups. Data are provided to demonstrate that field and clinical experiences include students from different socioeconomic groups, ethnic, racial, or language groups, and that candidates are assessed on their practice with these students. 4.2 Progress toward meeting the target level on this standard. The unit has identified this standard as a growth rather than target standard. 4.3 Feedback on correcting previous areas for improvement (AFIs). No areas for improvement were cited at the previous visit. 4.4 Areas of concern related to continuing to meet the standard. None 4.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Culturally relevant assessments. What assessments provide data on candidates developing proficiencies related to working with students from diverse groups? What are candidates perceptions of their development of diversity proficiencies? (2) Candidates experiences working with students from diverse groups. How does the unit track these placements? What is the diversity of the schools in which candidates have completed their field experiences and clinical practice? How does the unit know that candidates doing their student teaching in suburban schools have had urban experiences? (3) Aggregated data from teacher work samples. What data show that candidates are developing diversity proficiencies? (4) Strategies for candidate and faculty recruitment. What is the role of the diversity committee? 9
Standard 5: Faculty Qualifications, Performance, and Development Faculty are qualified and model best professional practices in scholarship, service, and teaching, including the assessment of their own effectiveness as related to candidate performance; they also collaborate with colleagues in the disciplines and schools. The unit systematically evaluates faculty performance and facilitates professional development. 5.1 Statement about the evidence. Evidence is provided through AIMS and linked to the exhibit list demonstrating that faculty members have doctorates or extensive experience. School faculty members are licensed in their field, and often hold additional certificates and endorsements. The majority of school-based faculty members hold the master s degree. Cooperating teachers are considered clinical faculty in the report; additional evidence is needed on the clinical faculty who are university faculty members supervising in the field. The unit provided evidence of strong evaluation of teaching, which includes peer observations. Specific examples of best practice are sometime indicated in the syllabi, and some information is provided in the HOLC report. Scholarly effort seems to be fairly even across faculty members. Service is described clearly at the college/university, school, and community level. Much of this service is unfunded. A clear, consistent faculty evaluation system is described, but examples of the use of these evaluations to improve a specific individual s performance are not apparent. A large number of opportunities for professional development are included, ranging from formal symposia to brown-bag discussions. 5.2 Progress toward meeting the target level on this standard. The unit has a clear strategic plan to improve faculty members professional knowledge and skills. This plan is aligned with activities that support faculty improvement. Individual faculty members are also engaging in initiatives to support faculty improvement. Technology is being used to improve the collection of data for annual reports. The strategic plan clearly indicates that progress is being made by the number of items that have been marked completed. 5.3 Feedback on correcting previous areas for improvement (AFIs). No areas for improvement were cited at the previous visit. 5.4 Areas of concern related to continuing to meet the standard. None 5.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Clarification of clinical faculty member and university supervisors. It appears that professional education faculty members do this supervision, but evidence needs to be clarified. (2) Clarification of best practices in teaching that are modeled by faculty members. How pervasive is the use of best practices? (3) Examples of the use of faculty evaluations to improve teaching, scholarship, and service. 10
Standard 6: Unit Governance and Resources The unit has the leadership, authority, budget, personnel, facilities, and resources, including information technology resources, for the preparation of candidates to meet professional, state, and institutional standards. 6.1 Statement about the evidence. The unit appears to have an organizational structure and leadership that supports its efforts. The unit has structures in place to encourage and support collaboration between faculty members in other units. The unit s budget is greater than another professional school and appears to be sufficient. Faculty members work loads are within those identified by the bargaining agreement, but clear information related to the amount of load assigned to student teachers is not provided. Facilities described appear adequate, and resources and technology appear to meet the program s needs. 6.2 Progress toward meeting the target level on this standard. In terms of budgeting, the college is piloting revenue sharing from community based programs, which may be a source of income for the unit. In addition, infrastructure issues are being systematically addressed. 6.3 Feedback on correcting previous areas for improvement (AFIs). No areas for improvement were cited at the previous visit. 6.4 Areas of concern related to continuing to meet the standard. None 6.5 Evidence for the Onsite BOE Team to validate during the onsite visit (1) Accuracy of academic calendars, catalogues, publications, grading policies, and advertising. (2) Sufficiency of budget to support candidates meeting standards. How adequate is the budget to support the needs of faculty and candidates? (3) The number of student teachers that are supervised by each clinical faculty member, and the subsequent load. (Supervision should not generally exceed 18 per semester per faculty member) (4) Adequacy of facilities and technology. Sources of Evidence Rhode Island College s Institutional Report Annual Reports and Program Reports in NCATE s Accreditation Information Management System (AIMS) Website and Exhibits of Rhode Island College 11