ABET Program Evaluator Training (Session 1 of 2)

Transcription

1 ABET Program Evaluator Training (Session 1 of 2) A-1 December 2005 Stephen J. Ressler, P.E., Ph.D. Who I Am Professor & Vice Dean for Education, U.S. Military Academy at West Point, NY Former CE Program Director at USMA Chairman, USMA ABET Committee ABET Evaluator ASCE Committees: Educational Activities Curriculum & Accreditation Academic Prerequisites for Professional Practice Faculty Development Instructor, ASCE ExCEEd Teaching Workshop

2 Administrative Notes Purpose & intended audience Attendance sign-in rosters ABET slides vs. my slides Pace, your questions, and my questions Breaks Homework Learning Objectives By the end of our two sessions, you will be able to: A-4 December 2005 Evaluate a program using the appropriate criteria Recommend an accreditation action Complete all required reports and forms

3 Plan for the Training Session 1: Program Evaluator Responsibilities Assessment Fundamentals Criteria - Basis for Accreditation Session 2: Decision Making The Accreditation Process Forms and Reports Program Evaluator Responsibilities A-6 December 2005 Program evaluators are ABET s most important asset. They are the face of ABET to the programs that we accredit. George D. Peterson Executive Director

4 You Provide As an ABET team member you provide knowledge concerning: Professional practice A-7 December 2005 Professional preparation Continuous quality improvement Your Commitment Your commitment is critical to a successful visit! A-8 December 2005 Exercise sound judgement. Base all decisions on the criteria. Do careful pre-visit preparation. Perform on-site visit thoroughly. Develop and communicate succinct conclusions. Participate in team decisions. Promptly submit all reports and forms. When in doubt or when a question arises, use you team chair as a resource.

5 Sequence of Events (1 of 4) As a PEV, you will: Complete your training by participating as an Observer on an evaluation visit. Work under the mentorship of an experienced PEV. Complete all forms and reports. Receive an assessment from the ASCE Committee on Curricula & Accreditation. Be assigned as a PEV the following year. Sequence of Events (2 of 4) As a PEV, you will: Receive notification of assignment to an evaluation team. Receive the program Self-Study, student transcripts, and other materials. Perform a complete evaluation of the program, based on the Criteria. Initiate a dialog with the evaluated Department Chair. Request additional materials, as required. Resolve as many issues as possible before the visit.

6 Sequence of Events (3 of 4) As a PEV, you will: Participate in the Evaluation Visit: Review student work and other documentation. Interview students and faculty. Evaluate facilities. Evaluate institutional areas, as directed by the Team Chair. Coordinate your findings with the entire team. Complete reports and draft Exit Statement Sequence of Events (4 of 4) As a PEV, you will: Participate in post-visit coordination: Submit final reports to Team Chair. Submit copies of all reports to ASCE for assessment of your performance. Be responsive to your Team Chair on all Due Process issues. Report travel expenses. More details in Session 2

7 What the Visit Team Produces A-13 December 2005 At the end of the visit Team submits a Visit Report Basis for the Draft Statement to Institution Criteria-based All visit forms feed into this document Includes statements of program strengths, concerns, weaknesses and deficiencies for each program based on documentation and observations from pre-visit and on-site Sample Draft Statement included in your course materials ABET CODE OF CONDUCT A-14 December 2005 Approved by the ABET Board of Directors in October 1999 Governs all volunteers and staff involved in ABET activities PEVs agree to abide by when completing on-line bio form Code of Conduct included in your course materials

8 Conflict of Interest A-15 December 2005 Expectations for ABET representatives: Behave in an ethical and professional manner Disclose real or perceived conflicts of interest Recuse yourself from discussions or decisions related to real or perceived conflicts of interest Confidentiality A-16 December 2005 Information supplied by institution and derived from the visit is for confidential use of ABET and its commissions ABET has specifically authorized Professional Societies to participate in the accreditation process General information about ABET and the Commissions is available on the ABET Web Site at

9 Conflict of Interest Exercise A-17 December 2005 Do this for homework, prior to Session #2 Assessment Fundamentals A-18 December 2005

10 Continuous Quality Improvement A-19 December 2005 A SYSTEMATIC PURSUIT OF EXCELLENCE AND SATISFACTION OF THE NEEDS OF CONSTITUENCIES IN A DYNAMIC AND COMPETITIVE ENVIRONMENT. Foundation of CQI is Assessment A-20 December 2005 Assessment of inputs & process only establishes the capability or capacity of a program Assessment of outcomes determines what is done with that capability Outcomes assessment improves: Institutional effectiveness Learning Accountability

11 Terms (Typically used by ABET) Objectives Outcomes Performance Criteria Assessment Evaluation Definition Statements that describe the expected accomplishments of graduates during the first few years after graduation. Statements that describe what students are expected to know and able to do by the time of graduation. Specific, measurable statements identifying the performance(s) required to meet the outcome; confirmable through evidence. Processes that identify, collect, use and prepare data that can be used to evaluate achievement. Process of reviewing the results of data collection and analysis and making a determination of the value of findings and action to be taken. Some other terms for same concept (could be used by programs) Goals, outcomes, etc. Objectives, standards, etc. Standards, rubrics, specifications, metrics, outcomes, etc. Evaluation Assessment Taxonomy of Approaches to Assessment Individual Level of Assessment (Who?) Group Competency-Based Instruction Assessment-Based Curriculum Individual Perf. Tests Placement Advanced Placement Tests Vocational Preference Tests Other Diagnostic Tests Program Enhancement Individual assessment results may be aggregated to serve program evaluation needs Learning/Teaching (Formative) Gatekeeping Admissions Tests Rising Junior Exams Comprehensive Exams Certification Exams Campus and Program Evaluation Program Reviews Retention Studies Alumni Studies Value-added Studies Accountability (Summative) Purpose of Assessment (Why?) K N O W L E D G E S K I L L S A T T I T U D E S & V A L U E S (Terenzini, JHE Nov/Dec 1989) B E H A V I O R Object of Assessment (What?)

12 Educational Objectives Institutional Mission Assess Evaluate Program Outcomes Measurable Performance Criteria Constituents Feedback for Continuous Improvement Program Learning Practices/Strategies Evaluation: Interpretation of Evidence & Data Assessment: Collection & Analysis of Evidence & Data Assessment for Continuous Improvement Environmental Factors Program Assessment Institutional Context Student Precollege Traits Coursework & Curricular Patterns Classes chosen; major Classroom Experience Pedagogy; Facilities; Climate; Faculty & Student Characteristics Out-of-class Experiences Co-curricular; coops; internships; support services Educational Outcomes Adapted from Terenzini, et.al. 1994,1995

13 Role of ABET Accreditation A-25 December 2005 The role of ABET accreditation is to provide periodic external assessment and evaluation in support of the continuous quality improvement program of the institution. Questions? PEV role and responsibilities: ABET Conflict of Interest and Confidentiality Policy Conflict of interest issues Fundamentals of assessment

14 Accreditation Basics A-27 December 2005 Objectives of Accreditation A-28 December 2005 (1) Assure that graduates of an accredited program are adequately prepared to enter and continue the practice of engineering (2) Stimulate the improvement of engineering education (3) Encourage new and innovative approaches to engineering education and its assessment (4) Identify accredited programs to the public

15 ABET Accredits Programs A-29 December 2005 All programs lead to degrees All paths of study must be accreditable A program is described by Objectives Outcomes Curriculum For purposes of accreditation review Transcript is primary evidence of degree Philosophy Institutions and programs define mission and objectives to meet the needs of their constituents enable program differentiation A-30 December 2005 Emphasis on outcomes preparation for professional practice Programs demonstrate how criteria and educational objectives are being met

16 Emphasis A-31 December 2005 Practice of continuous improvement Input of constituencies Process focus Outcomes and assessment linked to objectives Knowledge required for entry into the engineering profession Student, faculty, facilities, institutional support, and financial resources linked to program objectives Key Documents A-32 December 2005 Program Self-Study (not included in training packet) Criteria for Accrediting Engineering Programs (current and proposed revisions) Accreditation Policy and Procedure Manual PEV Report (forms)

17 Self-Study Report A-33 December 2005 A good program self-study should include: A complete description of how and the extent to which the program satisfies each of the criteria requirements: Students Program Educational Objectives Program Outcomes and Assessment Professional or Technical Component Faculty Facilities Institutional Support and Financial Resources Program Criteria ABET Definitions A-34 December 2005 The institution is free to define its own terms but must meet the philosophy of Criteria 2 and 3 definitions Make sure you understand their terminology Be aware the programs in a given institution may not use the same terms and definitions

18 ABET Definitions continued A-35 December 2005 Program educational objectives broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve. ABET Definitions continued A-36 December 2005 Program outcomes narrower statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire from the program.

19 ABET Definitions continued A-37 December 2005 Assessment one or more processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and program educational objectives. ABET Definitions continued A-38 December 2005 Evaluation one or more processes for interpreting the data and evidence accumulated through assessment practices. Determines the extent to which outcomes and objectives are being achieved and results in actions to improve the program.

20 Basis for Evaluation Criteria for the current visit cycle Accreditation policies and procedures for the current visit cycle Both on the ABET web site A-39 December 2005 PEV Report Curriculum Analysis Transcript Analysis Faculty Analysis Program Evaluator Worksheet A-40 December 2005

21 Purpose of the PEV Report Documentation (evidence) backs up the statement to the institution Filling out the forms moves the evaluator through the criteria thoroughly A-41 December 2005 Questions? Accreditation Basics: Objectives of Accreditation Self-Study Report PEV Report Review of Definitions

22 Accreditation Criteria A-43 December 2005 Standards to be applied with judgment! Criteria Where do they come from? A-44 December 2005 The ABET commissions and member societies representing the professions develop general and program criteria The commissions recommend criteria to the ABET Board The Board approves criteria

23 Criteria So what are they? Standards by which programs are evaluated for accreditation A-45 December 2005 Engineering Criteria I. General Criteria for Basic Level Programs II. Criteria for Advanced Level Programs A-46 December 2005

24 A-47 December 2005 I. Basic Level Accreditation Criteria 1. Students 2. Program Educational Objectives 3. Program Outcomes and Assessment 4. Professional Component 5. Faculty 6. Facilities 7. Institutional Support & Financial Resources 8. Program Criteria Students Criterion 1 A-48 December 2005 The program or its institution must: Evaluate student performance Advise students regarding curricular and career matters Monitor students progress to foster their success in achieving outcomes

25 Students Criterion 1 - continued A-49 December 2005 The program or its institution must have and enforce policies for transfer students policies for validation of courses taken for credit elsewhere procedures to assure that all students meet program requirements Student Transcripts A-50 December 2005 Provide direct evidence that the institution s program requirements are met Provide evidence that the curricular requirements of Criteria 4 and 8 (if applicable) are met

26 Transcript Evaluation A-51 December 2005 Determine if curriculum is being followed Note that curriculum may differ from the one being evaluated If no Worksheet is provided, ask if it exists Does institutional process assure that course substitutions meet ABET requirements? The transcript should indicate the title of the program being evaluated Consult team chair for programs in transition Issues for Criterion 1 A-52 December 2005 Problems with student advising (often cited with Criterion 5 - faculty) Ineffective and inconsistent advising Lack of understanding of curricular requirements especially if many options are available Ineffective monitoring No documentation of course substitutions or missing prerequisites

27 Issues for Criterion 1 - continued Problems with transfer students No documentation on acceptability of transfer credits (primarily for engineering topics courses) A-53 December 2005 Questions? Criterion 1

28 Program Educational Objectives - Criterion 2 A-55 December 2005 Program Educational Objectives: Broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve Consistent with the mission of the institution Allows differentiation between programs A-56 December 2005 Program Educational Objectives - Criterion 2 Each program must have Detailed published educational objectives consistent with the mission of the institution and these criteria Process based on needs of constituencies in which objectives are determined and periodically evaluated

29 A-57 December 2005 Program Educational Objectives - Criterion 2 Each program must have An educational program, including a curriculum that prepares students to attain program outcomes A process of on-going evaluation of the extent to which these objectives are attained and uses results to develop and improve the program outcomes Issues for Criterion 2 A-58 December 2005 Educational objectives not published or readily accessible to the public Educational objectives not related to institutional mission No evidence of constituency input in setting or periodic evaluation of objectives

30 Issues for Criterion 2 - continued A-59 December 2005 No process for evaluating the extent to which objectives are attained No data available on the attainment of educational objectives Results of evaluation not used to develop and improve the program outcomes A-60 December 2005 Questions You May Wish to Ask Objectives How were the objectives determined? Are they consistent with the mission statement? Are they measurable? How do you accomplish your objectives? What is your process to review and update? How do you know when you have achieved your objectives? Constituencies Who are your constituencies? How do you involve your constituencies in the continuous improvement process?

31 A-61 December 2005 Evaluation of Program Objectives The following tools are useful in evaluation of program objectives: Alumni Surveys Industrial/Professional Advisory Boards Employers Surveys and Placement Data Graduate programs at other universities Programs may choose to use these or other appropriate assessment methods Questions? Criterion 2

32 A-63 December 2005 Program Outcomes & Assessment - Criterion 3 Program outcomes: Narrower statements that describe what students are expected to know or be able to do by the time of graduation. Relate to skills, knowledge, and behaviors that students acquire from the program. The achievement of all outcomes indicates that the student is equipped to achieve the program educational objectives ABET designated (a-k) included in some way Program Outcomes and Assessment - Criterion 3 A-64 December 2005 Programs must demonstrate their graduates have achieved outcomes a to k Programs must have an assessment process with documented results that indicate the degree to which outcomes are achieved Programs must provide evidence that the results of the assessment process are applied to the further development and improvement of the program

33 Program Outcomes A-65 December 2005 Engineering programs must demonstrate that their graduates have: a. An ability to apply knowledge of mathematics, science, and engineering appropriate to the discipline b. An ability to design and conduct experiments, analyze and interpret data Program Outcomes A-66 December 2005 c. An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability. d. An ability to function on multidisciplinary teams

34 Program Outcomes (continued) e. An ability to identify, formulate, and solve engineering problems f. An understanding of professional and ethical responsibility g. An ability to communicate effectively A-67 December 2005 Program Outcomes (continued) A-68 December 2005 h. The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context i. A recognition of the need for, and an ability to engage in, life-long learning j. A knowledge of contemporary issues k. An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice

35 Issues for Criterion 3 A-69 December 2005 No evidence demonstrating one or more outcomes Outcomes not assessed objectively (student performance) Anecdotal versus measured results Reliance on course grades as assessment of outcomes Over-reliance on self-assessment (e.g., surveys) Issues for Criterion 3 (continued) A-70 December 2005 No systematic assessment process No process or process not documented Plans developed but not implemented Little or no faculty support for the process No evidence that assessment results are being applied to develop and improve the program Assessment results not used Assessment & development cycle not complete

36 A-71 December 2005 Assessment Tools and Measures The primary outcomes assessment should be based on direct measures of student learning Senior exit surveys, alumni surveys, and employer surveys as means of outcomes assessment are indirect measures. These data could be used as supportive evidence; their use as the primary means of assessment is discouraged. A-72 December 2005 Evidence to Be Provided Self-study should describe the materials that will be available for review during the visit to demonstrate achievement of the program outcomes The evaluators will be looking for results of assessment and evaluation. Evaluators will not assess student work to determine what outcomes were covered Taking a course does not guarantee achievement of outcomes

37 A-73 December 2005 Objectives versus Outcomes based on Criteria Definitions Objectives require longer term evaluation Will be based on needs of constituencies May be based on alumni and employer surveys Evaluation is required Program development is required Outcomes assessment is shorter term Should involve student work as the primary means of assessment Surveys and other opinion polls can be used as secondary evidence Outcomes should be measured against an expectation Closing the loop is required Questions You May Wish to Ask Outcomes What should students know and be able to do upon graduation? Are students familiar with the required outcomes? Processes What is your process to achieve your outcomes? How is it documented? What does it do? How do you know it does what you say it does? Who maintains and improves the process? What is A-74 December 2005 your involvement?

38 Questions You May Wish to Ask A-75 December 2005 Assessment Are data being collected for each outcome? What is measured? How often? How do you use the data you collect? Does your evaluation provide the information that you need to improve the program? What is your feedback process to continuously improve the program? What actions have been taken to improve the program as a result of the assessment process? Questions You May Wish to Ask A-76 December 2005 Results How do you demonstrate that outcomes are attained? What is your evaluation of the quality of the program? What evidence do you have to demonstrate that your efforts to improve the program are producing results? Based on your evaluation of assessment data, what are your plans for additional improvement?

39 A-77 December 2005 Level of Expectation Exactly which outcomes must each graduate attain? A system must be in place to ensure that all graduates have, to some minimum extent, achieved the required Criteria 3 and 8 outcomes and all elements of the Professional Component The level of achievement may vary among students, so long as the variation is consistent with program objectives What to Look For A-78 December 2005 Processes in place that provide for: Definition of desired, measurable outcomes Collection of data linked to the outcomes Analysis of data and evaluation of results Implementation of change Repeat cycle and review

40 What to Look For (continued) A-79 December 2005 Documentation of results and evidence that results are being used to develop the program, for example: Student portfolios Subject content examinations Performance observations Performance evaluations of internships and/or co-ops Keep in Mind A-80 December 2005 You don t have to be an expert on assessment. The institution must provide evidence that it has a working and effective system in place The institution must describe a clear relationship between program objectives, outcomes, and measurable indicators of success with required levels of achievement You are assessing the program based on the criteria and the strength of the evidence provided by the institution, not your own personal preferences

41 Questions? A-81 December 2005 Criterion 3 Professional Component Criterion 4 A-82 December 2005 Faculty must ensure that the curriculum devotes adequate attention and time to each component, consistent with the outcomes and objectives of the program and institution Preparation for engineering practice Major design experience Subject areas appropriate to engineering

42 Professional Component A-83 December 2005 Major Design Experience A culminating experience, based on knowledge and skills acquired in earlier course work Must incorporate appropriate engineering standards and multiple realistic constraints. A-84 December 2005 Professional Component Subject Areas One year of a combination of college-level mathematics and basic sciences (some with experimental experience) appropriate to the discipline One and one-half years of engineering topics, consisting of engineering sciences and engineering design appropriate to the student s field of study A general education component that complements the technical content of the curriculum and is consistent with the program and institution objectives

43 Criterion 4 - Issues A-85 December 2005 Quality of the major design experience No culminating experience - analysis or research instead of design; several courses with elements of design Multiple capstone courses with widely varying quality Design experience does not address multiple constraints Engineering topics satisfied by electives, but advising doesn t assure adequate coverage Questions? A-86 December 2005 Criterion 4

44 Faculty -Criterion 5 A-87 December 2005 Sufficient in number and competencies to cover all curricular areas Sufficient in number to accommodate adequate levels of student-faculty interaction, advising and counseling, service, professional development, and interactions with industrial and professional practitioners and employers Ensure proper guidance of the program and its evaluation, development, and improvement Criterion 5 - Issues A-88 December 2005 Insufficient number: To support concentrations, electives, etc. To provide student advising Poor faculty morale affecting program Lack of professional development Excessive workloads Retention/turnover rate Salaries as it relates to retention and recruiting (cited with Criterion 7)

45 Criterion 5 Issues - continued A-89 December 2005 Faculty Quality For teaching design (program criteria issues) Excessive reliance on adjuncts Heavy reliance on temporary faculty appointments, potentially jeopardizing program stability (cited with Crit. 7) Questions? A-90 December 2005 Criterion 5

46 Facilities - Criterion 6 A-91 December 2005 Classrooms, laboratories, and associated equipment must be adequate to support attainment of program objectives and provide an atmosphere conducive to learning Opportunities to learn the use of modern engineering tools Computing/information infrastructure to support scholarly activities of the students and faculty and the educational objectives of the institution Criterion 6 - Issues A-92 December 2005 Insufficient Space Overcrowded laboratories and classrooms Laboratories Unsafe conditions Inoperable equipment Lack of modern instrumentation Lack of funds for upgrading (cited with Crit. 7) Computing/Information Infrastructure Lack of funds for upgrading (cited with Crit. 7)

47 Institutional Support and Financial Resources - Criterion 7 A-93 December 2005 Institutional support, financial resources, and constructive leadership must be adequate to assure quality and continuity of the program Attract, retain, and provide for professional development of a well-qualified faculty Resources to acquire, maintain, and operate equipment and facilities Adequate support personnel Support of quality-improvement efforts Criterion 7 - Issues A-94 December 2005 Unstable leadership affecting programs Dean/Dept Chair positions open or filled by interim appointments for an extended period Frequent turnover of university administration and engineering school leadership Inadequate operating budget affecting: Acquisition and maintenance of laboratories and computing equipment Faculty salaries, promotions, and professional development affecting hiring and retention

48 Criterion 7 - Issues (continued) Insufficient Support Staff Teaching assistants Technicians for instructional laboratories, machine shops, and laboratory services Administrative/clerical A-95 December 2005 Questions? A-96 December 2005 Criteria 6 and 7

49 A-97 December 2005 Program Criteria Criterion 8 Each program must satisfy applicable Program Criteria Curricular topics Faculty qualifications Current Program Criteria are on the ABET server ( Must satisfy all Program Criteria implied by title of program CE Program Criteria 1. Program must demonstrate that graduates have: Proficiency in math, thru differential equations, prob & stats, calculus-based physics, and general chemistry Proficiency in at least 4 recognized CE areas Ability to conduct lab experiments in more than one recognized CE area Ability to perform design by means of design experiences integrated throughout the curriculum Understanding of professional practice issues

50 CE Program Criteria 2. Program must demonstrate that: Faculty teaching courses that are primarily design in content must be qualified by: Licensure or Education and design experience The program is not critically dependent on one individual. The ASCE CE Commentary Definitions and discussion of: Proficiency Recognized major civil engineering areas Engineering design Design integration Specific examples of how programs might demonstrate compliance Guidance on faculty qualifications CE Program Evaluators must read and understand this document!

51 CE Program Criteria - Issues Failure to ensure that all students attain proficiency in 4 CE areas Inadequate student understanding of professional practice issues Inadequate design integration Faculty teaching design who are not qualified to do so Faculty expertise inadequate to teach 4 CE areas Related Disciplines Construction Engineering ASCE is lead society Commentary available Architectural Engineering ASCE is lead society No Commentary yet Environmental Engineering AAES is lead society Combined degree titles must satisfy all applicable Program Criteria

52 Questions? CE Program Criteria A-104 December 2005 II. General Criteria for Advanced-Level Programs Completion of a program satisfying the general criteria for basic-level engineering programs One year of additional study An engineering project or research activity with a report that demonstrates: Mastery of the subject matter High level of communication skills

53 Session 1 Wrap-Up Print & sign the sign-in roster, and fax to Dion Coward at For homework: Study all remaining Session 1 slides. Formulate responses to the Conflict of Interest Exercise Read the ABET Decision Exercises Print the ASCE State Tech Report and have it available for our next session. Send questions to stephen.ressler@usma.edu Next session: Log in 15 minutes early! Review Conflict of Interest Exercise See you Review Criteria next week! ABET Decision Exercises The Accreditation Process & PEV Reports