Oregon Institute of Technology Computer Systems Engineering Technology Department Embedded Systems Engineering Technology Program Assessment 2014-15 I. Introduction The Embedded Systems Engineering Technology (ESET) program was proposed to OUS in spring of 2006 and approved in August, 2006. The curriculum for the ESET program is common with the hardware and software programs for the freshman year. The sophomore year of the ESET program has been constructed to mirror the track through both the Computer Engineering Technology (CET) and Software Engineering Technology (CET) programs, called the Dual Degree program. The ESET program junior year is when ESET students get instruction specific to topics of embedded systems engineering. These courses were taught for the first time in fall, 2008 on the Klamath Falls campus and soon after at the Wilsonville location. The full program is now offered to students at both locations. Current enrollment at the Klamath location is 23 students and 9 in Wilsonville for a total of 32 students. 1 There were five graduates of the ESET program in 2014-15. 2 II. Mission, Objectives and Program Student Learning Outcomes Program faculty met spring term 2015 and reviewed the mission, objectives and program student learning outcomes. Based on the recommendation from the Industrial Advisory Board solicited at the February 19, 2015 meeting (minutes included in Appendix A), the faculty affirmed the current program mission and objectives, and agreed to change the program student learning outcomes to match the ABET ETAC a-k. This change is not reflected in this report, but the change will be implemented beginning in fall 2015. The mission of the Embedded Systems Engineering Technology (ESET) Degree program within the Computer Systems Engineering Technology (CSET) Department at Oregon Institute of Technology is to prepare our students for productive careers in industry and government by providing an excellent education incorporating industry-relevant, applied laboratory based instruction in both the theory and application of embedded systems engineering. Our focus is educating students to meet the growing workforce demand in Oregon and elsewhere for graduates prepared in both hardware and software aspects of embedded systems. Major components of the ESET program s mission in the CSET Department are: 1 Oregon Tech Institutional Research Fall 2014 4 th week Enrollment Report http://www.oit.edu/facultystaff/institutional-research/majors-reports 2 Oregon Tech Institutional Research Completions Report http://www.oit.edu/faculty-staff/institutionalresearch/degree-data 1
I. To educate a new generation of Embedded Systems Engineering Technology students to meet current and future industrial challenges and emerging embedded systems engineering trends. II. To promote a sense of scholarship, leadership, and professional service among our graduates. III. To enable our students to create, develop, apply, and disseminate knowledge within the embedded systems development environment. IV. To expose our students to cross-disciplinary educational programs. V. To provide government and high tech industry employers with graduates in embedded systems engineering and related professions. Program Educational Objectives The Program Educational Objectives reflect those attributes a student of the ESET program will practice in professional endeavors. A. Graduates of the embedded program are expected to understand societal impact of embedded systems and technological solutions. B. Graduates of embedded degree program are expected to do hardware/software co-design for embedded systems. Graduates will continue to develop skills in analysis, approach, optimization, and implementation of embedded systems. C. Graduates of the embedded program are expected to obtain the knowledge, skills and capabilities necessary for immediate employment in embedded systems. Embedded Systems is a profession increasingly driven by advances in technology, therefore graduates are expected to obtain the necessary life-long learning skills to enable them to be able to adapt to a changing environment. D. Graduates of the embedded program are expected to develop a broad base of skills. These skills will prepare them for professional practice: 1) as embedded engineers, 2) participants in embedded development teams, and 3) effective communicators within a multidisciplinary team. E. Graduates of the embedded program are expected to acquire knowledge of management and marketing of embedded projects and products and to prepare for series production. Program Student Learning Outcomes Embedded Systems Engineering Technology baccalaureate graduates will be engaged in: 1. Application of mathematics including differential and integral calculus, probability, and discrete mathematics to hardware and software problems (Objectives C, D, E). 2. Application of project management techniques to embedded systems projects (Objectives C and D). 2
3. Application of knowledge of embedded systems engineering technology, along with some specialization in at least one area of computer systems engineering technology. (Objective D) 4. A broad education and knowledge of contemporary issues necessary to reason about the impact of embedded system based solutions to situations arising in society. (Objective A) 5. Identification and synthesis of solutions for embedded systems problems. (Objective B, C) 6. Design, execution and evaluation of experiments on embedded platforms. (Objective C, D) 7. Analysis, design and testing of systems that include both hardware and software. (Objective B, D) 8. Documenting the experimental processes and to writing of satisfactory technical reports/papers. (Objective D, E) 9. Delivery of technical oral presentations and interacting with a presentation audience. (Objective D, E) 10. Recognition for and the motivation to further develop their knowledge and skills as embedded engineering advances occur in industry. (Objective C) 11. Working effectively, independently, and in multi-person teams. (Objective D) 12. Professional and ethical execution of responsibilities. (Objective A, D) III. Three-Year Cycle for Assessment of Student Learning Outcomes Assessment activities for the ESET program began Fall, 2008. Table 1 presents planned learning outcome assessment on a three year cycle. The rows of this table correspond to the PSLO. The entries in the cells corresponding to the year represent mapping from the row (PSLO) to the ISLO. The number in the cells of the table corresponds to the ISLO defined for the OIT assessment cycle: ISLO 1: OIT students will demonstrate effective oral, written and visual communication. ISLO 2: OIT students will demonstrate the ability to work effectively in teams and/or groups. ISLO 3: OIT students will demonstrate an understanding of professionalism and ethical practice. ISLO 4: OIT students will demonstrate critical thinking and problem solving. ISLO 5: OIT students will demonstrate knowledge and understanding of career development and lifelong learning. ISLO 6: OIT students will demonstrate mathematical knowledge and skills. ISLO 7: OIT students will demonstrate scientific knowledge and skills in scientific reasoning. ISLO 8: OIT students will demonstrate cultural awareness. 3
Table 1: Baccalaureate Outcome Assessment Timeline # Learning Outcomes 09-10 10-11 11-12 12-13 13-14 14-15 1 2 3 4 5 6 7 8 9 10 11 12 The ability to apply mathematics including differential and integral calculus, probability, and discrete mathematics to hardware and software problems. An ability to apply project management techniques to embedded systems projects. Knowledge of embedded systems engineering technology, along with some specialization in at least one area of computer systems engineering technology. A broad education and knowledge of contemporary issues necessary to reason about the impact of embedded system based solutions to situations arising in society. The ability to identify and synthesize solutions for embedded system problems. The ability to design, conduct and evaluate the results of experiments on embedded platforms. The ability to analyze, design and test systems that include both hardware and software. The ability to document experimental processes and to write satisfactory technical reports/papers. The ability to make technical oral presentations and interact with an audience. The recognition for and the motivation to further develop their knowledge and skills as embedded engineering advances occur in industry. The ability to work effectively independently and in multi-person teams. An understanding of professional and ethical responsibility. 6 6 2, 3 2, 3 4 4 3, 8 3, 8 4 4 7 7 7 7 1 1 1 1 5 5 2 2 3, 8 3, 8 4
Outcomes to be assessed are listed below: 1. Application of mathematics including differential and integral calculus, probability, and discrete mathematics to hardware and software problems. 6. Design, execution and evaluation of experiments on embedded platforms. 7. Analysis, design and testing of systems that include both hardware and software. Target courses where the assessment tools were to be applied for the 2014-15 academic year are summarized in Table 2. Table 2: 2014 2015 Summary Courses of Assessment Application Outcome Courses Term 1 CST 315 Embedded Sensors & I/O CST 466 Embedded Security 6 CST 347 Real Time Operating Systems CST 455 System on a chip Design 7 CST 345 HW/SW Co-Design CST 373 Embedded Systems Develop III Fall Spring Spring Fall Winter Spring IV. Summary of 2014-15 Assessment Activities The following are the direct assessment activities that were accomplished during 2014-2015 academic year. Each activity is introduced with a description of the activity followed by a table that summarized the rubric criteria along with the rubric application results. Where available, the rubric used for assessment is shown in Appendix B. PSLO #1 Application of mathematics including differential and integral calculus, probability, and discrete mathematics to hardware and software problems. CST 315 Embedded Sensor Interfacing & I/O: Fall 2014 Data Collection Date: Fall 2014 Coordinator: Claude Kansaku Student work was collected to support this assessment, but as of the date of publication of this report the final assessment results have not be submitted and the faculty member responsible for assessing this work is out on sabbatical. This report will be updated with this assessment work when completed. 5
CST 466 Embedded Security: Spring 2015 Data Collection Date: Spring 2015 Coordinator: Phong Nguyen Assessment Method: Exam Though CST 466 is a required course in the ESET program, the spring term 2015 enrollment for this class contained no ESET majors. This outcome will be assessed using the same method and the same course spring term 2016 and this report will be updated with the results. PSLO #6 Design, execution and evaluation of experiments on embedded platforms. CST 347 Real Time Operating Systems: Spring 15 Data Collection Date: Spring 2015 This assessment was not completed as planned. This outcome will be assessed in CST 347, winter term 2016 and this report will be updated with the results. 6
CST 455 System on a chip design: Fall 2014 This assessment focuses on the design of a solution to a problem given as a take home final. There were five ESET students involved in this assessment. Data Collection Date: Spring 2015 Coordinator: Troy Scevers Assessment Method: A take home final project was given to the students. The project was to create a SoPC system using Hard-Core ARM processors inside of an Altera FPGA Performance Criteria Assessment Method Measurement Scale Minimum Acceptable Performance Results Hypothesis Proficiency (3) 5 of 5 Test Creation Proficiency (3) 5 of 5 Implementation Proficiency (3) 5 of 5 Outcome Proficiency (3) 5 of 5 Evaluation of results: Students were to successfully complete the assignment. The number of students in the sample space is small; however, there were no problems in successful completion of the assignment. Actions: No actions required at this time Closing the Loop: No actions from previous cycle 7
PSLO #7 Analysis, design and testing of systems that include both hardware and software. CST 345 Hardware/Software Co-Design: Winter 2015 The Hardware/Software Co-design course brings concepts of software and hardware design into a single activity resulting in more robust SOPC designs. There were five ESET students involved in this assessment their results are reported in the table below, other student in the course are not reported. Data Collection Date: Winter 2015 Coordinator: Troy Scevers Assessment Method: Junior level students were given a take home final were they were to design a digital thermostat system. They were given various pieces of hardware to complete this design. Performance Criteria Assessment Method Measurement Scale Minimum Acceptable Performance Results Knowledge Proficiency (3) 5 of 5 Analysis Proficiency (3) 5 of 5 Fabrication Proficiency (3) 5 of 5 Testing Proficiency (3) 5 of 5 Test Interpretation Proficiency (3) 5 of 5 Evaluation of results: Students were able to complete the take home final with a lot of work. The results shown are only for ESET students and do not include the results for CET students who also took the class. 8
Actions: No actions necessary at this time Closing the Loop: ESET students need more exposure to testing. They now take the ESET testing class in the senior year. I felt the students had an appropriate amount of testing background for this class. CST 373 Embedded System Design III: Spring 2015 CST 373 is the third term of the Jr. Project design sequence. Students from ESET and CET in this course work in project teams. The teams were evaluated and the results are included in the table below. This assessment focuses on the design of a solution to a problem. Data Collection Date: Spring 2015 Coordinator: Phong Nguyen Assessment Method: Projects (3 total) in CST 373 were assignments to analyze, plan, design, fab and test an embedded system complete with input, output, memory, microcontroller and control. Performance Criteria Measurement Scale Minimum Acceptable Performance Results by team 1-4 according to rubric 3.0 3 of 3 Analysis (Plan) of final project Design Project 1-4 according to rubric 3.0 3 of 3 Test Project 1-4 according to rubric 3.0 3 of 3 Evaluation of results: All three project did an outstanding job on all three criteria of this assessment Actions: None Closing the Loop: None 9
V. Summary of Student Learning PSLO #1 Application of mathematics including differential and integral calculus, probability, and discrete mathematics to hardware and software problems. Evaluation of results: will be updated fall 2016 once assessment has been completed. Actions: to be updated fall 2016 PSLO #6 Design, execution and evaluation of experiments on embedded platforms. Evaluation of results: results collected to date show student have met expectations, this section will be updated once all results have been collected. Actions: none at this time PSLO #7 Analysis, design and testing of systems that include both hardware and software. Evaluation of results: based on the two direct assessment of this outcome conducted in 2014-15, students performed above expectations. Actions: none 10
APPENDIX A Industrial Advisory Board Meeting Thursday, February 19, 2015 8:30 10am PV147 The meeting came to order at 8:35am. IAB members present were: Mark Bansemer (phone), Bhushan Gupta (phone), Michael Hoffman (phone), and Tavish Ledesma (phone). CSET faculty and staff present were: Todd Breedlove, Jay Bockelman (phone), Calvin Caldwell (Dept. Chair), Mike Healy, Phil Howard, Doug Lynn, Kevin Pintong, Troy Scevers, and Sherry Yang (phone). Also present was Charlie Jones (Dean of ETM). Welcome by Calvin Caldwell and introductions from those present. Announcements Introduction of new faculty; Phil Howard and Kevin Pintong and new CSET Lab Manager; Mike Healy. We are still searching for a new faculty for the SET Program. Program Updates Results from the ABET Accreditation Review in October: 1. One of the directives from ABET was to make available, copies of the PEOs (broad statements that describe what graduates are expected to attain within a few years of graduation) for the CET, SET and ESET programs, so that they could review and approve them. (documents attached). The current PEOs were approved by the attending IAB members and this process will be done at least once a year. Comments were that the PEOs seemed reasonable and no concerns were expressed by the members present. 2. Need documentation of our IAB meetings (minutes) Meeting minutes will be taken and documented. 3. Need documentation for the process of our assessment. One of the tools that we use for our documentation is SharePoint. We will be using a new version that will update this process. 4. In our recent ABET accreditation visit one concern was that we weren t regularly reviewing our ISLOs. Evidence stated that certain terms used in ABET s A-K were not apparent in our program ISLOs. Reviewing is being done by the department and we are in the process of reviewing our ISLOs at today s IAB meeting. 11
Program Concerns Certain terms such as analyze and respect for diversity were not found in our ISLOs 2 and 4. When mapping our ISLOs to ABET s A- K the language needed to be almost identical. As a department we have decided to use just ABET s A-K as other departments on campus are doing. We will assess relative to ABET s A-K. Every fall we will review ABET s A-K to see if we are still in line with them. We are able to add to ABET s A-K any other objectives we have. A motion was made by Calvin to accept using the A-K for student learning objectives. There were no objections. Motion passed. 1. Ways to increase student enrollment for the Embedded Systems Engineering Technology program. We have put together a new recruiting committee. Claude Kansaku is chairing that committee. We are doing external visits to various community colleges and high schools. One of the problems we have is explaining the Embedded Program to young people. Tavish Ledesma (Intel) said the demand for embedded has definitely increased. One of the things they do to promote embedded is to bring in recent college graduates or interns that are working in the industry to speak to and answer questions from high school and community college students about what they do in this field. He also said that there is a network of people at Intel that could be used and that he could gather some contacts. Todd Breedlove is working with the student CSET Ambassadors who join him on recruiting visits to high schools and community colleges. They also attend open houses for college preview days, Tech Trek, and other recruiting trips that we do to speak to students about our programs. Alumni ambassadors would be a good idea to use. Other suggestions: a. 2 minute videos from students about what they like about the program b. Which high schools we should target where there are more interested students c. Updating our marketing strategy for embedded d. Using new marketing terms associated with embedded e. Videotaping projects using embedded systems that can be shown during recruiting f. Targeting social media (Twitter, Facebook, Twitch, YouTube), which the students relate to 12
g. Students incorporating a marketing strategy into their projects h. Using recent college graduates i. In the Embedded program, discontinue CST102 from common core, which used in all three programs in the freshman year, and add a course that is more appropriate. j. RCC and KCC, whom we are working closely with, are in the process of implementing their CS degree. The program will transfer directly into our programs at OIT. Community colleges recruit from high schools, so that gives us access to more high school students k. Using OIT s Marketing Department resources that are available Other Topics and General Discussions 1. Mark Bansemer s (IGT-currently employs 100 OIT alums) philosophy on teaching: make things happen and have fun doing it when teaching students. 2. Michael Hoffman s impression of the Wilsonville campus: from an instructor s perspective is that it has changed tremendously. They have gone from older degree completion students to more high school grads. Because of their change of location from Portland to Wilsonville there are a lot more commuting students and there isn t the typical college atmosphere. The meeting adjourned at 10:00 am. djh 13
Appendix B PSLO#6 Design, execution and evaluation of experiments on embedded platforms. Rating Aspects Hypothesis Test Creation Implementation Outcome Excellent Score = 4 Hypothesis creation was clear and easy to understand. Tests designed correlated directly to hypothesis assumptions. Code written was easy to understand and performed designed tests. End results were analyzed and explained. Good Score = 3 Hypothesis had correlation to real-time systems; however, correlation was not clear. Tests correlated to hypothesis; however, they were difficult to understand and results were hard to extract. Implementation worked and performed tests but code was pearly designed. End results were analyzed; however, explanation did not draw from experimental results. Fair Score = 2 Poor Score = 1 Hypothesis was poorly formed. Hypothesis had no correlation to desired real-time scheduling behavior. Tests partially tested hypothesis Implementation was buggy and performed sporadically. End results were enumerated with no real correlation to the hypothesis. Test created did not test for hypothesis assumptions. Implementation was poorly done and did not perform tests. There was no analysis of outcome or explanation of results. 14
PSLO #7 Analysis, design and testing of systems that include both hardware and software. Rating Aspects Excellent (4) Good (3) Fair (2) Poor (1) Understanding Information Gathering Alternatives HW/SW Plan to Solve Carrying out the Plan Simulation, Testing, and Evaluating Prototype testing and Evaluation Student Understands the HW/SW problem and relates it to HW/SW Codesign solution Student collects a great deal of information, all relates to the Codesign problem Student has an acceptable technique for developing HW/SW solution. Sound rationale explaining their choice Come up with many strategies, decides on appropriate solution to each strategy. Tries new methods to solve the Codesign problem. Uses expanded HW/SW testing strategy. Student suggests other modifications or applications of the results after testing the Codesign solution Extensive analysis of testing including procedure, timing and evaluation Student Understands (can explain) the HW/SW Codesign problem and proceeds to the next step Student collects some information, most related to the Codesign problem Student has an rationale for choosing a HW/SW solution but rejects of other possible HW/SW solutions Come up with several strategies, decides on an appropriate solution Solves the Codesign problem using design, makes appropriate HW/SW testing strategy Student compare actual and expected results but does not repartition the HW/SW Prototype tested including evaluation but not extensive analysis of results Student needs some clarification from others to understand the HW/SW Codesign problem Student collects very little information, some relates to the Codesign problem Student has some rationale for choosing a HW/SW solution but fails to explore other possibilities Come up with a few strategies and requires assistance to select an appropriate strategy. Solves the Codesign problem without a HW/SW testing strategy. Student has Limited evaluation of Codesign without assistance, but requires help testing Codesign solution Prototype tested with no analysis of results Student does not have an understanding of the HW/SW Codesign problem Student does not collect any information that relates to the Codesign problem Student has vague or no rationale for choosing solution has not explored other possibilities Designs only one strategy, required assistance to evaluate strategy Attempts to solve Codesign problem with an inadequate HW/SW testing strategy Student requires assistance to evaluate and test the Codesign solutions Prototype tested with no analysis of results 15