Teaching Assessment and Rubrics

Teaching Assessment and Rubrics David GORE School of Technology Studies, Eastern Michigan University Ypsilanti, Michigan 48197, USA and Marie LEE School of Technology Studies, Eastern Michigan University Ypsilanti, Michigan 48197, USA and Autumm CAINES School of Technology Studies, Eastern Michigan University Ypsilanti, Michigan 48197, USA ABSTRACT This paper describes the pedagogical value of teaching assessment and rubrics through the simulation of actual IT business practices to undergraduate students at Eastern Michigan University. In replicating the acquisition of a telecommunication technology system, the simulation takes place across two courses. The Communication Transmission Systems course has students create a fictional telecommunications company and respond to an RFP adapted from industry. The Telephone Technology course creates a version of the RFP and evaluates the proposal responses from the previous course. This paper describes how the RFP is developed, the dynamics of how the two classes interface, as well as how the Communication Technology major at EMU address the changes taking place in the industry. There is a major focus on how students evaluate the incoming proposals by developing criteria for and applying a rubric, weighting those criteria, and averaging the scores to simulate actual business decision making. Keywords: Assessment, Rubrics, Telecommunications, Telephony, Communication, Transmission, Pedagogy. Introduction In today s highly dynamic world of Information Technology, our students need more than just theory and how-to skills. Traditional pedagogical approaches cover theoretical content through the use of textbooks and other materials, and many IT programs are providing students with opportunities for hands-on experience with the technologies that they will be using in the workforce. However, there is a gap between the theoretical and the practical. This gap is the evaluation and assessment of these technologies to assure that business needs are met at the best price. We see this gap substantially impacting the world of telecommunications and changing the role of the IT professional and telecommunications technician. As voice technology has become digitized, it can be sent over transmission lines as data, hence, the role of the IT professional and telecommunications technician have evolved and converged with one another. These professionals need to be proficient in the converged technologies, as well as the business practices to manage technology change within an organization. These changes require an understanding of infrastructure, billing, contracts, system features, and the individual needs of end users. These voice and data specialists need to articulate technology needs that will solve particular business problems in a very specific way, through an RFP and proposal response process, and then must objectively analyze solutions to make the best decision that will yield the greatest return on investment. In response to this educational and practical challenge, Eastern Michigan University (EMU) has created a detailed simulation that allows students to create and apply a rubric, as well as practice theory learned in a two course series, to a realworld telecommunication system acquisition. Through this simulation students are put in the role of a technician and asked to evaluate a local business or organization s telecommunication infrastructure. They then develop an RFP and create a rubric to weight each of the needs that they have identified. After being presented with several different proposals, they use the rubric to identify the best solution for the company s needs. Communication Technology at EMU The Communication Technology program at EMU is an undergraduate bachelor of science major. The curriculum is an interdisciplinary mix of the technologies of communication (i.e., computers, graphics, media, telephony, etc.) and general business skills. Students supplement their broad understanding of the interrelated technologies of the modern information age with a required cooperative work experience and approximately

16 hours of elective coursework. Graduates of this program typically enter the workforce at the entry level and move up to mid-management positions within 3 to 5 years. The program emphasizes technical skills necessary to gain entry-level employment in the IT area the student chooses. Along with the technical skills, most of the courses have very strong teamwork, interpersonal communication, written communication, problem solving, and presentation components. It is essential for graduates of the program to have a mastery of these skills before they attempt the job search process and enter the workforce. The proposals generated in the CMT 305 Communication Transmission Systems course are based on a modified RFP from the IT industry. This introduces the students to an actual situation and forces them to generate a proposal that is fairly realistic. While this experience is somewhat overwhelming due to its authenticity, it provides students with an experience they will not likely encounter in any other course. Furthermore, it provides the background for the generation of the RFP in CMT 408 Telephone Technology, the follow-up course. Creating a Request for Proposal (RFP) Simulation By incorporating a business simulation into the student s coursework they are given a sample of situations that they will actually encounter in industry. According to WordNet online dictionary, simulation is defined as, the act of imitating the behavior of some situation or some process by means of something suitably analogous (especially for the purpose of study or personnel training). The RFP Simulation in CMT 408 Telephone Technology teaches a complex business process through a series of classroom exercises, culminating in a presentation and bound, written assignment. The RFP simulation for a new telecommunication system is broken into 3 major sections as defined below in Table 1. Prepare for a New Telecommunications System Purchase Document the physical infrastructure Determine the use of each phone/data line Prepare a floor plan Assess telecommunications bills; perform an audit Assess Proposals Create scoring rubric Analyze given proposals with rubric Make a recommendation for purchase Prepare to Install a New Telecommunications System Create a call flow diagram for company Prepare questionnaires for individual departments Create help desk procedures Plan for downtime when system is cut from old to new Figure 1: Major Project Overview for CMT 408 Telephone Technology Concepts and content are presented though various classroom lectures and exercises. Students are directed to: - Document the technical baseline go to a company dmarc (telecommunications company point of demarcation in a building), write down all lines and circuits, take pictures, determine line and circuit use; collect telephone bills and audit charges; obtain or create current floor plan with location of all telephones, computers, fax machines, modems, other devices connected to a voice or data line, and unused wall jacks. - Assess need determine if all lines billed for are in use; discuss with CIO and management the three to five year technology plan; create technical questionnaire for end-users to determine current technology use, issues with connectivity, and unmet needs preventing them from performing efficiently; create interview questions to discuss needs with departmental managers. - Write the technical specifications gather technical needs and create the RFP document. In the real world, RFP documents are often borrowed and subsequently altered to meet a specific need. For the purpose of this course, technical need is documented through gathering the technical baseline and creating a plan for gathering the needs and wants of end-users. - Prioritize requirements determine the most important needs of their company. Itemized technical specifications in a section of the RFP are weighted on a scale of 1 to 5, with 5 being the highest priority. Exercises throughout the course illustrate that different companies can have different priorities and those priorities can

affect decision making. However, companies can be equally successful (or fail) with different priorities. - Determine the response parameters the RFP document should include how and when proposal responses should be received. This course presents common response parameters and the reasons a company may consider it for their RFP process. As technology changes, the way in which a company responds to proposals has also changed. Paper documentation is often substituted with electronic submissions. - Prepare for assessment (weighted priorities) proposals must be analyzed in the most objective manner possible. In order to maintain objectivity, and not be overwhelmed with the volume of responses, creating an assessment rubric is essential to sound decision making. Building the Rubric After the RFP has been developed, the students are directed to weight the items and points that are most important to their company. This is in preparation of the incoming proposals, none of the students have seen the proposals at this stage. The first step is to generate a list of those things that need to be included in the proposal in order for it to be in contention for the awarding of the contract. For a typical class the list generated might be as follows: 1. Is the Proposal Complete? 2. Are all the Key Technical Elements Present? 3. Are There Extra Features that Were Not Requested? 4. Is the Training Schedule Adequate & Flexible? 5. Is the Service Contract Included, Available, Suitable? 6. Is the Price Competitive? At this point a simple rubric is generated. As an example, three proposals from companies, CommTel Inc., IntelSystems Inc, and Cyber Voice will be assessed. Students are directed to simply determine if the proposal does or does not meet the requirement and award 1 point for yes and 0 points for no. The resulting rubric is: Proposal Key Tech Extra Training Service Price Total Complete Elements Features Schedule Comp. CommTel Inc. 1 1 1 0 1 0 4 IntelSystems 1 1 1 1 1 1 6 Cyber Voice 1 0 1 0 1 1 4 1 = Yes 0 = No Table 2: Simple Rubric Example According to the results of this rubric, IntelSystems has emerged as our winner. However, in the development of the assessment list it becomes apparent that not all of the items are of equal value, some of them are more important than others. So, the students are directed back to the original list of evaluation questions and instructed to weight each item based on the importance of that item as it relates to the proposal and the goals of the business. A scale of 1, least important, to 5, most important, is to be used. Weighting Factor 1. Is the Proposal Complete? 1 2. Are all the Key Technical Elements Present? 5 3. Are There Extra Features that were not Requested? 1 4. Is the Training Schedule Adequate & Flexible? 2 5. Is the Service Contract Included, Available, Suitable? 4 6. Is the Price Competitive? 3 By making this adjustment of the criteria to the above example and recalculating, the rubric now appears as: Proposal Complete Key Tech Elements X5 Extra Features Training Schedule Service Price Comp. X1 X1 X2 X4 X3 CommTel Inc. 1 1 1 0 1 0 11 IntelSystems 1 1 1 1 1 1 16 Cyber Voice 1 0 1 0 1 1 9 1 = Yes 0 = No Table 3: Rubric Example with Weighted Criteria Using Yes or No Total

The results do not change. In fact, Cyber Voice appears to be the least appealing proposal available. Once again the students are directed back to the rubric. By the applying the same reasoning that not all of the evaluation criteria have the same weight, it can also be Proposal Complete Key Tech Elements X5 Extra Features pointed out that not every question can be answered by a definitive yes or no. So this time the students are allowed to asses each proposal question using a scale of 2 for yes, 1 for maybe, and 0 for no. Introducing this intermediate option and applying the new values the rubric now looks like this: Training Schedule Service Price Comp. X1 X1 X2 X4 X3 CommTel Ind. 2 2 1 0 1 1 20 SmartSystems 2 2 1 2 1 2 27 Cyber Voice 2 1 2 1 2 2 25 2 = Yes 1 =Maybe 0 = No Table 4: Rubric Example with Weighted Criteria Using Yes, Maybe or No Total A very different result occurs. Cyber Voice is now in contention to be awarded the contract. This illustrates to the students exactly how important it is to set the evaluation criteria prior to the evaluation process. By doing so it ensures that you have written the RFP to reflect your real needs and also keeps the evaluation process focused on the results. The last step in this process points out that the above rubrics represent the efforts of just one individual. Here we have students combine the scores of each member of their team. For our example we are assuming a group of four people, Dave, Marie, Autumm, and Phil review the proposals and they will each have their own scores. Combining these into a new rubric results in the following: Dave Marie Autumm Phil Average CommTel Ind. 20 21 22 17 20 SmartSystems 27 22 27 26 25.5 Cyber Voice 25 26 28 23 25.5 Table 5: Averages of Rubric Scores We observe a tie between SmartSystems and Cyber Voice! Now the group must collaborate and come to consensus by debating and defending their scores. This last part of the exercise reinforces the teamwork skills necessary to be successful in the workplace. In practice the students taking part in the simulation are directed to create their own scoring rubric for the individual questions in selected segments of the RFP. This is a detailed exercise that allows the students to learn how to set up a scoring rubric that can be applied to very detailed as well as general proposal evaluations. Students are given actual proposal responses from the CMT 305 course and asked to evaluate specific sections based on their scoring criteria. Students are asked to determine a winner by the numbers, as well as justify the decision based on any other factors they may think are important. This assessment skill translates to many industries for decision-making and critical-thinking. Many companies do not have a formalized RFP process, however, these rubrics support sound decision making and the ability to justify purchases. However, many of the decision factors are not so clearcut, the students are also made aware of a confounding variable in this process that cannot be borne out by the numbers. Robert Potter (2004) calls this variable the emotional preference value which includes subjective, nonverbal criteria that relate to a buyer s level of comfort and trust. For our example, if there had been a strong working relationship in the past with Cyber Voice, it is quite likely that they might get the contract based on the final values determined by the rubric. This personal relationship, or prior business success, with this company might very well work in their favor. Obviously, a less than desirable prior working relationship could easily work against a company. These factors fall outside the seemingly cut and dry numbers in the rubric. The last step in the simulation is to award the contract. The student companies, having reviewed proposals from the Communication Transmission Systems class then make their decision known and notify the bidding company of their successful proposal. Academic Value The two course series of Communication Transmission Systems (CMT 305) and Telephone Technology (CMT 408) guide students through detailed simulations of the proposal and RFP processes. These simulations are rooted in real-world projects that have been adapted for use in the classroom. Students often interact with resources in the business world to help construct final

projects in each course. These courses were designed as constructivist learning environments where students work together as employees of their created company and are actively involved in learning through defining and solving problems set in real-world contexts. This differs drastically from traditional classes that delve into the theory of a content area without applicable practice in that field of study. Educational research studies have shown that students prefer doing over listening in a traditional lecture classroom (Bonwell, Eison 1991). Real world problems are typically ill-defined and have several possible correct solutions, or potentially no solution at all. Jonassen (1999) states, Students learn domain content in order to solve the problem, rather than solving the problem as an application of learning. The activities in the course direct the construction of learning as students find resources needed for the completion of the projects. Each activity is related to the larger project in the course, and the project in the Communications Transmission System course is related to the final project in the Telephone Technology course, and act as learning scaffolds to support the learners (Jonassen 1999). The simulation development for both courses was guided by Gagne s Nine External Events of Instruction (Gagne 1985): 1. Gain Attention 2. State Objectives 3. Elicit Prior Knowledge 4. Present Content 5. Provide Learning Guidance 6. Elicit Performance 7. Provide Feedback 8. Summarize 9. Retention and Transfer These events provided a checklist for the developer/instructor to ensure that learning will be effective and that objectives will be met by the instruction and activities. Success is achieved by engaging learners, requiring action, summarizing presented content, and providing practice and tools for students to incorporate the learning into real life settings. Simulation activities are: Structured with no right or wrong answers and encourage students to explore possible alternatives within the framework of the simulation Large and complex, requiring teamwork to accomplish the project Broken into small in-class activities and miniprojects to incorporate practice Concluded with the students concluding their semester long projects with a presentation to the class and a written proposal selection report which includes the individual components of the RFP. Building and the use of a rubric has broad application in a number of fields across many types of projects, but it is especially applicable to the field of information technology where businesses need to refresh their technology on a periodic basis to remain competitive. Students entering the workforce will be met with the challenges of needs assessment, proposal analysis and technology implementation. By providing these opportunities in an academic setting, students are free to engage in learning activities in a safe setting where their decisions will not impact the bottom line of a company. Conclusion In the CMT 408 Telephone Technology and CMT 305 Communication Transmission Systems courses at EMU, undergraduate students are taught how to combine IT and business skills with a series of labs and business simulations aimed at giving the students the opportunity to gain practical experience while making mock business decisions. EMU is preparing students for challenges by merging the hands-on lab opportunities, where the technical aspects of telecommunications are explored, with these simulations of real-world business situations, where students have to evaluate technologies on a larger scale and consider the changes that would affect an entire company. Through these two courses, the entire process of developing and responding to an RFP is simulated. By employing the use of rubrics in the simulation of evaluating proposals, students are given the opportunity to think about what factors would effect the actual implementation of these technologies and how important each of those factors would be given certain criteria. As voice communication becomes another application of the data network, it is not only technologies that are converging, but also skills. Industry is requiring that our students be able to understand the technologies involved, as well as be able to assess the need for and value of these technologies. EMU is meeting this need with the CMT 408 and 305 classes. As technologies change, EMU is addressing the needs of industry through the Communication Technology Advisory Committee (Gore, Lee 2007). In partnership with industry leaders, the committee is assuring that the CMT 408 and 305 courses, as well as the Communication Technology major as a whole, stays abreast of the technology needs and wants of industry so graduates can be prepared accordingly. With the advice, assistance, and direction of this committee, EMU continues to provide students with simulations in the CMT 408 and 305 courses that are true to life, giving them an opportunity to practice skills and understand consequences that will be applicable in their post graduate experiences. To continue to expand on the use of rubrics in business simulations, a graduate level course needs to be developed to detail and explore the process of creating

criteria that would be most effective in evaluating needs and creating a decision-making rubric which will increase the success of a business. References Bonwell, Charles C., Eison, James A. (1991) Active Learning: Creating Excitement in the Classroom. National Teaching and Learning Forum. Available [online]: http://www.ntlf.com/html/lib/bib/91-9dig.htm Gagne, R. (1985). The Conditions of Learning (4 th ed.). New York: Holt, Rinehart & Winston. Gendron, M., Jarmoszko, A.T. (2003) Teaching Data Communications and Networking to School of Business Undergraduates: A Pedagogical Framework for Juxtaposing Theory and Practice. In Proceedings of Informing Science IT Education (InSITE) Conference, 2003. Gore, D. & Lee, M. (2007). VoIP: A Curriculum Approach to Technology Convergence. In Proceedings of ED-MEDIA 2007 World Conference on Educational Multimedia, Hypermedia & Telecommunications. Jonassen, David. (1999) Designing Constructivist Learning Environments. Instructional-Design Theories and Models: A New Paradigm of Instructional Theory Volume II. Edited: Charles M. Reigeluth. Publisher: Lawrence Erlbaum Associates, Inc. Mahwah, NJ. Potter, R.A. (2004) Winning an RFP. The CPA Journal 74.4. simulation. (n.d.). WordNet 3.0. Retrieved October 08, 2007, from Dictionary.com website: http://dictionary.reference.com/browse/simulation