ASSESSMENT REPORT FOR GENERAL EDUCATION CATEGORY 1C: WRITING INTENSIVE

ASSESSMENT REPORT FOR GENERAL EDUCATION CATEGORY 1C: WRITING INTENSIVE March 28, 2002 Prepared by the Writing Intensive General Education Category Course Instructor Group

Table of Contents Section Page I. Introduction and Purpose of the Assessment... 1 II. Assessment Methodology... 1 Selection of GECCIG Members... 1 Drafting of the Assessment Rubric... 1 Selection of Student Writing Samples... 2 Training of GECCIG Members... 3 Assessment Process... 3 III. Assessment Results... 4 Inter-rater Reliability... 4 Descriptive Statistics... 5 Summary of the Data Analysis... 6 IV. Recommendations for Improving the GECCIG Assessment Process... 7 V. Recommendations for Improving the Writing Intensive Program... 8 VI. Conclusion... 9 List of Tables Title Page Table 1: Definition of the Scoring Rubric... 2 Table 2: Fall 2001 Writing Intensive Courses & Origin of Student Portfolios... 3 Table 3: Inter-rater Reliability of Writing Skills... 4 Table 4: Inter-rater Reliability of Organization Skills... 5 Table 5: Inter-rater Reliability of Sources and Evidence... 5 Table 6: Average Essay Ratings... 5 Table 7:Writing Skills... 6 Table 8: Organization Skills... 6 Table 9: Sources and Evidence... 6 i

I. Introduction and Purpose of the Assessment The assessment of Writing Intensive course writing assignments is part of a Minnesota State University, Mankato-wide effort to determine if students are learning what is being taught in the general education courses and to determine if the general education program is meetings its stated objectives. The information obtained as a result of the assessment process can be used to provide state agencies, governmental bodies, disciplinary accrediting groups, and faculty in departments offering general educations with outcome-based data to determine the degree to which the objectives of the general education program are being achieved. Finally, as with all assessments, the over-arching purpose is to improve students learning. The General Education Program at MSU,M stipulates that all students must successfully complete one writing intensive course. The 2001-2002 Undergraduate Bulletin stipulates that the goal of the writing intensive requirement is as follows: Students will continue to develop skills taught in Composition, applying them in the context of a particular discipline. Students will be able to: (a) use writing to explore and gain a basic familiarity with the questions, values and analytical or critical thinking methods used in the discipline; (b) locate, analyze, evaluate, and use source material or data in their writing in a manner appropriate to intended audiences (popular or within the discipline (p.25). Currently there are 33 courses that satisfy the writing intensive category, representing the following 11 academic disciplines: biology, English, history, humanities, music, philosophy, political science, computer engineering, social work, women s studies, and urban and regional studies. The objective of the General Education Category Course Instructor Group (GECCIG) was twofold: to develop a cross-discipline assessment protocol and to evaluate the extent to which student are achieving the aforementioned goal and objectives of the writing intensive program. II. Assessment Methodology Selection of GECCIG Members In the fall of 2001 a meeting was held of representatives from each of the departments offering a writing intensive course during the current semester. GECCIG membership was determined consisting of five faculty representing the departments of English, history, philosophy, political science, and social work. Membership on the Writing Intensive GECCIG was voluntary. In addition, the GECCIG was cochaired by two members of the General Education Committee. Drafting of the Assessment Rubric The writing intensive assessment rubric is predicated on the two objectives of all writing intensive courses. The co-chairs of the GECCIG drafted the initial rubric. A series of revisions were undertaken based upon the feedback of the five GECCIG members in order to enhance both the reliability and validity of the measure. The rubric was designed to assess students skills in the areas of basic writing skills, organization, and sources and evidence. A rating scale consisting of three rankings was selected to delineate the quality of the writing (refer to Table 1). A score of 1 indicated that the essay represented work that is marginal to poor, falling short of the rubric in significant ways. A rating of 2 indicated that the essay represents competent work, meeting the rubric. The rating of a 3 indicated that the essay while not flawless, demonstrated very good work. 1

Table 1: Definitions of the Scoring Rubric Writing Area Rating of a 1 Rating of a 2 Rating of a 3 Assessed Writing Skills The essay exhibits a pervasive lack of clarity and incoherence; or its errors in grammar, spelling, and punctuation often interfere with meaning; or errors are more numerous than are acceptable in college level writing. The essay contains some gaps in clarity or consistency; or contains enough errors in grammar, spelling and punctuation to sometimes interfere with meaning; or despite not interfering with meaning, more errors than are consistent with very good The essay is consistently clear and concise; exhibits a grasp of college-level writing skills; contains few errors in grammar, spelling and punctuation. Organization Sources and Evidence Selection of Student Writing No clear statement of objectives or sense of purpose. There is no clear development of ideas and arguments relevant to the purpose or there are serious and consistent inadequacies in these areas. A scarcity of credible sources and evidence and/or sources are not evaluated or shown to be relevant to the work s objectives. college writing. Writer seems to have a clear idea of what he or she wants to accomplish and what is required in order to accomplish it, but has not fully articulated this or has not successfully or consistently carried it off. Work may contain uneven statements of its objectives, development or relevant ideas and arguments, analyses, or conclusions Work uses a balance of credible sources, but gaps exist in critical evaluation, or credibility, or in demonstration of relevance to the work s objectives. Work contains a clear statement of its objectives, a proper development of relevant ideas and arguments. Key questions and proposed analyses are clearly formulated. Its conclusions are supported by cogent reasons. Work uses a balance of credible sources and evidence, is critically evaluated and relevant to the work s objectives. Twenty-one writing intensive courses were offered in the fall of 2001. An additional 12 courses satisfy the writing intensive general education category, however were not offered during the assessment period. Refer to Table 2 for a listing of courses. Departments offering and/or individual faculty teaching the 21 writing intensive courses were instructed to determine which assignments and/or activities in each course provided the most useful information for determining if the category learning outcomes were being met. The assignment was one already required for the course, a separate or additional assignment was not required for assessment purposes. Course instructors were directed to submit the pre-selected writing assignment from each student whose Tech ID ended in eight. The writing samples represented work that had undergone a series of revisions and was the final version of the assignment. Student samples were received from eight departments. Refer to Table 2 for a listing of departments and writing intensive courses. Writing assignments were collected from a total of 86 students. Of those writing assignments collected, a sample of 50 were selected for evaluation by the GECCIG. An attempt was made to ensure that each grouping of 10 writing assignments included an equal number of essays from each of the department. However, this was not always possible because some assignments did not have course identifiers. 2

Table 2: Fall 2001 Writing Intensive Courses Department Course Name and Number Biology Biology of Women (102) Introduction to Biotechnology (103) English Introduction to Poetry & Drama (112) Introduction to Prose Literature (113) Perspectives: Human Diversity & Literature/Film (211) Perspectives: World Literature/Film (212) Perspectives, Ethics and Civic Responsibility in Literature/Film (213) Perspectives in Literature/Film (214) Introduction to Creative Writing (242) History Ancient World Civilization in 1500 (170) European History to 1648 (180) Humanities Humanities Traditions (280) Global Perspectives and Humanities Traditions (282) Philosophy Introduction to Philosophy (100) Introduction to Ethics (120) Culture, Identity, and Diversity (205) History of Philosophy: Ancient (334) Political Science Thinking About Politics (103) Social Work Introduction to Social Welfare Services (190) Urban & Regional Studies Community Leadership and Service Learning (230) Training of GECCIG Members GECCIG members underwent a half-day training on the assessment process. The co-chairs of the GECCIG provided a tutorial on the rubric. The co-chairs for the pre-assessment training selected five different types of writing samples from the departments of biology, philosophy, humanities, political science and social work. Each of the five members of the GECCIG independently rated the writing samples. Upon completion of the independent ratings, a discussion ensued among members to determine the degree of agreement and to determine the reasons for disagreement on the rankings. Minor changes in the scoring rubric were recommended and subsequently made based upon the results of the preassessment. Assessment Process The 50 writing samples were divided into five groupings of ten. Each of the five GECCIG members was assigned to independently rate two of the five groupings, totally 20 writing samples. This process resulted in each writing sample being read and rated by two members of the GECCIG. Upon completion of the formal assessment process the GECCIG discussed the overall quality of the writing samples and assessment process. 3

III. Assessment Results Inter-Rater Reliability The results are based upon the assessment of the two independent evaluators of 49 writing assignments in three areas: writing skills, organization, and sources and evidence. Each of the three writing areas was assessed on a scale from one to three. In order to determine the reliability of the methodology of using two independent raters to assess the quality of writing, crosstabulation was calculated and the degree of inter rater reliability was determined using the Spearman Rank Statistic. The Spearman Rank Statistic provides a correlation coefficient based on the degree of perfect rater agreement, resulting in a more stringent analysis of inter-rater agreement. There was perfect rater agreement on 29 or 59 percent of the 49 essays rated in the category of writing skills. Bolded numbers in Table 2 represent perfect rater agreement. Raters differed by one rating on 18 or 37 percent of the 49 essays. For example, on two of the essays, rater 1 gave a score of 2 and rater 2 gave a score of 1. On only two essays was there a two-point discrepancy between raters. Rater 1 rated the two essays a 1 and rater 2 a 3, meaning that one rater assessed the essays as not meeting the standard of college level writing skills and the other rater determined that the essays had clearly met the standard of college-level writing. The Spearman Rank correlation coefficient of.50 indicates a moderate degree of inter-rater reliability. However, it is important to note that the raters were evaluating writing assignments in a wide variety of disciplines, the majority in disciplines other than that of their own area of expertise. In addition, the assessment process was in a formative stage, meant to establish a baseline for subsequent evaluation of general education writing intensive course essays. Given the interdisciplinary nature of the process, it is reasonable to expect that there would be some level of disagreement, especially when one rater was more familiar with the subject matter of the essay than the other rater. Table 3: Inter-Rater Reliability of Writing Skills Rater 2 Rater 1 Rating of a 1 Rating of a 2 Rating of a 3 Total Rating of a 1 8 6 2 16 Rating of a 2 2 15 6 23 Rating of a 3 0 4 6 10 Total 10 25 14 49 Table 3 presents the results of the analysis of essay organization. There was perfect rater agreement on 27 or 55 percent of the 49 essays rated in the category of organization. Bolded numbers in Table 3 represent perfect rater agreement. Raters differed by one rating on 19 or 39 percent of the 49 essays. On three essays there was a two-point discrepancy between raters. Rater 1 rated the two essays a 3 and rater 2 a 1 and on one essay rater1 assigned a 1 and rater 2 a 3, meaning that there was a difference of opinion regarding whether the organization of the essay reflected the college-level standard as defined in the rubric. The Spearman Rank correlation coefficient of.51 indicates a moderate degree of inter-rater reliability. Again it is important to note that the strength of inter-rater reliability was likely impacted by having raters from different disciplines (i.e., philosophy, english, social work, psychology and history) assessing essays from an array of disciplines (e.g., social work, biology, philosophy, political science). 4

Table 4: Inter-Rater Reliability of Organization Skills Rater 2 Rater 1 Rating of a 1 Rating of a 2 Rating of a 3 Total Rating of a 1 8 7 1 16 Rating of a 2 3 8 4 15 Rating of a 3 2 5 11 18 Total 13 20 16 49 In the area of sources and evidence, the number of perfect matches between raters was the lowest. There was perfect rater agreement on 25 or 51 percent of the 49 essays. Bolded numbers in Table 4 represent perfect rater agreement. Raters differed by one rating on 22 or 45 percent of the 49 essays. On two essays there was a two-point discrepancy between raters. Rater 1 rated an essay a 3 and rater 2 a 1 and on one essay rater1 assigned a 1 and rater 2 a 3, meaning that there was a difference of opinion regarding whether the use of credible sources and evidence reflected a college-level standard as defined in the rubric. The Spearman Rank correlation coefficient of.53 indicates a moderate degree of inter-rater reliability. Again, it is likely that the results are impacted by the interdisciplinary nature of the assessment model. Table 5: Inter-Rater Reliability of Sources and Evidence Rater 2 Rater 1 Rating of a 1 Rating of a 2 Rating of a 3 Total Rating of a 1 8 9 1 18 Rating of a 2 6 6 2 14 Rating of a 3 1 5 11 17 Total 15 20 14 49 Descriptive Statistics In order to evaluate the overall quality of the essays a primary rater was selected for each of the 50 writing assignments. Each of the five raters was a primary rater on ten of the writing assignments. Means and frequencies were calculated. Table 5 presents the mean scores and standard deviations for the three areas assessed. Table 6: Average Essay Ratings Writing Area Assessed Mean S.D. Writing Skills 1.98.71 Organization Skills 2.10.79 Sources and Evidence 1.94.79 The total average score of all 50 of the essay in the writing skills area was 1.98 (one a scale of 1 to 3 with 3 representing the highest possible score), with a standard deviation of.71. Table 6 represents the results of the frequency analysis of writing skills. Almost one-quarter of the essays were determined to demonstrate very good work in the area of writing skills, with three-quarters at least meeting basic college-level competency in writing skills. Approximately one-quarter of the essays were determined to not meet the minimum standard for college level writing. 5

Table 7: Writing Skills (N=50) Rating & Definition Number Percent 1 Marginal work, falls short of college level standards. 13 26 2 Competent work, meets college level standards. 25 50 3 Not flawless, but demonstrates very good work at the expected college-level. 12 24 Total 50 100 The total average score of all 50 of the essays in the organization skills area was 2.10, with a standard deviation of.79. The results of the frequency analysis are presented in Table 7. Over one-third of the essays were determined to demonstrate very good work in the area of organizations skills. An additional 38 percent were determined to meet basic competency at the college-level. Similar to the area of writing skills, one-quarter of the essays did not meet reflect college level writing in terms of organization skills. Table 8: Organization Skills (N=50) Rating & Definition Number Percent 1 Marginal work, falls short of college level standards. 13 26 2 Competent work, meets college level standards. 19 38 3 Not flawless, but demonstrates very good work at the expected college-level. 18 36 Total 50 100 Overall, the students as a group demonstrated the least competency in the area of sources and evidence. The total average score of all 50 of the essays in the sources and evidence skills area was 1.94, with a standard deviation of.79. Table 8 represents the results of the frequency analysis of sources and evidence. Slightly over one-third of the essays did not meet the standard for college-level writing. Almost 40 percent met the basic requirement for college level writing, with an additional 28 percent reflecting very good work in using credible sources and evidence to support the written work. Table 9: Sources and Evidence (N=50) Rating & Definition Number Percent 1 Marginal work, falls short of college level standards. 17 34 2 Competent work, meets college level standards. 19 38 3 Not flawless, but demonstrates very good work at the expected college-level. 14 28 Total 50 100 Summary of the Data Analysis It is difficult to definitively interpret the results given several compounding factors that may be impacting the validity of the findings. One of the factors has to do with the raters and methodology. The raters represented five different academic disciplines; knowledge of the subject matter may have impacted the results. In addition, essays were assessed from disciplines in which none of the raters was familiar. Another factor to consider is the actual scoring rubric. For example, the differences in the results of the three areas assessed may reflect differing degrees of difficulty in evaluating basic writing skills, organization skills, and sources and evidence. Finally, the type of writing assignments varied (e.g., term paper, chemistry laboratory report, creative writing). It is possible that some of the writing assignments assessed were more easily evaluated using the scoring rubric. Upon completion of the assessment process, evaluators discussed the strengths and challenges of the process. Concern was raised that what was being measured may be the writing skills the students had developed in high school rather than the development or improvement of those skills due to MSUM writing intensive courses. In addition, because of limited information about the nature of the specific course assignments and the details of the instructor s standards, evaluators concurred it was somewhat difficult to confidently evaluate organization 6

and sources and evidence. However, these limitations are not significant enough to discount the results of the assessment. The results of the inter-rater reliability analysis indicate a moderate degree of reliability between the two independent raters assessing each essay. The Spearman Rank correlation coefficients obtained in each of the three areas can be used as a baseline by which to measure improvements in the assessment process (e.g., refinement of the rubric, training of raters) of subsequent evaluation of general education writing intensive course essays. The percent of perfect matches between raters was 51 percent or more in all three areas. While there was some level of disagreement between raters, extreme differences of opinion, as indicated by a two-point spread between raters, occurred infrequently. This result indicates the overall positive merits of the assessment process and lends credible support for the validity of the findings pertaining to students writing abilities. The results of the quality of the writing assignments indicate that 74 percent of the students who wrote the essays evaluated, were writing at least at a minimally acceptable college-level in the areas of basic writing skills and organization. In addition, two-thirds of the students were using a balance of credible sources and evidence that met the minimum standard for college-level writing. Very good work was demonstrated by at least one-quarter of the students in all three areas assessed, with organization skills determined to be the main area of strength. However, slightly more than one-quarter of students represented in the sample fell short of the college-level standard in writing skills and organization, with one-third falling below the standard in providing credible sources and evidence to support their assertions. IV. Recommendations for Improving the GECCIG Assessment Process Upon completion of the assessment process and tabulation of results, GECCIG members discussed strategies for improving the interdisciplinary general education writing intensive program assessment process in subsequent evaluative undertakings. Recommendations for improving the process are as follows: 1. Writing intensive course instructors should receive a copy of the scoring rubric in advance in order to determine the most appropriate type of writing assignment to select for submission. 2. Instructors of all writing intensive courses offered during the assessment period must submit all student essays meeting the sampling criteria. 3. Writing intensive course instructors must ensure that the final version of writing assignments, reflecting revisions made in response to faculty comments, be submitted for evaluation. 4. More specific information from the individual faculty or department needs to be supplied about the nature of the assignment and the relevant criteria for quality. Methods for supplying the information include providing a written narrative and a sample of a high quality assignment, and/or a brief presentation to the GECCIG. 5. Rubric revision. The rubric may need some revision to be more coherently applicable to diverse academic fields (e.g., biology, philosophy, social work) and assignments as disparate as lab reports, creative writing, and discipline based analyses. Some of the problems applying the rubric may be overcome by more applicable selection of writing assignments. Consideration should be given to modifying the rating from a 3-point to a 4-point scale. 7

The standard writing skills taught in English 101 should be provided to the GECCIG to aid in the modification of the rubric, thereby creating greater consistency and coordination between the two general education programs that focus on student writing skills. 6. The GECCIG should strive to improve the inter-rater consistency. All of the aforementioned recommendations would be beneficial towards improving the inter-rater consistency. The pre-assessment training should be more expansive and should entail rating and discussing the results amongst the entire GECCIG of a larger number of sample writing assignments. The assessment process should not begin until the GECCIG has scored a Spearman Correlation Coefficient of.60 or better in each of the writing skills areas. 7. A larger sample of student essays should be assessed. 8. The GECCIG should include a representative from each of the departments in which writing assignments are being evaluated. V. Recommendations for Improving the Writing Intensive Program Upon completion of the assessment process and tabulation of results, GECCIG members discussed strategies for improving the General Education Writing Intensive Program. The recommendations for improving the program are predicated in the dual-focus of the writing intensive course, that is improving writing skills and teaching the subject matter of the course. Instructors have an additional burden to ensure that students who are also taking the course to fulfill a requirement of the major not only achieve the writing intensive program objectives, but also master the subject matter, which is often a foundation for subsequent courses. Instructors need additional resources in order to achieve the lofty objectives set by individual departments and MSU,M as a whole. Recommendations for improving the Writing Intensive program are: 1. English Composition 101 should be a prerequisite for enrolling in a writing intensive course. 2. Students should be required to fulfill the writing intensive course requirement within three semesters following the completion of English Composition 101. 3. English Composition 270 or 271 or a second writing intensive course should be a graduation requirement for all students. 4. A forum needs to be created to create greater communication and collaboration between English Composition 101 instructors and writing intensive course instructors. 5. A forum for writing intensive course instructors should be created and offered early in the fall of each academic year. The purpose of the forum should include developing skills in designing, teaching and evaluating writing assignments. 6. Faculty in courses not designated as writing intensive should make every effort to emphasize discipline specific critical thinking techniques, the proper use of scholarly sources, and college-level writing skills. 7. A fully staffed MSU,M Writing Intensive Resource Center should be created and made available to all students in writing intensive courses, with special resources for students for which English is a second language and for others with unique writing challenges. 8. Incentives and recognition are needed for faculty and departments committed to the General Education Writing Intensive Program. Possibilities include: 8

Lower variable-credit targets for departments with writing intensive courses. An FTE multiplier for writing intensive courses. Overload-pay or reduced teaching loads for faculty with writing intensive course responsibilities. Optional extra duty days at the beginning of each semester for course preparation and revision. Additional MSU, M funded faculty development opportunities. An General Education Writing Intensive Course Instructor award should be created and given on an annual basis. The award should be given during at a university-wide forum (e.g., spring convocation). A widely utilized and visible location on campus should be selected to display the name of recipients. A cash prize and/or a one-time scholarship in the recipient s name, for a student in the recipient s department, should be awarded. 9. Financial resources are needed for departments to hire qualified and MSU, M trained graduate assistants and upper-level undergraduate students to assist in reviewing drafts of writing intensive course assignments. 10. The results of the first writing intensive evaluation should be widely circulated and discussed in multiple forms. MSU,M commitment to the centrality of writing should be addressed by the President at faculty-wide and student-wide forums (e.g., fall convocation). VI. Conclusion The Committee has put forward recommendations to improve the assessment process and to increase inter-rater reliability, thereby making this assessment more meaningful across disciplines. These improvements should assist future assessors in being able to speak with greater confidence about the writing skills of our students and the impact that writing-intensive general education courses are having on the enhancement of those skills. With regard to substantive issues, the Committee notes with concern that a substantial number of those writing assignments sampled did not evidence satisfactory writing at the college level. The substantive recommendations of the Committee are aimed at making writingintensive general education courses more effective by providing greater coordination with the English Department on composition courses. The Committee wishes to call the attention of the university community to the fact that the improvement of writing skills and critical thinking among our students requires a commitment by the entire community. Resources should be made available to facilitate greater faculty participation in writing and critical thinking courses, as well as to make additional tutorial resources available to those student populations which may need somewhat more help in acquiring college level writing skills. Finally, the Committee strongly urges the university community to foster discussion of our common objectives and the challenges we face in enhancing the writing skills and critical thinking of our students. Submitted by: Richard Liebendorfer, Associate Professor, Philosophy Department, GECCIG Co-Chair Anne O-Meara, Professor, English Department, GECCIG Co-Chair Terry Davis, Professor, English Department, GECCIG Member Nancy Fitzsimons-Cova, Assistant Professor, Social Work Department, GECCIG Member Margaretta Handke, Professor, History Department, GECCIG Member Jacqueline Vieceli, Professor, Political Science Department, GECCIG Member Hal Walberg, Professor, Philosophy Department, GECCIG Member 9