Victor M. Catano a & Steve Harvey b a Department of Psychology, Saint Mary s University, Halifax, Nova

Transcription

1 This article was downloaded by: [Bibliothèques de l'université de Montréal] On: 10 December 2012, At: 07:08 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: Registered office: Mortimer House, Mortimer Street, London W1T 3JH, UK Assessment & Evaluation in Higher Education Publication details, including instructions for authors and subscription information: Student perception of teaching effectiveness: development and validation of the Evaluation of Teaching Competencies Scale (ETCS) Victor M. Catano a & Steve Harvey b a Department of Psychology, Saint Mary s University, Halifax, Nova Scotia, Canada, B3H 3C3 b School of Business, Bishop s University, Lennoxville, Quebec, Canada Version of record first published: 15 Jun To cite this article: Victor M. Catano & Steve Harvey (2011): Student perception of teaching effectiveness: development and validation of the Evaluation of Teaching Competencies Scale (ETCS), Assessment & Evaluation in Higher Education, 36:6, To link to this article: PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

2 Assessment & Evaluation in Higher Education Vol. 36, No. 6, October 2011, Student perception of teaching effectiveness: development and validation of the Evaluation of Teaching Competencies Scale (ETCS) Victor M. Catano a * and Steve Harvey b a Department of Psychology, Saint Mary s University, Halifax, Nova Scotia, Canada B3H 3C3; b School of Business, Bishop s University, Lennoxville, Quebec, Canada CAEH_A_ sgm / Assessment Original Taylor Dr. vic.catano@smu.ca VictorCatano and & Article Francis (print)/ x & Evaluation Higher (online) Education A major criticism of student evaluations of teaching is that they do not reflect student perspectives. Using critical incidents job analysis, students identified nine teaching effectiveness competencies: communication, availability, creativity, individual consideration, social awareness, feedback, professionalism, conscientiousness and problem-solving. The behaviourally anchored Evaluation of Teaching Competencies Scale is a highly reliable (alpha =.94), unidimensional measure that correlated strongly with an instructor-related composite of the Students Evaluation of Educational Quality (SEEQ, r =.72), but not to a SEEQ composite related to instructor assigned work (r =.04, N = 195). The results are discussed in the context of other measures of teaching effectiveness and transformational leadership theory. Keywords: student evaluation of teaching; teaching effectiveness; leadership; teaching competencies Universities commonly use student evaluations of teachers (SETs) in soliciting feedback about teaching performance and, in fact, many schools rely on SETs to make decisions related to faculty retention, promotion, tenure and merit pay (Jacobs 2004). This latter use is controversial in that the majority of rating forms used to obtain the SETs have unknown or problematic psychometric properties and may not measure what they purport to measure (Simpson and Siguaw 2000). Indeed, the use of SETs for making faculty career decisions has raised many questions concerning the reliability, validity and potential bias associated with these evaluative measures (Simpson and Siguaw 2000). In response to these concerns, many unionised faculty associations prohibit their use in their collective agreements with their employers. Despite ongoing criticism (e.g. Jacobs 2004), SETs continue as a key performance indicator of teaching effectiveness and quality with considerable effort directed at making SETs reliable and valid (Marsh and Roche 1997; Toland and De Ayala 2005). Despite faculty criticism, it seems unlikely that university and college administrators will abandon SETs (Penny 2003). Penny notes that many of the SETs are ill-designed and lack construct validity, thus making any inferences drawn from them questionable. Indeed, sustained research on the construct validity of SETs has been lacking (Penny 2003). We accept that SETs will continue to be used to evaluate teaching performance; as such, a measure that is fully grounded in the job requirements of teaching stands the *Corresponding author. vic.catano@smu.ca ISSN print/issn X online 2011 Taylor & Francis DOI: /

3 702 V.M. Catano and S. Harvey best chance of attaining construct validity and adequately meeting the needs of those using SETs. As a measure that may be used for administrative purposes, the SET should assess critical functions of the job of teaching. Most SETs are simply derived from opinion of expert teachers and do not necessarily measure important job components. Items generated by faculty, while valid and informative, may not capture aspects of teaching that are important from a student s perspective, or that reflect student values. There is the need for student involvement at the first stage, without limiting their role to reviewing the traits or behaviours suggested by master teachers. Almost every existing SET measure has been built from the perspective of teachers acknowledged as having teaching expertise and they rarely include items generated by students. The two studies presented here outline the systematic development of a new SET following techniques frequently used in the practice of industrial and organisational psychology to establish effective job performance measurement systems. Study 1 focuses on the development of an SET measure grounded in complete and verified job-requirements; Study 2 focuses on convergent validity by correlating the resulting SET with other measures of teaching effectiveness. Study 1 The first potential problem with existing SETs is that they are not assuredly jobrelated and do not necessarily capture the work that is performed as part of the job. As such, the starting point in the development of any SET should be the a priori identification of the job dimensions that comprise the job of teaching. SETs have not made use of industrial and organisational psychology for identifying, a priori, job dimensions or competencies involved in teaching. The recognised best practice is to begin by conducting a thorough analysis of the job in question (Gatewood, Feild, and Barrick 2008). Accordingly, Study 1 used the Critical Incident Technique (CIT; Anderson and Wilson 1997; Gatewood, Feild, and Barrick 2008) to identify the major dimensions associated with effective teaching and the behavioural anchors that could be used in creating a measure to assess teaching performance. The CIT is a qualitative assessment procedure first developed by Flanagan (1954) and used in organisations for many different purposes. In the CIT, subject matter experts (SMEs) are asked to recall examples of either effective or ineffective behaviours and to report the antecedents of the behaviour, the behaviour itself and the consequences of the behaviour. This relies on information already possessed by the SMEs. Following the generation of scenarios, each incident is rated on the degree to which it represents an effective or ineffective behaviour on the part of the protagonist. When SMEs generate a sufficient number of incidents, a representative overview of the job emerges. The incidents are then edited into meaningful performance items that become the basis for a behavioural measure of the performance. SMEs independently sort the edited incidents into predetermined categories or allow the categories to develop empirically through the sorting process. If there is agreement from a majority or more of SMEs on the classification of an item into a category, the item is retained as an exemplar of the category or dimension. The CIT does not require extensive training on the part of the SMEs to generate critical incidents. The CIT has been used to create structured interviews, to design curricula, to evaluate services, to establish performance requirements, or as in the present study, to create a performance measurement instrument (Anderson and Wilson 1997). Our purpose

4 Assessment & Evaluation in Higher Education 703 was to obtain views from students and therefore we used primarily undergraduate students as SMEs, as the perspectives of teachers were well known through the existing SETs. Method Participants With instructor permission, SMEs were recruited directly from senior-level university classes in the arts, science and business faculties. The intent of the recruitment was to obtain experienced undergraduate students who were studying in a variety of disciplines. The end result was the participation of 65 (41 females) undergraduate students 1 from diverse areas and eight (four females) graduate students who were completing a master s degree in I/O psychology. The University s Research Ethics Board (REB) approved the study. Procedure The student SMEs attended CIT workshops in groups of five to 10 that were run by one of the graduate students. The standardised workshop materials followed the guidelines proposed by Anderson and Wilson (1997). These materials presented the rationale for the study, the goals and objectives of the workshop, and the requirements of the participants. The participants were given tips on writing critical incidents and examples of well-written and poorly written critical incidents. After receiving their instructions, the SMEs were asked to write as many critical incidents of effective or ineffective behaviour as they could in a two-hour time period. The SMEs were given a stack of critical incident report forms that asked the SMEs to describe: the situation that led up to the incident; what the teacher did in the situation; and the outcome or results of the teacher s actions. The SMEs also rated the incident on a seven-point scale from 1, highly ineffective, to 7, highly effective. The SMEs did not place any personal information on the report forms. The workshops generated 496 critical incidents. Next, each workshop leader typed and edited the answers to the three questions into a one paragraph narrative and corrected spelling and grammar. The eight graduate students then reviewed, as a group, all of the incidents with respect to their usability. In doing this, the graduate students followed the criteria laid out by Anderson and Wilson (1997); 74 items were dropped because they were redundant in duplicating another item and 84 were dropped because the content was too general and did not identify specific behaviours or outcomes, or did not address the three questions. All items were dropped by consensus. This procedure resulted in the retention of 338 edited critical incidents. Analysis Sorting of incidents. The CIT requires analysts to sort the retained incidents into categories or themes. The categories can be formed empirically or derived from predetermined dimensions. We used 12 teaching dimensions that had been identified in previous research (see Table 3) to guide the sorting of the incidents. The eight graduate students felt that 12 categories were sufficient to classify all of the incidents. Next, each graduate student independently sorted each of the retained incidents into one of the 12 categories. The results of these independent classifications were consolidated

5 704 V.M. Catano and S. Harvey into a master list. Generally, 60 70% inter-rater agreement for critical incident retention is considered appropriate for research purposes (Pulakos 1997). We accepted incidents as belonging in a category if five of the eight graduate students agreed on the assignment. This process resulted in the retention of 159 items and reduced the number of categories from 12 to nine. We labelled the nine categories: communication, availability, creativity, individual consideration, social awareness, feedback, professionalism, conscientiousness and problem-solving. Rating of critical incidents. The eight graduate students independently rated the retained critical incidents in terms of teaching effectiveness using a seven-point rating scale. Based on the mean rating, an incident was classified as representing low (M 2.49), moderate (2.50 M 5.49) or high (M 5.50) levels of teaching effectiveness. This rating format is the recommended classification procedure (Pulakos 1997) for developing behavioural summary scales (BSS). A BSS uses the critical incidents to develop behavioural anchors that are descriptive of low, moderate and high performance on the dimensions of interest (Pulakos 1997). Nine BSS were developed, each anchored by three behavioural descriptors, representing low, moderate or high levels of teaching effectiveness. Whenever possible, critical incident mean ratings that most closely approached 1.0, 4.0 and 7.0 were chosen to anchor one of the nine scales. Only items with a standard deviation up to and including 1.50 were deemed to have an acceptable range distribution. If effectiveness ratings were the same for two or more critical incidents, the incident with the smaller standard deviation was used (Pulakos 1997). The underlying principle of a BSS is to create a general picture that represents the specified effectiveness level of a dimension; therefore, after the statistical criteria for inclusion were satisfied, critical incidents at the same effectiveness level were combined to make the essence or meaning of the competency as understandable and explicit as possible; that is, the anchor statements were composites of behaviours listed in critical incidents that were given the same effectiveness rating. Results Table 1 presents the means and standard deviations of the low, moderate and high effectiveness anchors. In those cases where a composite was used, the values represent the averages of the combined incidents. Table 2 presents the final set of teaching effectiveness scales for the nine competencies, including a definition of each competency and its behavioural anchors. We used a seven-point scale and allowed multiple values for each effectiveness category for greater variance in assigned ratings. Content validation Table 3 compares the teaching competencies found in the present study to those reported by Cohen (1981), Fulton (1996) and Marsh and Roche (1997), which were derived from teaching masters opinions. We compared not only the label given to the dimensions but also to the assigned definitions. Accordingly, seven of the nine teaching competencies identified through our job analysis map onto at least one dimension identified in the previous studies. Individual consideration mapped on to

6 Assessment & Evaluation in Higher Education 705 Table 1. Means and standard deviations of behavioural summary statements used to anchor the nine teaching competencies. Means (SD) of scale anchors Competencies Low Medium High Availability 1.13 (.36) 2.67 (.50) 6.78 (.44) Communication 1.89 (.93) 2.88 (.64) 6.00 (.71) Conscientiousness 1.56 (.73) 2.78 (.78) 6.63 (.74) Creativity 1.66 (.87) 3.33 (1.14) 6.44 (.73) Feedback 1.89 (1.05) 3.33 (.71) 6.75 (.46) Individual consideration 1.78 (1.09) 2.89 (1.17) 6.67 (.5) Professionalism 2.00 (.50) 4.00 (.93) 6.38 (.74) Problem-solving 2.22 (.83) 3.67 (.71) 6.44 (.73) Social awareness 1.39 (.71) 5.33 (1.12) 6.12 (.73) dimensions identified in all three of the other studies while conscientiousness and professionalism were similar to dimensions in two of the other studies. Communication, creativity, social awareness and feedback matched a dimension reported in one of the other studies. Two competencies, problem-solving and availability, are new and were not identified in previous SET research. As a final assessment of content validity, we mapped the 28 items from the Teachers Behaviour Checklist (TBC; Keeley, Smith, and Buskist 2006) on to the nine dimensions. All 28 items mapped on to eight of the nine competencies; however, the distribution was not uniform. Five to eight TBC items mapped on to communication, conscientiousness and creativity while two or three items mapped on to individual consideration, professionalism and social awareness. One item from the TBC mapped on to availability and feedback ; none of the TBC items mapped on to problem-solving. Discussion The purpose of Study 1 was to develop an SET measure based on a job analysis. We identified nine competencies that students considered important for teaching effectiveness that for the most part captured dimensions of teaching effectiveness identified by master teachers (Cohen 1981; Fulton 1996; Marsh and Roche 1997; Keeley, Smith, and Buskist 2006). The similarities provided preliminary support for the content validity of the new SET. The students identified two additional dimensions of problem-solving and availability; dimensions that are clearly important to students and represent conscientious teaching. The scales represent the perspective (i.e. language) of students, which Cunningham and MacGregor (2006) found to be important for face validity. The more interesting aspects of Table 3 are the dimensions identified in previous studies that do not map on to the nine competencies. Most of these relate to the course mechanics, the instructor s style or the student s perception of the instructor s knowledge of the course material. We believe it is the unmapped dimensions that lead to controversy surrounding the use of SET. For example, why should a student taking an introductory course be assumed competent to speak about whether the instructor has

7 706 V.M. Catano and S. Harvey Table 2. Evaluation of teaching competencies scale. (1) Availability: To take necessary measures in order to be accessible to others Low Medium High The instructor presents office hours on the syllabus, yet is unavailable during specified times, and contact information is not available 1 2 When the instructor is unavailable during office hours, the professor compensates by using teaching assistants to take care of student concerns The instructor is regularly available to provide teaching assistance beyond required lecture time and office hours 6 7 (2) Communication: To display verbal and written eloquence and flexibility based on the type of audience, to communicate with clarity, precision and purpose, and to take the time to listen to others and decipher relevant points Low Medium High The instructor has a monotone voice, depends on written notes when lecturing and has difficulty answering student questions 1 2 Sometimes instructions, lectures and assignments are a bit unclear The professor speaks clearly, listens to students intently and precisely answers students questions 6 7 (3) Conscientiousness: To produce high quality work by paying strict attention to details, perceiving what others have overlooked and rejecting whatever does not meet the criterion of excellence Low Medium High The instructor often arrives unprepared and/or without the proper materials. Course materials are ill prepared and/or outdated 1 2 The instructor does not cover all the material outlined in the course syllabus and sometimes seems a bit unprepared The instructor provides a detailed and complete syllabus, covers all necessary information and is very well prepared 6 7 (4) Creativity: To promote innovative ideas and new processes by being flexible in personal attitudes, rules or ways of operating Low Medium High The instructor uses the same teaching style in every class, and is inflexible to change 1 2 At times, the instructor varies the lecture style, such as by showing a video The instructor uses multiple teaching methods, such as applied assignments and group discussions 6 7

8 Assessment & Evaluation in Higher Education 707 Table 2. (Continued). (5) Feedback: To react verbally or in writing to the behaviour or performance of others. To suggest corrective measures when necessary Low Medium High The instructor marks assignments without providing specific information as to the points that were lost 1 2 The instructor provides feedback to students at a point when there are few opportunities for improving their final grades The instructor provides timely, detailed feedback, thereby helping the students to progress beyond the course requirements 6 7 (6) Individual consideration: To show sensitivity and empathy by accommodating the needs of others, and to provide encouragement and personal attention towards others Low Medium High The instructor does not provide individual assistance to students, and appears uncompassionate and hurried when encountered for help 1 2 The instructor is usually available for help but is sometimes reluctant to provide individual assistance or consideration to students The instructor goes far beyond what is expected to help individual students, such as using extra time to further student motivation and education 6 7 (7) Professionalism: To demonstrate honesty and integrity by being congruent in words and actions, displaying a sense of fairness and justice and maintaining confidentiality of information received Low Medium High The instructor changes components of the course without consulting students or grades work using completely unfair methods 1 2 The instructor is generally fair and just, but can be inconsistent at times when grading tests and assignments or when discussing course objectives The instructor is completely fair and just, such as seeking complete consent when changing course components and fully clarifying exam and assignment requirements 6 7

9 708 V.M. Catano and S. Harvey Table 2. (Continued). (8) Problem-solving: To analyse problems, seek suggestions for effective solutions and decide on corrective measures to be taken Low Medium High The professor refuses to reach consensus with the class when issues are raised, such as the appropriateness of syllabus content or making new arrangements for assigned work 1 2 The professor generally deals with student concerns effectively, yet solutions are not always universally accepted, such as omitting material to compensate for lost time The professor effectively deals with issues that impede learning, such as rearranging lab time due to class oversize or facilitating group discussions when the material is not clearly understood 6 7 (9) Social awareness: To be in tune with the thoughts and feeling of others and be tactful while maintaining discipline Low Medium High The professor fails to maintain classroom civility and does not recognise discomforts or disapproval he or she creates 1 2 The professor generally maintains civility, but sometimes singles out students for their mistakes and treats some sensitive issues awkwardly The professor seems to completely understand the feelings of the students, whereby sensitive topics are discussed tactfully and civility is maintained without singling out individual students 6 7

10 Assessment & Evaluation in Higher Education 709 Table 3. Comparison of the ETCS competencies to previously identified teaching dimensions. ETCS Cohen (1981) Fulton (1996) Marsh and Roche (1997) Individual consideration (Shows sensitivity and empathy to needs of others) Conscientiousness (Pays attention to details; organised) Professionalism (Demonstrates honesty and integrity in dealing with students/grading) Communication (Verbal, written and listening skills) Creativity (Promotes innovative ideas through use of different teaching styles) Social awareness (Sensitive to student feelings while maintaining discipline) Feedback (Provides verbal or written comments on students work) Problem-solving (Deals in an effective way with issues impeding classroom learning) Availability (Accessible to students) Rapport (Empathy, friendliness) Structure (Planning and organising) Feedback (Concern with students work in progress) Respectful and caring attitude Individual rapport (Friendly to students; availability) Organisation/clarity (Well-prepared materials, facilitates note-taking) Fairness in grading Examinations/grading (Fair exams, tested material in course) Communication skills Openness to different teaching methods Receptive to critical thinking and criticism Skill (Mastery of subject) Expert knowledge of subject Breadth of coverage (Presents different points of view/ theories)

11 710 V.M. Catano and S. Harvey Table 3. (Continued). ETCS Cohen (1981) Fulton (1996) Marsh and Roche (1997) Interaction (Feel free to participate in class) Difficulty (Workload demands placed on students) Group interaction (Feel free to participate in class) Workload/difficulty (Workload demands placed on students) Enthusiasm Enthusiasm (Dynamic, energetic, use of humour) Assignments/readings (Course readings and assignments seen as valuable) Learning (Value of and interest in course)

12 Assessment & Evaluation in Higher Education 711 covered all possible theories or points of view on the course topic? Group participation/interaction assumes participation is possible in every class, even large lectures with 500 students. Is the requirement to use humour just capitulation to the Dr Fox effect? Similarly, why should effectiveness depend on the difficulty of the course material and the workload? The comparison of the nine competencies to the TBC is also instructive. Nineteen of the 28 items map on to three of our dimensions suggesting that qualities related to communication, conscientiousness and creativity are overrepresented in the list of behaviours generated by teaching masters. The TBC qualities related to the competencies of availability, feedback, individual consideration, professionalism and social awareness are underrepresented while problem-solving is overlooked completely. Moreover, it is important to note that the job analysis presented here has identified those aspects of teaching that students, not teaching masters, believe are critical to teaching effectiveness. Using the competencies developed here may overcome some of the negative views towards existing SETs. Study 2 Study 2 examined the reliability, validity and factor structure of the new Evaluation of Teaching Competencies Scale (ETCS). We used the Students Evaluation of Educational Quality (SEEQ), one of the better and more popular standardised SETs, to examine the construct validity of the ETCS. The SEEQ is a 40-item scale developed to measure nine dimensions associated with teaching effectiveness plus a section on course load/difficulty and another on course demographics. One of the SEEQ subscales is used to obtain an overall rating of the professor and the course. We expected that the ETCS would positively correlate with the SEEQ and with its dimensions because of the overlap with SEEQ dimensions (see Marsh and Roche in Table 3). Students were asked to think of one professor whom they had taken a course from within the past 12-month period, but from whom they were not currently taking a course, and to complete both SETs holding that professor in mind. They were asked not to identify either the professor or the course in any way, a requirement of the REB. The REB would not allow us to administer the ETCS along with the SEEQ as part of the normal teaching evaluation process as there was concern that the additional measure would influence the SEEQ results, which are used in renewal, tenure and promotion processes. Method Participants and procedure A total of 195 students (98 females; 97 males) at an Atlantic Canadian university participated in the study. Participants ranged in age from 18 years to 60 years, although 87% of the students were between 18 and 25 years old. Most (52.5%) of the participants were in their second year of university, with 65% reporting a grade-point average (GPA) of 2.51 or higher. Approximately, 70% were in commerce disciplines with 30% in arts and science. The study had received approval from the university s REB. Participants completed the ETCS, the SEEQ (Marsh 1983) and a demographic questionnaire.

13 712 V.M. Catano and S. Harvey Results ETCS Factor structure. There was a high degree of correlation between the nine competencies with correlations ranging from.41 to.75, with half of the correlations greater than or equal to.64. A principal components factor analysis extracted one general factor with all nine competencies loading on that factor with component weights of.75 or greater. The factor accounted for 68% of the variance. Reliability. We considered the nine competencies, based on the factor analysis, as comprising one scale, the ETCS. The internal consistency of the ETCS was excellent (alpha =.94), with the item-total score correlations ranging from.73 to.89. Validity. Table 4 presents the correlations among the SEEQ subscales together with their Cronbach alphas. There is a high degree of correlation among the first seven SEEQ subscales. The ETCS composite correlated significantly with all of the SEEQ subscales. The significant correlations ranged from.16 to.66. Only the correlations with course difficulty/workload were negative. We replicated these results using a set of 35,568 SEEQs used for actual course evaluations at the participants university. The results parallel those found in the sample and are presented in the upper triangle of Table 4. Based on a principal components analysis of both the sample and population SEEQ scores, we formed three composites: (1) one based on all nine SEEQ subscales (SEEQ-9); (2) one based on the first seven subscales (SEEQ-7) of learning, individual rapport, enthusiasm, examinations, organisation, breadth and group interactions; and (3) a composite based on assignments and difficulty/workload (SEEQ-2). Table 5 presents the correlation matrix between these three variables, the ETCS and the demographic items. The ETCS correlated with the SEEQ-9 (r =.69) and the SEEQ-7 (r =.72), but not with the SEEQ-2 (r =.04). Like the total ETCS score, the SEEQ-7 did not correlate with the SEEQ-2 (r =.12). Both GPA (r =.30) and gender (r =.30) correlated moderately with the ETCS while gender, but not GPA, correlated weakly with the SEEQ-7 (r =.18) and SEEQ-9 (r =.18). In all cases, female students and those with higher GPAs provided higher ratings on either the ETCS or the SEEQ and its composites. Discussion Study 2 showed that the ETCS is a reliable and valid measure of teaching effectiveness, when compared to the SEEQ. Although the measure consisted of nine independently devised scales, the factor structure was unidimensional. This outcome is very similar to other studies where well-developed, multiple dimensions appear to have one underlying factor (Pulakos and Schmitt 1995; Huffcutt et al. 2001). It is arguable whether these results represent a halo effect or an assessment of a general performance factor underlying the multiple items. We believe that the ETCS is best considered as a global measure of teaching effectiveness with the nine different competencies contributing to the overall reliability of the scale. Both the composite ETCS and SEEQ-7 measures provide good estimates of teaching effectiveness. The scores from the individual competencies or subscales provide useful information on areas that may need improvement but by themselves

14 Assessment & Evaluation in Higher Education 713 Table 4. Intercorrelations among the SEEQ subscales and relationship to the ETCS Alpha (N = 195) Alpha (N = 35,568) 1. Learning.62**.67**.63**.70**.64**.57**.59**.15* Individual rapport.55**.73**.65*.67**.62**.68**.50* Enthusiasm.60**.64**.63**.72**.66**.66**.52** Examinations.61**.68**.53**.72**.62**.59**.59** Organisation.60**.65**.55**.71**.71**.63**.61** Breadth.58**.58**.51**.53**.61**.64**.56**.14* Group interaction.56**.72**.56**.54**.52**.66**.53** Assignments.48**.37*.25**.49**.51**.39**.33**.17* Workload/difficulty.05.19*.18* *.14* ETCS.51**.66**.55**.58**.63**.55**.56**.31**.16*.94 *p <.05; **p <.01. Notes: Correlations below the diagonal are from participants in Study 2 (N = 195). Correlations above the diagonal are from a population of university SEEQ cases (N = 35,568).

15 714 V.M. Catano and S. Harvey Table 5. Correlations between the ETCS and SEEQ composites and items. Mean SD ETCS a SEEQ SEEQ ** SEEQ **.31**.98* 5. GPA ** Age * 7. Gender.30**.04.18*.18*.34**.11 *p <.05; **p <.001 (two-tailed; N = 195). Note: a ETCS is measured on a seven-point scale. should not be used for decision-making purposes. The composite scores give better, more reliable estimates of teaching effectiveness. Our results also suggest that while both the ETCS and the SEEQ assess teaching effectiveness, they are two independent measures that assess different aspects of teaching effectiveness. The data in Table 5 clearly show that neither the ETCS nor the SEEQ-7 correlates with the SEEQ-2. This supports our argument that course factors should not form part of measures that evaluate the performance of the instructor. An assessment of the course mechanics such as assignments and workload may be valuable, but those evaluations should not be included in an overall assessment of teaching performance. It is not very surprising that gender and GPA correlated with both the ETCS and the SEEQ. Gender has had a mixed history where sometimes the sex of the respondent is important and other times it is not (Marsh and Roche 1997). In the present case while the correlations with gender were significant in both cases, the strength of those correlations did not indicate strong relationships. Both McKeachie (1997) and Greenwald (1997) found that grading leniency was related to ratings of teaching effectiveness. In this study, we did not ask students about their expectations but rather to think about one professor they had in the past when answering the questionnaire. We think it is more likely that the students would have recalled a very favourable experience with a professor who gave them a good grade. This is particularly the case as the average GPA of the students in the study is relatively high. Based on published data at the students school, the all-university average GPA was 2.64 compared to 3.10 for students participating in Study 2. Our conclusions on the validity of the ECTS would have been strengthened if we had been permitted to collect data from the ECTS along with the SEEQ as part of the ongoing evaluation process. General discussion Penney (2003) has argued that one of the shortcomings of research SETs is that not enough attention has been paid to the construct of teaching effectiveness and the psychometric properties of the rating forms used to measure that construct. The ETCS addresses both of these issues by first using well-established methodologies in industrial/organisational psychology to identify competencies that students felt embodied effective teaching and then by developing a psychometrically sound rating scale by various empirical tests. The nine competencies that we discovered overlap with

16 Assessment & Evaluation in Higher Education 715 dimensions identified in other attempts to measure effective teaching, but the ETCS omits several contentious dimensions and it includes two that other measures have ignored: problem-solving and availability. Based on the factor analysis, there appears to be one latent dimension that is measured by the nine competencies. We would argue that this latent, statistically derived dimension is teaching effectiveness. Rating scales used as part of SETs have been frequently criticised for not providing a standard definition of what good or poor means. Individuals may have different perceptions and rate the same instructor as good and poor when the students are observing the same behaviour. Behaviourally anchored scales like those in the ETCS use empirically derived examples of good and poor behaviours to define what is meant by good and poor. Behaviourally anchored rating scales provide psychometrically sound information. The use of the behavioural anchors fosters consistency in responding and is responsible for the high level of reliability we found with respect to the ETCS. What is an effective teacher? Bass (1985; Avolio and Bass 1988) theory on transformational leadership is one of the most influential leadership theories. Transformational leaders build exchange relationships with their followers and engage the followers emotionally to build identification, commitment and trust in the leader or the mission. They easily communicate the importance and values of stated objectives and the goals they set for their followers. Very good teachers are transformational leaders (Harvey, Royal, and Stout 2003). Teachers serve as role models who inspire their students and stimulate their intellectual curiosity. Transformational teachers sacrifice themselves for the advancement of their students. They engage students in and out of the classroom to develop intellectually and to internalise the values associated with discovery of new knowledge. They set goals and expectations for their students and motivate them to meet those challenges. While rewards (grades) are important, they do not use rewards as a means of influencing behaviour. Motivation is accomplished through trust and commitment in the teacher. Transformational leaders develop a sense of attachment among their followers to their school. Comparison of the ETCS content to measures of transformational leadership (Avolio and Bass 1988) suggests that most of the competencies identified by students for the ETCS are components of transformational leadership. We expect future studies to show a strong relationship between ETCS and measures of transformational leadership such as intellectual stimulation, inspiration and trust. Conclusions The ETCS is a reliable and valid measure of teaching effectiveness that represents the competencies that students have identified as important to good teaching. It overcomes many of the problems associated with SETs. It has a clear unidimensional factor structure composed of nine competencies. It is short and concentrates on factors under the control of the teacher. It does not confound evaluation of teaching with evaluation of the course or workload/assignments. Some will argue that these last two dimensions are important components. We do not disagree; we simply believe that they should be evaluated separately from the evaluation of teaching. We

17 716 V.M. Catano and S. Harvey encourage other researchers to test and refine the measure across other samples and populations. Acknowledgements We wish to thank the following students who were involved in various stages of this project: Paul Angelopoulos, Sebastien Blanc, Cinthia Branco, Sebastien Houde, Carla MacLean, Chris Mahar, Tammy Mahar, Amy McMurray, Marie-Helene Michaud, Martin Royal, Karene Saad and Mike Teed. We could have not done this research without their dedicated participation. Note 1. The percentage of female undergraduate participants, 63%, is consistent with the percentage of women in arts programmes, 60% and slightly higher than the overall university enrolment of 55%. Notes on contributors Victor M. Catano is a professor and chair of the Department of Psychology, Saint Mary s University. He is a fellow of the Canadian Psychological Association and a past recipient of its award for Distinguished Contributions to Education and Training. He has also received the Canadian Society of Industrial and Organizational Psychology s award for Distinguished Contributions to Industrial and Organizational Psychology. He has published in Personnel Psychology, Journal of Applied Psychology, International Journal of Selection and Assessment, Journal of Occupational Health Psychology and the Journal of Dental Education, among others. He is the first author of Recruitment and selection in Canada, now in its fourth edition. He has acted as a consultant for many organisations. Steve Harvey is dean of the Williams School of Business at Bishop s University. His research activities have touched upon several topics in organisational behaviour and human resource management over the years. His main interests are in the area of work-related stress and psychological health and well-being, including the experiences of employees from particular groups such as teenagers and older workers. He also conducts research in the areas of work attitudes, workplace aggression, leadership, and spirituality in management. He has published research in diverse international journals such as Work and Stress, Journal of Occupational Health Psychology, Journal of Business and Psychology, Canadian Psychology, Personality and Individual Differences and Social Behaviour and Personality, among others. References Anderson, L., and S. Wilson Critical incident technique. In Applied measurement methods in industrial psychology, ed. D.L. Whetzel, and G.R. Wheaton, Palo Alto, CA: Davies-Black. Avolio, B.J., and B.M. Bass Transformational leadership, charisma and beyond. In Emerging leadership vistas, ed. J.G. Hunt, B.R. Baloga, H.P. Dachler, and C. Schriesheim, Lexington, MA: Lexington Books/DC Heath. Bass, B.M Leadership and performance beyond expectations. New York: Free Press. Cohen, P.A Student ratings of instruction and student achievement: A meta-analysis of multisection validity. Review of Educational Research 51: Cunningham, J.B., and J.N. MacGregor The echo approach in developing items for student evaluation of teaching performance. Teaching of Psychology 33: Flanagan, J.C The critical incident technique. Psychological Bulletin 41: Fulton, W How can we use course evaluations to improve teaching and the curriculum. (accessed January 20, 2002). Gatewood, R.D., H.S. Feild, and M. Barrick Human resources selection. Mason, OH: Thompson/South-Western.

18 Assessment & Evaluation in Higher Education 717 Greenwald, A.G No pain, no gain? The importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology 89: Harvey, S., M. Royal, and D. Stout Instructor s transformational leadership: University student attitudes and ratings. Psychological Reports 92: Huffcutt, A.I., J.A. Weekley, W.H. Wiesner, T.G. DeGroot, and C. Jones Comparison of situational and behavior description interview questions for higher level positions. Personnel Psychology 54: Jacobs, L.C Student ratings of college teaching: What research has to say. best/multiop/ratings.shtml (accessed February 27, 2008). Keeley, J., D. Smith, and W. Buskist The teacher behaviors checklist: Factor analysis of its utility for evaluating teaching. Teaching of Psychology 3: Marsh, H.W Multidimensional ratings of teaching effectiveness by students from different academic settings and their relation to student/course/instructor characteristics. Journal of Educational Psychology 75: Marsh, H.W., and L.A. Roche Making students evaluations of teaching effectiveness effective. American Psychologist 52: McKeachie, W.J Student ratings: The validity of use. American Psychologist 52: Penny, A.R Changing the agenda for research into students views about university teaching: Four shortcomings of SRT research. Teaching in Higher Education 8: Pulakos, E.D Ratings of job performance. In Applied measurement methods in industrial psychology, ed. D.L. Whetzel and G.R. Wheaton, Palo Alto, CA: Davies-Black. Pulakos, E.D., and N. Schmitt Experience-based and situational interview questions: Studies of validity. Personnel Psychology 48: Simpson, P.M., and J.A. Siguaw Student evaluations of teaching: An exploratory study of the faculty response. Journal of Marketing Education 22: Toland, M.D., and R.J. De Ayala A multilevel factor analysis of students evaluations of teaching. Educational and Psychological Measurement 65: