Teaching critical thinking and problem defining skills

Transcription

1 Teaching critical thinking and problem defining skills Jens J. Kaasbøll Abstract Department of Informatics, University of Oslo P.O.Box 1080 Blindern, N Oslo, Norway jens.kaasboll@ifi.uio.no URL: Tel: Fax: April, 1998 Published in Education and Information Technologies Vol.3, No.2, 1998, pp.1 17 Critical thinking and skills in defining problems have been amongst the goals of a computers and society course that had its credits reduced and student number increased. In an attempt to prevent worsening the students learning, four measures were taken. The results actually improved from a failure rate of 21.5% to 0.7%. This is mainly due to a tighter project structure and additional student work. Reduced course material and improved teacher preparation did not seem to have any effects, while more focused project teaching may have contributed. Further improvements may be gained through time estimation in the problem definition process. Keywords Higher Education, creativity, interdisciplinary, problem solving, social issues, teaching methods 1. INTRODUCTION Employers and managers complain that computer science graduates lack practical competence, eg, for writing and teamwork (Hartmanis et al., 1992). Denning (1992) argues that the engineers of the future need more communicative skills, and that the students should be given more responsibility for their own learning. Scragg et al (1994) argue that students also have to learn creativity, and that this comes from insight into the facts, methods, and paradigms of a the field. We have observed similar lacks of skills among students who embark on their Master s thesis. Then they have to define their own research problem in cooperation with their supervisor, find their methods, and question the literature and their own empirical data. This requires both skills in critical thinking and the creativity needed to define their own problem. We will call the latter problem defining skills.

2 2 In most of the undergraduate courses, the teacher has defined the student assignments, which can be solved within the time given, at the department s computer, and by means of the method lectured the same week. This kind of teaching does not specifically support critical thinking and problem defining skills. In the Department of Informatics, University of Oslo, we have for more than twenty years also taught an undergraduate course in computers and society where critical thinking has been encouraged. Students have had to get exposed to the world of practical tasks outside the university, define their own problems based on the ill-structured situations they encountered, work with the problem without knowing how the conclusion will look like, and write a comprehensive report of their work (Kaasbøll, 1981; Nygaard, 1982; Øgrim, 1991). However, only a minority of the undergraduates used to attend this course. Consequently, many students started working on their Master s thesis without this training. Therefore, the Department decided to make this course compulsory for all students wanting at least one year of informatics studies. Concurrently, the credits for the course were reduced from 0.5 workload of a 13 weeks semester to 0.3, putting much more stress on the efficiency of the course. The number of students increased from approximately 60 to 143, giving the lecturers less time for interaction with each student. We made several efforts to tackle this problem, the major ones being preparing the teachers, tightening the course structure, and providing more project directed teaching. The research problem addressed in this paper is how these efforts were effective in improving computer science students skills in critical thinking and problem definition. Since we actually changed the course in the process of finding answers to the research question, we carried out action research. As often happens in action research, action takes precedence over research, making it difficult to interpret the effects of the actions. Recent evaluators of adult education suggest information to be collected from many sources (Athanasou, 1995), something we tried to achieve. Our sources consisted of written student assignments, grades given to the students by external examiners, examiners statements, informal reports from the tutors and the students during the course, and answers to questionnaires. Unfortunately, the response rates to the questionnaires were 40% and 34%, disabling most generalisations from the responses. Therefore, the responses have been used only when no better information has been available. Previous research in teaching critical thinking and problem definition skills is described in the next section. Section 3 describes the actions taken to change the course, and section 4 reports that the number of students who passed the course increased significantly compared to previous years. After a discussion of whether this measure reflects improvements in students abilities of

3 3 critical thinking and defining problems, the effects of the different changes are discussed in sections PREVIOUS RESEARCH Training students in the skills needed to set up and carry out their own research project can of course be done through a course having this as its main activity. In fact, we run the course this way. In addition to this similarity between means and ends, educational science provides another reason for choosing this teaching strategy Principles of instruction Critical thinking and the ability to define problems both belong to the category of general cognitive skills. Pascarella and Terenzini (1991) have carried out a survey of research on what students learn at college. They refer to research which suggests that institutions with a broader curriculum influence positively the development of general cognitive skills (p.137). They define critical thinking, as follows: the individual s ability to do some or all of the following: Identify central issues and assumptions in an argument, recognise important relationships, make correct inferences from data, deduce conclusions from information or data provided, interpret whether conclusions are warranted on the basis of the data given, and evaluate evidence or authority. (p.118) Even if identify central issues and recognise important relationships also are parts of problem definition, their concept of critical thinking does not include the ability to act in illstructured situations, which is necessary in order to define problems. We adapt this definition of critical thinking, except that identify central issues and recognise important relationships are given less emphasis, since these are also included in problem defining skills. Pascarella and Terenzini (1991, p.144) only found one experiment that significantly correlates a particular form of instruction with positive influence on critical thinking, being the integrated teaching of humanistic disciplines that otherwise were taught separately. Problem defining skills constitutes one out of five features of practical competence (Pogson and Tennant, 1995). This is the competence needed to cope with real world situations, as opposed to intelligence, which correlates with the ability of theoretical studying. Problem defining skills do not seem to correspond to one category of cognitive skills. However, the categories reflective judgment and conceptual complexity seem to be preferable abilities. Conceptual complexity is the extent to which a person is capable of attending to a large variety of cognitive stimuli and organising her or his dealings with the external environment in increasingly abstract, complex, and varied ways (Pascarella and Terenzini, 1991, p.126). Reflective judgment has

4 4 been measured through subjects having to justify their position in an ill-structured problem, which is represented by two contradictory points. A specific curriculum where courses were organised inductively from the concrete to formal reasoning influenced conceptual complexity positively (Ibid., p.142, 145). The more general factors of student involvement and student - teacher interactions also have positive influence on general cognitive skills. In summary, the following principles of instruction seem to foster critical thinking and problem defining skills: Student involvement Student - teacher interaction Inductive teaching Integration of disciplines 2. 2 Problem based learning Problem based learning is an inductive educational strategy that aims at increasing student involvement and integrating knowledge from different sources, and it requires interaction between the student and teacher. Therefore, this strategy may be effective for learning the skills at which we aim. A main principle of problem based learning is that the students work on real life problems or constructed problems that mimic the complexity of the practical world. In such situations, the students have to think critically through all the information available to sort out the relevant. When students have the opportunity to define their own problems, they become more involved in their work, and this involvement increases motivation for learning. Since students are assumed to obtain a more profound understanding of the subject area, assessment of problem based learning should focus more on the students skills in handling an ill structured situation than on recalling the textbook. Problem based learning or similar approaches have been used in software development courses. The literature confirms that this approach to learning is feasible. Kirsch (1996) report of a human-computer interface course, where students could choose their problem themselves. They were given three major assignments plus weekly explorations of techniques to be used in the project. Kirsch gives an enthusiastic report of the students achievements. However, he provides no rigorous comparison with more traditional teaching. Ryan and Quinn (1994) report of a course in which the students are given an incompletely specified design task. The students evaluate the course positively, and an increasing number of students has been attracted to the course. Baldwin (1996) reports from courses conducted as discovery learning, where the instructor sets the project goal. The students in a computer graphics course following these principles were able to construct a sophisticated ray tracer and a 3-dimensional game, while students at previous lecture based courses could hardly make up a

5 5 ray tracer for a single object and a 2-dimensional game. In Adams s (1993) course, the instructor has the role of project manager, who plans deliverables and weekly due dates and reviews. Even if the students have a more subordinate role in this course, Adams reports that the students became engaged. These experiences from computer science education unanimously give a positive evaluation of the principles of problem based learning. However, the evidence is episodic rather than scientific. It is worth noticing that frequent follow ups on the students progress are mentioned as an important part of the course structure by most of the authors. 3. COURSE STRUCTURE The computers and society course has usually been taught by two lecturers. The lecturers responsible for the new course had also been teaching the old course a total of four times during the five last years. In addition, the course had tutors, each responsible for one or two classes of students. Three of the five tutors had teaching experience from the old course. The students have to complete a project in groups of 4-6, supervised by their tutor. The typical student in the course is male, between 20 and 25 years, his mother tongue is Norwegian, and he has completed 2 semesters of study in informatics and one semester with mathematics and statistics. The students carry out field studies of computer systems in organizations. The students have to find suitable organizations and arrange contacts themselves. Summaries of previous projects can be found in (Kaasbøll et al. 1993) and (Kaasbøll and Øgrim, 1994) Changed conditions The main changes of the conditions for the course were reduced credits, and the fact that it became mandatory for the computer science students. The latter means that students who are not interested in this particular subject also attend the course. Among the 57 students who responded to the survey made after one month of teaching, six ticked lack of interest as their main reason for putting less work than required into the course. Without any knowledge about the one hundred students who did not respond, it is not possible to say how widespread the low motivation was. However, in an optional course, one would not expect that any student had low motivation. Apart from students motivation, we have no reason to suspect any other changes of the population of students attending the course Course changes in response to changed conditions The number of lecturers, the ratio of students per tutor, and the required project work were unchanged. The modifications mate to adapt to changing conditions concerned course material, structure, teacher preparation, and sessions. An overview of changes is given in this section, while more details are provided later, where we try to explain their consequences.

6 6 The material of the new course consisted of parts of a textbook on management research (Easterby-Smith et al, 1991) and a textbook on management and technology (Levin et al, 1994). The remaining material was available through the World Wide Web (Øgrim and Kaasbøll, 1996) ). A collection of articles used in the old course was abandoned in the new one. The course schedule allocated two hours for lecturing (reduced from four) and three hours in class per week for thirteen weeks. There were six lectures concerned issues within subject of computers and society, and five lectures aimed at guiding the project work. In addition there were three class exercises concerning the subject and three aimed at the project. The remaining six classes were used for supervision of the projects. Given an average of forty hours studying time per week, the 0.3 full time credits over 13 weeks gives a total of 156 hours of student work during the course. Assuming that the students spent ten hours on subject related and activities of no relevance for the project, the time allotted for the activities in the course can be separated as shown in Table 1. The previous course was scheduled to occupy twice the amount of time, including a 3-4 weeks period of time allotted for study between delivering the project and the oral exam. The exam was abandoned in the new course, and replaced with grades based on the project work. In total the amount of time for project work was reduced with 33%, and the time for project work outside the scheduled lessons was reduced by 43%. In addition to these changes, we also initiated and carried out training of the tutors prior to the course. Four of the five tutors attended this training. Table 1. Student hours allotted for course activities. Previous course New course Activities Project Subject Project Subject Lecturing Class exercises Supervision Organised teaching in total Unorganised studying (rough estimate) Total Total for the whole course ASSESSMENT 4. 1 Criteria for grading The projects were marked according to four main areas: subject matter, empirical study, reporting and working process. These areas were further divided into sub areas, see Table 2. Both subject matter and empirical study had to be passed in order to obtain a passing grade. Problem defining skills are covered in the sub area problem definition under empirical

7 7 study. It emphasises that the problem is grounded. The sub area of subject matter requires ability to recognise important issues. relevance under Analyses of data, comparing sources, and drawing conclusions require the skills of critical thinking, and these are included in empirical study. However, critical reference to other literature is not included, since this is categorised under the report area of assessment. With this exception, the areas subject matter and empirical study cover problem defining skills and critical thinking as we intended in the course, and also, as far as we can see, in accordance with (Pascarella and Terenzini, 1991). The previous course had no such detailed list of requirements. However, the requirements made up for the new course were based on the our experience of assessment in the old course, such that the students were assessed in the same main areas. The level of expectation of student performance remained, except in the process area, where the new requirements constituted a distinct rise of expectation. In the previous course, the students were given pass or not pass based on their project and an oral examination where both the project and subject matters were discussed. In the new course, only 20% of the students were given an oral examination, and this examination concerned only their project. The Faculty s numerical grading scale consisting of 31 steps of passing grades from 1.0 to 4.0 was used. Main areas Subject matter Table 2. Criteria for grading projects. IT = Information Technology Sub areas Criteria Insight in relations between IT and society Relevance Reflection on the influence of IT, and understanding of others relation to IT Of interest to specific groups Empirical study Report Problem definition Coherence between problem and empirical method Quality of data collection Coherence between data and analysis / conclusion Coherence between problem and data / conclusion Structure and disposition Quality of presentation The problem and the partial problems are grounded, precisely formulated, and can be answered during the course Methods and selection of units to be studied are grounded in the problem, and reasons for not choosing other methods are given Follow the methods as thoroughly as time permits. Different sources are compared. Analysis and conclusion are based on data Data is relevant for problem, and conclusion answers problem Logically grounded sequence of chapters which focuses the main points Documents method, presents data, and conveys units of study, analysis and conclusion in sufficient depth and precision

8 8 Language Well formulated and easy to read Process 4. 2 Results Presentation and analysis of other literature Organization of work Management of controversies Planning and following up Constructive self critique Refer others work, relate to it, and show its function in own work Account for roles, areas of responsibility and division of tasks, and explain why the group decided this structure based on individual differences and the principle that one ought to learn the skills of which one is ignorant. Corroborated with quotations from minutes of group meetings. Account for how the group has handled controversies and possible conflicts. Corroborated with quotations from minutes of group meetings. Account for how the group has handled delays and changes of plans. Account for what the group would have done otherwise given they could redo the project. The students were organized in 28 projects, and some report titles were: Commerce on the Internet a survey of Norwegian net users attitudes towards buying and selling on the net Change of an application system at Telenor Research and Development Will increase the possibility for changes of power structures? Who cares where you come from? Employers conception of IT education Out of 143 students delivering their projects, only one student failed. This student was expelled from a group and was tested in an oral exam. The average grade of those who passed was 2.6. A comparison with the total number of students who took the previous course in the five last years is given in Table 3. Table 3. Assessment of students. The previous course columns show the accumulated numbers of the last five years. Number of students Number of project groups Previou s course New course Previous course New course Passed Failed Total Since the actual distributions of the pass/fail grades of both the old and new courses are known, the chi-square test between the two will show whether they are correlated. Chi-square of the distributions calculates to A score above 6.64 indicates with probability.99 that the distributions are not correlated.

9 9 Since the grading is based on projects rather than students, the corresponding number of projects is shown to the right in Table 3. Chi-square of the projects is 7.25, also being above the.99 level. Hence, we conclude that the proportion of students who passed the new course has improved significantly compared to the old course. We set out to improve students critical thinking and problem defining skills, and we have seen a significant improvement of student s ability to pass the course. In order to pass, the areas of assessment called subject matter and empirical study have to be passed. We have argued that these areas include problem definition and critical thinking, except critical reference to other literature. We therefore conclude that with this exception, the students who passed have the required skills in critical thinking and problem definition. In order to show an improvement in these skills, we must also establish that those students who failed in the previous course failed in these issues. The requirements for the new course were based on the old, with no intention of changing the areas of subject matter, empirical study, and report. The failures in the old course were commonly due to poor coherence between problem definition, method, analysis, and conclusion, and the poor coherence could often be tracked down to poor problem definition. Therefore, we believe that the grades constitute a valid measure of critical thinking and problem defining skills. The projects were graded by examiners external to the university. Out of the four examiners of the new course, two had also been examiners for several years in the previous course. Together, the two experienced examiners marked ten projects, all of which passed. This number is too small to yield any statistically significant increase in the number of projects they marked as passed. Chi-square corresponds to the.9 level of confidence. One of the experienced examiners says: The five reports I had were in general at a much higher level than before. None of them were bad, and one was exceptionally good. Adding the fact that the only student who failed was judged by the new examiners, makes us believe that the result also is a reliable measure of student improvement. Taking the reduction in credits and increase in the number of unmotivated students into consideration, we were very pleased with the students improvement. We identify the following factors as possible reasons for the improvement, the first four of which constitute our deliberate changes, and we will subsequently discuss each factor. Reduced course material Improved teacher preparation Tighter project structure More project focused teaching

10 10 Increased amount of student work 5. REDUCED COURSE MATERIAL The course material was reduced to partly make up for the reduced credits of the course. Our impression from previous years is that the students put their main effort on reading between the end of the project and the oral exam; see our estimate of unorganised studying in Table 1. Since they presumably spent little time on reading during the project period of the old course, there was little time saved for project work this way. Some of the responses to the open questionnaire reflected this, eg, I don t understand how one can remove one book and say this is a 3 credit course. The project report is the same. Same amount of work. We therefore rule out reduction in course material as a reason for the improved results. 6. PREPARING THE TEACHERS We arranged a ten days pre-course for training the tutors two months before the course commenced. Four of the five tutors attended the course, three of whom had acted as tutors before. We spent fourteen hours together with the students, partly lecturing, but mainly discussing the issues with the tutors. The main assignment negotiated was to make up guidelines for tutors for how to tackle the most difficult part of their teaching. The guidelines they wrote up were useful, and the tutors were awarded a good mark by the external examiner. The tutor who did not follow the training course supervised 9 projects with a total of 49 students, all of whom passed. This is in a better result than the group of tutors who attended the training course had! However, we cannot generalise any finding based on one tutor. In addition to the training course, we had weekly teacher meetings with the tutors, where we discussed the students progress and possible ways for the tutors to react to actual problems. From the educational science, we know that recurrent sessions on a topic are more effective than one shot training. The weekly discussions may therefore explain why the tutor who did not follow the training course had such a good result. This is also in line with Carbone et al s (1996) observation that regular discussions between lecturers and tutors on the tutoring sessions of a programming course have a greater impact on teaching quality than a preliminary course for tutors. Another possible explanation is that the course gave too little practical training on how to guide students during a project. We have no knowledge of the selection of students in different classes, and only one tutor did not follow the course. Consequently, we find no effect of the training course on the tutors competence, as judged by the results of their students. However, the findings suggest that the weekly meetings are important.

11 11 7. TIGHTENING PROJECT STRUCTURE We tightened the structure of the project in four major areas: time schedule, project organization, criteria for evaluation and numerical grading Frequency of deliverables In the previous course, there were three written and one oral, mandatory progress reports on the project during the semester. Experiencing that the students did not spend their time effectively during the first half of the project, we doubled the number of written deliverables. We also tried to make them like draft versions of sections of their final report, eg, problem formulation, units of study, method, and results. The emphasis in the previous course had been more on status reports on the students understanding. The first questionnaire that was handed out after a month showed that 2/3 of the students who answered said that they spent more than the allotted 12 hour per week on the course. This indicates that the students had a high level of activity from the start of the project. Many comments on the deliverables were positive, eg: Our tutor had more assignments than the lecturers had decided. Could have had even more One needs to start working on the project at an early stage. The tutors who checked and accepted the deliverables were unanimously in favour of the tight control, saying that the students thereby were forced to carry out the tasks required in their projects. The intermediate reports were also effective in giving early warnings of projects that were not on track. During the weekly teacher meetings, we had numerous discussions on which actions to take, based on the students reports. Also the external examiners emphasised that the groups that worked consistently throughout the semester produced a better report than those who started off late. It seems that the frequent deliverables, at least every two weeks, helped the students through the projects. Other comments suggest that the schedule was too rigid: Too many deliverables, many of which were irrelevant for the project. They [the deliverables] were too inflexible to fit each group. These comments may indicate that the deliverables offer the students too little freedom in their scheduling of work tasks at a particular stage. However, we do not think it had major impact on their choice of project and research method, since the deliverables requested general issues like problem formulation, literature, units of study, etc. To provide more flexibility, one could require the students to submit a partial product every two weeks, and leave the actual choice of deliverable to the students and the tutors.

12 Project organization The students were required to set up a structure for their project work. Sets of work tasks had to be assigned to roles, and the persons could take up roles, some of which might be changed during the course. They were required to adapt roles from which they could learn new skills. They were also required to agree on procedures of work, in particular how to make decisions and how to follow up when plans were not met. In the previous course, the students were encouraged but not required to set up a formal project organization. Responses to the first questionnaire indicated that the groups who had decided upon a clear leadership had structured their work more tightly than those who did not want to have a leader. Whether having a clear leader role or not, the responses indicated that the more engaged and extrovert students had a greater saying in decisions. In the previous course, project reports with poor cohesion, consequently making the students fail, were often produced by groups with major internal conflicts. Although one group had a split off of one member, other conflicts in other groups were resolved or controlled. Considering general knowledge of conflicts, we find it likely that groups who have agreed on procedures for decision making and internal control are more able to prevent and control conflicts. Consequently, we believe that the tighter project organization helped students learn more, through preventing and controlling possible obstacles to learn Criteria for assessment Mature learners are more inclined to aim at goals of learning than youngsters, who rather follows the teacher s instructions timely. We therefore assumed that introducing rather detailed criteria for grading the would have an impact on what the students tried to achieve. The assessment shows that the students met a majority of the criteria at an acceptable level. Two observations indicate that the students also considered the criteria during their projects. The reports in general had a better cohesion between problem, description, discussion and conclusion than before. They also included the important elements which are assumed to be in a report. The criteria for evaluation were obviously used a lot during writing, making some of the reports a little bit too mechanical. And what was not included in the criteria was not in the reports, even if it could have fit their topic. (One of the external examiners) There is obviously a risk that the students work becomes more influenced by standard criteria than the projects problem. Although the criteria seem to have prevented poor reports, students who master their projects should not feel that they get bad marks due to not following the guidelines in every respect.

13 13 The first criterion, Reflection on the influence of IT, and understanding of others relation to IT was less operational than most of the other criteria. It is our general impression that the students on the new course showed no progress as compared to the previous course in this respect. The other criteria on subject matter, empirical study and report were more operational, and the students seemed able to meet them. A way to try to achieve improved results also in this respect would be to reformulate the criterion in more operational terms. The criterion may be easier to fulfil if expressed like, eg, Explain how your findings are aligned with our general knowledge of the influence of IT in society and Report how the persons in your empirical study conceived and understood the IT. The other indication of that the criteria were effective on student behaviour, was a rather frank suggestion for improvement found among the responses to the final questionnaire: Nearly all the process criteria are completely behind target! Consequently the work load and the requirements become overwhelming, so the students tailor a report according to these requirements. Lie a little bit here and there... This admitting of opportunistic behaviour indicates that the students regarded the criteria so seriously that they even tried to cheat in order to fulfil them. In addition, the response also indicates that the process criteria should be reconsidered completely. When students admit fabricating data in this respect, one may also wonder whether they have conducted the empirical work described in their reports. The tutors followed the progress of the students work closely, without becoming suspicious of any cheating. In fact, we believe that it is easier to cheat in courses where the students can copy reports over the Internet than in this course where work progress is monitored frequently. Although we do not know whether the criteria or the frequency of had greater impact on student learning, the observations strongly suggest that the criteria have aided the students Numerical grading Changing from pass/fail to the numerical scale was done to make the students trying to achieve more than just a poor passed, and to honour those who did well. Out of the 43 students who responded in the first questionnaire that they spent more than the allotted time on the course, grading was their most important motivation. This result indicates that the numerical grades were effective in making the students work hard. 8. PROJECT-DIRECTED TEACHING Project relevant class teaching was the only type of teaching that had more hours. In some of the previous courses, the projects were focused on one area of study, eg, Internet applications. Then some of the lessons concentrated on that particular area, while all the lessons in the new course were aimed at general skills useful for the project work.

14 Problem definition Russel et al (1994, p.64) suggest to teach problem defining skills in two phases. First, the students select a topic, carry out a literature study, and write up their understanding of the area. Second, they develop the question they would like to research. Our approach mixed problem definition and empirical studies to a larger degree. During the first week, the students were required to write up a page on the area of interest and the problem on which they wanted to work. Afterwards, they explored the empirical area and read some literature, whereupon they refined and possibly revised their questions. In a training session on project definition, the students were given two problem formulations from earlier reports, and the following criteria that a problem definition should fulfil: New and partly unknown Motivating for the students Of interest to others Can be realised Manageable during the course They compared the given examples with these criteria, and then proceeded to assess their own project. Subsequently, they reworked their problem definition several times during the project period, often after feedback from the tutors on deliverables. In the first questionnaire response, this session was the only main activity that the students thought had contributed to their skills in defining problems. In response to the second questionnaire, some students said that they would have liked there to be more guidance on defining problems, separation into partial problems, and how to limit their work. The training should probably focus more on the process of defining problems. Russel et al (1994, pp.65-67) mention drawing the situation and second hand reports as suitable techniques. Research on the effects of teaching directed towards cognitive skills and intellectual growth suggests that a cumulative set of mutually reinforcing experiences over an extended period of time is better than a single lesson (Pascarella and Terenzini, 1991, p.159). Instead of generating more sessions, the students recurrent reworking of their own problem definition may therefore be more effective in learning problem defining skills. Requirements for resubmitting the problem definition could be included in two or three of the deliverables Critical thinking Lectures and class lessons on research methods and analysis of results were carried out. The latter was added in the new course. Methods of empirical inquiry include reading documents, hands-on experience with the systems, studying logs of use, observations of use, interviews, and possibly also questionnaires. The students were encouraged to try at least two methods of inquiry, and most of them followed that advice.

15 15 Student response to the last questionnaire mentioned particularly that they would have liked more training on how to structure a report. This should also be included in teaching. We cannot evaluate the contributions of the individual sessions on the overall performance of the students. The effects of each class exercise were discussed at the weekly teacher meetings. In addition to minor hints for improving the exercise, the tutors typically said: I don t think the students became very good at interviewing, but, you know, every training session helps. Our judgment is that there should be a training session for each major assessment criterion. 9. AMOUNT OF STUDENT WORK The planned time for unorganised project work was reduced from 149 to 85 hours, see Table 1. Thirty-seven out of the 45 students responding to the questionnaire say that the work load exceeded the credits. The 28 students who answered the same question in the last year of the previous course also complained about too high work load. The report of high work load had increased, but not significantly, for the new course. Unfortunately, only a minority of the students have responded to the questionnaires. Introduction of numerical grading was mentioned as a reason for the students eagerness. We believe that the hours put into the project work was not reduced much compared to the old course, and that the students effort contributes to explaining their good results. Increasing the credits is a simple solution for aligning the amount of work with the course requirements. However, this solution may require other changes of the curriculum. If wanting to reduce the requirements such that less time is needed to complete, we should have had more knowledge on which tasks that consume most time. Our general impression is that much time is spent on coordination, empirical inquiry, and report writing. Reducing group size will reduce the need for coordination. However, there will also be fewer persons to share the burden, and the group will be more vulnerable to quitting of group members. Some groups carry out a large number of interviews or they administer a questionnaire, while others are capable of passing with four interviews and reading appropriate literature. A way of reducing workload can be to consider the amount of empirical study needed more thoroughly during problem definition. Less empirical material could also reduce the length of the reports, which often reach the limit of fifty pages. 10. CONCLUSION Managers and researchers have called for computer science graduates to have better skills in teamwork, writing, and communication, and that they also should take more responsibilities for

16 16 their own learning. Problem defining competence constitutes the skills that are needed to cope with real world situations, where the above mentioned skills are important ingredients. In order to strengthen students learning of critical thinking and problem defining skills in a course, four main changes were introduced: reduced course material, improved teacher preparation, tighter project structure, and more project focused teaching. While we found no direct effects of the two first mentioned changes, tighter project structure seemed to be the main cause of the actual improvement observed. The credits of the course were reduced, and we believe that the students worked more than the scheduled twelve hours a week, and that their extra efforts also contributed to the improvement of learning. Approaches reported in the literature (Adams, 1993; Baldwin, 1996; Kirsch, 1996) follow up the projects in a weekly or bi-weekly frequency. Our study confirms that tight follow up is essential for students learning. When we allowed students to define problems of their own, we required the problems to be new and partly unknown, motivating for the students, of interest to others, realisable, and manageable during the course. Considering the high workload reported by the students, the students problems did not fulfil the last of these requirements. A way of improving this issue could be to provide the students with estimates of the time required for single tasks like making an interview and finding a literature reference. Even if the actual time spent may differ greatly from the estimate, work that exceeds allotted time considerably should make the students rethinking their problem. In a software engineering course, estimation techniques could be used in the same manner. Detailed criteria for assessment prevented poor reports. However, the individuality of projects should also be honoured, even if this implies that some criteria are not met. Pascarella and Terenzini s (1991) literature review concluded that student involvement, studentteacher interaction, inductive teaching, and integration of disciplines were effective in fostering general cognitive skills. These principles were followed to the extent possible, and we have found no indications that they should be abandoned. While learning critical thinking has attracted research, less knowledge exists on how students acquire problem defining skills. We have confirmed that the motivational factors (interesting for students and for others), and the research factors (partly unknown and realisable) are valuable for learning problem defining skills. Our case also indicates that management of the problem solving activity should be taken into account in problem definition. If not, students may drown in tedious work that does not necessarily foster learning. While software project assignments seem to be common in information systems teaching, we have only encountered one other case (Kirsch, 1996) where the students had to find the

17 17 problems themselves. In most of the areas taught in informatics, eg, programming, software engineering, systems analysis and design, human-computer interaction, communication, multimedia, there are ample opportunities for letting students both define and solve problems, thus increasing their responsibility for learning and strengthening their motivation. The study reported in this paper shows that student defined projects are feasible provided tight follow-up by the teachers. 11. REFERENCES Adams, E.J.(1993) A Project-Intensive Software Design Course. SIGCSE Bulletin, 25, 1, Athanasou, J. (1995) Issues in the evaluation of adult education. In Understanding adult education and training. Foley (ed.) Allen & Unwin, St.Leonards, Australia Baldwin, D. (1996) Discovery learning in computer science. SIGCSE Bulletin, 28,1, Carbone, A.; Mitchell, I.; Macdonald, I. (1996) Improving teaching and learning in first year Computer Science tutorials. In Proceedings of the thirteenth annual conference of the Australian Society for Computers in Learning in Tertiary Education. Edited by A. Christie, P. James and B. Vaughan. Adelaide, South Australia Denning, P. (1992) Educating a New Engineer. Communications of the ACM. 35,12,83-97 Easterby-Smith, M., Thorpe, R., and Lowe, A. (1991) Management Research: an Introduction. Sage, London Hartmanis, J. and Lin, H. (eds) (1992) Computing the future: a broader agenda for computer science and engineering. National Academy Press, Washington, D.C. Kaasbøll, J. (1981) Computers and Society: A Part of Informatics. In Education on Data Processing and Cooperation in Computer Technology. Raasio and Fontell (eds.), Helsinki, National Board of Vocational Education in Finland, Kaasbøll, J.; Braa, K.; Bratteteig, T. (1993) User Problems Concerning Functional Integration in Thirteen Organizations. In Human, Organizational, and Social Dimensions of Information Systems Development D. Avison, J. Kendall and J.I. DeGross (eds) IFIP Transactions A-24, North Holland, Amsterdam, Kaasbøll, J. and Øgrim, L. (1994) Super-Users: Hackers, Management Hostages, or Working Class Heroes? A study of user influence on redesign in distributed organizations. In Precedings of the 17th Information systems research seminar in Scandinavia. Edited by P. Kerola, A. Juustila, J. Järvinen. University of Oulu, Department of Information Processing Science, Research Paper Series A 21, Part II, Kirsch, R.P. (1996) Teaching OLE automation: a problem-based learning approach. SIGCSE Bulletin, 28,1,68-72 Levin, M.; Fossen, Ø. Gjersvik R.(1994) Management and Technology. Introduction to organization and management. (In Norwegian) Universitetsforlaget, Oslo Nygaard, K. (1982) Computers and Society. In Teaching Informatics Courses: Guidelines for Trainers and Educationalists. Jackson, H.L.W. (ed.), North-Holland, Amsterdam Pascarella, E.T. and Terenzini, P.T. (1991) How College Affects Students: Findings and Insights from Twenty Years of Research. Jossey-Bass, San Francisco Pogson, P. and Tennant, M. (1995) Understanding adult learners. In Understanding adult education and training. Foley (ed.), Allen & Unwin, St.Leonards, Australia Russell, A.L., Cready, D., Davis, J. (1994) The use of contract learning in PBL. In Reflections on Problem Based Learning. Australian Problem Based Learning Network, Chen, S.E.; Cowdroy, R.M.; Kingsland, A.J. and Ostwald, M.J. (eds.), Sydney, 57-72

18 Ryan, G.L. and Quinn, C.N (1994) Cognitive apprenticeship and problem based learning. In Reflections on Problem Based Learning. Australian Problem Based Learning Network, Chen, S.E.; Cowdroy, R.M.; Kingsland, A.J. and Ostwald, M.J. (eds.), Sydney,15-33 Scragg, G.; Baldwin, D.; and Koomen, H. (1994) Computer Science Needs an Insight-Based Curriculum. SIGCSE Bulletin. 26, 1, 94 Øgrim, L. (1991) Project Work in System Development Education. In Information system, work and organization design, P. van den Besselaar, A. Clement and P. Järvinen (eds) North- Holland, Amsterdam, Øgrim, L. and Kaasbøll, J. (1996) Computers and Society Course. (In Norwegian) URL: 18

19 19

20 20

21 21

22