Developing a Performance Measurement System for University Central Administrative Services

Transcription

1 Higher Education Quarterly, DOI: /j x Volume 63, No. 3, July 2009, pp Developing a Performance Measurement System for University Central Administrative Services Marika Arena, Michela Arnaboldi, Giovanni Azzone and Paola Carlucci, Department of Management, Economics and Industrial Engineering Politecnico di Milano Abstract Central administrative services have recently received increasing attention from practitioners and academics due to the challenging need to both manage scarce resources and provide high-quality services. In this context, performance measurement systems (PMSs) may assume a central role, although an unresolved debate remains on the claimed benefits of accountability and the difficulties that have emerged in defining and managing proper measures. This paper contributes to this debate by presenting the results of a study in which a PMS for central administrative services has been developed and tested through an action research approach drawing on actor network theory. The experiment was carried out in 15 Italian universities and five areas of services were dealt with: student support, research support, accounting, human resources, and logistics and procurement. The highly participative method resulted in a comparable system with a complete set of cost and quality indicators across the participating universities. These data proved to be useful at managerial and policy level, providing insights on the presence of scale effects and on the relative importance of quality dimensions for users of services. Participating in the project encouraged the university staff to use indicators in decision making. Introduction During the last decade, there has been increasing pressure on higher education to use resources more effectively and efficiently, not only in teaching and research activities but also in support services. These services and their accountability have recently been acknowledged to be 2008 The Authors. Journal compilation 2008 Blackwell Publishing Ltd, 9600 Garsington Road, Oxford, OX4, 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

2 238 Higher Education Quarterly crucial both as a budget line and as a contributor to primary activities (see Hill, 1995; Aldridge and Rowley, 1998; Casu and Thanassoulis, 2006; Brown and Brignall, 2007). This research was carried out in Italy, which has been characterised by a fundamental reform process in higher education. The first step in reforming the system was a law that was approved in 1993 (Law n.537/ 1993), which defined new principles to determine resources obtained from the central government. A second important step was made in 1999 (Ministerial Decree n.509/1999), when the central government increased the autonomy of universities to decide on their teaching offers and started to distribute monetary incentives to improve their services. This, added to a continuous trend to reduce funds from the central government, has also increased competition to acquire new and better students. Teaching performances and reputation have in fact remained central to attract students; however, support services are becoming increasingly important as facilitators in students careers and academic work. In this context, the paper presents a study of central administrative service accountability that endorses a participative approach, aimed at implementing and defining the significance of the measurement system for university members. The approach was developed in two distinct phases: system design and system application.the system includes three dimensions: the costs of services, the quality provided and the quality perceived. The first and the second dimensions required the elaboration and discussion of data, which were collected specifically by the participants (administrative managers and officers). The investigation of the quality perceived, however, required the involvement of the service clients, in particular students and teaching staff, with an overall participation of 12,044 students and 1,226 academics. The paper is divided into four sections: a reflection on performance measurement systems (PMSs) and the perspective adopted; a discussion of the specific context of Italian universities; the research approach adopted for the study; a description of the research model and a discussion of the results with some conclusions. Conceptual perspective: performance measurement and actor network theory Measuring performances in the public sector has been a central theme for researchers, practitioners and politicians for over two decades, with a number of contributions criticising the implemented and recommended

3 PMS for University Central Administrative Services 239 systems. However, PMSs continue to be designed, tested, developed and sometimes abandoned. This contradiction has led many authors to consider that performance measurement is not an end in itself and to wonder why measure performance? in the public sector ( Hatry, 1999; Johnsen, 2005). Behn (2003) provided a literature review and identified eight reasons for measuring performances: to evaluate, to control, to budget, to motivate, to promote, to celebrate, to learn, to improve. Although the list may be contested or different, his review explains the still vivid interest that exists in attempting to define a PMS. This awareness of the opportunity of introducing higher accountability as an instrument for managers is counterbalanced by the difficulties involved in developing a measurement system. These difficulties may be related to three stages of defining a PMS: (1) the identification of a proper set of measures and target values; (2) the implementation of the operational system; and (3) the actual use of the system. Most of the failures encountered in literature appear to be related to the separation of these stages from the context in which they are implemented and the PMS distance from the managers who are supposed to act upon them. This complexity and the need to study the phenomenon as a whole has led actor network theory to be endorsed as a valuable framework; in this light, the research studied the development of a PMS, the creation of knowledge and the evolution of the social space surrounding the experiment. Reference was made to the initial contributions to actor networks and the sociology of translation (Callon, 1986; Bijker, Hughes and Pinch, 1987; Latour, 1987) and their subsequent developments, which have been applied and translated into different fields of research (Lowe, 2001; Doolin and Lowe, 2002; Lodh and Gaffikin, 2003; Gendron, Cooper and Townley, 2007). From the perspective of actor network theory, society is seen as being constituted by diverse groups of people, who always interact with technology, machines and objects. This is a complex interplay among the heterogeneous elements of what is called technoscience, which constructs our society and organisations (Knorr-Cetina, 1997). This interpretation suggests that society and technology cannot be conceptualised as separate entities (Latour, 1994), but they need to be studied together in their mutually influencing evolution. In this light, the process of innovation (in this study the development of the PMS) is seen by actor network theory supporters as a process of translation in which scientists search for allies and consensus in society. In Latour s (1987) words, translation is the interpretation given by fact-builders of their interests and that of the people enrolled (p. 108). Translation is the way to build technical or

4 240 Higher Education Quarterly scientific objects that is [...]the simplest means of transforming the juxtaposed set of allies into a whole that acts as one, to tie the assembled forces to one another, that is, to build a machine (Latour, 1987, p. 128, emphases as in the original). In studying the process of innovation, the distinction between technological and social innovation is blurred; humans and non-human actors are equally important and it is only through their interaction and the construction of a socio-technical network that innovations can propagate.this equality is a controversial issue of Latour s (1987, 1994) philosophy, which has both supporters and opponents (Lowe, 2001; Harris, 2005). However, the symmetry between humans and nonhumans might not be intended as an equal status among them, but it suggests placing attention on the interaction between technologies and social actors and their reciprocal changes as an original idea progressively tries to become an innovation.this is the interpretation adopted in our study as it is only by studying the interaction between the university managers, researchers and technological devices (the PMS) that it is possible to understand the PMS that has been obtained and the general consensus that has been reached on the data. Italian universities and the reforming context An explanation of the general context in which Italian universities are set is important to understand the environment and the challenges of the project. The reform of the public sector has been pursued by governments in western countries since the early 1980s; actions have been undertaken at different levels and universities are a component part of this process. The Italian university system reform process was started in the mid-1990s, with the introduction of a new financing system, which represented a real break from the past. A law drawn up in 1993 (n. 537/93) defined new rules to determine the resources that could be obtained from the central government and this helped contribute to budgetary devolution. Previously, the Italian system was characterised by the centralised role of the Ministry of the University, which was in charge of defining the allocation of resources to each university; personnel and research funds were directly paid by the central government and then assigned locally to the faculties of each university. The previously mentioned 1993 reform introduced three main changes: (1) the definition of a single-line budget of financial resources, which included all the items previously assigned directly to specific cost centres (e.g. faculties); (2) the introduction of re-balancing mechanisms, which assign an increas-

5 PMS for University Central Administrative Services 241 ing part of the resources on the basis of the standard costs per student; and (3) the introduction of incentive mechanisms, based on the achievement of results defined by the central government. Furthermore, the same law introduced two evaluation mechanisms for universities. First, at a local level, universities have to define an internal committee that determines the fairness of the management of the resources, research productivity and teaching results. Second, at a central level, the government created a national committee to evaluate the university system, which assesses the results of academic institutions and controls their development plans. These reforms have helped shape a more autonomous and competitive system in Italy and were inspired by pacesetters like the UK, although some differences still exist from the Anglo-Saxon model. The main differences between the two systems are related to three issues (Agasisti and Catalano, 2006): the autonomy of institutions, the role of the state as the main provider of funds and its role as regulator. Regarding autonomy, UK universities have almost complete autonomy in teaching and in recruiting staff; in Italy, however, staff recruitment is carried out through a public competition that must be approved by the central government. There are also some limits in teaching autonomy in Italy, where courses are again subject to approval from the central government. As far as the financing system is concerned, both Italy and the UK adopt a lump sum approach with a unique budget line; however, state financing in Italian universities is, on average, much higher (80 per cent) than in UK universities (OECD, 2005). As for the regulator role of the state, while the UK emphasises the quality of ex-post evaluation, the central government in Italy also carries out an ex-ante regulation of the programmes. Table 1 summarises the main differences between the UK and Italy. Although there are some limitations compared to the UK, the devolution of decisions has offered new opportunities to Italian universities, but at the same time it has led to challenges for their management and academic boards and has highlighted the many difficulties involved in overseeing their administration. In this context, the first requests were made by university managers and the Ministry of the University to carry out an experiment on a PMS in central support services, an area that had (systematically) been neglected by the internal evaluation committee. With the financial support of the Ministry of the University, a first project, involving 10 universities, was undertaken in 2000 to experiment a cost-accounting model.this first project was successful but at the same time highlighted the need to extend the system not only to cost but also to quality performance.

6 242 Higher Education Quarterly TABLE 1 The main difference between UK and Italian higher education reform measures Phase UK Italy Recruitment Free Public competition, approval by the central government required Teaching autonomy Free Approval of programmes by the central government Allocation Lump sum Formula based Lump sum Formula-based Regulating role of the state Emphasis on ex post evaluation Ex post evaluation Ex ante evaluation of programmes and strategy The research approach The adopted research approach is a consequence of the need to implement and test a PMS for university support activities, and it reflects the conceptual perspective adopted in the study.the lack of available frameworks on this topic led to the choice of an action research method. Interest in action research first developed from the social sciences and underwent a new renovation impulse in the 1980s, also in higher education studies (Cohen and Manion, 1980; Suojanen, 1999). Since its origins, action research has witnessed growing recognition of its contribution to the understanding of practices and its provision of theoretical insights. The central issue of this approach is the interplay between science and practice, which simultaneously influence the theoretical conceptualisation and the practical rules of the studied phenomena (Argyris, Putnam and Smith, 1985).This method is appropriate for the scope of this study and to translate the key concepts of Latour s (1987) work and actor network theory. The two central contributors to action research, Argyris and Kemmis, consider it a way of connecting the critical thinking of researchers to the emancipation to change processes (Argyris, Putnam and Smith, 1985; Kemmis, 1985). Furthermore, action research entails the participatory involvement of actors within a studied organisation in all of the research phases: it builds an exchange of information and knowledge and it enhances participants competencies, through their participation in building facts and acting with technologies (Greenwood and Levin, 1998).

7 PMS for University Central Administrative Services 243 Considering performance measurement and accounting, the use of real data to explore these phenomena is not new. Lapsley and Llewellyn (1995) described a new approach to case-study research, which explores practitioners reactions to specific elements and actions. The authors suggest the use of this method for the reflection of conceptualising on embedded accounting information in organisational processes. The approach adopted here draws on this previous experience, and it was designed to define a measurement system that was agreed upon with the university managers. The model was designed with the suggestions and applied to 15 universities, the results were compared and the significance of the specific system for the university members was examined. The 15 universities have 35 per cent of the total number of students in the 61 Italian state universities. Accordingly, the approach was divided into four phases: (1) the design; (2) the implementation; (3) the discussion of data; and (4) the observation of the spread of ideas. The first phase was devoted to the definition of the network, which includes the participant infrastructure (humans) and the technical device (non-humans) (Latour, 1987). The second phase concerned the application of the PMS in the participant organisations. The third phase was aimed at presenting and discussing the data within the project arena. Finally, fourth phase was aimed at observing the use and effects of the experimented system, outside the project space. Here the data collection was carried out through interviews with managers and officers in two different periods, just after the closure of the project and then 10 months after that. Overall, the research was carried out over a two-year period from January 2004 to December The endorsed approach implied a continuous and close interaction between the research team and the organisations involved (Table 2). TABLE 2 The research phases No. Phase Timing Researcher s role Participant s role 1 Creating the network From January 2005 to May 2005 Agent of change Involved in the design 2 Acting the network From June 2005 to December 2005 Agent of change Involved in the experiment 3 Observing the January 2006 Observer Passive spread of ideas From October 2006 to December 2006 (interviewed)

8 244 Higher Education Quarterly The results: from ideas to action The results of this study are reported in four sections: (1) the analysis of the incremental creation of the network, in which the actors (researchers and university managers) interact and design the PMS; (2) an examination of the problems of the PMS implementation; (3) a discussion of the obtained data, highlighting the benefits at managerial and policy level; and (4) the analysis of the use of data outside the boundaries of the network. Creating the network: actors and the PMS Latour (1987) and actor network theorists have emphasised the importance of networks for the success of an innovation; in their view, a network is made by both human actors and the technology under development (non-human). This perspective is particularly appropriate for the implementation of a PMS in higher education, in which the distance between the system, its proponents and the users has been recognised to be one of the main reasons for PMS failure. In a simplified way, Figure 1 University A Research group University N PMS University B University Figure 1 A simplified representation of the network

9 PMS for University Central Administrative Services 245 shows the network, in which the universities interact among each other, with the technology (the PMS) and with the research group, which acts as the translator.this infrastructure is at the basis of all the phases in the PMS construction, from its design to its use after the end of the study. To ensure a high level of interaction and commitment, participation in the project was voluntary and self-financed by the universities. The proposal was made at the end of 2003 and 15 universities agreed to participate in the experimentation; pseudonyms were given for reasons of confidentiality (Table 3). There were two main actors (or groups of actors) in the universities: the scientific referee and the technical support. The scientific referees were administrative directors or delegated top managers, and they were mainly involved in the design phase and in the discussion and revision of the results. The level of participation in the study was high right from the early stages. Discussions were held with top managers to decide not only on the type of performances that should be included in the system (e.g. cost or quality) but also on the areas of support services in which they were more interested. These preliminary discussions were useful for the research team to help design an initial model for the PMS, which was based on three dimensions, cost, objective quality and subjective quality, in five major central administrative service areas: student support services, research support services, accounting, human resources, and logistics and procurement. After this straightforward decision, some doubts were raised on the possibilities of comparing the support services of the 15 institutions as they are organised in different ways. To overcome this problem, and drawing on previous experiences (Arnaboldi and Azzone, 2006), the research group proposed an activity-based model ( Kaplan, 1983; Johnson and Kaplan, 1987; Bhimani and Brimson, 1989; Mitchell, 1994, 1996). With this model, the measurement basis became the activity (e.g. payroll payments) instead of the organisational units in which each university is organised. This approach can be graphically clarified in Figure 2, which gives several calculation examples and shows not only the benefits but also the implications on the design and data collection. This early negotiation helped resolve the comparability issue, from the very first stages of the project, and also led to the managers and officers being more interested in actively participating in the design of the system and the protocol for the data collection, as the following comment shows:

10 246 Higher Education Quarterly TABLE 3 The main attributes of the organisations Universities University dimension Geographic Number of location students (a.a ) Number of PhD students (a.a ) Number of teachers ( ) Number of workers ( ) 1 University A 94,416 1,870 2,999 2,872 North 2 University B 30, South 3 University C 66, ,606 1,643 South 4 University D 65,581 1,427 2,290 1,793 South 5 University E 10, South 6 University F 36,489 1,072 1, South 7 University G 28, South 8 University H 38, ,137 1,002 North 9 University I 60,271 1,503 2,195 2,162 North 10 University L 63,526 1,287 1,961 2,525 South 11 University M 21, , North 12 University N 25, North 13 University O 14, North 14 University P 17, North 15 University Q 22, North Total 597,239 12,983 18,773 17,869

11 PMS for University Central Administrative Services 247 Administrative director Finance Accounting ACCOUNTING ACTIVITIES in the project space Human resources Logistics Figure 2 The relationship between the project activities and the university organisational structure At the beginning, we were worried about the possibility of comparing our universities, which we know are organised differently. The presentation of the activity approach convinced us that the comparison was possible but we also realised that we needed to define a detailed and shared protocol to collecting data. I asked my people to pay particular attention to the technical design in order to interact with the team; this was crucial to understand what the data meant. (Administrative director at University C) Starting from this awareness, the process of translation of the PMS was based on a continuous interaction between the researchers and the university managers, and in this way it was possible to design the three dimensions of the system: costs, objective quality and subjective quality. The first measurement dimension, cost, required the definition of three elements: the cost pools, the activities and the drivers. The definition of the cost pool was straightforward, as it was possible to take advantage of previous projects in Italian universities (Arnaboldi and Azzone, 2006). The definition of the activities was instead a significant reason for discussion and translations in the project network; the interaction with the scientific and technical referees helped to better define the whole set of activities performed and to specify the collection protocol to guarantee comparability of data. Finally, the cost model introduced the drivers, which are the measures that are used to calculate a unit cost per output. The changes to the initial proposal made by the research team were marginal, but the interaction again served as a process of learning for the managers. The second and third measurement areas are related to effectiveness and adopt both objective and subjective measures. The first set of

12 248 Higher Education Quarterly indicators (objective effectiveness) has the aim of measuring the level of services actually offered by different universities (e.g. response time, service availability). The second set of indicators (subjective measures) has the aim of measuring the perception of the service quality by the users (e.g. satisfaction about the waiting time, service availability, information support). As far as objective effectiveness is concerned, a first proposal was circulated in the network and several comments were collected. The discussion was long and some managers proposed a much longer list of indicators; again the fact-building approach helped to show the possible results and the difficulties involved in carrying out an effective comparison among universities. The set of measures for objective effectiveness was significantly reduced after the discussion to include both quantitative and qualitative measures.the quantitative information refers mainly to the existence of specific services (e.g. frequently asked questions for students, counselling, etc.). On the other hand, qualitative information ensures a better understanding of the actual services provided, although more subjective dimensions, such as different procedures to enrol new students are also included. The discussion on subjective quality was based on the formulation of a questionnaire aimed at investigating user satisfaction. This instrument included closed questions against a four-point scale. 1 As in the previous steps, the first proposal, formulated by the research group, was discussed with the referees from the different universities and was revised according to their comments and suggestions. Again, the confrontation helped not only to tailor and refine the questionnaire but also to improve the managers competencies and awareness of measuring performances. A final question, investigating the overall service quality, was eventually included in the questionnaire to support the analysis of the relevance of each item on user satisfaction (see Appendix for translation of questionnaire). For subjective quality, the identification of the samples and the conducted survey were differentiated according to the users: the students or the teaching staff. A stratified sample was selected to monitor student satisfaction, which incorporated students of different ages (newly enrolled and graduating students) and from different faculties (scientific and humanistic faculties). The sample selection resulted in 14,300 questionnaires being distributed to the 15 universities. The survey of academic satisfaction targeted the whole teaching staff in each of the 15 universities who were asked to answer the questionnaire.

13 PMS for University Central Administrative Services 249 Acting the network: the implementation of the system The model presented in the previous section was implemented in the 15 universities involved in the study; the results of the experiment are presented and the operational are difficulties discussed. The operational difficulties encountered in the application were different for the three areas (efficiency, objective and subjective performance). Considering the application of the cost model, the universities were facilitated by previous projects; however, there were still some difficulties in data collection. The most critical part of the activitybased exercise was the definition of the percentage of time devoted to the activities; first, the university staff who were not involved in the design had difficulties in placing their activities in the correct section and understanding the terminology defined in the project space; second, they tended to see the collection of time percentage as a control device, which could affect their position within the organisation. The role of the technical referees was crucial in this situation and showed how participation in the meetings, in the development of the model and in the implementation itself enhanced their understanding of the PMS. The main issue of the implementation of the objective quality model was the relevance of qualitative information within the PMS. Participants frequently asked for reassurance over the definition of the qualitative dimensions, for example, the description of procedures. Although caution was adopted in phase (1), in order to try to achieve a high standardisation of the possible answers, the analysis was difficult. Some dimensions were excluded from the analysis to guarantee a significant comparison. Finally, the main problems concerning the subjective measures were related to the data collection. For the student satisfaction survey, the resulting number of questionnaires that resulted was suitable for processing was influenced by contingent situations. In two universities (Universities H and P), some professors did not allow the questionnaires to be distributed during their lessons; this led to fewer questionnaires being collected than was initially agreed upon. In another two universities (Universities A and M), there were a high number of unacceptable questionnaires because of errors in filling them in; the average response rate was 84.2 per cent. The data collection for the teaching staff survey involved various difficulties. In several institutions, for example, some respondents decided not to take part in the research project because they feared a possible lack of anonymity; the average response rate was 6.5 per cent.

14 250 Higher Education Quarterly Discussing the data In this section, the obtained data are presented with reference to three areas: the costs, the quality (subjective and objective), and the overall map of costs and quality. In the discussion of the data, the implications both at a managerial and a policy level have been highlighted. The first result area concerns costs; the collected cost results were compared with three specific indicators: the total cost of the activities, the unit cost of the activities and the service unit cost (obtained by dividing the total cost of the activities for each service by a unique driver). Table 4 shows an example of the unit costs for human resource services. Examining the first activity in Table 4 (teaching staff competition 2 ), university managers have the possibility of comparing their costs with other universities. For example, University G, with the highest costs, realised that a reduction of its cost on the average (1,825.6 per competition) would lead to a significant saving, as the following comment shows: I knew that we spent too much on competitions but we did not have a reference value. Last year we organised 40 competitions. If we had been able to have a lower cost, let s say the average value, we would have saved nearly 1,400 per competition, that means a total potential saving of nearly 56,000. We need to analyse the activities in depth but this is impressive. (Top manager at University G) The lowest cost for each activity is usually obtained by different institutions (Table 4): there is no best institution with the lowest cost in all activities. The adoption of the activity-based approach and the careful definition of the protocol also made it possible to define an overall efficiency measure for the five services. The calculation was made summing the total costs of the activities in the area (e.g. student support) divided by a unique driver (e.g. the number of students). Table 5 shows the unit costs, the difference from the minimum and the possible savings. A regression test on the university size was made, which highlighted the lack of scale effects. Examining these data, two implications emerge. These numbers offered the managers an overall comparison of the costs of the service, which may be used at the aggregate level to position the universities; second, at a more detailed level, the numbers indicate specifically which specific activities are particularly costly. Two results are relevant at the policy level. First, this comparison can offer target values for the unit cost of support activities, which may be

15 PMS for University Central Administrative Services 251 TABLE 4 The unit costs of the activities of the Human Resource Services Staff competition Administrative management for teaching staff Administrative management for professional staff Pension and Staff leaving indemnity External job authorization Teaching contract management Payroll management Training for professionals /competition /teaching staff /professional staff /file /person /contract /person /training hour A 2, B 1, C 1, D 1, E 1, F 1, G 3, H 1, I 1, L 1, M 2, N 1, O 1, P 1, Q 1, Mean 1,

16 252 Higher Education Quarterly TABLE 5 The overall unit costs for the student support services Unit cost [ /student] Unit cost Variance (actual cost minus minimum cost) [ /student] Driver [n. of students] Potential saving (variance*driver) [ ] University A , ,379 University B ,061 1,430,886 University C ,796 1,684,394 University D ,581 2,515,918 University E , ,058 University F ,489 1,265,227 University G ,770 0 University H ,610 1,885,420 University I ,271 1,457,578 University L ,526 1,277,107 University M , ,142 University N ,420 1,688,714 University O ,510 1,798,017 University P , ,534 University Q ,856 2,341,435 Minimum Cost Total saving = 19,132,810 used to steer a better use of government resources. Second, the results suggest that the size and the geographical position of the universities (northern against southern) are not related to higher or lower efficiency levels. The second area of data is the quality, which includes both the subjective (Table 6) and the objective measures (Table 7). The results of the student satisfaction survey provided a significant basis for comparing the performances of the universities. At the managerial level, each university not only had the possibility of seeing their position in relation to the overall satisfaction but also to understand what the most critical factors were for their students. Table 6 presents the scores of the survey; the universities are ranked in decreasing order with respect to the overall satisfaction. If we analyse university O, for instance, it can be seen that although it has a positive position in the overall ranking (first column), some parameters are below the average results: for example, front-end timetable and the waiting space.

17 PMS for University Central Administrative Services 253 TABLE 6 Subjective quality: descriptive statistics Overall satisfaction Staff competency Staff helpfulness Front-end timetable Automatic tools Waiting time Waiting space Information reliability Simplicity of forms Accessibility of forms Timeliness Consistency of information Coherence of information D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 C O E N Q P A M H F D L I G B Total

18 254 Higher Education Quarterly Weight University TABLE 7 Objective performance scoring system 0.33 Response time 0.33 Automation 0.33 Informative support University A University B University C University D University E University F University G University H University I University L University M University N University O University P University Q OVERALL SCORE Examining the overall survey data, an interesting result emerges from a comparison of the scores given by the students to each factor and their relevance on the overall satisfaction. 3 Figure 3 shows the graphical representation of these two variables. The x-axis indicates the attribute importance, measured by the standardised coefficients and the y-axis shows the performance (satisfaction about service quality). The performance and importance values divide the matrix into four quadrants.the position of each item in the matrix indicates whether student satisfaction attributes constitute major or minor strengths and weaknesses. Quadrant I contains factors with high performances and low importance. The resources directed here may be excessive and could be potentially employed more effectively elsewhere (consistency of information provided in different moments; coherence with information available elsewhere; simplicity of forms and documents; accessibility of forms). Quadrant II shows attributes considered very important by the students and where universities demonstrate high levels of performance. Figure 3 shows that no factor falls in this area. Quadrant III includes attributes considered very important for students but where the performance of the universities is fairly low.the factors positioned here are the most critical and include staff competency, staff aid, timeliness, waiting time and front-end timetable. Finally, quadrant IV shows the attributes

19 PMS for University Central Administrative Services Information reliability 2.80 Quadrant I Quadrant II Perceived effectiveness score Consistency of information Simplicity of forms Coherence of information Accessibility of forms Quadrant IV Automatic tools Waiting time Timeliness Staff helpfulness Quadrant III Staff competency 1.80 Waiting space Front-end timetable Importance of factors Figure 3 The importance performance matrix where universities perform badly but that students do not consider critical, such as the availability of automatic tool and the quality of waiting spaces (Harvey, 1994; Harvey et al., 1997). The analysis of the objective performance was based on the construction of a scoring system, which assigned predetermined scores, which were discussed within the project network, to each indicator. The indicators were then grouped into three main areas, namely response time, automation and informative support. A synthetic index was calculated for each area as an average of the different individual indicators. At the end, a single score was introduced to rank the objective performances of the universities. The results of the objective performance were also compared to those of the subjective performance, both at an aggregate level, considering the overall score and with respect to the single items (i.e. waiting time). This comparison helped the university managers identify areas with significant gaps between the level of service provided, given by the objective measure and the students perceptions. Such a situation could have been determined by faulty communication towards the students concerning the services actually offered by the administrative staff, which do not appear to be aligned with the students expectations. The final area of the results is dedicated to the overall picture of performances. A map was built for each of the five service areas, using

20 256 Higher Education Quarterly 2.0 C 1.8 Quality score O Q N H B D P F E I L M A G Variance from the minimun unit cost ( student) Figure 4 Map of the overall performances two items: the unit cost of the service (Table 4) and the sum of the subjective and objective quality scores. Figure 4 shows the performance map for student support services. The figure highlights an efficiency-effectiveness frontier formed by three good practices : university G has the lowest cost but also the lowest level of quality; university C has the highest quality score but a higher cost per student; finally, university A is in the middle with a good combination of quality and efficiency. This picture helped managers understand their position and to reflect on future investments, as this comment shows: Seeing our university [O] in the map was extremely helpful. We have invested significantly in the quality of services for students and I m happy about the results: our students are satisfied, the automation level is high and the information support resulted to be one of the best within the network. But the variance of our actual cost from the benchmark is high and maybe what we need to do now is to just maintain the level of service without searching for more and more improvements. (Administrative director at University O) Looking at the map from a policy point of view, the variety of positions is not surprising; it reflects a context of increased autonomy where universities may chose whether to support services at the minimum cost with an acceptable level of quality, or search for high-quality service but with higher costs.

21 PMS for University Central Administrative Services 257 Observing the spread of ideas The use of the performance measures in the organisational routine was different for the 15 universities; in particular, there were differences in the range of information used, in the organisational role of the users and in the goals. Some universities used all the information provided during the laboratory experiments, while others tended to focus on specific areas or specific areas of measurement. For example, universities H, M and O used all the data provided and even asked for further analysis. Some universities tended to focus on costs and some others on the quality perceived. The actors who acted upon the measures were mainly the administrative directors and the top managers; however, in four universities, the data were also required and used by the academic bodies and an internal evaluation committee. 4 The most interesting difference that emerged, however, concerns the goal pursued with the PMS. First, some universities used the data for internal reports to show the position of the university in comparison to other institutions; in these cases, no actions usually followed these publications. Two universities used the measures to analyse the organisational performance. University H in particular activated an internal participative analysis process, involving top managers in the administration; this process highlighted possible areas of improvement, which were inserted in the top managers evaluation system. Two universities used the results of the laboratory for external accountability; one published the results of the student satisfaction survey and this was followed by some focus groups with the students.the rector of University A used the data on costs during his public speech at the opening of the academic year. Finally, three universities made use not only of the collected data but also of the developed system. One university used this information to enlarge the exercise to decentralised structures (departments) while the other two used it to carry out a replication of the study. In these last cases, the universities tried to increase the involvement of lower levels of staff in the experimentation in order to enhance their understanding of the systems and to train them on the use of the model. Discussion and conclusions Higher education system reforms in Italy have drawn attention to the issue of accountability and performance measurement. Although the importance of accountability is uncontested, the debate on the use,

22 258 Higher Education Quarterly usefulness and possibility of measuring public-sector performances is still at the centre of research discussions. This paper contributes to this debate by presenting the results of a study that has developed a PMS for central administrative services in Italian universities. The initial idea of the study was set in a context of reform, to respond to the desire of both the university and the Ministry of the University 5 to carry out an experiment on university support services. The study drew on Latour s (1987, 1994) work and, more generally, on the actor network theory to develop the PMS. In particular, some key concepts that were appropriate to enhance the study were endorsed: the participative involvement of humans in acting and revising technologies; the use of fact-building to enhance the project network; the observation of ideas in actions that are internal and external to the project space, after the end of the project. The results of the study concern three main features: (1) the developed PMS; (2) the utility of the data obtained; and (3) the method adopted to develop the systems. The first main result of this study is the developed PMS, which is novel from two points of view. First, the system is complete and covers both the costs and the quality of the services provided. Previous studies focused on one specific element, in particular cost, without providing a complete picture of CAS performances. The complete quality and cost map allows different (un)intended choices of universities to manage their services to be seen and the balance between higher costs and higher quality to be verified. Considerations on quality can also be drawn from two different points of view, measuring objective indicators on the services provided (e.g. waiting time) and measuring the satisfaction of the users. This dual perspective has not only highlighted differences among the universities in both dimensions but it has also shown differences among the universities in the users perception of the services for the same provided quality.the second novel aspect of the PMS developed in this study concerns the fact that it can offer comparable data. This is a major issue for benchmarking services in universities, which usually have different organisational structures. The problem has been overcome in this study by adopting an activity-based model; with this model, the fundamental unit for measuring performances and then comparing data became the activity. Each area (e.g. student support services) did not refer to the organisational structure present in each university but to a series of activities that were defined and specified in a protocol by the participant universities themselves. In this way, each university knew, for example, that data for the payroll payments were collected with reference to the same protocol and that efficiency and quality performance could

23 PMS for University Central Administrative Services 259 be reliably compared. This advantage was possible because of the great effort that was made both in the design phase and in the data collection. In the design phase, the university managers, the officers and the research group prepared a detailed and shared protocol to prevent possible bias in the comparison. Data collection was then carried out with an ad hoc exercise in which the technical referees collected the data, regardless of their organisational structure. The second major result is the utility of data. Three main sets of data were obtained: the costs, the quality and the overall performance map. All of these analyses proved to be useful and meaningful to managers at an organisational level, but they could also be of benefit at a policy level. At a managerial level, the data on costs and quality, taken singularly, offered each university the possibility of seeing its position in comparison to other universities in the network. Furthermore, the entities with the minimum costs and the highest quality performance could become a target for further improvement. The details provided by the measures also made it possible to investigate which activities were particularly costly or which areas of quality were most critical. The comparison of the objective and the perceived quality also highlighted potential gaps between the level of services offered and students perceptions. Areas with higher gaps could perhaps benefit from an enhanced communication about the services provided. Finally, the overall picture of quality and costs created an interesting moment of self-reflection for the top managers: seeing their position in the map, they saw their strategy (or the absence of strategy) translated into performances. As testified by the administrative director of university O (with the highest quality but very high costs in student support services), seeing the position in comparison to other institutions helped him understand that they should stop investing in quality. In his words: maybe what we need to do now is just maintain the level of service without searching for more and more improvements. The data could also be interesting at a policy level.three main issues have emerged from the data. First, the cost data, for the five areas of services included in the study, showed differences in the efficiency levels and provided the possibility of simulating the possible savings for the systems if the inefficient institutions were to reduce their costs. Furthermore, the regression analysis showed that there were no significant scale effects in any of the five mapped areas. A second interesting result emerged from the data on quality; the overall analysis of the importance performance matrix could offer indications to policy makers on the factors which are considered important by the students and in which