The DUCKLING Project. Summative External Evaluation Report

Transcription

1 The DUCKLING Project Summative External Evaluation Report Final October 2010 Andrew K Comrie 0

2 Contents Executive Summary Introduction External Evaluation Focus Methodology What has been done/achieved, to what quality and how efficiently? What has been learned? How has the learning been acted on or fed back in? Did the DUCKLING Project meet its Aims and Objectives? Has the DUCKLING Project achieved its intended outcomes and outputs and shared these across the Institution and with the wider HE Community in the UK? Key Messages for University of Leicester Key Messages for the JISC and the wider HE Community Appendix Kerson Associates Ltd 1 25 October 2010

3 Executive Summary Conclusions This has been a well executed project that has very successfully achieved its aims and objectives. The Beyond Distance Research Alliance (BDRA) have worked proactively and effectively with two Schools across three distance learning Masters programmes to inform and drive the re-design of the programme s delivery models. The BDRA team have proved invaluable in promoting and encouraging academic staff to try alternative delivery approaches and have supported staff as they have grappled with the development of technology enhanced activities. The BDRA research role has ensured that the new delivery models implemented have been systematically evaluated. As a result, there is robust evidence and lessons learnt from the project that evidence positive impact to the learner experience and efficiencies in programme delivery that will prove invaluable to the University when trying to encourage other work-based distance learning programmes to adopt similar approaches in their own delivery models. The Teaching Fellow model adopted has provided an effective link between BDRA and its contributing programme teams. This linkage has ensured that academic staff have been able to access just-in-time support as they have developed knowledge and skills in the use of the technologies being introduced to change their delivery methods. The Teaching Fellows, with both their knowledge of the subject being taught and technologyenhanced learning and teaching have been able to work very effectively as Ambassadors in the Schools to promote, encourage and support academic staff to introduce technologies to their teaching methods. Overall the DUCKLING project has delivered its intended outputs and outcomes and has resulted in a significant change to the way the MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology) are now delivered as part of standard practice. The changes progressed under DUCKLING has complimented and informed other changes to programme delivery that were already in progress. The DUCKLING has been very active and effective in disseminating the DUCKLING project to its stakeholders and the wider HE Community. There is evidence to show that there is commitment to continuing dissemination and the sharing of DUCKLING s experience beyond the end of the project. Key Messages for University of Leicester Kerson Associates Ltd 2 25 October 2010

4 1. DUCKLING has demonstrated clearly that the BDRA plays a key role in learning and teaching change initiatives in the University. Their insight to learning futures is key to inform future re-designs of programme delivery and to provide timely support to staff when they embark on technology enhanced curriculum delivery changes. BDRA s research role is crucial to ensuring that changes to curriculum delivery models are systematically evaluated and that evidence of impact and lessons learnt are recorded. This is unique and often missing from e-learning projects in other institutions. This serves three purposes for the University: 1) it provides evidence insight and exemplars for other Schools planning different learning futures. The Carpe Diem process continues to be an effective way of transferring knowledge and experience of technology enhanced learning and teaching practice to other programme teams 2)it ensures that the University has a rich source of data that evidences the impact of their Learning Innovations Strategy in practice. 3) it ensures that the university s research work in e-learning research continues to be visible and continues to inform learning futures across the University and in the Higher Education Sector both nationally and internationally. 2. Teaching Fellows have proved effective Ambassadors for change to learning and teaching. Their knowledge of their subject discipline coupled with expertise in technology enhanced learning and teaching has ensured their credibility to both academic and technical/research teams alike. This has enabled them to provide convincing arguments and cases to engage and support academic staff in learning and teaching change initiatives. The University should consider how they might continue this role beyond the life of the DUCKLING project to further cascade DUCKLING s work in support of university wide implementation of the learning Innovation Strategy. 3. There is evidence from the project that the use of new technologies is resulting in an evolution of the moderator role in distance/work-based curriculum delivery models where associate tutors are being used effectively to enhance support to remote learners cost effectively. This should be examined further and considered more widely for distance learning delivery across the Institution. 4. DUCKLING has demonstrated that projects of this kind can produce open educational resources (OERs) which can provide a cost-effective way of sharing resources and expertise to assist the transformation of practice in other disciplines. Kerson Associates Ltd 3 25 October 2010

5 Key Messages for the JISC and the wider HE Community 1. DUCKLING has started to demonstrate the relationship between the cost of technology enhanced curriculum development and impact. It is clear that some technologies such as podcasts are emerging as low cost/high impact, whilst other technologies have higher development costs with less impact on the learner experience. The Beyond Distance Learning Research Alliance team should be encouraged and supported to progress this work and share it more widely with the HE community. 2. Voice boards have emerged as an unplanned and unintended outcome from the DUCKLING project. Early trials have demonstrated their potential to enhance the learner experience especially in relation to work-based distance learning. JISC should consider working with the Beyond Distance Learning Research Alliance team to explore ways of funding further studies in this area. Kerson Associates Ltd 4 25 October 2010

6 1. Introduction The DUCKLING Project (Delivering University Curricula: Knowledge, Learning and Innovation Gains) is one of 25 projects funded by the JISC s Transforming Curriculum Delivery programme. The project ran between November 2008 until October The focus of the DUCKLING project was on advanced delivery, presentation and assessment processes to enhance the experience of work-based learners studying remotely. The project set out to demonstrate the marriage between sound approaches in delivery with new technology and work-based pedagogies for learning support, communication and assessment of professional adult learners from commencement to completion of the programme of study. DUCKLING Revised Project Plan (2009). Work-based/professional development programmes from two disciplines were selected to pilot the enhanced delivery models developed during the project: MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology) This report provides a summative external evaluation of the project. 2. External Evaluation Focus The external evaluation of the DUCKLING project has been structured around three central questions as defined in the JISC six-step evaluation model 1 : What have we done/built/achieved, to what quality, and how efficiently? What has been learned or confirmed through development activities? How has the learning been acted on or fed back in? The summative external evaluation report also confirms: The extent to which the project aims and objectives have been met. The extent to which the project has achieved its intended outcomes and outputs and shared these with project stakeholders and the wider HE community in the UK. Key messages for the University, the wider HE community and/or the JISC. 1 df Kerson Associates Ltd 5 25 October 2010

7 3. Methodology The methodology used to externally evaluate the DUCKLING project included: a. Structured interviews with the Project Team and Programme Delivery Teams running the MA Applied Linguistics and TESOL, MSc in Occupational Psychology and the MSc in Psychology in Work. b. Structured telephone Interviews with students c. Literature Review. The literature review took into account the project website and key documents including: Project Plan Interim Reports for the JISC Final Report (Draft) for JISC DUCKLING Research Reports Interim External Evaluation Reports Steering Group Meeting Minutes Benchmarking Reports A set of structured questions were developed and used during external evaluation visits with the project and programme delivery teams to capture information about the project to enable the external evaluator to assess the extent to which the project was progressing in accordance with planned activities, aims, objectives and planned outcomes and deliverables set out in the project plan. Three interim evaluation reviews were conducted during the life of the project in July 2009, January and March Interim Evaluation reports were produced at each Interim Review point confirming progress and giving recommendations, as appropriate, aimed at providing the Project Team with formative feedback intended to help inform future planning and activity. Interim reports were structured around three central evaluation questions: What has the project done/built/achieved, to what quality and how efficiently? What has been learned or confirmed through development activities? How has the learning been acted on or fed back in? Each report considered the three central evaluation questions and summarised findings to date and made recommendations to the project team intended to inform and shape the project s progress and maximise the project s outcomes and outputs. Kerson Associates Ltd 6 25 October 2010

8 4. What has been done/achieved, to what quality and how efficiently? The DUCKLING project aimed to identify new opportunities for enhancing curriculum delivery for work-based programmes and to develop a range of technology-based responses. DUCKING targeted three post-graduate programmes: MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology) with the objective of aligning learning experiences with students work-related needs enabling active, situated and work-based learning. Five key challenges in curriculum design and delivery were identified at the start of the project from consultation with staff, students and employers. These were: 1. Improving learner engagement with the course content by bringing the materials to life and offering a variety of teaching approaches. 2. Improving learner support by offering enhanced guidance, support and feedback in a variety of media formats. 3. Enhancing flexibility and mobility in programmes aimed primarily at time-poor, work-based learners. 4. Reducing learner isolation through the provision of additional opportunities for student-student, student-tutor and student-e-moderator interactions. 5. Enabling skills development and transferring theory into practice for work-based learners. The project aimed to address these challenges and work was progressed through ten work-packages as detailed in the Revised Work Plan (2008). These activities have delivered the following outcomes and outputs: Work Package 1: Project Setup, Recruitment and Training The activities listed in Work Package 1 were completed satisfactorily and resulted in the following outcomes and outputs: The project team was established which included a research team, teaching fellows and Project Champions in the Schools of Psychology and Education.. The Project Steering Group was established and chaired by Christine Fyfe, Pro Vice Principal at the University of Leicester. The Steering Group included by internal representation at senior level and external representation. Participation in JISC programmes level activities. The project plan was finalised and submitted and approved by JISC. Kerson Associates Ltd 7 25 October 2010

9 Work Package 2 and 3: Review Current Processes and Practices and Consultations with Stakeholders The activities listed in Work Packages 2 and 3 were completed satisfactorily and resulted in the following outcomes and outputs: Information about current processes and practices was gathered by the project s Research Associate, Dr Ming Nie, in association with the two Teaching Fellows for Education and Psychology using a range of data collection methods including student online questionnaires, semi-structured interviews with students, analysis of student progression data, informal meetings and discussion with members of the programme team and semi-structured interviews with employers. The information was analysed and findings presented in Baseline Reports for both subject disciplines participating in the project. Work Package 4: Understanding and Characterising Changes and Planning Action The activities listed in Work Package 4 were completed satisfactorily and resulted in the following outcomes and outputs: A number of surveys, semi-structured interviews and cognitive mapping interviews were carried out during initial podcast, e-book reader and Second Life (SL) pilots and used to determine the learners experience of the technologies and how they were used in their learning. In the final year of the project, WIMBA Voice Board emerged as a technology to further enhance work done with podcasting. Voice boards were trialled with a small group of 6 learners on the education programme and the student experience tracked and analysed. The research has been written up and the reports used to inform conference papers and journal articles written by the Research Associate and project team. Findings have been used to inform further pilots during the life of the project and have informed future developments planned to sustain work post-project. Work package 5 and 6: Appropriate Curriculum Delivery Presentation 1 and Optimising Delivery The activities listed in Work Packages 5 and 6 were completed satisfactorily and resulted in the following outcomes and outputs: Two Curriculum delivery examples have been produced illustrating the curriculum delivery models used by the Education and Psychology programme teams before and after DUCKLING. The psychology team have produced more than 100 Podcasts in six categories for the post-graduate psychology programmes. A summary of Psychology podcasts can be found at the DUCKLING website Some of the podcasts were submitted to another JISC funded project run by the Beyond Distance Learning Research Alliance Kerson Associates Ltd 8 25 October 2010

10 (OTTER) which was exploring the creation of Open Educational Resources (OERs). These podcasts are released as OERs, ( Education programme team have produced a series of podcasts to supplement content of three modules. A summary of these podcasts can be found at the DUCKLING website There is evidence emerging that the podcasts produced by the Psychology team to support masters dissertation is resulting in improved quality of dissertations and retention. A number of post-graduate students from both Psychology and Education participated in Second Life trials. The psychology team created a virtual oil rig platform (available from the Media Zoo Island to teach health and safety. Full details are available from The Education, students were encouraged to visit virtual language labs where English as a second language was being taught. Full details are available from Despite small numbers of students volunteering to participate in these trials, there being some considerable technical barriers associated with accessing Second Life and the cost of implementing Second Life being high with a low impact return, the trials have given the programme teams an opportunity to review the potential for virtual worlds in the course delivery. The education team successfully demonstrated that Second Life, although often viewed for synchronised delivery can be used asynchronously, flexibly and inexpensively when combined with other learning tools (such as VLE discussion forum) for project work. As a result, the education programme team will continue to give their students opportunities to use and evaluate the potential of Second life in teaching languages as an option in future programmes as part of the Computer Assisted Language Learning module. Whilst the experience has helped the psychology team identify where virtual worlds could be used in their course delivery models in the future, they do not plan to continue with Second Life in its current form. This work has produced a series of Second Life training guides, resources which have been added as OERs to the OER repository produced as part of the OTTER programme and the work has helped to inform both technical and teaching developments in a new project (SWIFT ) funded under the HE Academy National Teaching Fellow initiative. The use of e-book readers have been successfully piloted by both education and psychology students. Despite the project experiencing difficulties in negotiating copyright for core texts for the courses in e-book format with publishers, by pre-loading course content and resources in e-book format to the readers has enabled the project to demonstrate the extent to which mobile technologies such as e-book readers give distance/remote and work-based students flexibility of choice of where and when to study and mobility. There is also evidence emerging to show that this delivery approach is encouraging learners to print less. Whilst it is unlikely that the programme teams will continue to provide e-book readers for their courses post-project, it has highlighted the need to produce course materials in the future in different formats (e.g. e-pub format) so that materials can be downloaded to portable devices should students wish to use resources flexibly. Kerson Associates Ltd 9 25 October 2010

11 The Psychology team has already converted all module materials into epub and delivered them via Blackboard VLE for students to drag and drop onto their e- book readers. This development will continue and be cascaded to others. As a result the project has produced a guide on converting Word documents into epub format which has been made available as an OER Other resources produced include An e-book reader video: A recording of a discussion session with postgraduates at the University of Leicester via the Graduate School Media Zoo about e-books and e-readers, An unplanned outcome for the project has been the emergence of voice boards as a technology straddling the benefits of podcasting. As a result the project team have built trials of Wimba voice boards into the project with a small group of 6 volunteer students on the education programme. Feedback from students was collected via the voice board itself and analysed. The voice board was being used to generate interaction between students and get them to discuss subject-specific concepts and issues, in the context of the formative assessment tasks. These tasks had previously been carried out by students individually in the form of text-based portfolios. Verbal feedback for students and staff gathered during the evaluation suggest that voice boards are enhancing practice. For example students cited that the opportunity to hear tutors and other students, share work and gain feedback helped to overcome feelings of isolation often experienced by distance learners. Staff are finding that voice boards are enabling increased interactivity between students which is often difficult to achieve with remote learners. Both students and staff think that interactive voice board activities (v-tivities) are the best way to structure student discussion and for formative assessment on voice boards. As a result of the voice board trials, e- moderation models are being enhanced. The work emerging from this project will inform the revision that is currently underway of Professor Salmon s e-moderating text book. In addition, the project team have produced a Voice Board user guide and a guide for e-moderating on voice boards. Kerson Associates Ltd October 2010

12 Work package 7: Data Analysis and Modelling The activities listed in Work Package 7 were completed satisfactorily and resulted in the following outcomes and outputs: Cognitive Mapping has been used for interviews with students involved in the SL and e-book reader studies as a way of triangulating responses from surveys and confirm responses. This methodology has helped the research team to identify in a more focussed way the changes in learning behaviour for individual students and to map these in a way to identify the core causes that has brought about the change(s). Further analysis of individual maps and responses has usefully helped the research team to identify key concepts and themes emerging from trials using the technologies being explored. This work is supporting the development of seven case studies that illustrate the changes made to curriculum delivery models by the Psychology or Education programme teams during the project and guidelines for other staff and managers in the HE sector interested in developing flexible and learner centred, technology-enhanced work-based curriculum delivery models. The project team have continued to liaise with the JISC Curriculum Design/Delivery programme teams and practitioners in the sector to develop an alternative curriculum lifecycle model that supports effective, scalable and sustainable work-based distance delivery. Full details of the model are available from ( At the time of writing this evaluation report it remains unclear how this model might influence the future development of the delivery models for work-based and distance learning in the University of Leicester and more widely in the sector. This is something that needs further consideration by the project team. Work Package 8: Planning for Embedding and Sustainability The activities listed in Work Package 8 were completed satisfactorily and resulted in the following outcomes and outputs: Feedback from practitioners in both the Education and Psychology teams evidence that the experiences from DUCKLING are being embedded and that resources such as podcasts will be continued to be used and further developed post project. Whilst not all of the technologies trialled during the project will be continued, programme teams have learnt from the experience and are evaluating how key lessons learnt can be progressed. For example, whilst the psychology team will not providing e-book readers for future iterations of their programmes, they will be creating content on an ongoing basis in different formats so that students can download them to portable devices. Kerson Associates Ltd October 2010

13 There is evidence that staff from both programme teams are promoting their work internally and externally. For example the education team has given a presentation on DUCKLING to the wider school in the presence of the Vice Chancellor. The Education team has got a bid for university funding to progress podcasting work in association with the university s Distance learning forum. All programme members of the psychology team are now using podcasts in their respective modules as standard practice. Team members have prepared presentations and conference papers to other psychologists and internally, there is interest from the criminology and forensic science programme teams in the work progressed in DUCKLING. Work Package 9: Project Evaluation The activities listed in Work Package 9 were completed satisfactorily and resulted in the following outcomes and outputs: Three Interim External Evaluation Reports providing formative feedback to the project team. A summative External Evaluation Report was produced at the end of August A distance learning benchmarking group, facilitated by Prof Paul Bacsich (external consultant) and coordinated by Jaideep Mukherjee (representing the DUCKLING project & the University of Leicester) and involving seven partner institutions has been established: University of Leicester, UK (Lead) University of Liverpool, UK University of Southern Queensland, Australia Massey University, New Zealand Thompson Rivers University, Canada Lund University, Sweden KTH (Royal Institute of Technology), Sweden. At the time of writing this report, six out of the seven partners had completed the benchmarking exercise using the Pick & Mix model which has been updated for this purpose. Further benchmarking activities were planned although not concluded. It is anticipated that the planned Benchmarking report will be complete however, by the end of November 2010 as agreed. Kerson Associates Ltd October 2010

14 Work Package 10: Dissemination The activities listed in Work Package 10 were completed satisfactorily and resulted in the following outcomes and outputs: The project website and other communication/dissemination tools (e.g. Project Blog) have been created and maintained during the life of the project:. DUCKLING website: DUCKLING blog: A full list of all the project s dissemination activities is provided at A journal article based on Psychology Podcasting, published in ALT-J, ( titlelink). Two Interim Reports for JISC have been produced and submitted with positive feedback from the JISC programme team and critical friend ( ing.aspx). A draft version of the final report was submitted at the end of August and the project team are on course to submit the final report and completion report by the 31 October A Benchmarking Report is due by the end of November 2010), Other resources and outputs developed have included: A summary of how DUCKLING technologies addressed the original challenges A cost-benefit analysis of DUCKLING technologies A DUCKLING booklet: providing an overview of DUCKLING methods and key findings A document about the development of a consultancy task, produced using video podcast and run as part of the 2010 Occupational Psychology (OP) Course Conference. An overview document about how DUCKLING technologies were integrated into the two Psychology programmes. Whilst all DUCKLING deliverables and outputs will be made available to the wider HE sector, this was work in progress. Plans were progressing well for deliverables to be made available from the DUCKLING website by the end of the project in October Kerson Associates Ltd October 2010

15 5. What has been learned? The key learning points from the DUCKLING project include: Programme Teams Both the education and psychology programme teams have engaged actively and purposefully with the project and the technologies being piloted. As a result there is evidence that staff have new knowledge and skills in the use of podcasting, e-book readers, Second Life and voice boards. Experimentation has enabled staff to test if the technologies being used were able to assist them to address the key issues and challenges identified at the start of the project. Whilst not all the technologies used will continue beyond the life of the project, staff at least now have first hand experience of these technologies and clearly understand their potential in curriculum delivery. Importantly, however, programme-wide innovations are now being introduced and embedded into curriculum delivery models, cascaded to colleagues and are becoming standard practice in delivery models. The role between tutors, moderators and teaching fellows have evolved during the project and are resulting in enhanced support during course development and delivery. Students There is strong evidence that the DUCKLING project has enhanced the student learning experience, increased the level of personalisation, and improved flexibility of the curricula to support and accommodate the mobile learning needs of distance work-based learners. The University and the wider HE Community The project has demonstrated the effectiveness of the learning technologies trialled in enhancing learning and teaching. Sound evidence is emerging in support of podcasting as a low cost/high impact technology widely regarded by staff and students alike. Whilst it is unlikely that the programme teams will continue to provide e-book readers for their courses post-project, it has highlighted the need to produce course materials in the future in different formats (e.g. e-pub format) so that materials can be downloaded to portable devices should students wish to use resources flexibly. The Psychology team has already done so. Programme teams have had opportunities to trial virtual world tools (Second Life) and explore how they can be used in their course delivery models. This work has informed the development of an elective module in the Linguistics programme and helped to inform both technical and teaching developments in a new project (SWIFT, funded under the HE Academy National Teaching Fellow initiative. The evidence-based research that has underpinned this project is helping support change across the university and give staff confidence to embrace change. Kerson Associates Ltd October 2010

16 6. How has the learning been acted on or fed back in? The project team and programme teams have actively disseminated their work throughout the project. The following evidences this: Project Website and Blog. The DUCKLING website and Blog have been regularly updated throughout the project. Key outputs from the project are available from the website. DUCKLING website: DUCKLING blog: JISC Programme/Cluster Activities. The DUCKLNG project team have participated actively in JISC Curriculum Delivery/Design Programme Meetings and Cluster meetings with their Critical Friend Peter Chatterton. Steering Group. The project team have engaged effectively with the project Steering Group which has been chaired by Christine Fyfe, Pro Vice-Chancellor, Student Experience and University Librarian. This has ensured DUCKLING achievements and challenges have been understood and supported at senior levels in the University. Other Projects run from the Beyond Distance Research Alliance (BDRA). The project has both drawn from earlier research work carried out by the BDRA team and has contributed resources and further insights to projects that have either been running in parallel with DUCKLING (e.g. the OTTER project where a number of outputs produced from DUCKLING were submitted to OTTER as OERs) or projects that have proceeded DUCKLING (e.g. SWIFT where experiences of developing and using Second Life for Learning and teaching has informed the development of virtual genetics labs where students can explore, experiment and evaluate situations in risk free interactive activities within the virtual laboratory. Such activities would not replace but rather enhance the effectiveness of real-life laboratory work. Dissemination to the Wider HE Community. The project team and programme teams have very proactively presented papers and written journal articles about their work and the lessons learnt. A full list of all the project s dissemination activities is provided at Guidelines and Resources. A range of guidelines, models and resources have been developed aimed at others planning to embed technologies into workbased learning programme delivery. These are available from Cascade of work to other programme teams and disciplines in University of Leicester. There is evidence that the project team and programme teams are actively cascading their work to others in the university. This is being done through Carpe Diem workshops and through presentations to other interested programme teams. Kerson Associates Ltd October 2010

17 7. Did the DUCKLING Project meet its Aims and Objectives? Overview The DUCKLING project aimed to identify new opportunities for enhancing curriculum delivery for work-based programmes and to develop a range of innovative technology based responses. These were to be piloted with three postgraduate distance learning demonstrator programmes in psychology and education. The DUCKLING Objectives were to: a) Enhance distance and work-based delivery with three innovative, technologymediated approaches to learning. b) Engage key stakeholders throughout the delivery, development and piloting c) Be flexible enough not to require re-validation by enabling subject teams to design once and delivery many times d) Inform the two key curriculum delivery-orientated elements of the University of Leicester s new Distance Learning Strategy. e) Inform senior managers, strategy and policy makers in the University and across the Sector. Key Findings a) Enhance distance and work-based delivery with three innovative, technology-mediated approaches to learning. The project has successfully introduced four new technologies to the delivery of modules in the MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology): Podcasts E-book Readers Second Life Wimba Voice Board b) Engage key stakeholders throughout the delivery, development and piloting. The project has had a dedicated Project Team who has actively sought to engage key stakeholders including: Programme team members teaching on the MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology). Students studying remotely from the university on the MA Applied Linguistics and TESOL and MSc in Occupational Psychology and MSc in Psychology of Work Kerson Associates Ltd October 2010

18 Employers in ELF (English Language Teaching) and Occupational Psychology Other projects funded within BDRA Senior Managers and policy makers in the University of Leicester. The wider HE Community c) Be flexible enough not to require re-validation by enabling subject teams to design once and delivery many times There is evidence to show that programme teams are designing resources and tools for one programme and/or mode of delivery and now re-using them in other programmes. For example, resources and tools such as podcasts are being cascaded and used by other programme teams and tools developed for distance work-based delivery are being evaluated and used for campus-based delivery. d) Inform the two key curriculum delivery-orientated elements of the University of Leicester s new Distance Learning Strategy. DUCKLING s work has produced a strong evidence base that demonstrates the realisation of the University s Learning Innovations Strategy in practice. Evidence from evaluating the student experience of the four technologies piloted during this project is showing that the technology enhancements made to the delivery model of three programmes have enhanced the work-based distance learner experience and importantly for a dual-mode university such as Leicester, programme teams are now recognising that the models developed initially for distance learners, can be used effectively for campus-based deliver too. e) Inform senior managers, strategy and policy makers in the University and across the Sector. The Beyond Distance Learning Research Alliance (BDRA) has taken an active role in promoting the work of DUCKLING to academic staff across the University using their CARPE DIEM workshops. The staff from the programme teams have also actively promoted their work to colleagues in their Schools and to experts in their disciplines in other institutions. The Project team have actively participated in JSC programme meetings and online events and have offered ideas and comments informing the development of JISC s Design Studio. They will also contribute resources to the Design Studio as part of their final deliverable from the project. The project team have actively disseminated their work. This is evidenced on their project website where all dissemination activities are recorded Kerson Associates Ltd October 2010

19 Conclusion This has been a well executed project that has very successfully achieved its aims and objectives. The Beyond Distance Research Alliance (BDRA) have worked proactively and effectively with two Schools across three distance learning Masters programmes to inform and drive the re-design of the programme s delivery models. The BDRA team have proved invaluable in promoting and encouraging academic staff to try alternative delivery approaches and have supported staff as they have grappled with the development of technology enhanced activities. The BDRA research role has ensured that the new delivery models implemented have been systematically evaluated. As a result, there is robust evidence and lessons learnt from the project that evidence positive impact to the learner experience and efficiencies in programme delivery that will prove invaluable to the University when trying to encourage other work-based distance learning programmes to adopt similar approaches in their own delivery models. The Teaching Fellow model adopted has provided an effective link between BDRA and its contributing programme teams. This linkage has ensured that academic staff have been able to access just-in-time support as they have developed knowledge and skills in the use of the technologies being introduced to change their delivery methods. The Teaching Fellows, with both their knowledge of the subject being taught and technology-enhanced learning and teaching have been able to work very effectively as Ambassadors in the Schools to promote, encourage and support academic staff to introduce technologies to their teaching methods. Kerson Associates Ltd October 2010

20 8. Has the DUCKLING Project achieved its intended outcomes and outputs and shared these across the Institution and with the wider HE Community in the UK? Overview The outcomes and outputs from the DUCKLING project were: Outputs A transferable model for effective, scalable inclusive and sustainable workdistance-based curriculum delivery within a dual-mode university, with discipline based comparisons Exemplar delivery curricula for each pilot/demonstrator programme. With relevant documentation An inventory of examples of technology-enhanced, re-usable and reversionable approaches to solutions to WB delivery challenges, with associated evidence of use, to foster transferability. Guidelines for HE Practitioners and managers for developing flexible, informed and learner-centred WB curriculum delivery processes in their contexts. Interim and Final reports and case studies, as required by the funder A report-based on two annual benchmarking studies ( ) involving two or more dual-mode HEI offering work-based distance learning across Europe. A community of practice of work-based deliverers from dual-mode UK universities Outcomes Modelling curriculum delivery enhancement: Enhanced curriculum delivery processes to support flexible delivery meeting diverse and changing learner requirements. Improved understanding of effective curriculum delivery for work-based learning and distance informed by all key stakeholders Improved understanding of research to practice curriculum delivery processes and how they can be supported by technology to transform learning opportunities for students learning and earning and achieve strategic objectives. Evidence of learners achieving their goals via flexible, well designed, learners and employer informed delivery processes. An enhanced approach to WB curriculum delivery relevant to other subject areas, institutions and stakeholders. Contributions to national policy in curriculum design at pedagogic and institutional levels. Effective dissemination Kerson Associates Ltd October 2010

21 a. Sector wide dissemination of findings and engagement with key stakeholder communities b. Evidence for the value of annual DL benchmarking exercises across comparable HEIs Development in dual-mode universities of exemplary remote learning delivery Sustainability and development across the Sector. Key Findings Section 4 of this report highlights the key outputs and outcomes achieved from this project. Conclusions Overall the DUCKLING project has delivered its intended outputs and outcomes and has resulted in a significant change to the way the MA Applied Linguistics and TESOL (School of Education) and MSc in Occupational Psychology and MSc in Psychology of Work (School of Psychology) are now delivered as part of standard practice. The changes progressed under DUCKLING has complimented and informed other changes to programme delivery that were already in progress. The DUCKLING has been very active and effective in disseminating the DUCKLING project to its stakeholders and the wider HE Community. There is evidence to show that there is commitment to continuing dissemination and the sharing of DUCKLING s experience beyond the end of the project. Kerson Associates Ltd October 2010

22 9. Key Messages for University of Leicester 5. DUCKLING has demonstrated clearly that the BDRA plays a key role in learning and teaching change initiatives. Their insight to learning futures is key to the university to inform future re-designs of programme delivery and to provide timely support to staff when they embark on technology enhanced curriculum delivery changes. BDRA s research role is crucial to ensuring that changes to curriculum delivery models are systematically evaluated and that evidence of impact and lessons learnt are recorded. This is unique and often missing from e-learning projects in other institutions. This serves three purposes for the University: 1) it provides evidence insight and exemplars for other Schools planning different learning futures. The Carpe Diem process continues to be an effective way of transferring knowledge and experience of technology enhanced learning and teaching practice to other programme teams 2)it ensures that the University has a rich source of data that evidences the impact of their Learning Innovations Strategy in practice. 3) it ensures that the university s research work in e-learning research continues to be visible and continues to inform learning futures across the University and in the Higher Education Sector both nationally and internationally. 6. Teaching Fellows have proved effective Ambassadors for change to learning and teaching. Their knowledge of their subject discipline coupled with expertise in technology enhanced learning and teaching has ensured their credibility to both academic and technical/research teams alike. This has enabled them to provide convincing arguments and cases to engage and support academic staff in learning and teaching change initiatives. The University should consider how they might continue this role beyond the life of the DUCKLING project to further cascade DUCKLING s work in support of university wide implementation of the learning Innovation Strategy. 7. There is evidence from the project that the use of new technologies is resulting in an evolution of the moderator role in distance/work-based curriculum delivery models where associate tutors are being used effectively to enhance support to remote learners cost effectively. This should be examined further and considered more widely for distance learning delivery across the Institution. 8. DUCKLING has demonstrated that projects of this kind can produce open educational resources (OERs) which can provide a cost-effective way of sharing resources and expertise to assist the transformation of practice in other disciplines. 10. Key Messages for the JISC and the wider HE Community 3. DUCKLING has started to demonstrate the relationship between the cost of technology enhanced curriculum development and impact. It is clear that some technologies such as podcasts are emerging as low cost/high impact, whilst other technologies have higher development costs with less impact on the learner experience. The Beyond Distance Learning Research Alliance team Kerson Associates Ltd October 2010

23 should be encouraged and supported to progress this work and share it more widely with the HE community. 4. Voice boards have emerged as an unplanned and unintended outcome from the DUCKLING project. Early trials have demonstrated their potential to enhance the learner experience especially in relation to work-based distance learning. JISC should consider working with the Beyond Distance Learning Research Alliance team to explore ways of funding further studies in this area. Kerson Associates Ltd October 2010

24 Appendix 1 The following people have contributed to the external Evaluation of DUCKING Individual Prof. Gilly Salmon Jaideep Mukherjee Dr Alejandro Armellini Dr Ming Nie Gabi Witthaus Ray Randall Sue Harrington Pam Rogerson-Revell Diane Davies Claire Hardy Adam McRoy Timothy Nicoll Anna Smith Title Professor of E-Learning, Beyond Distance Learning Research Alliance Project Manager Senior Learning Designer, Beyond Distance Learning Research Alliance Research Associate Teaching Fellow (Education) Project Champion /Programme Leader (Occupational Psychology) Course Team (Occupational Psychology) Course Team (Education) Course Team (Education) Teaching Fellow (Psychology) Education Student Education Student Psychology Student Kerson Associates Ltd October 2010