IMPACTFUL, QUANTIFIABLE AND TRANSFORMATIONAL?

IMPACTFUL, QUANTIFIABLE AND TRANSFORMATIONAL? EVALUATION OF THE IMPROVING QUALITY TOGETHER (IQT) NATIONAL LEARNING PROGRAMME Report for 1000 Lives Improvement Service, Public Health Wales Mark Llewellyn, Jennifer Hilgart, Marcus Longley and Nicola Davey Welsh Institute for Health and Social Care University of South Wales July 2015

ACKNOWLEDGEMENTS Thanks are due to the Improving Quality Together (IQT) team of 1000 Lives Improvement Service within Public Health Wales for commissioning this study and providing direction throughout. Thanks are also due to the IQT leads across the health organisations in Wales. As with any such project, this study was only possible thanks to the contributions of the participants who took part either as research participants or in helping to supply data to the research team. Their willing engagement with the study, openness and honesty is gratefully acknowledged. The report analyses the findings generated during the course of the study. The conclusions we come to are based on our understanding of the evidence presented to us at this point in time and any errors of interpretation are solely due to the authors. We trust that the independent analysis of the data will help to ensure that IQT continues to develop and evolve, and that it continues to respond to the challenges facing the NHS in Wales. Dr Mark Llewellyn, Professor Marcus Longley, Jennifer Hilgart and Nicola Davey 1 Welsh Institute for Health and Social Care (WIHSC) July 2015 1 Nicola is an experienced pharmacist and quality improvement practitioner who specialises in service improvement and is an Associate of WIHSC. She provides consultancy services to providers and commissioners in quality improvement, patient safety, medicines management, and clinical governance and risk. With the Quality Improvement Clinic she provides opportunities for virtual engagement to enhance impact and delivery during and after face-to-face encounters. Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 1

EXECUTIVE SUMMARY The study was commissioned by the Improving Quality Together (IQT) team of 1000 Lives Improvement Service within Public Health Wales and the Welsh Institute for Health and Social Care (WIHSC) at the University of South Wales was asked to undertake an evaluation of the national programme. STRUCTURE OF IQT Improving Quality Together (IQT) is the latest in a line of quality improvement initiatives that began nationally in 2013. The Programme consists of four levels: 1 IQT Bronze which is open to all staff and contractors. IQT Bronze comprises e-learning modules that can all be completed in less than two hours. There is a national target that 25% of the NHS Wales workforce completes this qualification; 2 IQT Silver involves a combination of further learning and experience of delivering a small service improvement in practice. There are different approaches to delivering IQT Silver across Wales. Some health organisations provide IQT Silver as a complete package, meaning that they deliver training to learners and require them to complete a project, and have it assessed, before they receive their qualification. In other organisations, however, they split the qualification such that it is possible to gain a Silver training or foundation level qualification for completing the classroom element, which can then be augmented by a further qualification if learners subsequently complete a project. 3 IQT Gold is designed for those who have received the IQT Silver Award and aims to offer more advanced systems level improvement knowledge, tools and techniques, and there is a Gold Network which brings together leaders in quality improvement across Wales to share learning and approaches to further embedding IQT; and 4 IQT at Board level which aims to spread the common language of quality improvement by training all board members to bronze level, and raising awareness of how IQT relates to the everyday business of health organisations in Wales. PURPOSE OF THE STUDY The purpose of this study was to provide an independent and objective evaluation of IQT, and was undertaken through using a multi-methods approach using four key methods: 1 A bilingual online survey was developed for IQT Bronze and Silver learners to complete; 2 Semi-structured interviews with members of NHS staff from across Wales who were at some level of IQT Silver training; 3 A focus group of the Gold Network to discuss their perspectives on IQT as a programme; and 4 A range of data was collected from organisational leads for IQT across Wales. This included information in the form of storyboards (see Appendices I-VI) which provided a snapshot of their experience, data on case studies which demonstrated the impact of IQT in bringing teams and staff together, and information on the numbers of learners within their organisations IQT KEY NATIONAL PERFORMANCE DATA There are some significant uncertainties in the numbers of those completing IQT in Wales, so the following should be treated with caution. That said, The IQT Bronze 25% target has not been achieved to date, although more than 13,000 staff have undertaken the qualification, which equates to around Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 2

16% of the workforce. There are four health organisations who have achieved above the target level. There is no national target for numbers at IQT Silver level, but more than 1,200 people have completed IQT Silver training, which equates to 1.5% of the NHS Wales workforce. These numbers are understandably lower for those who have completed IQT Silver projects, with around 150 staff members completing a project. IQT BRONZE SURVEY FINDINGS 620 people responded to the IQT Bronze survey. Key findings from the analysis were: Respondents were generally positive about the IQT Bronze training, with a majority agreeing that they learnt something new about quality improvement and that they feel confident to put their training into practice; When asked if they had put their training into practice in their workplace, 50% said yes, and 50% said no; The main barriers for not putting Bronze IQT training into practice were lack of time and the fact that some staff only did the training because they were told to do it; When respondents did report that they had put their learning into practice, the most commonly reported impacts were streamlined processes and procedures with improved efficiency. Some respondents also reported on the benefits of having a consistent approach to quality improvement which enhances understanding of improvement and facilitates team-based working; 65% (n=395) of respondents reported that they had considered IQT Silver but had done nothing about it yet, whilst 143 (24%) were not interested in any further quality improvement training; There were some differences between health organisations in the proportion of respondents who had training via the various methods (E-learning versus face-to-face). However, there were no differences amongst any of the outcome statement variables when they were examined by mode of training; When the responses from those who had done IQT Bronze only because it was required training were compared with the responses from the rest of the cohort, significant differences were found in all outcomes. For example, those who had only done IQT because they were told to do it, were significantly less likely to have a positive response to the outcome statements and more likely to have neutral or negative responses; There were also significant differences in the proportion of Bronze learners who reported having put their learning into practice. 32% (68/211) of those who had only completed IQT Bronze because it was required of them reported having put their learning into practice. This compares to 59% (240/409) in the remainder of the cohort; and 133 respondents provided a free-text written comment and the general themes that emerged from these related to the importance of support from managers and other colleagues when undertaking quality improvement initiatives. It was also highlighted that when IQT Bronze is seen as a tick-box exercise it may deter people from implementing the training in practice. IQT SILVER SURVEY FINDINGS 92 people responded to the IQT Silver survey. Key findings from the analysis were: 37% of those who completed the IQT Silver survey were in administrative or clerical roles; Most of the respondents undertook IQT Silver either because they were interested in quality improvement or to further the knowledge they gained from IQT Bronze training; Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 3

Ten people reported that they did not intend to start an IQT Silver project, most of whom gave lack of time as a reason. One individual reported that the project they chose was too large to manage; Respondents were asked to report the extent to which they agreed or disagreed with statements about the assessment for the IQT Silver projects. A majority either strongly agreed or tended to agree that the amount of work required for the assessment was manageable (87%) and that the assessment process was fair (86%). The statement with the highest proportion of respondents who disagreed related to the clarity of the assessment process, with 15% disagreeing that there was clarity about how the assessment would be done; Most of the respondents (n=74, 81%) felt part of their organisation s improvement efforts. Over half (n=48, 53%) had had completed other improvement projects following IQT Silver. 77 (84%) reported having aspirations to build on their improvement knowledge; and Many of the free-text comments provided by Silver learners related to the quality of the trainers for the IQT Silver taught sessions and the need for clarity about the assessment. INTERVIEWS WITH SILVER LEARNERS Key findings from the analysis of the interviews related to the catalysts and barriers to the success of IQT training and implementing quality improvement in practice. The table (below) provides a summary of the commonly reported enablers and, conversely, the common barriers to success. Enablers Effective training Enthusiasm/motivation Support for implementing QI Team-based approach with QI tools and language Belief in the value of IQT Ability to influence others Barriers Ineffective training Lack of support from managers / other staff Attempting a large and often unachievable change Lack of time/resource Lack of clarity about the assessment Resistance to change from other staff DISCUSSION Taking all of the findings together, there are a number of important points in discussion that were raised: There is an important role for IQT in delivering the policy agenda in Wales, and helping and supporting organisations in achieving their strategic and operational goals; Evidence from elsewhere provides interesting insights into what Wales might do next, with the message that time, measurement and cultural change are important dimensions to consider; It is possible and desirable to measure the impact of IQT (where this is possible), although change in complex systems is difficult to discern. The critical mass, catalyst, delivery and sustainability and legacy domains provide a framework within which metrics might be established; Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 4

IQT: THE FOUR DOMAINS OF IMPACT CRITICAL MASS Numbers and % of people who have completed IQT at either Bronze, Silver (or post-silver) levels has risen and continues to grow CATALYST Skills from IQT mean that people have achieved aims demonstrably more quickly, and converted ideas into achieved change at a higher rate than otherwise DELIVERY Post-projects and QI activity, days are saved due to poor and inefficient processes having been addressed and remedied SUSTAINABILITY and LEGACY Champions/ experts on QI with a common language and vocabulary are now dispersed across Wales MOST QUANTIFIABLE/EVIDENTIAL LEAST QUANTIFIABLE/EVIDENTIAL Momentum around the further development of IQT is central, and ways need to be found to do this which may include the development of qualifications for individuals who have already completed IQT Silver Project level; The networks within IQT and elsewhere are crucial to its success, and that clarity of purpose is important for such networks. Additionally there needs to be a review of how internal networks within health organisations read across into national networks; and IQT has the potential to reach parts of the NHS that other programmes struggle to do, and that harnessing that potential is imperative. CONCLUSIONS AND RECOMMENDATIONS Ten recommendations are made, and they are grouped against the four domains of impact for IQT critical mass, catalyst, delivery, and sustainability and legacy. They are written against the context that IQT has done a good job so far, but that there is always room for improvement, as IQT learners, teachers and managers will know better than anyone else. CRITICAL MASS 1 Continue to build from success and branch out into new areas It is important to recognise the contribution that IQT has made, and to celebrate the success of standardised improvement methods across an entire health system. It is also central to acknowledge its further potential to transform the safety and efficiency of services, and that whilst this does require improvement to occur within IQT (like engaging with F2 medics, or tighter definition around the higher levels of QI practice) it does not require wholesale change. 2 Measures need to be identified that will allow for the spread and impact of IQT to be evidence-based There has been a paucity of robust data on IQT. This is understandable at one level at the system level, change in complex environments is difficult to measure, although the impact and outcomes for individual learners is more easily comprehended. This is not easy to do but is something a Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 5

national QI programme must aspire to the four domains of impact as identified in the previous chapter provide a basis for this and should be explored. That said it is unacceptable that a more complete, robust and reliable register of those that have completed IQT Bronze and IQT Silver at the national level (and sometimes within health organisations) does not exist, and this should be addressed immediately. 3 The national focus on the numbers of IQT Bronze learners should be maintained, and the momentum to train as many people at IQT Bronze level should be increased The national target for 25% of the NHS Wales workforce to be trained was helpful in raising the profile of IQT at the outset. People sat up and took notice, but this has now drifted, and there are no real consequences or sanctions for organisations who are below target. Accordingly, leaders within IQT should give due consideration to re-shaping the national target to make it a more meaningful measure. At the same time the momentum around increasing the numbers of IQT Bronze learners must be maintained, and greater effort needs to be placed on ensuring people are properly motivated to complete Bronze, without resorting to making this appear as though it is only something they have to do because it is required of them. CATALYST 4 Prior learning needs to be addressed more effectively A consistent criticism of IQT is that is has not done an effective job of recognising prior learning and the QI CVs of learners. Whilst there has been a mechanism for doing this in place, it has not been well understood or widely used. There is now an opportunity to further instil momentum around IQT by developing a set of criteria through which prior learning could effectively be recognised and reflected. This is especially important in motivating those who were part of the successful Transforming Care programme to migrate into IQT. 5 Improve the mechanisms through which people might be brought into IQT More could be done to allow learners access to important information about IQT whether at the national level or within health organisations. It is very difficult to navigate from the 1,000 Lives Plus webpage to the IQT page, and there is scant information about IQT learners, what they have done, and, for example, whether they would be prepared to act as informal coaches and mentors to new learners on the web pages of health organisations. This should be rectified. 6 Issues around the capacity of qualified trainers and inconsistency around assessment needs to be addressed There are not enough people training IQT in Wales, and the quality of the training is variable. Similarly assessment processes are inconsistent, and need to be brought more closely into line to ensure that qualifications mean the same thing in different parts of Wales. These processes need to be tied into the quality assurance functions mentioned under Recommendation 9. DELIVERY 7 The priorities of health organisations need to be more formally aligned and connected with those of IQT and quality improvement more generally In order for the activities of those undertaking QI activities in Wales (whether as part of IQT or not) to be fully optimised and sustained, these need to be formally linked to the top-level priorities and core business of health organisations. This would ensure that innovative improvement practice was mainstreamed but that also health organisations would be making the most of the skills of its staff. Health organisations are starting to harness generic IQT capacity in the organization by selecting IQT projects which specifically address corporate priorities. More could often be done to support Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 6

this by maintaining an accurate register of those staff with IQT (and equivalent) skills and experience, their particular strengths and previous projects. 8 An advanced programme for IQT should be available One of the means by which IQT could be re-energised and the momentum maintained, is to formalise a post-iqt Silver level in Wales. Many learners feel frustrated that as individuals there is nothing after their Silver level which is clearly specified, and an advanced practitioner qualification would address this, whether developed in Wales on a bespoke basis or commissioned from elsewhere. This could complement the development of the internal Faculties for quality improvement, but be focused at the level of individual learners. SUSTAINABILITY AND LEGACY 9 Reconfigure the relationships, responsibilities and networks within and without IQT Greater clarity needs to be brought to the relationships between the Welsh Government, 1,000 Lives Plus, the National IQT Team and the 10 health organisations in Wales. The current way in which IQT is being implemented and delivered has moved significantly from the way it was originally intended, and the relationships and responsibilities need to be reconfigured accordingly. There are a number of networks that were developed as IQT evolved. Most important here is the role of the Gold Network which has done an excellent job in establishing IQT across Wales. However its role and purpose now needs to be reviewed and clarified, especially in relation to a series of other networks (both inter- and intra-health organisations, and potentially around training, assessment, mentoring/coaching) that could be established. This might mean re-shaping the current Gold Network so that it becomes a Quality Improvement Community of Practice for Wales, which would incorporate IQT, but other forms of quality improvement, both within and without the health sector. This Community of Practice could be supported by regional Faculties for quality improvement on the footprints of the forthcoming public service boards, but primarily located within the health boards and organisations. A possible (but not specified) outline of how IQT might be configured in the light of the discussion of the relationships, responsibilities and networks above is provided in Figure 1 (below). Certain functions are suggested, the most important of which is for IQT to develop a systematic way to peer review IQT curricula to ensure that they are standardised and quality assured, for example on the different forms of IQT Silver that have been developed. 10 Forge more effective links with those outside health across public services, especially in social care There is a well-established community of quality improvers across the public sector in Wales. Whilst there has been some read-across between IQT and these networks, this has been somewhat piecemeal. This should be addressed and a formal mechanism developed for an effective relationship between IQT and equivalents outside health, especially in social care, to be established. This should help to remove one of the impediments for more effective integration of health and social care not having a common language around quality improvement. Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 7

Figure 1 Possible structure for quality improvements networks NHS WALES WELSH GOVERNMENT SOCIAL CARE AND OTHER PUBLIC SERVICES Public Health Wales (incl. 1,000 Lives) Public bodies Health Boards and Trusts 1,000 Lives Improvement Service Steering Group IQT National Programme Manager Academi Wales Internal Faculties of Quality Improvement Activities could include: Record keeping Developing capacity of trainers Coaching/mentoring Connections to local priorities as determined by the IMTP and other key strategic plans and drivers Quality Improvement Community of Practice Functions could include: Purpose = Share, learn and connect between health organisations Membership: Open to all those at Silver Project level on the proviso that people are active members who give and get Associate and Full members? Chaired by IQT National Programme Manager Activities could include: Keeping lists of coaches and mentors across Wales Developing train the trainer capacity Co-producing events Peer review /QA of IQT curricula - Silver and above Links to industry and commerce All-Wales Continuous Improvement Community DELIVERY OF IQT Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 8

THREE KEY MESSAGES IQT is going well. The tasks now are to finish the roll out, and ensure sustainability; start to realise the full benefit of IQT by linking it more effectively to the core business of NHS bodies; and start to expand and refine the core approach to meet other aspects of the external environment. In closing, the following three key messages bring together all of the evidence that has been gathered, and provides a succinct summary of the conclusions from the research. 1. IQT is well on the way to achieving its key objectives, and progress now needs to be maintained and consolidated. Many parts of the NHS are now approaching a critical mass of staff who are attuned to the concepts and possibility of service improvement, with a significant cohort of trained practitioners who can lead local projects. This is built on many previous drives to develop quality improvement, a shared vocabulary and grammar of improvement, reflected glory from national and international endorsement, and an early set of tangible achievements. The numerical targets may not have been achieved on time, but their realisation is now within grasp. The task now is to complete this roll out, consolidate and sustain the expertise in place, and address the other issues set out below. 2. The infrastructure and people in IQT can contribute more to the NHS agenda. Health organisations are now starting to develop different ways of harnessing the expertise developed by IQT. This requires an awareness of who has been trained and their respective skills, and a capacity to free up the appropriate resource to meet development priorities, wherever they occur in the organisation. Different health organisations have different approaches to this - all need further development. 3. IQT needs to embrace the integration and co-production agendas. IQT is designed for NHS Wales and for clinicians and managers; its impact will be enhanced if it embraces the broader quality improvement movement across Wales' public services. In addition, there is growing recognition - through Prudent healthcare and other initiatives - of the power of designing and delivering services (and therefore quality improvement) in partnership with patients and the public. This may require some adaptation to IQT processes, and an appropriate way of equipping and supporting lay people in their contribution. Evaluation of IQT for 1000 Lives Improvement Service July 2015 Page 9

Welsh Institute for Health and Social Care University of South Wales, Glyntaf Campus, Pontypridd, CF37 1DL wihsc.southwales.ac.uk wihsc@southwales.ac.uk 01443 483070