Deliberative methods in policy research Lessons from the regulated industries K A ELA SCOT T - H EAD OF DEMOCRAT IC I N N OVAT ION, I N VOLVE SARA DAVIDSON - DELIBERAT IVE R ESEA RCH LEAD, I PSOS M ORI
About the project Commissioned by the Consumer Futures Unit of Citizens Advice Scotland MORI In January 2017 Three discrete strands of work: Energy, Water and Post Dual focus: sector-specific policy insights AND comparative analysis of different deliberative methods
Policy questions ENERGY - What elements of incentives and new regulation would be most likely to encourage homeowners in Scotland to invest in improving the energy efficiency of their homes? WATER - How engaged are consumers in matters related to water and the environment? and Do consumers think there should there be similar service standards across urban and rural areas and if so, to what degree? POST - What aspects of the Post Office Ltd Outreach network are most important to consumers in remote and rural areas, and why?
Methodological questions Which deliberative methods are most effective at identifying and understanding consumer preferences, motivations and priorities within the regulated industries? Where can deliberative methods be best applied for consumer engagement within the regulated industries? What additional benefits/added value, if any, does deliberative engagement bring over other qualitative methods e.g. focus groups?
What is Deliberative Engagement? Qualitative research methods in which participants are supported to develop informed opinions about a topic through a process of learning, dialogue and public reasoning (i.e. deliberation).
Choice of deliberative methods Grounded in preliminary work for CFU reviewing HOW deliberative engagement methods had been used in the regulated industries internationally, including: how effectively the methods has answered research questions if they added new insight or knowledge regarding consumer preferences; whether they provided outputs that were useful to policy and decision-makers; how they achieved impacts for consumers. Consumer Reference Groups / Customer Forums Citizens Advisory Forums Deliberative Focus Groups General Deliberative Workshops Repeated Structured Dialogues Distributed Dialogues Citizens Juries Citizens Assembly Deliberative Mapping Participatory Strategic Planning Online Deliberations.
Choice of deliberative methods 16 different consumer engagement processes in February/March 2017 Focus Group (2hrs) Motherwell Focus Group (2hrs) Perth Energy Water Post Structured Dialogue (6.5hrs) Motherwell Citizens Jury (9hrs over 2 days) Perth Focus Group (2hrs) Inverness Focus Group (2hrs) Glasgow Structured Dialogue (6hrs) Inverness Structured Dialogue (6hrs) Glasgow Citizens Assembly (6hrs) Edinburgh Focus Group (2hrs) Melrose Focus Group (2hrs) Peterhead Structured Dialogue (6hrs) Melrose Structured Dialogue (6hrs) Peterhead 3 on-line Sounding Boards (2hrs each, split over 2 days)
Recruitment of participants Participants generally recruited using a using a face-to-face (door-to-door and in-street) free-find approach by Ipsos MORI s in-house team Quotas to ensure representative pool of consumers in terms of sex, age, working status and social grade Additional sector-specific criteria: Energy: home ownership, dwelling type, attitudes to energy efficiency. Water: general interest, and level of engagement with environmental matters. Post: mix of users and non-users of Post Office Outreach All participants received a monetary thank-you for taking part in the workshops ranging from 30 to 150.
Participants In total 257 people from across Scotland participated in the engagement events. Focus Groups 17 Energy Water Post Structured Dialogue 24 Citizens Jury 18 Focus Groups 19 Structured Dialogues 36 Citizens Assembly 77 Focus Groups 15 Structured Dialogues 36 On-line Sounding Boards 20 Total Number 59 Total Number 132 Total Number 66
Comparative methodology findings
Making things difficult for ourselves.. Methodological purity Stuck to the key defining features of each method While each event was designed to cover the same content each process was designed to ensure it didn t just repeat the same exercises /activity (but give more time therefore ensuring more detailed results) Repeated processes When the same process was repeated (as far as possible) they were run concurrently with different facilitation teams, to reduce the likelihood of assumptions or previous results (or learning from experience) influencing a process with a different group Evaluation The responsibility for content analysis and methods analysis was (largely) split between the 2 organisations Participant evaluations were extensive asking for comments on the process as well as the experience of taking part Assessments were also sought from observers and the facilitators about the effectiveness of the process
Consistency of findings. Despite the different methodological approaches the topic findings were largely consistent within each of the 3 industry strands: Veracity of the deliberative process Confidence to extrapolate from deliberative outputs
Variations in findings What varied was: the depth of understanding of consumer preferences, motivations and priorities the type of outputs produced e.g. collective recommendations, majority preferences or trends revealed through analysis
Learning: dealing with complexity Providing information to participants about the topics themselves was a key feature in all of the workshops. A range of different presentation techniques were used including neutral presentations of the issues, case studies, short scripted inputs from the facilitators (at tables or in plenary) and written handouts. Citizens Jury however was different: Expert led information Presenting arguments Opportunity to discuss and question
Learning: the role of information In the Structured Dialogues by contrast considerable time was given over to participants to find their own way into the topics Group exercises Story cards Brainstorming Working together to explore the complexities of the subject, the implications for themselves and others and co-create an understanding of the issues i.e. a process of building DIALOGUE
Deliberation & forming conclusions In the Focus Groups and the Structured Dialogues participants, largely, took part as individuals. Focus Groups role: Responding to stimulus Deliberative component added (in most cases ranking exercises)
Deliberation & forming conclusions Structured Dialogues used a wide range of methods to build up the participant s level of involvement in the discussion and gradually increase the deliberative demands being placed on participants. discussing, in small and large groups written responses post-it note brainstorming surveys voting negotiated ranking developing recommendations The response time if your water is cut off unexpectedly The response time to sewerage flooding on the street % saying variation might be acceptable 59% 57% The response time to sewerage flooding in your home or on your property 51% The amount of notification given for planned water cuts 44% The promises made about customer service 28% None of the above 23% Base: All citizens assembly participants who gave an answer (75)
Role participants were asked to play. Sounding Boards Told that they were selected precisely because of their geographical location in communities where specific services were operated Encouraged, between meetings, to speak to others in their community and bring their thoughts into the room
Role participants were asked to play. Citizens Assembly 77 participants made aware that they were there as a representative sample Explained that they were there to make recommendations on behalf of the wider community
Role participants were asked to play. Citizens Jury Also informed that they were a representative sample Only group that were explicitly given a task / problem to solve
What did the research show? 1. That ordinary members of the public can engage in informed deliberation on complex subjects 2. That deliberative engagement methods can produce information that is relevant and useful to policy makers 3. That there is a veracity to deliberative engagement processes undertaken with a mini-public that means the results can be extrapolated 4. That deliberative engagement produces different types of information than polling or focus groups (and at times different results)
Impact of learning and engaging with the opinions of others 40% Jurors attitudes towards regulation 30% 20% 10% 0% 19% 19% 38% 19% 6% Strongly support Support Neither support or not Don t support Strongly opposed Day 1 - Pre-deliberation
Impact of learning and engaging with the opinions of others 40% Jurors attitudes towards regulation 30% 20% 10% 0% 19% 6% 19% 22% 38% 11% 19% 17% 6% 28% Strongly support Support Neither support or not Don t support Strongly opposed Day 1 - Pre-deliberation Day 2 - Post-deliberation
When to use deliberative methods Deliberative methods are best used when a policy question: has multiple possible answers; involves complex issues; where there are potentially conflicting beliefs/ values; or when decisions will require making trade-offs between differing options.
Value added Where traditional public engagement tools, such as opinion polls or customer surveys tend to measure top of the head public views, deliberative public engagement provides policy and decision-makers with a different sort of information: Informed and considered opinion from engaging with factual information and the views/experiences of others Deeper understanding of public priorities - and the values and reasons behind them Understanding of where consensus is and is not possible
Considerations on choosing a method. Determining the best method relies on: the type of question being asked the type(s) of outputs that are needed who needs to be involved the role participants are being asked to play how the results are going to be used
Thank you Kaela Scott- Head of Democratic Innovation, Involve Kaela@Involve.org.uk Sara Davidson - Deliberative Research Lead, Ipsos MORI Sara.Davidson@ipsos.com