Downloaded from UvA-DARE, the institutional repository of the University of Amsterdam (UvA)

Transcription

1 Downloaded from UvA-DARE, the institutional repository of the University of Amsterdam (UvA) File ID Filename Version uvapub: Thesis final SOURCE (OR PART OF THE FOLLOWING SOURCE): Type PhD thesis Title Shaking up the Cost Benefit Analysis process: Issues and directions for improvement when assessing integrated spatial transport plans through a cost benefit analysis Author(s) E. Beukers Faculty FMG: Amsterdam Institute for Social Science Research (AISSR) Year 2015 FULL BIBLIOGRAPHIC DETAILS: Copyright It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content licence (like Creative Commons). UvA-DARE is a service provided by the library of the University of Amsterdam ( (pagedate: )

2 Els Beukers Shaking up the Cost Benefit Analysis process This dissertation focusses on the Cost Benefit Analysis (CBA) processes when assessing integrated spatial transport plans, using the Netherlands as a case in point. It answers the following research question: What process issues occur when assessing integrated spatial transport plans through a CBA in the Netherlands, and how can the CBA process be improved? Providing insight into CBA process issues and identifying ways to improve on them, increases the relevance of this research for practice. One such improvement was found in using the CBA for learning purposes, instead of solely as a final assessment. Furthermore, five clear interventions were articulated, providing a toolkit with which to build communication and trust between plan owners and evaluators. Shaking up the Cost Benefit Analysis process Els Beukers

3 1 Shaking up the Cost Benefit Analysis process

4 Shaking up the Cost Benefit Analysis process Issues and directions for improvement when assessing integrated spatial transport plans through a cost benefit analysis Els Beukers 2

5 Shaking up the Cost Benefit Analysis process Issues and directions for improvement when assessing integrated spatial transport plans through a cost benefit analysis ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit van Amsterdam op gezag van de Rector Magnificus prof. dr. D.C. van den Boom ten overstaan van een door het college voor promoties ingestelde commissie, in het openbaar te verdedigen in de Agnietenkapel op donderdag 29 januari 2015, te 10:00 uur door Elisabeth Beukers geboren te Lelystad 4 5

6 Promotiecommissie Promotor: Prof. dr. ir. L. Bertolini Copromotor: Dr. M.C.G. te Brömmelstroet Overige leden: Prof. dr. A. Khakee Prof. dr. G.P. van Wee Prof. dr. J. Woltjer Prof. dr. W.G.M. Salet Prof. dr. J. Grin Dr. J. de Vries Faculteit der Maatschappij en Gedragswetenschappen Table of content Acknowledgements Summary...15 Samenvatting...23 Prologue...29 CHAPTER 1: Introduction A problematic use of CBAs Transport appraisal: The changing context Overall research design Research question 1: Finding CBA process issues Research question 2: Theoretical improvement process issues Research question 3: Testing the theoretical improvements Triangulation Outline of this dissertation...53 CHAPTER 2: CBA process issues Abstract Introduction Research methodology Results of the focus group sessions Results of the open interviews Interpreting the research findings Closing the inductive research cycle: Planning versus economics Conclusion: mistrust and defective communication Graphic design: Niessen & De Vries This research has benefited from the financial support provided by Platform 31 (Nicis). CHAPTER 3: Theoretical improvements of CBA process issues...85 Abstract Introduction: Unsuitable assessment Communication, trust, and the use of CBA as a learning process Improving communication and building trust between experts An applicable approach for CBA processes Conclusion

7 CHAPTER 4: Testing in an experiential case study Abstract Introduction Research methodology Experiential Case 1 A CBA of station area development in Amsterdam Experiential Case 2 A CBA of tramway routes in the Utrecht region Conclusion and reflection CHAPTER 5: Testing in an ex-post case study Abstract Introduction Theoretical framework: Communication, trust, and learning Research design and methodology Case 1: CBA process of Schiphol Airport development Case 2: CBA process of the IJmuiden sea lock Conclusion and reflection CHAPTER 6: Overall conclusions and reflections Overall conclusions Methodological reflection Recommendations: From theory to practice 193 APPENDIX I APPENDIX II REFERENCES

8 Acknowledgements There is an African saying that says you need a village to raise a child. The same could be said for PhD candidates: you need an academic village to do PhD research. I was lucky to find a wonderful academic village at the University of Amsterdam, with people who share a passion for doing academic research. I had a great time with my roommates (Wendy Tan, Guowen Dai, Federico Savini, Andrew Switser, Koen Raats, De Jung, Paul Chorus, Rick Vermeulen, Bas Hissink-Muller) in the several buildings where we were located, and the other members of the mobility group PUMA (Ren Thomas, Roel ter Brugge, Thomas Straatemeier, Lucas Harms, Antonio Ferrerra, Julio Soria Lara). You formed the great setting for sharing preliminary findings and ideas, and gave constructive feedback that provided the energy boosts to go on. Furthermore, I had the pleasure of having very warm and inspiring colleagues at the planning group (Willem Salet, Leonie Jansen-Janssen, Nick Smit, Sebastian Demski, Caroline Uitenbroek, Stan Majoor, Jochem de Vries, Mendel Giezen, Anita Blessing, Willem Boterman, Wouter van Gent, Johan Gomes), and the faculty of Human Geography, International Development Studies, and Planning (Joos Drooglever Fortuin, Barbara Lawa, Guida Morales, Marianne van Heelsbergen, Marian Hasan, Puikang Chan). You were always there for all kinds of academic and non-academic questions, or just to chat. I also found my academic village outside of the University of Amsterdam through the rich conferences of AESOP, at the International Evaluation and Planning group, at the University of Groningen, and at Aalborg University in Denmark; Niels Heeres, Marije Hamersma, Laksmi Darmoyono, Patrick Driscoll, Morten Skou Nicolaisen, thanks a lot for the cooperation. However, I was most lucky with my PhD supervisors, Luca Bertolini and Marco te Brömmelstroet, who have guided me through the difficult process of writing a PhD by being both critical and supportive. In this guidance, you formed a great duo: Marco, you helped me to focus on methodological matters, and Luca, you prevented me from losing track of the overall research setup or forgetting to communicate to the outside world of practice. Furthermore, although writing a PhD is a solitary job, you were always involved and gave me the feeling that we were in this together. Besides our intense meetings at the university, we have had several Skype meetings to continue our conversations 11

9 across the globe. I cherished your flexibility and the faith you had in me. Also, the cooperation with the researchers of the Technical University of Delft (Bert van Wee, Jan Anne Annema, and Niek Mouter) for organizing the OBRRI meetings and the final CBA seminar resulted in many valuable memories of vibrant discussions that proved that planners meeting economics can be much fun. This concerns also the wonderful cooperation with Nick Smit, who managed the research project practically, and helped to juice things up when necessary. I was very happy to hear that you are going to do a PhD research, yourself, and are willing to fill in the role of paranimf. I also got a lot of support outside of academia. For example, from the many devoted CBA and planning practitioners who allowed me to interview them, or participated in the OBRRI meetings and the conducted workshops. For this, I am very thankful. Special thanks goes as well to Coen Mekers, Fokko Kuik, Bart van der Heijden, Bertus Postma, Will Klercx, Cees-Jan Pen, Johan Visser, Niek van de Heiden, Eric Verroen, and Zenzi Pluut for sharing your interesting experiences, personal stories, cooperation, and everlasting interest in this research. I would also like to thank my former colleagues at Decisio, Niels Hoefsloot and Kees van Ommeren, who taught me about the CBA and then remained involved long after. Furthermore, this thesis would not have been possible without the great help of Huan Hsu. My dear friend, editor, neighbor, and paranimf who got more and more interested in my research after every paper he has edited. Additionally, I want to thank my (new) colleagues at Balance, and especially Jasper Schaap, with whom I cooperate to complete the learning cycle of Kolb and Fry (1975) by applying insights from this PhD research into planning practice (leading to new concrete experiences, et cetera). Finally, when I started this PhD research on CBA process issues, I received much skepticism from people who wondered if doing a PhD would be boring, working on the same project for at least four years, and if a PhD about the CBA would be very boring. In my opinion, though, the opposite is true: writing this dissertation was the most exciting and challenging experience, and the theme of the problematic use of the CBA for assessing integrated spatial transport plans remains fascinating to me. Furthermore, although the research indeed lasted for many years five in total each phase was characterized by different demands, skills, and interactions. These five years were therefore very intense, with many difficult and glorious moments, and I am grateful for all of it. Els Beukers Amsterdam, December 14 th 2014 I also owe a lot to my friends and family for their support and understanding of the demanding job of writing a PhD, and for providing the necessary distraction. I especially want to thank my parents, parents-in-law, Irene, Gea and Fredu for being there and taking care of Louise, although I have the strong impression that this pleasure was all on their side. My greatest thanks, though, goes to my partner David Jablonowski. You supported me to start this PhD research in the first place, helped me through the moments of losing faith, and took care of Louise whenever you could. But mostly, I thank you for bringing joy to my life every day. It is a pleasure to spend it with you

10 Summary Shaking up the Cost Benefit Analysis process Issues and directions for improvement when assessing integrated spatial transport plans through a cost-benefit analysis Introduction The cost benefit analysis (CBA) would seem to be an attractive assessment tool for planning and evaluating transport (or other kinds of) plans. It promises to structurally expose all expected effects on society, translate these effects in monetary terms, and assess whether an investment might be worthwhile by giving a benefit-cost ratio. Based on these attributes, the CBA has become a widely used instrument for the appraisal and evaluation of large transport projects in most OECD countries. In the Netherlands the role of the CBA in decision-making has become increasingly important: since 2007, a CBA has been obligatory for local and regional spatial transport plans that require national funding. The CBA, though, is also a controversial tool. Especially when assessing complex spatial transport plans that aim to influence fundamental but hard to grasp aspects of societal challenges like sustainability, or quality of life, that resist easy translation into monetizable pieces. Although many of these controversies arise from process related matters, academic discussions tend to focus on content aspects solely. This knowledge gap is problematic, because it remains to be seen whether solving the technical, content-related issues alone will be enough to reverse the antagonistic attitude that many planning actors have towards the use of the instrument itself. Furthermore, considering the necessary development of integrated transport plans concerning multiple goals and involving a variety of actors, process issues are becoming even more relevant. This dissertation therefore aims to close this knowledge gap by focusing on CBA processes in assessing integrated spatial transport plans in the Netherlands. Moreover, through realistic 15

11 evaluation, this dissertation aims to identify ways to improve the process issues at play. Research design The following central research questions are answered: What process issues occur when assessing integrated spatial transport plans through a CBA in the Netherlands, and how can the CBA process be improved? These questions are answered through three sub-questions: (1) What process issues do CBA participants perceive of CBAs of integrated land-use transport plans in the Netherlands? (2) What are promising mechanisms and interventions for improving the perceived CBA process issues? (3) Which interventions improve the perceived process issues, and through which mechanisms? The adopted research design is prescription-driven rather than description-driven, and follows the experiential learning cycle of Kolb and Fry (1975): observing and reflecting on concrete experiences, developing abstract concepts to make sense of them, and testing the gained insights in new situations, in order to induce change in concrete experiences. The research first investigated perceptions of CBA process issues through in-depth interviews and focus groups sessions. Secondly, promising improvements for the perceived CBA process issues were found through a literature review of deliberative planning and organizational learning studies, resulting in an overview of interventions and related mechanisms. In a third and final step, the working of these interventions and mechanisms were tested by combining the strengths of a control rich-context poor experiential case study, and a context rich - control poor ex-post case study. CBA process issues Focus group sessions and open interviews were used to shed light on issues in the CBA process as perceived by several types of CBA participants, see table I. Many of these perceptions indicated large differences between the worlds of plan ownersand-planning and evaluators-and-economics. This dichotomy helps to explain and understand the three problems connecting most of the issues that were expressed: the distrust between plan owners and evaluators towards each other and the plan or instrument which they represent, how this leads to a communication deficit, and a perceived use of the CBA as a final assessment whereas a use for learning is desired. Table I: Perceived issues in the CBA process 1. Deficient communication among participants: experienced as a conflict between plan owners versus evaluators 2. Fear that hard effects dominate over soft effects in decision making. 3. CBA as a black box: without explanation, the tool is perceived as not understandable. 4. Parties act strategically towards CBA input (composition of the plan) and output to make the plan score according to their desires. 5. The criteria of testing CBA reports are unclear and the testing process is unstructured. 6. Even though CBA may not be suited for each assessment question, CBA is being used because it is compulsory in the planning procedure. 7. There is too little attention for process matters: time pressure; underestimation of need for discussions on assumptions and starting points; CBA too late in planning process. 8. CBA is perceived being used as a final judgment, which is indicated as wrong, because the CBA will never give the whole picture, and it leaves out the possibility for learning. 9. There is too little room for uncertainties and nuance in the decision making process. Theoretical improvement of CBA process issues Literature on evaluation studies emphasize that assessments of complex plans should facilitate a learning process in which planning ideas, means, and ends are progressively constructed by participants. Such an approach entails active and engaged participants who communicate with and trust each other, which are intertwined concepts. These insights are contrasting to the found CBA process issues. It is therefore stated that, to improve the process issues, the levels of interpersonal communication and trust need to be increased in order to achieve the participants desired outcome of using CBA as a tool to support learning processes. This formed the basis for the conceptual framework, as illustrated in figure I. Furthermore, a literature review of the research fields of deliberative planning and organizational learning was conducted to identify the most promising interventions and mechanisms for improving communication and trust between plan owners and evaluators. Five interventions were identified, proposing to: (1) create a dialogue space where plan owners and evaluators meet faceto-face; (2) share and discuss the plan and CBA together; (3) be prepared to share and discuss knowledge together; (4) have the interaction guided by a moderator; (5) use a dialogue mode (sketch, simulation, storyboard, role play) presented as work in progress

12 Figure I. Conceptual framework of interventions for improving communication and trust between plan owners and evaluators as conditions for using the CBA for learning The actual effectiveness, though, of these interventions in practice has not been tested, let alone in the context of CBAs of integrated transport plans. To improve the academic rigor and practical relevance of such theory-derived interventions and related mechanisms, testing them in the context of their intended use is fundamental. Furthermore, by focusing on both the if (does it work?) as well as the how (how does it work, or not work?), testing also allows for improvements to be made in the fit of the interventions to the specific characteristics of CBA processes. The interventions were therefore tested in a control-rich, context-low experiential case study, and a control-low, context-rich ex-post case study. The experiential case study Two sequential experiential cases were set up to test the working of the communication and trust building interventions in a situation close to practice. This was found in two early staged CBAs of integrated transport plans: Case 1 in Amsterdam (the capital and largest city of the Netherlands), and Case 2 in Utrecht (the fourth largest city of the Netherlands). The findings from Case 1 showed diverse results in whether the five interventions helped to improve communication and trust between plan owners and evaluators. Four possible reasons were identified for the limited effectiveness of the interventions in Case 1, which were used to refine the application of the interventions in Case 2. These refinements were to: devote more time to the discussion; also prepare the participants for the content of the discussion in advance; ask the moderator to encourage participants to be explicit and avoid jargon; use a dialogue mode that is related to planning and CBA rationale. Compared to Case 1, Case 2 indeed resulted in higher measured communication and trust effects of the interventions for both plan owners and evaluators. The ex-post case study A second test of the interventions took place through a contextrich ex-post case study that investigated in finalized CBA processes how the interventions influenced communication and trust, and a use of the assessment for learning. For this, two contrasting cases were compared: a case in which the five interventions were likely present (the CBA process of Schiphol airport), and a case in which they probably were not (the CBA process of the IJmuiden sea lock). The analysis of these cases showed contrasting findings for their communication and trust conditions, that were largely corresponding with the expected influence of the five interventions. The analysis of the Schiphol case signalized perceptions of a large variety of mostly positive communication and trust mechanisms, and outcomes of communication and trust. The IJmuiden sea lock case showed more negative communication and trust mechanisms, and a lack of communication and trust in the early phases. Nevertheless, participants in the sea lock case appeared better able to communicate with and trust each other as the plan and assessment matured, and by reaching a shared sense of urgency. Furthermore, both cases showed several moments of learning large and small about the plan and the CBAs, through which the alternatives and assessments evolved during the process. Whereas the learning moments in the sea lock case seemed influenced by the succession of different CBAs, the participants in the Schiphol case seemed able to learn within the dialogue process of the same CBA. However, for the moments of deeper learning in both cases, communication and trust seemed important conditions, notwithstanding how this communication and trust was achieved. The cases also showed that this learning was strongly influenced by the wider planning and decision making processes. In the Schiphol case, the strong and politically supported position of the residents influenced the establishment of communication, trust, and learning. In the sea lock case, the shared sense of urgency supported the establishment of communication and trust, and learning. The assessment, planning and decision making processes thus seem strongly interwoven, especially when large changes in the alternatives and assessments were discussed. Compared with the experiential case study, the ex-post case study confirmed that a communication and trust building strategy supported the ability to share meaning (that is to communicate with and trust each other), and that lacking such a strategy could hamper this. However, it further illustrated that this could also be established without a deliberative communi

13 cation and trust strategy, for example through reaching a shared sense of urgency and political pressure. The ex-post case study also underlined the influence of the wider planning and decision-making context, on communication and trust, and the learning use of the CBA. Nevertheless, the ex-post case study showed that independently from how communication and trust was gained, establishing these seemed forming an important condition for deeper learning to occur. Overall conclusions and recommendations This dissertation gave insight into the problematic interaction between plan owners and evaluators in CBA processes when assessing integrated spatial transport plans in the Netherlands. Furthermore, it emphasized the importance of communication and trust as mechanisms for using the CBA for learning. A communication and trust-building strategy was formulated from relevant academic literature. This strategy was critically assessed through experiential and ex-post case studies. The findings of these studies indicated that the strategy indeed seems to improve communication and trust. However, further refinements to the strategy are necessary to make it sensitive to the assessment context. Also, its effectiveness is influenced by wider planning and decision-making processes. This dissertation concludes with five recommendations: 1. Pay attention to establishing communication and trust between plan owners and evaluators in CBA processes; 2. A communication and trust-building strategy in assessment practices based on the five interventions can help; 3. However, such a strategy it should be applied cautiously, like through pilots; 4. Furthermore, when using such a strategy in assessment practices, it should be made context-sensitive; 5. Be aware of the learning potential of the CBA, especially when assessing complex plans

14 Samenvatting Het maatschappelijke kosten- batenanalyse proces opgeschud Knelpunten en mogelijke verbeteringen van het analyseproces van integrale ruimtelijk-infrastructurele plannen in een maatschappelijke kosten- batenanalyse Inleiding De Maatschappelijke Kosten- Batenanalyse (MKBA) is een veel gebruikt instrument voor het afwegen van grote infrastructurele plannen in, bijvoorbeeld, de meeste OECD landen. Sinds 2007 is in Nederland een MKBA zelfs verplicht voor infrastructurele en ruimtelijke plannen waar (co-) financiering van de Rijksoverheid nodig is. Daarmee is positie van de MKBA in de Nederlandse ruimtelijke ordening verder versterkt om de ontwikkeling van en besluitvorming over ruimtelijk-infrastructurele (en andere) plannen te ondersteunen. De insteek van de MKBA is om een gestructureerd overzicht te geven van alle verwachte effecten voor de maatschappij en deze effecten te vertalen in monetaire waarden; kosten en baten. Het kosten-batenratio geeft vervolgens aan of de beoogde investering de moeite waard kan zijn. De MKBA is echter ook een zeer controversieel instrument. Vooral wanneer complexe, ruimtelijk-infrastructurele plannen worden geanalyseerd met aspecten die nauwelijks in monetaire termen te vangen zijn, zoals een verbetering van de leefbaarheid of het versterken van de internationale concurrentiepositie. De controverse rond MKBAs hangt vaak samen met procesknelpunten. Toch richt de academische discussie zich veelal op de inhoudelijke problemen. Deze leemte is problematisch, omdat het onwaarschijnlijk is dat de controverse rond de MKBA minder wordt wanneer alleen de technisch-inhoudelijke problemen worden opgelost. Ook zal het belang van procesaspecten verder toenemen bij analyse van integrale plannen waarbij meerdere doelen worden nagestreefd en verschillende actoren betrokken zijn. Dit proefschrift richt zich op deze kennisleemte en stelt analyseprocessen van integrale ruimtelijk-infrastructurele plannen in MKBAs in Nederland centraal. Doelstellingen van dit 22 23

15 proefschrift zijn om procesknelpunten te identificeren, mogelijk verbeterrichtingen te verkennen en te testen. Onderzoeksopzet De centrale onderzoeksvraag is als volgt: Welke procesmatige knelpunten vinden plaats wanneer integrale ruimtelijk-infrastructurele plannen worden geanalyseerd in een MKBA, en hoe kunnen deze knelpunten worden verbeterd? Deze vraag is beantwoord door drie sub-vragen: (1) Welke procesmatige knelpunten ervaren betrokkenen wanneer integrale ruimtelijk-infrastructurele plannen worden geanalyseerd in een MKBA? (2) Wat zijn veelbelovende mechanismen en interventies om deze knelpunten te verbeteren? (3) Welke interventies verbeteren de ervaren knelpunten en door welke mechanismen? De onderzoeksopzet heeft een voorschrijvend in plaats van alleen een beschrijvend karakter en volgt de ervaringsgerichte leercyclus van Kolb en Fry (1975). In deze leercyclus vindt leren plaats door: observatie van en reflectie op concrete ervaringen; ontwikkelen van abstracte concepten om de observaties en reflecties te begrijpen; en het testen van de abstracte concepten in nieuwe situaties waardoor nieuwe concrete ervaringen ontstaat, die vervolgens weer geobserveerd kunnen worden. In dit proefschrift zijn de percepties van MKBA-procesknelpunten onderzocht door diepte interviews en focusgroepsessies. Vervolgens heeft er een inventarisatie plaats gevonden in planningen organisatieliteratuur van kansrijke mechanismen en interventies om de gesignaleerde knelpunten te verbeteren. De werking en effectiviteit van deze mechanismen en interventies zijn in de derde onderzoeksfase getest op twee aanvullende manieren: in een ervaringsgerichte case studie (controle rijk en context arm) en een ex-post case studie (controle arm en context rijk). MKBA-procesknelpunten De ervaren procesknelpunten zijn samengevat in tabel I. Deze percepties wijzen erop dat het proces wordt gekenmerkt door twee tegenstrijdige werelden: die van de planeigenaren en van de MKBA-analisten. Deze dichotomie verklaart drie probleemvelden waaraan de meeste ervaren knelpunten gerelateerd zijn: Wantrouwen tussen planeigenaren en MKBA-analisten naar elkaar en naar de MKBA dan wel het plan (1); Gebrekkige onderlinge communicatie: te laat, te weinig en te weinig gestructureerd (2); Het ervaren gebruik van de MKBA als oordelend instrument, terwijl een lerend gebruik gewenst is (3). Tabel I: Ervaren knelpunten in het MKBA-proces 1. Gebrekkige communicatie tussen planeigenaren en analisten. 2. Angst dat harde (te monetariseren) effecten domineren over zachte (niet goed te monetariseren) effecten in de besluitvorming. 3. De MKBA als black box : zonder nadere toelichting wordt de MKBA als onbegrijpelijke ervaren. 4. Strategisch gedrag om het plan te laten scoren zoals politiek wenselijk: zowel met betrekking tot de MKBA-input (kenmerken van het plan) als de MKBA-output. 5. De criteria van second-opinions zijn onduidelijk en het proces verloopt ongestructureerd. 6. De MKBA is niet altijd als het beste afwegingsinstrument gezien, maar wordt toch gebruikt omdat het verplicht is. 7. Er is te weinig aandacht voor procesaspecten waardoor tijdsdruk ontstaat, de noodzaak om aannames en uitgangspunten te bespreken wordt onderschat, de MKBA wordt te laat in het plan proces ingezet. 8. De MKBA wordt oordelend gebruikt, wat als onjuist wordt ervaren omdat de MKBA nooit het complete plaatje geeft, en de mogelijkheid om te leren uitsluit. 9. Er is te weinig ruimte voor onzekerheden en nuances in het besluitvormingsproces. Theoretische verbetering van het MKBA proces Evaluatieliteratuur benadrukt dat voor de analyse van complexe plannen een leerproces gefaciliteerd moet worden waarin ideeën, meningen en doelstellingen gevormd worden. Dit vraagt om actieve en betrokken deelnemers die met elkaar communiceren en elkaar vertrouwen; twee gerelateerde concepten. Deze inzichten staan haaks op de procesknelpunten die in de eerste fase van dit onderzoek naar voren zijn gekomen. Om de gesignaleerde procesknelpunten te verbeteren, wordt daarom gesteld dat de MKBA als leerinstrument moet worden gebruikt, waarvoor de onderlinge communicatie en vertrouwen moet worden vergroot (zie figuur I). Uit de literatuurstudie blijkt dat het opbouwen van communicatie en vertrouwen tussen verschillende type experts kan worden gestimuleerd door de volgende vijf interventies; (1) creëer ruimte voor ontmoeting, (2) deel en bespreek het plan en de MKBA gezamenlijk en gelijktijdig, (3) wees voorbereid op deze ontmoeting, (4) laat de interactie begeleiden door een moderator, (5) gebruik een discussiemiddel (schetsen, storyboards, rollen spel) waardoor duidelijk dat het plan en de analyse werk in ontwikkeling is. Ondanks dat deze interventies gebaseerd zijn op omvangrijke studies, is de werking ervan in de context van 24 25

16 een MKBA-proces niet duidelijk. Om inzicht te krijgen op of en hoe de interventies werken in MKBA-processen zijn ze daarom getest: in een ervaringsgericht case studie en een ex-post case studie. Figuur I. Conceptueel model van verbetering van het MKBA proces door interventies die de communicatie en het vertrouwen tussen planeigenaren en analisten vergroten, zodat de MKBA meer als een leerinstrument wordt gebruikt. Ervaringsgerichte case studie De interventies zijn toegepast in twee cases die een praktijksituatie simuleerden van een pril MKBA proces van een integraal ruimtelijk-infrastructureel plan: in Amsterdam en in Utrecht. De eerste casus liet matige uitkomsten zien van de gemeten effecten van de interventies waarvoor vier verklaringen werden gevonden. Deze verklaringen zijn gebruikt om de interventies verder aan te scherpen voor de toepassing in de tweede cases door: meer tijd te nemen, de deelnemers niet alleen op gedrag maar ook inhoudelijk voor te bereiden, te benadrukken dat de moderator let op het vermijden van jargon en expliciet te zijn, en een discussiemiddel te gebruiken die aansluiten bij zowel de wereld van de MKBA als van de planning waarvoor de effectenarena is gebruikt. De uitkomsten van de tweede casus laten meer onderlinge communicatie en vertrouwen zien. Ex-post case studie In de ex-post case studie zijn twee contrasterende cases van afgeronde MKBA processen vergeleken: een casus waarin de vijf interventies waarschijnlijk aanwezig waren (het MKBA proces van de ontwikkeling van Schiphol); en een casus waarin deze interventies dat waarschijnlijk niet waren (het MKBA proces van de zeesluis IJmuiden). De cases zijn geanalyseerd op communicatie en vertrouwen interventies mechanismen en effecten op een lerend gebruik van de MKBA. De communicatie en vertrouwen interventies en mechanismen in de cases komen grotendeels overeenkomen met de verwachting. De Schiphol casus wordt gekenmerkt door een communicatie strategie en laat veelal positieve communicatie en vertrouwen mechanismen zien. In de IJmuiden casus ontbreekt een dergelijke strategie en zijn meer negatieve communicatie en vertrouwen mechanismen gesignaleerd. Echter, ook in deze casus neemt in de loopt der tijd de communicatie en vertrouwen toe als het plan verder is ontwikkeld en er een gedeeld gevoel van urgentie ontstaat. Verder laten beide cases verschillende leermomenten zien. In de IJmuiden casus lijkt het leren vooral gestimuleerd door de vele opvolgende MKBA s en gerelateerde analyses die zijn gemaakt, terwijl het leren in de Schiphol casus lijkt plaats te vinden in de dialoog waarbinnen de MKBA is gemaakt. Echter, voor beide cases lijkt een hoog niveau van onderlinge communicatie een vertrouwen cruciaal om meer diepgaand te leren, ongeacht hoe dat niveau wordt bereikt. De cases laten ook zien hoe leren sterk wordt beïnvloed door de plan- en besluitvormingsprocessen, die met het analyseproces verweven zijn. In aanvulling op de ervaringsgerichte case studie bevestigt de ex-post case studie dat de vijf interventies de communicatie en vertrouwen beïnvloeden. Echter, ook wordt aangetoond dat een goed niveau van communicatie en vertrouwen kan worden bereikt zonder een gerichte communicatie strategie, bijvoorbeeld door een gedeeld gevoel van urgentie of politieke druk. Conclusie en aanbevelingen Dit proefschrift heeft het knelpunt van problematische interactie tussen planeigenaren en MKBA-analisten in Nederlandse MKBA processen verder verhelderd en onderstreept het belang van goede communicatie en vertrouwen en het gebruik van de MKBA als leerinstrument. Om dit te stimuleren is een strategie met vijf samenhangende interventies samengesteld die samen als een MKBA-dialoog kan werken. Deze interventies zijn getest in een ervaringsgerichte en een ex-post case studie. In deze tests zijn de interventies verder aangescherpt, waardoor de aansluiting op de context van een MKBA-proces wordt vergroot. Ook is de invloed van het plan- en besluitvormingsproces verder verduidelijkt. Dit proefschrift sluit af met vijf aanbevelingen: 1. Besteed aandacht aan het opbouwen van communicatie en vertrouwen tussen planeigenaren en analisten vroeg in het MKBA proces. 2. Een strategie om communicatie en vertrouwen op te bouwen, zoals de MKBA-dialoog, kan daarbij helpen; 3. Echter, de toepassing van een dergelijke strategie moet behoedzaam worden gedaan, bijvoorbeeld door pilots; 26 27

17 4. Als een dergelijke strategie wordt gebruikt, moeten de vijf interventies worden aangepast op de MKBA-context; 5. Wees bewust van het potentieel van de MKBA als leerinstrument. Prologue This dissertation emphasizes the importance of a better understanding of Cost Benefit Analysis (CBA) processes. However, as I observed, process aspects are often overlooked in CBA reports. Sometimes a CBA report contains a prologue in which the actors involved or the development of the assessed alternatives are briefly mentioned, and sometimes one can read between the lines about process aspects. Mostly, though, CBA reports do not mention process aspects at all. So as not to have the same limitations in this dissertation, this prologue gives some insights into the process of this dissertation. Why: A fascination with CBA In 2008, I worked at a consultancy firm in Amsterdam where I was mainly involved with developing strategic spatial economic plans. At another subdivision of this firm, the focus was on conducting CBAs. This is where the economists worked, whereas at my subdivision most employees had expertise in planning and geography. One day, the CBA division asked me to participate in a CBA assignment of several road extensions in a rural area. I was asked to assess some qualitative and un-monetizable effects, and guide the flows of information; the input (information about the project) that was needed from the client. During the long car rides to this CBA client, I could ask the firm s main CBA expert everything I wanted to know about this evaluation tool. Through him, I received an intense private CBA course, learning its many ins and outs, the terminology, and the economic rationality behind the main calculation principals. The further involved 1 and informed I became, the more fascinated and amazed I got. Fascinated, because a CBA seemed to make it possible to value a large range of effects, and to support decision making about difficult investments by giving a structured overview. However, the shortcomings of a CBA also seemed quite evident, and about this my amazement continued to grow. There were many uncertainties about the assumptions that formed the fundament for the calculations. Furthermore, I experienced that qualitative, non-calculable effects did not receive the same attention as the effects that were calculable. Whereas I was supposed to write at most ten lines about, for example, the value of the loss of nature as a result of building of a new road, the economists involved were working for days to value, for example, the expected travel time savings. Moreover, the interaction with the client was often problematic. Difficulties arose when the client needed to deliver the required information; 28 29

18 plans kept changing or the information given was too broad. Furthermore, it was not easy to explain the CBA to the client, or to make an acceptable assessment. Notwithstanding these limitations, it was also clear to me at that time that the position of the CBA in Dutch planning was and probably would remain strong, since a CBA was mandatory for projects that require national funding. Apparently, my problematic experiences with the CBA were not isolated. In 2009, I read in a newspaper about a PhD position in a research program of the University of Amsterdam that would focus on the problematic use of the CBA in Dutch planning, called OBRRI (Ontwerp en Beoordeling Regionale visies Ruimte en Infrastructuur (In English: design and assessment of regional visions for spatial and infrastructural developments). This research seemed to offer the ultimate opportunity for me to develop a better understanding of why the use of CBA appeared problematic, and to possibly contribute to its improvement. With whom: Involving CBA practitioners This dissertation is thus part of the OBRRI research program, directed by the University of Amsterdam and in cooperation with the Technical University of Delft. The program had two focuses: on content issues 2 and on process issues, the latter of which this dissertation is the result 3. Furthermore, this program had the specific aim to develop scientific knowledge that would be valuable in planning practice. The research was therefore guided by a consortium of three governmental parties, all in the Netherlands (the municipality of Amsterdam, the municipality of Rotterdam, and the province of Gelderland), and the Dutch research institute NICIS (now Platform 31). Moreover, CBA practitioners were continuously involved throughout the research, including representatives of the Ministry of Infrastructure and Environment, the Dutch Bureau for Economic Policy Analysis (DBEPA), the Netherlands Environmental Assessment Agency (NEAA), and several CBA consultants. For this purpose of involving CBA practitioners, the OBRRI group would meet three times per year to share and discuss preliminary research findings. These meetings formed, in my opinion, a great opportunity to actively discuss the research and compare the findings with practitioners experiences. Moreover, these meetings forced me to leave the academic ivory tower and make sense of abstract knowledge. However, sharing preliminary findings with very critical practitioners was not always easy. Although the participants knew the research was still ongoing, there were expectations to come up with new insights every time. These meetings were therefore quite time-consum- ing. Furthermore, the academic output of journal articles was not appealing to planning practice. Therefore, two reports with a practical orientation were also written 4. Nevertheless, the plusses far outweighted the minuses. This apparently was also felt by the consortium and other participants, because their interest in the research and loyalty towards these meetings remained high through the end. How: What did not work This dissertation gives a precise description of the conducted research. However, it reflects mostly on everything that worked, whereas not all the efforts made were effective. For example, in 2011 I had the opportunity to temporarily live and work in New York City (NYC), United States (US). Considering that the CBA (or Benefit Cost Analysis) is originated in the US, NYC seemed to offer an interesting context in which to conduct a case study on how the CBA was used in US spatial transport planning, with a specific focus on communication and trust aspects between plan owners and evaluators. However, conducting a case study in NYC was very difficult for me; I was clearly not familiar with local customs and practices, and most importantly I did not have a network of CBA and planning practitioners. In NYC, the practitioners I contacted had one thing in common: they appeared to be very busy and did not have time to talk to a PhD student from Amsterdam. Outside of NYC, though, the planning practitioners seemed more relaxed and willing to cooperate. However, from the conversations I had with these practitioners, I got the impression that the challenges in US spatial transport planning were very different from the Dutch, because the focus there was mostly on maintenance and safety issues. These insights made me decide not to conduct research outside of the Netherlands. The type of research I was conducting asked for an in-depth understanding of personal experiences of CBA practitioners. Access (Bryman, 2008, p. 461) was thus very relevant, and Dutch planning practice was the most accessible context for me. Furthermore, I conducted a pilot of the experiential case study to find the most suitable approach for testing the theoretical CBA process improvements. For this pilot, I cooperated with the planning division of the city department of Amsterdam South, which was about to assess a complex spatial transport plan using a CBA: the possible development of a car garage under a canal in a densely populated residential area close to the touristic Amsterdam Museum Square. To test the theoretical process improvements, I designed an experiential setup with a test and control group, asked the participants to perform according to the theoretical CBA process improvement, and tried 30 31

19 to measure the effects. However, although the participants felt the meeting in which the test was performed was valuable to them, this experiential setup appeared far too complex to generate reliable findings. Nevertheless, this pilot was very important for me in developing the experiential case study that did work, and through which the theoretical CBA process improvements were tested, as described in Chapter 4. A PhD research process This dissertation is thus the result of a research process that is influenced by many decisions large and small, intended and unintended developments, academic and practical considerations, and personal dynamics. The final result, though, is consolidated in this document. NOTES 1 I contributed to two large CBAs. One to prioritize regional road investments that aimed to decrease traffic jams and the local livability by relocating roads outside the villages. The second CBA assessed an investment in a national biodiversity center. 2 The research with a content focus was conducted by Niek Mouter and supervised by prof.dr. Bert van Wee and dr. Jan-Anne Annema from the Technical University of Delft. 3 The research with a process focus was supervised by prof.dr.ir. Luca Bertolini and dr. Marco te Brömmelstroet from the University of Amsterdam. 4 Beukers, E., Bertolini, L. and te Brömmelstroet, M. (2011). Knelpunten in het MKBA-proces, Nicis, Den Haag. Beukers, E., Bertolini, L. and te Brömmelstroet, M. (2013). De MKBA dialoog, Platform 31, Den Haag. 32 1

20 CHAPTER 1: Introduction 35

21 1.1 A problematic use of CBAs The cost benefit analysis (CBA) would seem to be an attractive assessment tool for planning and evaluating transport (or other kinds of) plans. It promises to structurally expose all expected effects on society, translate these effects in monetary terms, and assess whether an investment might be worthwhile by giving a benefit-cost ratio. Based on these attributes, the CBA has become a widely used instrument for the appraisal and evaluation of large transport projects in most OECD countries (Haezendonck, 2007; Mackie, 2010; May et al., 2008; Odgaard et al., 2005; Rotaris et al., 2010; Vickerman, 2000). In the Netherlands, the role of the CBA in decision-making processes has become increasingly important. Since 2000, it has been obligatory for large transport plans partially funded by the Dutch national government (Annema et al., 2007; De Jong and Geerlings, 2003; Eijgenraam et al., 2000). Since 2007, it also has been obligatory for the assessment of integrated spatial transport plans, because of the merging of the governmental budgets for these sectors (Ministry of Transport and Water Management and Ministry of Housing, Spatial Planning, and the Environment, 2009). This means that all local and regional spatial transport plans that require national funding require a CBA assessment. Although a positive CBA balance is not a formal requirement for approved funding, these planning regulations do give the CBA a central role in Dutch planning and decisionmaking processes (Rienstra, 2008). For example, the 2009 CBA (DBEPA & NPAE, 2009) of a possible new rail connection between Amsterdam and Almere across the IJ lake to support the economic development of both cities was the central topic of the public debate in the national parliament (Ministry of Transport and Water management, 2009); in the involved city councils (Gemeente Almere, 2009); in national, regional, and local media (Almere vandaag, 2009; Trouw, 2009; NRC, 2009); and, among and within some of the involved institutions (DBEPA, 2009; Geurs et al., 2011). In addition to being extensively used, the CBA is also a controversial tool (Annema et al., 2007; Jong & Geerlings, 2003; Mackie, 2010; Naess, 2006; Mackie & Preston, 1998). Especially when assessing complex spatial transport plans that aim to influence fundamental but hard to grasp aspects of societal challenges like sustainability, or quality of life, that resist easy translation into monetizable pieces (Ziller & Phibbs, 2003; Vaughan et al., 2000; van Wee and Molin, 2012). Aside from these content challenges, research has shown that assessing complex transport plans in a CBA often comes with severe process challenges (Sager & Ravlum, 2005; Martinsen et al., 2010; Eliasson & Lund- berg, 2010; Veisten et al., 2010). Planning practitioners seem to perceive CBAs as a final judgment of a plan s fulfillment of CBA criteria, leading some planners to see it as just another obstacle to overcome instead of an opportunity for learning and improving the plan (Page et al., 2009; ECTM, 2004; Savelberg et al. 2008; Rienstra, 2008). Again, this process issue seems to be aggravated when it comes to integrated plans and their attendant group of diverse actors. Although these studies shed some light on the problematic use of CBAs in complex planning processes, there remains an opportunity for a more in-depth understanding of possible causes of process issues when assessing spatial transport plans in CBA. This knowledge gap is problematic because it remains to be seen whether solving the content-related issues alone will be enough to reverse the antagonistic attitude that many planning actors have towards the use of the instrument itself. Furthermore, considering the necessary development of integrated spatial transport plans that concern multiple goals and involve a variety of actors, process issues are becoming even more relevant (Banister, 2008; Bertolini et al., 2008; Curtis, 2008). This dissertation therefore aims to narrow this knowledge gap by focusing on CBA processes in assessing integrated spatial transport plans. Moreover, this dissertation aims to identify ways to improve the process issues at play, since understanding a problem is only halfway of solving it. The second step is to develop and test (alternative) solutions. (Van Aken, 2004, p. 220). The overall research questions of this dissertation are therefore: What process issues occur when assessing integrated spatial transport plans through a CBA in the Netherlands, and how can the CBA process be improved? Besides practical considerations, there are two reasons why the Netherlands was chosen as the context for this study. First, as mentioned above, CBA is extensively used in Dutch planning for the assessment of transport and integrated spatial transport plans. Second, two Dutch studies in 2008 indicated the existence of CBA process (and other) issues (Savelberg et al., 2008; Rienstra, 2008). The Netherlands therefore seems to form a suitable context for an in-depth investigation of CBA process issues when assessing integrated spatial transport plans. Before further elaborating the research setup, a better understanding of the changing context of CBA assessments is necessary. The remainder of this introductory chapter is therefore structured as follows. Paragraph 1.2 discusses the emergence of integrated spatial transport plans and its consequences for (CBA) assessment processes. Then, the central research 36 37

22 question, research design, and methodological choices are addressed in paragraph 1.3. This introductory chapter closes with an outline of this dissertation in paragraph 1.4. Figure 1.1. Contrasting approaches to transport planning (Banister, 2008, p. 75) 1.2 Transport appraisal: The changing context For a deeper understanding of the CBA process issues in the next chapters, this paragraph sketches out the changing context in which CBAs take place. First, the urge to integrate the domains of transport and land-use planning is highlighted, after which is described how the contrasting approaches of the technical-rational and deliberative-rational models affect (CBA) assessments Linking the domains of transportation and land use planning Integration of spatial and transport planning is widely seen as a promising and urgent direction for addressing dilemmas of sustainable mobility (Banister, 2008, Bertolini et al., 2008; Curtis, 2008; Meyer and Miller, 2001; Te Brömmelstroet, 2010a). However, linking these domains is extremely difficult because of their fundamental differences. Whereas transport planning tends to have a technical-engineering and economics-based character and is dominated by instrumental rationality, with like-minded indicators and modes of thinking, land use planning (today) is less technical, more visionary, and has embraced the paradigm of communicative rationality (Te Brömmelstroet, 2010a; Willson, 2001). For instance, transport planners prefer to express themselves through numbers as a precise form of language, implying objective knowledge (Willson, 2001, p. 1). Land use planners, however, tend to stress that there is no such thing as objectivity (Allmendinger, 2002) and are oriented towards reaching intersubjective understanding (Allmendinger, 2002; Healey, 1992; 2003; Huxley, 2000; Huxley and Yiftachel, 2000). The contrasting characteristics of transport and spatial planning are further emphasized by Banister (2008) in Figure 1.1. In this figure, he places integrated spatial transport planning, labeled as sustainable mobility, opposite conventional transport planning. Furthermore, the scheme places characteristics of the CBA on the left side as components of conventional transport planning, as they are an economics-based evaluation that uses a modeling approach and uses traffic forecasts as the main source of information for further calculations (Mishan and Quah, 2007). Integrated spatial transport plans, meanwhile, are characterized by the conditions on the right, and, according to Banister, are best assessed through a multiple-criteria analysis Limits of the technical-rational model This dichotomy between conventional transport planning and integrated spatial transport plans has large consequences for the theory and practice of CBAs when assessing such integrated plans, as discussed by Owens et al. (2004). In their view, among others (Khakee, 2003), appraisal evolved from more traditional and instrumental approaches, the technical-rational model, towards a more communicative approach, the deliberative-rational model. Parallel to the characteristics of Banister s conventional transport planning, the technical-rational model promises to produce objective and value-free knowledge, and assumes a process in which scientific advice, grounded in a positivist epistemology, translates straightforwardly into the substance of policy, and a separation of powers is deemed to exist between neutral, authoritative experts and the decision makers they advise (Owens et al., 2004, p. 1945). Owens et al. (2004) gave three arguments why the technical-rational model is limited and might even be of danger. First, they discussed that the assumption of a unitary decision maker is an oversimplification of modern democratic policies, and that analysis rarely informs policy in the simple linear manner envisaged. Second, they indicated that ethical and political choices masquerade as technical judgments, reinforcing prevailing 38 39

23 norms and existing structures of power. Third, they argued that the dangers of technical rationality are at their most acute (and have been manifest most clearly) when the issues at stake are transscientific, in the sense that science is unable to converge upon a solution and/or the relevant problems are unstructured, such that proponents cannot agree on problem framings, let alone upon a constructive search for solutions (Ownes et al., 2004, p. 1946). These latter two limitations resonate in a study by Handy (2008) that brings these arguments closer to the practice of CBAs. Handy studied what goals get the most weight in the planning process and found that: - goals that have performance measures that can be forecast using travel demand models get the most weight in the process; - goals that have performance measures that cannot be forecast using the travel demand model are unlikely to be forecast using other techniques and get less weight in the planning process; - goals without performance measures get the least weight in the planning process. In other words, goals that are difficult to measure and forecast, like increasing livability or sustainability, could get lost along the process (by not being debated, or not having arguments informed by assessment and/or research) if the assessment approach used is dominated by travel demand models. As such, Handy described how the entrenched use of travel demand models to forecast system performance (as a component of the technical-rational model) determines planning processes toward congestion relief, despite the adoption of a broader range of sustainability goals, and despite the overall agreement that the predict and provide transport planning paradigm tends to be outdated (Meyer and Miller, 2001). Handy, therefore, emphasized that if the new (sustainable) goals are truly important, then new planning tools and/or (process) approaches are needed. She stated that transportation planning agencies need to develop performance measures for all their goals, not just the traditional measures, and must clearly match performance measures to goals. Furthermore, she argued that if some measures cannot be forecast, then agencies must find a way to give equal weight to concerns with and without forecasts, even if this means downplaying the role of forecasts in the process (Handy, 2008, p. 124) The communicative-rational model as the way forward? In reaction to the described limitations of the technical-rational model, the communicative-rational model was developed with particular attention to social practices, norms, behavior, and language (Allmendinger, 2002; Healey, 1992; Huxley, 2000), and on reaching inter-subjective judgments through a process of argument and debate (Owens, 2004). Furthermore, Allmendinger (2002, p. 86) emphasized that communicative perspectives posit a much more socially, historically, and locally embedded understanding underlining the context of planning. However, despite new rhetoric, criticism of the technicalrational model has had rather little impact on the conceptions or practices of appraisal, and seems to have failed to provide a coherent alternative. Deliberative and inclusive processes are upheld as an alternative to technical rationality but seem difficult, expensive, time consuming, and (to the discomfort of decision makers) potentially inconclusive. (Owens et al., 2004, p. 1950) This dichotomy between technical-rational and communicativerational approaches, though, is less absolute than it appears (Owens, et al. 2004), since the approaches share characteristics. For example, technical procedures also facilitate deliberation and learning, although not always intentionally. Annema et al. (2007) described, for instance, how conducting multiple CBAs in transport planning in the Netherlands seemed to have led to further improvements of the assessed plans, which they thought could be interpreted as unintended learning processes. Furthermore, like Owens et al., Innes (1998) stated that a potential new planning model should not dismiss calculations and quantitative analysis. Instead it should take into account that technical information should become gradually embedded in the understandings of the actors in the community (including planning experts). It seems indeed likely that ex-ante evaluations like the CBA will remain important in the design of and decision making on spatial transport plans. Politicians, policy makers, private parties, and the public should be informed how a discussed plan could contribute to a desired future that remains debatable and unknown (Couclelis, 2005), and what the expected effects, costs, and benefits could be. As argued above, it is necessary, however, that ex-ante evaluations of integrated plans be able to provide the information needed for complex settings; not only technical information, such as travel time savings, but also (transscientific) effects on complex goals like economic diversity or sustainability. Assessing integrated spatial transport plans with complex and diverse (transscientific) goals thus asks for a communicative approach in addition or parallel to a technical approach. As such, 40 41

24 attempts to integrate communicative elements into or alongside instruments of the technical approach seem to be a step towards finding a balance between the two, adjusted for contemporary demands (Ziller and Phibbs, 2003; Haezendonck, 2007). However, this search for balance is far from finalized, considering the ongoing tensions in planning practices of the appraisal of complex, integrated plans (Page et al., 2009; Stoeglehner et al., 2009; Bond et al., 2012; Morrison-Saunders et al., 2014). 1.3 Overall research design This dissertation builds on the issues sketched out in the previous paragraph, using the Netherlands as a case in point, and considers: - what process issues occur when the CBA is used for assessing complex plans, like integrated spatial transport plans? - and how can the CBA process be improved? Research questions The overall research questions are: What process issues occur when assessing integrated spatial transport plans through a CBA in the Netherlands, and how can the CBA process be improved? These questions are answered through three sub-questions: 1. What process issues do CBA participants perceive in CBAs of integrated spatial transport plans in the Netherlands? 2. What are promising mechanisms and interventions for improving the perceived CBA process issues? 1 3. Which interventions improve the perceived process issues, and through which mechanisms? 2 The concepts in these research questions are defined as follows: The CBA process is considered as the process in which a plan is assessed through the CBA methodology, involving different CBA participants. This process includes the formation and use of CBA inputs (e.g., project alternatives, assumptions about project effects and monetary values, future scenarios), and output (the formal CBA report). The start and end of a CBA process is not always clear. Sometimes, several assessments are made that together form the CBA process. Process issues are issues that relate to how (where, when, why, with whom, and in which manner) CBA participants interact, and how (where, when, why, by whom, and through what method and technique) information is processed. CBA assessments aim to answer a priori the question whether a project (or a number of projects) should be undertaken, and which one (or ones) should be selected. Furthermore, a CBA aims to answer whether, by undertaking the project rather than not undertaking it or undertaking an alternative, net benefits will accrue to a society consisting of all the individuals who reside or work within the area in question (Mishan and Quah, 2007, p. 5). Several techniques are employed in CBA to estimate quantitatively the impacts for each alternative and to assign monetary values to project benefits and costs (Meyer and Miller, 2008). Besides extensive assessments, CBAs can be made based on fixed numbers, for example, giving a more indicative evaluation. Integrated spatial transport plans are transport plans with wider spatial, social, and economic development goals. These plans typically have a mixed local, regional, and national scope, and involve various actors. An improved CBA process is expected when the process issues are mitigated. A perception represents how something is regarded, understood, or interpreted. Interventions are concrete actions that are needed to trigger specific mechanisms, and the mechanisms reveal how something works and how the different elements function in relation to each other. Before addressing the overall research epistemology in the next paragraph, it should be noted that, with its focus on assessment processes, this dissertation could be placed in the tradition of Faludi s typology of planning theory which makes a distinction between process and substance. Thereby, process and procedures are seen as the natural business of planners (Faludi 1973, in: Allmendinger, 2002). However, although this distinction was dominant in planning theory for several decades and has yet to be fully replaced by a satisfactory alternative (Allmendinger, 2002), it has been noted that process and content are hardly separable (De Bruijn et al., 2002). Furthermore, from a post-positivist point of view the ability to separate facts and values is rejected, as well as the distinction between substance (analysis) and procedure (process) (Allmendinger, 2002, p. 84). As such, this dissertation will entail several examples of the inseparability between content 42 43

25 and process; for instance, the process issues will be shown as emerging when difficult-to-measure effects (like livability) are at stake Research epistemology: Design science The three research questions are not only focused on understanding CBA process issues, but also on contributing towards an improved CBA assessment of integrated transport plans through developing and testing (alternative) solutions to the identified issues. As such, this dissertation is part of a recent tradition in planning studies with design-oriented research approaches (Te Brömmelstroet, 2010b; Hoetjes, 2010; Chorus, 2012; Straatemeier, 2008; Tan, 2013), that aim to mitigate the utilization problem of explanatory sciences (Van Aken 2004) and the oft-described gap between theory and practice (see, for example, Willson, 2001). According to Van Aken (2004), prescription-driven research (instead of description-driven) is necessary for developing research products that can contribute to solving (rather than just understanding) problems (as required in fields such as engineering, medicine, law, or management). Such research should aim for the development of technological rules for solving problems that are systematically tested within the context of their intended use. Outcomes should be abstract and scientifically grounded knowledge that can be used as design exemplars to solve a class of problems, instead of a specific planning problem, the latter being the domain of practitioners. Straatemeier et al. (2010) followed this reasoning, and emphasized that one cannot cope with complexities of planning practice just from an outside and academic position. To systematically test planning innovations within the context of their intended use, they made use of the experiential learning cycle of Kolb and Fry (1975) as a framework to develop and reflect upon planning innovations in interaction with practice. This experiential learning cycle (learning through experience), illustrated in Figure 1.2, follows four iterative steps: (1) concrete experiences from practice are (2) observed and reflected upon, (3) leading to the formulation of abstract concepts (like a potential planning innovation), that are (4) tested in new situations, which (1 ) lead to new concrete experiences, and so on. Figure 1.2 The experiential learning cycle (Straatemeier et al. 2010, adapted from Kolb and Fry, 1975) The learning cycle provides a useful framework for understanding how planning practice and research potentially (and ideally) could be linked more strongly. Now, planning practitioners tend to focus on the top side of the cycle by observing and reflecting on their daily concrete experiences and applying their insights in new situations, forming new concrete experiences. However, they seldom relate this to a more fundamental, theory-based understanding of problems and solutions. On the other hand, planning research has a stronger focus on the bottom side of the cycle, focusing on forming abstract concepts, often formulated as recommendations, derived from observation of and reflection on practice. However, they seldom test these in practice. Thus, the cycle is rarely completed as the structural testing of academic insights into new situations hardly happens, and practitioners tend not to step back to more critically and fundamentally reflect on their decisions and actions (Straatemeier et al., 2010, p. 581). This might be a lost chance for planning research to help solve planning problems. Furthermore, Straatemeier et al. (2010) argue that the development of practically useful knowledge in planning could be improved if the relationship between planning research and practice is enforced through completion of the learning cycle. This, though, requires change from both practitioners and researchers. Researchers need to engage more in practice (in concrete experience ) and practitioners need to engage more in research (in forming abstract concepts ). (Straatemeier et al., 2010, p. 581) 44 45

26 1.3.3 Research methodologies As mentioned above, this dissertation aims not only to understand CBA process issues, but also contribute to their improvement. Therefore, the experiential learning cycle was followed, enabling a link between planning research and practice, and CBA practitioners were involved throughout the entire research. As illustrated in Figure 1.3, this dissertation entails three research phases that together form the academic half of the learning cycle. Figure 1.3. Three research phases parallel to the four stages of the experiential learning cycle Table 1.1 gives an overview of how the four steps of the learning cycle correspond with the three research questions, what methodology was used to answer these questions, how CBA practice was involved in each step, and where this was reported in this dissertation. Besides including the concrete experiences from the involved CBA practitioners into the research, the activities described in Table 1.1 also enabled the practitioners to observe and reflect on their own experiences. Furthermore, the findings of each research phase were shared with various CBA practitioners throughout the research 3. The involved CBA practitioners thus might have been inspired by the academic findings, which they could have applied in new situations in their daily practice, leading to new concrete experiences (dashed arrow in Figure 1.3). However, this particular spin-off was not monitored. Paragraphs 1.4 to 1.6 discuss the methodological choices that were made for answering the three research questions. In Paragraph 1.7, attention is paid to the use of triangulation throughout this dissertation. Specific methodological details, though, can be found in chapters 2 to 5. Furthermore, although the research findings are going to be discussed in the corresponding chapters, some key findings are briefly mentioned in order to clarify the methodological choices made. This chapter concludes with Paragraph 1.8, an overview of the outline of this dissertation. 1.4 Research question 1: Finding CBA process issues The first research question focused on finding issues in CBA processes when assessing integrated spatial transport plans (illustrated in Figure 2), for which an inductive and grounded approach was applied. This approach was necessary for two reasons. First, there is limited fundamental knowledge on the process-related issues of CBA. Second, at the start of this research some explanations for the problematic use of the CBA were already formulated by the guiding consortium (see Prologue). Therefore, to prevent bias and avoid a tunnel vision of limiting the empirical findings to known issues (since there is only limited academic knowledge on them), it was necessary to follow an inductive approach in this phase. This inductive approach implied that the research was conducted without formulating theory-based hypotheses in advance. Instead, the theoretical conceptions were directly derived from the empirical data and related to theory as a final step (Bryman, 2008, pp. 9 13). Furthermore, the research focused on a broad range of perceptions of CBA practitioners (ranging from the actual CBA evaluators, to plan owners, to residents confronted with a CBA in the planning process) on CBA processes, instead of directly on process issues. From this pluralistic view on CBA processes, and by using Grounded theory (Lewins and Silver, 2007), the process issues were reconstructed with a minimum influence of assumptions and hypotheses from the researcher. The CBA process perceptions were collected through 22 indepth interviews and 3 focus groups (in which 10, 9, and 7 CBA practitioners participated, respectively). Although the CBA community in the Netherlands entails more individuals than those involved in this research 4, theoretical saturation was reached for finding the most prominent process issues. Triangulation between the two research methods was important for this (see Chapter 2 for further details)

27 Table 1.1 Relating the four stages of the experiential learning cycle, the three research questions, the methodology used, how CBA practice was involved, and where reported in this dissertation Step in learning cycle Concrete experiences Observation and reflection Forming abstract concepts Testing in new situations Corresponding with which subquestion? RQ 1: What process issues do CBA participants perceive in CBAs of integrated spatial transport plans in the Netherlands? RQ 2: What are promising mechanisms and interventions for improving the perceived CBA process issues? RQ 3: Which interventions improve the perceived process issues, and through which mechanisms? Answered through which research methodology? In-depth interviews and focus groups Literature review Experiential case study (control-rich, context-low) Ex-post case study (controllow, contextrich) How were CBA practitioners involved? The research topic of this dissertation was suggested by the guiding consortium of CBA practitioners, based on their concrete CBA experiences Perceptions of CBA practitioners were collected to identify possible process issues The theoretical process improvements that were found in the literature review were discussed with the guiding consortium in order to preliminarily assess their potential practical relevance CBA practitioners experienced the theoretical CBA process improvements in the experiential case study Where reported in this dissertation? Prologue Chapter 2 Chapter 3 Chapters 4 and 5 The findings were synthesized into three CBA process issues: (1) communication deficits and (2) distrust between the two prominent CBA practitioners (plan owners and CBA evaluators), and (3) an overall desire to use the CBA more for learning instead of solely as a final assessment. These three issues corresponded with the assessment literature in that a learning use of assessments remains an important contemporary challenge, and that optimal communication and trust seem to be crucial conditions in achieving this learning use. This research phase was finalized through the construction of a conceptual framework that formed the potential direction for improvement of CBA processes when assessing integrated spatial transport plans, illustrated in Figure 1.4. This framework directed the search for process improvements in the second phase. Figure 1.4. Conceptual framework of how to improve CBA process issues 1.5 Research question 2: Theoretical improvement process issues To answer research question 2, academic literature was analyzed for concrete interventions and mechanisms on how to improve communication and trust between plan owners and CBA evaluators (see Figure 1.4). The literature review focused on the research fields of deliberative planning and organizational learning: two fields with rich ideas for improving communication and trust between different types of experts. The literature selection followed two criteria: most frequently cited publications and publications that provided specific, innovative ideas on communication and trust building. In processing these publications, all lessons on how to improve 48 49

28 communication and trust between different kinds of experts were first identified and then grouped based on the similarities in their reasoning. This resulted in 5 interventions for improving communication and trust between plan owners and evaluators and 30 measurable mechanisms that reveal how these interventions work. These interventions would be tested within the context of their intended use in the third research phase. 1.6 Research question 3: Testing the theoretical improvements This research question forms the key part of this dissertation: systematically testing theory-based CBA process improvements within the context of their intended use (see Figure 1.3). This context consists of processes in which integrated spatial transport plans are assessed through a CBA, and more specifically the interaction between plan owners and evaluators within these processes. However, systematically testing (applying and monitoring) the five CBA process improvements within, for example, an ongoing CBA process would be difficult, considering that: - The five interventions were not tested yet in a CBA process (and if at all tested very limitedly in other practices), which made it uncertain as to how they would affect the assessment, planning, and decision-making, and therefore risky; - It is unlikely that all involved participants would cooperate and act along the five interventions; - CBA processes tend to have a long time frame, which creates a practical constraint for this research. Therefore, to test the five interventions a two-way strategy was followed. First, the five interventions were tested in an experiential case study, developed by Straatemeier et al. (2010), that formed a close-to-real situation. Second, a classic ex-post case study was conducted of two CBA processes: one in which the five interventions likely were present and one in which they likely were not. These two research methods were supplementary as they enabled the testing of the interventions in a context-low but control-rich setting (the experiential case); and a context-rich but control-low setting (the ex-post case) The experiential case study Straatemeier et al. (2010) developed the experiential case study to test planning innovations in a controlled and academic setting that represented the context of its intended use. They proposed an iterative cycle of selecting a testing situation involving the intended users, planning and implementing the planning innova- tions, monitoring its effects, reflecting on the results, and developing knowledge to be tested and refined in subsequent testing contexts. This approach allows practitioners and researchers to experience and reflect upon the interventions, which can be used to further improve the interventions, and could be tested and further improved in a next case. In this research, the five CBA process interventions were tested in two sequential experiential cases: a CBA of station area development in Amsterdam (Case 1), and a CBA of tramway routes in the Utrecht region (Case 2). In these cases, the intended users (plan owners and CBA evaluators) were involved. The effects of the interventions were measured through the 30 mechanisms that were supposed to be triggered by the interventions, as defined in the second research question. Furthermore, the reflections on the first case were used to further improve the interventions. The refined interventions were then tested in the second case. Thus, the two cases together formed an iterative cycle (illustrated in Figure 1.5). Figure 1.5 The iterative cycle within the experiential case study with two sequential cases The ex-post case study The control-rich, context-low approach of the experiential case study was complemented with a context-rich, control-low approach by conducting a classic ex-post case study. The main focus of the ex-post case study was to make sense of human action in a CBA process without clear boundaries between the as

29 sessment process and its wider planning and decision-making context, taking an interpretive stand (Bryman, 2008, p. 384). The setup of the ex-post case study was to compare two cases: one case with the interventions and one without. Varying the interventions as the independent variable would allow for the assessment of their effect. The perceptions of the plan owners and CBA evaluators involved were captured through narrative interviews that focused on the stories and recollections of past experiences (Giezen, 2012). The selection of interviewees was stratified to ensure that both evaluators and plan owners were represented. Furthermore, the interviewees were asked to reflect on whether the list of interviewees represented the breadth of standpoints in the case. This supported the inclusion of all relevant perspectives and attainment of theoretical saturation (Bryman, 2008, p. 416). As mentioned above, the experiential case study and the ex-post case study formed the third research phase of testing the theoretical process improvements. Therefore, this research phase is finalized through triangulating the findings of the two, which is discussed in Chapter Triangulation In this dissertation, triangulation is often used for increasing the quality of the research and establishing valid and reliable findings. A closer look at this technique might therefore be relevant. When studying social phenomena, triangulation is often recommended to increase its (internal) validity. Through triangulation, a research question is analyzed from different perspectives, which increases confidence in research data and gives a clearer understanding (Thurmond, 2001, p. 254). Guion et al. (2011) describe five types of triangulation: 1. Data triangulation, when different sources of information are used; 2. Investigator triangulation, when different investigators are involved in the analysis using the same qualitative method; 3. Theory triangulation, when multiple perspectives (often from different disciplines) are used to interpret a single set of data; 4. Methodological triangulation, when multiple qualitative and/or quantitative methods are used; 5. Environmental triangulation, when using different locations, settings, and other key factors related to the environment in which the study took place, such as the time, day, or season. In this research, all five types of triangulation are used. Methodological triangulation is used to answer research question 1: conducting in-depth interviews and focus group sessions. The findings of both methods are then analyzed to find the areas of consistency and inconsistency. Theoretical triangulation takes place between the two academic fields that are reviewed to answer research question 2. Methodological triangulation is exercised to answer research question 3 by testing the theoretical improvements in the experiential and ex-post case studies. Triangulation also takes place within these latest two research methods. The experiential case study is monitored through several means, in which both investigator, methodological, and environmental triangulation occur. First, by involving another observer, other than the first researcher who conducted this study. Second, by using several research techniques: questionnaire, observation, video analysis, group discussion, and reflective interviews. Third, by interviewing the participants some weeks after the experiential case was conducted, enabling them to reflect on their experiences in a different time, place, and social setting. Furthermore, data triangulation is relevant for the ex-post case study by collecting the perspectives of multiple sources: interviewing different stakeholders and analyzing the related assessment reports and literature. 1.8 Outline of this dissertation The remainder of this dissertation is organized as follows (see also Appendix II): Chapter 2 reflects on the search for CBA process issues. This chapter is a reprint of the article Why Cost Benefit Analysis is perceived as a problematic tool for assessment of transport plans: A process perspective that was published in the Journal of Transportation Research Part A (2012) 4. Chapter 3 gives a review of deliberative planning and organizational learning literature, synthesizing the theoretical interventions for improving the perceived CBA process issues. This chapter is a reprint of the article Using cost benefit analysis as a learning process: identifying interventions for improving communication and trust that was published in the Journal of Transport Policy (2014) 5. Chapters 4 and 5 form the third research phase. Chapter 4 discusses the testing of the theoretical interventions in an experiential case study. This chapter is a reprint of the article An assessment of Interventions for improving communication and trust in Cost Benefit Analysis processes, accepted for publication in the journal of Impact Assessment and Project Appraisal 6. Chapter 5 addresses the ex-post case study. This chapter is a reprint of the article Communication and trust as conditions for using 52 53

30 Cost Benefit Analysis for learning: Comparing the CBA processes of Schiphol Airport and the IJmuiden sea lock, which is under revision 7. Chapter 6 forms the concluding Paragraph of this dissertation. In addition to overall conclusions and reflections, this chapter triangulates the findings of the experiential and ex-post case studies. Directions for further research are also discussed. NOTES 1 This sub-question is reformulated in chapter 3 as: What are promising interventions and mechanisms for improving CBA process issues specifically, improving communication and trust between plan owners and evaluators? 2 This sub-question is reformulated in chapters 4 and 5 as: Do the interventions derived from the theoretical literature improve communication and trust conditions between plan owners and evaluators in CBA processes, and through which mechanisms? 3 The research findings were, for example, presented at the economics advisory board of the Ministry of Transport and Environment, at the national planning agency for the environment, at the municipality of Amsterdam, at the province of Gelderland, and at the Dutch Knowledge platform for mobility. 4 Beukers, E., Bertolini, L., M., Te Brömmelstroet, Why cost benefit analysis is perceived as a problematic tool for assessment of transport plans: a process perspective. Transp. Res. A: Policy Pract., 46, p Beukers, E., Bertolini, L., M., Te Brömmelstroet, Using cost benefit analysis as a learning process: identifying interventions for improving communication and trust. Transport policy, 31, p Beukers, E., Bertolini, L., M., Te Brömmelstroet, 2014, An assessment of Interventions for improving communication and trust in Cost Benefit Analysis processes, Impact Assessment and Project Appraisal, DOI: / Beukers, E, submitted, Communication and trust as conditions for using Cost Benefit Analysis for learning: Comparing the CBA processes of Schiphol Airport and the IJmuiden sea lock 54

31 56 2

32 CHAPTER 2: CBA process issues Why Cost Benefit Analysis is perceived as a problematic tool for assessment of transport plans: A process perspective 59

33 ABSTRACT Academic discussions on Cost Benefit Analysis (CBA) as an appraisal instrument for integrated spatial transport plans tend to focus on its technical aspects. However, many issues of CBA also arise from process related matters, especially when assessing integrated plans. Using an inductive research design, we explored how these process related issues play out in Dutch planning practices. In two applied research techniques, focus group sessions and open in depth interviews, we focused on process related issues as perceived by CBA participants ranging from plan makers to CBA testers. This article presents the different perceptions of issues in CBA processes. Through these collected perspectives, we found that these issues are multi-layered and present a number of fundamental dilemmas. After relating our empirical data to theory, we conclude that the biggest challenge lies in decreasing the level of mistrust and communication deficits revealed between plan owners and CBA evaluators, and the perceived use of the CBA as a final assessment instead of an opportunity for learning

34 2.1 Introduction CBA as a problematic instrument Cost Benefit Analysis (CBA) is a widely used ex-ante evaluation tool to support the decision making on infrastructure plans and others. Its aim is to provide an integral overview of the (estimated) costs and benefits of alternative plans, and to translate them as much as possible into monetary terms for comparison (Brent 1996). Based on these attributes, the CBA has become a widely used instrument for the appraisal and evaluation of large infrastructure projects in many countries (Haezendonck 2007; Mackie 2010; May et al. 2008; Odgaard et al. 2005; Rotaris et al. 2010; Vickerman 2000). In the Netherlands, the role of CBA in the decision-making process has become increasingly important. It was already obligatory for large infrastructure plans partially funded by the Dutch national government (Annema et al. 2007; De Jong and Geerlings 2003; Eijgenraam et al. 2000). Since 2007, it also has been obligatory for the assessment of integrated spatial infrastructure plans, because of the merging of the governmental budgets for these sectors (Ministry of transport and water management and Ministry of housing spatial planning and the environment 2009). This means that all local and regional spatial infrastructure plans requiring national funding need to go through a CBA assessment. Through this, the national government prioritizes proposed plans and decides which ones are funded. Although a positive CBA balance is not a formal requirement for approved funding, these planning regulations do give the CBA a central role in Dutch planning and decision-making processes. Although widely used, CBA is also contested as being inadequate for appraising transport-related plans (Annema et al. 2007; De Jong and Geerlings 2003; ECMT 2004; Mackie and Preston 1998; Naess 2006; Priemus et al. 2008; van Wee 2006). Academic literature identifies a number of aspects of CBA that underlie this critique. These include disputable calculation methods for translating soft variables like quality of nature into money, or leaving these effects out of the analysis altogether (Mackie and Preston 1998); missing information about winners and losers (the distribution effects) and ignoring equity issues (Ackerman and Heinzerling 2002); missing information about expected synergy and agglomeration effects (van Wee 2006); poorly constructed reference cases (Annema et al. 2007; De Jong and Geerlings 2003; Naess 2006; Wee 2006); poor incorporation of uncertainties (Salling and Banister 2009; Ševcíková et al. 2011); and too much focus on how infrastructure can help solve traffic bottlenecks (i.e. decreased travel time) and too little on how it can support a vision for spatial economic developments (ECMT 2004; van Wee et al. 2006). The last problem is perhaps not surprising if we consider the intrinsic difficulty of measuring such effects. According to Mackie (2010), it is very difficult to appraise the effect of investment in infrastructure on the regional economy. Moreover, Mackie states that: The interaction between transport and the wider economy, and its treatment in appraisal, is one of the most lively current topics (Mackie 2010, p.19). Although these are important issues, it is questionable whether solving these technical, content-related issues alone will be enough to reverse the antagonistic attitude that many planning actors have towards the use of the instrument itself. CBAs, especially when applied to integrated spatial and transportation plans, may cause several tensions and frustrations in planning practices. Different participants who operate in CBA processes do not agree on how the analysis should be understood and used, and it appears unclear what role CBA should be allowed to play in infrastructure decision-making processes. As such, we can state that along with the aforementioned technical aspects, there might be process-related issues that cause the controversy surrounding the CBA in transport planning. There is, however, only limited attention given to process-related issues in academic debate Process-related problems of the CBA A report of the European Conference of Ministers of Transport (ECMT, 2004) addresses several process-related problems of CBA, observing that: planning actors blame it for not being transparent; being used too late in the planning process; and being used as a final assessment (or sword of Damocles ) without the possibility of improving the underlying plan or vision. The report stresses that although robust economic appraisal is necessary for infrastructure project development, assessments should not be seen as a blunt yes or no. They should be used instead to draw out issues and propose ways forward. They should also provide a mechanism for drawing stakeholders into a consensus as to the fundamental problems a project is to address, the alternatives available and the solutions preferred (ECMT 2004, p. 8). Haezendonk (2007) also addresses the importance of involving stakeholders in CBA processes, which does not seem to be a natural element in current CBA practices. However, the incorporation of stakeholders in the CBA process could be difficult, because the CBA is based on welfare theory and compensation criteria, whereas each stakeholder has its own set of costs and 62 63

35 benefits (Macharis in: Haezendonck 2007). Savelberg et al. (2008) notice that in Dutch planning practice, planners feel frustrated if a CBA does not give an understandable and recognizable output. This may happen if the plan aims for difficult-to-assess effects like increasing the livability or the economic competitiveness of a region. Mackie (2010) confirms this struggle by pointing at a discrepancy between the views of planners and transport appraisers. Planners want to know how integrated spatial and infrastructural projects influence effects such as induced land-use changes and economic activity, questions which are difficult to answer with a CBA. According to Savelberg et al. (2008), this discrepancy is especially frustrating if the CBA is used as judgment in the decision-making process. The latter seems, however, an issue in itself, as Sager and Ravlum (2005), Martinsen et al. (2010) and Eliasson and Lundberg (2010) show that it remains unpredictable to what extent and in what way CBAs influence decision making if at all. Other political processes seem to be more relevant. Although the abovementioned publications shed some light on process-related issues, it remains unclear what exactly happens during the process of applying CBA in transport planning practices and what possibly causes the process-related issues. What are the issues at play that hinder the CBA process? With our research, we aim to gain more insight into these issues and in a later phase how they can be overcome. The central research question of this chapter is therefore: What process issues do CBA participants perceive in CBAs of integrated spatial transport plans in the Netherlands? To answer this question, two research techniques are combined: focus group sessions and open in-depth interviews. Because there is limited fundamental knowledge on the process-related issues of CBA, an inductive and grounded approach was applied. This means that the research was conducted without formulating theory-based hypotheses in advance. Our theoretical conceptions are directly derived from our data and related to theory as a final step (Bryman 2008, p. 9-13). In the following Paragraph, we describe the research methodology that was used to collect empirical data. In Paragraphs 2.3 and 2.4, the research findings of the focus group sessions and in-depth interviews are presented. After comparing the results of both research techniques, the article continues in Paragraph 2.5 with a reflection on and interpretation of why these issues might develop by examining several underlying dilemmas. Before ending the article with a conclusion and discussion on future directions for research in Paragraph 2.7, the findings and dilemmas are linked to the wider literature on competing rationalities in planning in Paragraph Research methodology To gain more insight into the CBA process and which underlying process issues are perceived by the CBA participants, a two-stage inductive research method was applied, with focus group sessions and open in-depth interviews. In both methods, participants in CBA processes for integrated infrastructure projects were asked to reflect on their experiences with the CBA instrument. Both the focus groups and the interviews followed an open technique. In doing so, we avoided the tunnel vision of limiting the empirical findings to known issues (since there is only limited academic knowledge on them) and allowed for the opportunity to explore latent issues. Moreover, this in-depth method offered the opportunity to explore how issues in the CBA process came into existence and why they were perceived as problematic. In the next paragraphs, we will explain our research design and other methodological choices Trustworthiness The research design reflects the understanding that trustworthiness is important to achieving worthwhile qualitative studies (Lincoln and Guba (1985) in: Bryman, 2008, p. 377); Guba and Lincoln (1994) in: Bryman, 2008, p. 377). Trustworthiness is subdivided in four criteria: credibility, transferability, dependability, and confirmability. Credibility parallels with internal validity, and stresses if the research findings reflect the multiple accounts of the social reality investigated. To meet this criterion, the findings were triangulated by using two methods to answer the research question. These methods showed an overlap in the perceptions of issues in the CBA process, which is a strong signal that the research was able to gain insight into the complexity of the CBA process. Transferability, the second criterion, relates to external validity. However, qualitative findings tend to be oriented to the contextual uniqueness and significance of the aspect of the social world being studied (Bryman, 2007, p. 378). Therefore, qualitative researchers should produce thick descriptions, in order to provide others to make judgments about the possible transferability of the findings to other milieu. This was established by giving a detailed description of the research findings within this chapter and publishing our research findings in international and Dutch planning practice journals (Beukers et al., 2011a; 2011b; 2012a; 2012b). In order to meet the criterion of dependability and leave open the possibility to have the research audited, the interview recordings, transcripts, and codes used in the systematic analysis were precisely documented. The fourth criterion of confirmability, that relates to objec

36 tivity, was met by following an open and grounded research approach which limited the influence of personal values and theoretical inclinations (Bryman, 2008, p. 379). The aim of this research thus was not to find generic truths, but to gain more insight into CBA processes and the process issues as perceived by its actors. Therefore, all of our findings reflect perceptions, which are by definition subjective. Sometimes, these perceptions contradict the formal role of the CBA in Dutch planning practice. Nonetheless, CBA participants perceive these issues in daily practice and may change their behavior and expectations accordingly. This makes these perceptions highly relevant as a study subject. The following Paragraphs will present the analysis and interpretation of the perceptions from the focus group sessions and in depth interviews. Paragraphs 2.2.2, 2.2.3, and will give more insight into the two research methods used and the analysis process Focus group sessions Focus groups are used to get information about the preferences of participants on a specific theme in a relatively short period (Bryman, 2008). Additionally, through the social interaction between the participants, values and norms can be shaped (Morgan, 1997). In 2009 and 2010, three focus group sessions were organized with 7 to 10 planning participants who have dealt with the CBA. The group included civil servants who were involved on a local, regional, and national level in making plans which had to be analyzed in a CBA, professionals that were responsible for guiding the CBA process, and professionals that conducted CBAs. In each focus group session, participants discussed one specific integrated spatial and infrastructural plan which was recently subjected to a CBA. During the sessions, the participants were asked to voice their perceptions on these assessment processes, based on their own experiences with CBAs. These discussions identified different roles in the process of the CBA assessment in transport planning. Based on that, seven roles were defined. They are those that: 1. Make and represent plans to be analyzed in a CBA: Plan owners; 2. Advise in the CBA process on how plans should be made and how the CBA should be applied: CBA advisors ; 3. Apply the CBA: CBA makers ; 4. Test if the CBA is applied correctly according to governmental manuals on the CBA: CBA testers ; 5. Apply for national funding. This role is represented by regional politicians: Funding applicants ; 6. Lobby for or against an analyzed plan: Lobbyists, and; 7. Reflect academically on the CBA: Academics Open in-depth interviews In the second research approach, 22 professionals (of whom one also participated in a focus group session) with experience in CBA assessments of integrated spatial and infrastructure planning were interviewed in depth. The interviews focused on the seven types of CBA participants defined above, which enabled us to distinguish which issues were perceived by which type. An open interview technique was used (Bryman 2008). After clarifying the role of the interviewee in the CBA process, each interview started with one basic question: What is (are) your perception(s) of the CBA process? This question was followed up with clarifying questions of how, why, when, and who. Each interview session lasted about one to one-and-a-half hours. The aim was to bring all (positive and negative) perceptions about the CBA process to the surface, without structuring it too much based on assumptions or hypotheses. Since we aimed to generate hypotheses, the average number of three individuals per role was deemed sufficient. Furthermore, the coherence in the findings (i.e. when no new perceptions emerged in new interviews) showed that theoretical saturation was achieved (Bryman 2008, p. 462) Process of analysis The focus group sessions and interviews were analyzed based on the principles of grounded theory. Again, this technique ensures a minimum influence of assumptions and hypotheses from the researcher (Lewins and Silver 2007). The interviews were analyzed through a systematic process of transcribing, inductive coding, and several rounds of grouping the codes (as illustrated in figure 2.1). Due to practical constraints, the focus groups were analyzed based on detailed reports and coded more loosely per session. The coding of the interviews resulted in more than 1500 unique codes

37 Figure 2.1: From a cloud of codes to a meaningful interpretation of issues in CBA processes: Example from the analysis of interviews for the role of plan makers 2.3 Results of the focus group sessions The three focus group sessions were organized with small groups of 7 to 10 planning participants who had dealt with the CBA and experienced a CBA process. These participants represented the following roles in the CBA process: - Plan makers; - CBA advisors; - CBA makers; - Academics. The sessions focused on perceptions of the CBA process. In each session, the sharing of these perceptions was triggered by discussing a specific integrated spatial and infrastructural plan that was subjected to a CBA assessment. During the focus groups sessions, five major process issues emerged, as summarized in table 2.1. Table 2.1. Summary of CBA process issues expressed by CBA participants in focus groups sessions 1 CBAs are used as a final assessment so that opportunities for learning and optimizing plans are lost 2 CBA processes are perceived as time-consuming 3 CBAs are perceived as rigid or used in a rigid way 4 A perceived unbalanced attention in CBA between monetized effects and non-monetized effects 5 Perception of a lack of communication between plan owners and evaluators resulting in the feeling of the CBA as a black box For each of the CBA participant roles, the codes were grouped into themes. For example, within one role, the theme of CBA input: the use of norms was revealed, into which all codes related to perceptions on this theme were grouped, as illustrated in figure 2.1. Thereafter, the themes were approached with more focus to select only the codes on issues. Following the example in figure 2.1, we selected the codes (20 in this case) which contain perceptions on the issue disagreement on CBA-input: use of norms. Together with the other selections of codes referring to issues (perceptions on process issues related to difficult-toquantify effects or the complexity of calculation), these codes gave insight into the issues related to the theme of CBA input: use of norms. These codes on perceptions of issues were interpreted together, as represented in figure 2.1, step 4. This procedure was done for each group of CBA participant roles, which made it possible to compare the results. This process of analysis made it possible to understand and give meaning to the 1500 codes Perceptions of process-related CBA issues Participants experienced that the national government uses the CBA balance (cost-benefit ratio) to decide whether or not to invest in a certain regional project (i.e. as a final assessment). This perception deviates from the official role of the CBA in Dutch planning, which is to support not to determine the decision making on regional infrastructural projects to be financed by the national government. In the perception of the focus groups, this role is stronger than officially intended and experienced as judgmental. This was noted by the participants as being negative, especially if they were not able or allowed to use the CBA outcome to improve the plan and to learn from the CBA. Although planning practice sometimes shows several rounds of making and assessing plans with the CBA and adjusting the plans, participants explicitly stated that this is unintended and a result of tough negotiations and struggles among bureaucrats and politicians. This was experienced as time-consuming or retarding, as stated by one participant, and tedious by another

38 Another perceived issue is that the CBA was experienced as being used in a rigid way by the so-called evaluators, the professionals who conduct and test the CBA calculations. Some participants experienced that there was little or no possibility of discuss how effects are treated in the calculation and that the evaluators were too strict in what they do. According to those participants, this resulted in the feeling that elements of the plan were left out and that the CBA did not reflect, for example, the synergetic effects of the plan as a whole. Moreover, it was negatively perceived if the evaluators paid more attention to the effects which are more easy to calculate and monetize; i.e. the hard or tangible effects (Meyer and Miller, 2001) such as investment and maintenance costs and travel time benefits. According to some CBA participants, this may lead to a lack of attention on the effects which are not easy to calculate and monetize and sometimes are mentioned only as pro memory items, the soft or intangible effects (Meyer and Miller, 2001) such as decreases or increases in livability or spatial quality. A final and general perception from the focus groups pertained to the struggle to cooperate between involved parties in the CBA process and the sense of not understanding each other. On the one hand, there are the plan owners or visionaries: civil servants, policy makers, and politicians with complex ambitions containing many hard-to-quantify spatial and social elements and political beliefs. On the other hand, there are the evaluators, (mostly) researchers with financial or economic backgrounds and focused on the generation of scientifically sound information. The discrepancy between these two groups is perceived as resulting in a lack of listening to and learning from each other. It was expressed several times during the sessions that in the CBA process, too little time and effort is invested in getting to know one another in order to ease these cooperation and communication problems. Because of this, the CBA is felt as an unreliable and unpredictable black box by the plan owners. 2.4 Results of the open interviews In the second research step, we aimed to gain more in depth insight into the nature of these issues in the CBA process, how they might have come into existence, and why they are perceived as problematic. The following paragraphs will describe and compare the issues that appeared from the interview analysis, articulated by each type of participant Perceptions on issues in the open interviews Table 2.2 summarizes the issues perceived by the seven types of CBA participants. CBA participant Table 2.2: Issues in CBA processes as perceived by the seven types of CBA participants Perceived issues in CBA processes Plan owners - Contradiction between visionaries and evaluators : not understanding each other; - CBA disregards political sensitivities: makes assessment less useful in decision-making; - Ambiguities on CBA input: demands are unclear; - Too little attention in CBA on soft effects leading to mistrust towards CBA and evaluators; - Influence of CBA on decision making is unclear: judgmental or not? CBA advisors - CBA is used too formally/rigidly by the evaluators; - Stakeholders are not structurally involved; - Testing of CBA output is organized ad hoc; - Cooperation between regional and national parties is ambiguous; - Civil servants influence the use of CBA in decision-making, but do not understand it correctly; - CBA professionals conduct, advice, and test each other s CBAs: it is a small world; - Preparation phase of CBA is unclear; - CBA is not always the right instrument. The monetary approach is too narrow; - Too little dialogue on CBA methodology; - CBA is not transparent enough; - The importance of a CBA process is underexposed; - CBA (input and output) is strategically used in decisionmaking process; - CBA is used late in the process to judge plans: leaves out the possibility for optimizing plans; - Numbers receive too much attention in CBA; - CBA disregards political sensitivities which makes it less useful in decision-making process; - Information and advice in decision-making process are too narrow and too black and white. There is too little room for uncertainties

39 CBA makers - Quality of CBA input is not always correct; - Balance of cost-benefit ratio is too dominant in decisionmaking process; - CBA input and output is used strategically; - Too little time is scheduled to assess integrated plans and synergy effects. CBA testers - Plan makers and funding applicants are committed by forehand, regardless of output of the CBA assessment; - Testing criteria are diffuse ; - CBA is not the right instrument to assess every plan. Funding applicants - CBA is wrongly used as judgment; - Information in a CBA is too limited to support decision making. Lobbyists - CBA is too narrow to capture all effects; - CBA input and output is used strategically; - Stakeholder groups are involved selectively by governmental parties. Academics - Debates in hard-to-assess plans, like integrated spatial transport plans, are not balanced; - There is too little room for uncertainties in the decisionmaking process; - Deficits on presentation of CBA output. These are not understandable for non-economists; - CBAs are conducted too late, when plans are already fixed and politicians committed. Opportunities for learning are missed. However, making CBAs earlier in the planning process seems impossible due to the detailed information needed and political sensitivities Differences and similarities in perceptions As table 2.2 shows, it appeared that the CBA advisors perceive the most issues, possibly because they oversee the CBA process as a whole, instead of the more narrowed views of the other participants on a specific phase in the process. Several issues are shared by more types of CBA participants, whereas other issues are contradicting, see table 2.3. The similarities and differences in the perceptions of the CBA participants on these nine themes reveal give a more in depth understanding of the perceived issues and how they may arise. For example, differences and similarities on the theme of insufficient communication are illustrated by the finding that while both CBA makers and testers did not express issues related to communication aspects, plan owners and advisors strongly emphasized this as being an important set of issues. Additionally, the plan owners and CBA advisors criticized the rigid attitude of CBA testers in relation to the communication issue in the CBA process. A similar contradiction appeared around the issue of too little opportunity for discussion on the use of norms and assumptions in the CBA. Again, this problem was mainly emphasized by plan owners and advisors, but not mentioned by CBA makers and testers. Although the CBA makers and testers may be unaware of a lack of communication, the analysis showed that it is perceived that these parties give little effort to stimulate active communication with other participants in the CBA process. Similar differences emerged related to the theme of soft (non-monetary or intangible) effects. Plan owners, advisors, CBA makers, funding applicants, and lobbyists worry about the matter of (a lack of) inclusion of soft effects into the CBA. They fear that soft, intangible effects do not receive the same amount of attention as hard, monetary or tangible effects in the calculation and in the decision-making process. Although CBA makers mentioned this problem and stated that hard effects should not dominate, advisors blamed them for not paying enough attention to non-monetary effects. The notion that the CBA is a black box was mainly expressed by those participants who are not involved in the CBA process on a daily basis: plan owners, advisors, and lobbyists. Black box refers to an opaque instrument, the generation of whose output is unclear for non-daily users. CBA makers recognized that the CBA may be perceived as a black box. This was problematic for them if they had to cooperate with inexperienced plan makers who had to deliver the plan to be analyzed. Perceptions of participants showed much agreement on the possible strategic use of CBA input and output. Strategic use refers to manipulation of the CBA input and selective use of CBA output as well as ignoring CBA output if this output does not correspond with the political preference. It is striking, however, that this issue was not mentioned by funding applicants and hardly at all by plan owners. According to all other participants, these parties tend to behave especially strategically. Apparently, these parties do not experience their own strategic behavior as problematic. According to CBA makers, strategic behavior can be reduced by independently testing CBAs. The CBA makers and testers themselves pointed out that testing of CBAs is done too infrequently, and often in an unstructured, unorganized manner. The timing of the involvement of the testing committee is recalled as especially problematic in current practice, because the committee members are mostly involved only at the end of the CBA process when the assessment is already finalized. Also stated 72 73

40 by CBA testers as problematic was that the testing criteria are diffuse. Their explanation for this diffusion is that the first CBA manuals with testing criteria were written solely for infrastructure plans and not for integrated spatial transport or other complex plans. This broadening of the scope of plans resulted in an increase in the amount of extra manuals and reports on how to conduct a CBA, specified for the more complex plans. The CBA testers stated that this broadening of manuals and criteria make it more difficult to test CBAs in a consistent way. For all roles except the lobbyists, who did not mention this topic, it emerged that it is very difficult (if not impossible) to appraise complex or integrated plans by using CBA. Area development, regional, and land use plans are also mentioned as very difficult to be assessed by a CBA. Participants mentioned that the CBA is not always the optimal instrument to appraise these type of plans and does not always fit the needs of the planning process. Whereas on the one hand the CBA gives too much information, on the other it shows a deficit of information related to what was mentioned as political sensitivities. Still, the CBA is compulsory for integrated plans that require national funding, according to Dutch government regulations. The notion that the CBA is not always the right instrument for analysis of complex plans has a side-effect in that, according to CBA makers and testers, plan owners and funding applicants tend to downplay the CBA and its output. They underestimate the time needed to reach agreement on the exact layout of the plan to be analyzed and other inputs. This downplaying and underestimation of the CBA process appeared to result in allocating too little time and means to this process, and assessing the plan late in the planning process when the plan is already fixed. In contrast to this apparent downplaying, plan makers did stress the importance of a CBA, because they saw a negative CBA balance as a potential risk factor in the decisionmaking process for blocking the plan s realization and aimed for a positive balance. This relates to the perceived use of the CBA as a final assessment. Although CBA testers did not mention an opinion on this theme, the other participants expressed that the CBA should not be used to judge plans. They mentioned that the CBA might be too restricted and does not always give a comprehensive picture of an analyzed plan. Because of the CBAs restrictions, participants stated that it is not possible to weigh pros and cons based solely on the CBA. At the same time, however, it remained unclear for all participants how and to what extent exactly the CBA influences decision making and how it is used in the actual decision-making process. Furthermore, these Table 2.3: Perceived issues in the CBA process experienced by different CBA participants Issues in CBA process 1. Deficient communication among participants: experienced as a conflict between plan owners versus evaluators 2. Fear that hard effects dominate over soft effects in decision making. 3. CBA as a black box: without explanation, the tool is perceived as not understandable. 4. Parties act strategically towards CBA input (composition of the plan) and output to make the plan score according to their desires. 5. The criteria of testing CBA reports are unclear and the testing process is unstructured. 6. Even though CBA may not be suited for each assessment question, CBA is being used because it is compulsory in the planning procedure. 7. There is too little attention for process matters: time pressure; underestimation of need for discussions on assumptions and starting points; CBA too late in planning process. 8 CBA is perceived being used as a final judgment, which is indicated as wrong, because the CBA will never give the whole picture, and it leaves out the possibility for learning. 9. There is too little room for uncertainties and nuance in the decision making process. Perceived by Plan makers; advisors; funding applicants; lobbyists; academics Plan makers; advisors; CBA makers; funding applicants; lobbyists; academics Plan makers; advisors; CBA makers; lobbyists Plan makers; advisors; CBA makers; CBA testers; lobbyists; academics Advisors; CBA makers; CBA testers Plan makers; advisors; CBA testers; funding applicants; lobbyists; academics Advisors; CBA makers; CBA testers; lobbyists Plan makers; advisors; CBA makers; funding applicants; lobbyists, academics Plan makers; advisors; CBA makers; academics participants regret the missed opportunity to use the CBA for learning, for example for optimizing the assessed plan based on the CBA output. However, the academics mentioned that the CBA is often used late in the planning process, when the assessed plans are already fixed, which make a learning use of the CBA hardly possible. Finally, plan makers, advisors, CBA makers, and academics found it problematic that decision-making processes for transport plans demand unrealistically clear and unambiguous information. Participants stated that decision makers simply 74 75

41 want to get advice stating if a plan should be conducted or not, preferably with analytical reports to support that decision. The government seems to be under pressure to deliver the right answers in accordance with the political preference of the decision maker. Because of these attitudes, participants perceived that there is too little room for nuance and uncertainties in the decision-making process. This nuance should be taken into account, because a CBA as well as other kinds of analyses can never give exact answers and should not suggest that it can. 2.5 Interpreting the research findings The focus group sessions and open interviews shed light on issues in the CBA process as perceived by the seven types of CBA participants. Although the difference in information richness yielded by the two techniques is evident, they both pointed in the same directions. The findings of both techniques will be confronted and triangulated in Paragraph Furthermore, the interpretation of the issues as perceived by the different types of participants revealed some dilemmas, which will be discussed in Paragraph Triangulation of the findings The overlap between the findings is evident when the same words are used to formulate the mentioned issue. The use of CBA as a final judgment was, for example, mentioned as an issue during both the focus group sessions and in the interviews, as well as the missed opportunity using the CBA for learning. In the interviews, however, it became clear that other issues are closely related, which sheds some light on why this is considered to be so problematic. For example, the issue that a compulsory CBA is not always the right assessment tool for every plan and research question may increase the problematic experience that CBA is used as final assessment, as well as the perceived emphasis on hard effects that are easier to calculate over the soft effects that are difficult to calculate. The interviews also signaled that the use of CBA as a final assessment may cause strategic behavior by the plan owners. If the CBA is experienced as the last hoop in the process to get the necessary financial resources for the realization of the plan, the plan owners will do their best to jump through it (instead of learning from the entire experience and improving their plans and strategies). This could include presenting the plan as beautifully as possible, maybe even unrealistically, to gain that desired positive CBA balance. The issue of imbalance in the attention given to the calculation of soft effects in comparison to the hard effects could in both the focus groups and interviews also be related to other issues. This issue seemed especially problematic when the CBA gives surprising (low) output, which plan owners do not recognize in their plan. Then plan owners tend to start looking for soft effects like increasing livability, which may be very important to them, and if they were calculated correctly and received enough attention. Often, such effects are then found to be put in the pro memory box, which adds to the frustration. In the occasion of an unexpected and incomprehensible output, plan owners may then even question the entire instrument and start wondering what the CBA actually calculates (without getting clear cut answers). In other words, they then experience the CBA as a black box which is used too rigidly. So, the experience of an imbalance in the attention given to soft and hard effects in CBA also relates to the experience of CBA as a black box and the notion that the instrument is used rigidly. These discussed issues also point at the existence of two distinct groups in the CBA process: the plan owners on the one hand and the evaluators on the other. This distinction relates to the issue of insufficient communication and cooperation between these two groups. The interviews showed that the evaluators find that too little time is dedicated by the plan owners to conduct a CBA, and therefore too little attention is given to the process. In the focus groups sessions, however, the CBA process was typified by the plan owners as being too time consuming. This may be related to their experience of the CBA as an obligation and final assessment, for which plan owners are not inclined to dedicate much time and effort, and which they often start late in the planning process. Nonetheless, neglecting the process elements of the CBA seems to give a contrary result, because of the necessity to discuss CBA input and assumptions used early in the process. If this is not done sufficiently, participants start to question the used input later on, which may cause severe delays. All these issues indicate insufficient communication and cooperation between the participants from opposing directions. In general, the issues that came forward in both the interviews and the focus groups were overlapping, supplementary, and interconnecting. Only two issues mentioned in the interviews were less reflected in the focus group sessions. Namely, 1) the suboptimal functioning of the testing of CBA reports, which was characterized as being unstructured and done too late and 2) the insufficient room for uncertainties in planning and decision making. Besides showing the complexity of the processrelated issues, the findings of both research techniques brought forth four dilemmas

42 2.5.2 Dilemmas that emerged A first dilemma relates to the disagreements on how soft effects should be treated in the CBA and in the CBA process. Some participants stated that as long as soft effects are not monetized, they will not receive the same attention in the assessment and decision making as hard effects that can be monetized. Others, however, expressed the opinion that monetizing soft effects should not be done at all, since this will create false certainty. These opposing opinions on the difficulties of dealing with soft effects in the CBA are expressed randomly among the different roles of participants. It is striking that these apparent fundamental differences related to the issue of soft effects, and indirectly to the appropriate possible solution (putting effort into monetizing soft effects or not), seem to paralyze the discussion on this theme. A second dilemma relates to the first. On the one hand, plan owners and funding applicants expressed their worries about soft effects receiving too little attention vis-à-vis hard effects in the decision-making process. On the other hand, the interview analysis showed that there is hardly room for uncertainties in the decision-making process and exact numbers are favored. These two observations seem difficult to unite, because soft effects can be particularly uncertain and difficult to capture in concrete figures. It seems that from these observations, participants criticize the procedures in which they play a part and which they even help preserve. The third dilemma is formed at the one hand by the aim of the CBA testers to keep the CBA scientifically pure, as expressed in their more or less rigid behavior. On the other hand, participants acknowledged that the CBA is not always the right instrument to fully analyze all types of plans, especially complex spatial infrastructural plans. It might be questioned if it is fair to require a CBA for complex spatial transport plans, as the Dutch government does, even though the instrument is not suited to fully assess these plans. The interviews showed that participants find several ways to deal with this dilemma or inconsistency. However, these ways were pointed out by other participants as issues. In practice, for example, governmental regulations seemed to be not as strict as they appear, and other, lighter versions of a CBA or CBA-like assessments are conducted. Furthermore, participants may act strategically by just fulfilling the formal requirements and trying to adjust the CBA input in such a way that it will score well in CBA terms. Alternatively, they may downplay the CBA output by emphasizing its deficits and rejecting all of its results. Taken together, these reactions have the effect of diminishing the intended role of the CBA to support the decision-making and planning process. A fourth and final dilemma came from the academics perceptions. They stated that the CBA is applied too late in the planning process, when plans are already designed and political positions entrenched. Because of this, it is no longer possible to use the CBA to improve plans or learn from its outcomes, which directs the use of the CBA as solely being a final assessment. Participants, except CBA testers, in the focus groups and the interviews firmly expressed that they disapprove of such judgmental use. Although the tardiness of the CBAs application is problematic, it also seems hard to do it much earlier. The academics perceived that attempts to apply the CBA earlier in the planning process failed because politicians did not want to reveal their positions then. Furthermore, the CBA requires so much detail that plans already have to be thought through comprehensively before a CBA can be made. 2.6 Closing the inductive research cycle: Planning versus economics As a final step, in this Paragraph the inductive research findings are related to theory to enable us to identify relevant patterns. Although it is not possible to identify a simple hierarchy among the several mentioned fundamental issues and dilemmas, the theme of communication deficits and the apparent gap between plan owners (visionaries) and economists (evaluators) forms a core problem in the CBA process. Many perceptions indicate large differences between the worlds of plan owners-andplanning and economists-and-economics. These differences can cause difficulties when these two worlds have to cooperate and (therefore) communicate. As a potential bridge between the two worlds, the CBA is proposed as an economic instrument built on economic logic, with economic terms and methods. A plan, however, is based on plan owners terms and logic, and the two do not match one another. It is also worth noting that plans come into existence in a process in which plan owners operate largely separately from economists (and others), since in an early planning phase (economic) appraisal techniques are minimally used if at all. This results in a situation in which an optimal plan in terms of plan owners logic still has to jump through the economic hoop of the CBA. In this sense, it is not surprising that assessing a plan owners plan through an economic method of appraisal causes friction. A problematic confrontation of different systems thus seem to happen when making a CBA of integrated spatial transport plans. This confrontation is explained by Owens et al. (2004) as a clash between the technical rational approach and 78 79

43 the communicative rational approach. In short (and at the risk of oversimplification), the technical rational approach represents the positivist belief that rational knowledge exists and can be provided by technical instruments (such as the CBA) operated by experts. The communicative rational approach, on the contrary, is fundamentally a reaction to this positivist thinking. Its basic premise is that knowledge is a construction whereby the role of the planning expert is to create a space or place for different actors to communicate and to let inter-subjective knowledge come into existence (see for example Huxley and Yiftachel 2000). Within this dichotomy, the CBA and its prescribed use seems to be an exponent of the technical rational approach, whereas the transport planning domain, especially when including wider regional economic visions, seems to increasingly tend towards communicative rational approaches (Durning 1999; Te Brömmelstroet and Bertolini 2011; Willson 2001). Also, Handy (2008) recognizes that transport planning as a discipline has changed and that this affects the effectiveness of the appraisal methods being used. The goals in transport planning are no longer dictated through predicting and providing. Nowadays, goals for transport planning are far more complex in nature and less shared among different actors. Hence, appraisal methods which are aimed to support the old goals (such as easing congestion) do not automatically fit the demands of the changing planning paradigm (such as enhancing accessibility and sustainability). Te Brömmelstroet (2010), Hatzopoulou and Miller (2009), and Amekudzi et al. (2009) explored the lack of the use of integrated models in planning practice, even though there is a growing need for upto-date models which reflect different priorities like sustainability. As the findings of this study indicate, the distinction between plan owners and economists in the CBA process does not only result in misunderstanding and a lack of communication, but also leads to a situation of mistrust. Apparently, the plan owners presume that economists do not understand plan owners values, like the increasing of livability, as they do, so they fear these values do not get the same attention as those effects that the economists do understand; the participants call this the contrast between soft and hard effects. Also, the experience of CBA as a black box is an expression of mistrust. Plan owners are anxious if the CBA assesses their plan appropriately and this anxiety is increased if they do not exactly understand the calculation methods being used. Furthermore, this mistrust seems to induce plan owners into operating tactically and strategically, which seems to happen more often in transport planning (Naess 2011; Wachs 1989). Moreover, plan owners appear to feel resistance towards the CBA and therefore postpone it and devote little time and attention to the assessment and its necessary processes. On the other hand, the evaluators exhibit suspicious behavior, too, if they act rigidly towards the assessed plan. The consequence of all this seems to be that a crucial opportunity for learning, and thus making better plans, that CBA can in principle offer, is getting lost, and its sole use is as a final assessment. 2.7 Conclusion: mistrust and defective communication This article started with mapping the problematic relation between the growing importance of the CBA instrument and its controversy in Dutch planning practices. It emphasized that CBA literature tends to focus on content-related issues instead of the CBA process and possible related issues. The aim of the research presented in this article was to narrow this knowledge gap on CBA processes and to gain insight into perceptions on process-related issues by CBA participants in Dutch planning practice. Through our inductive research approach, several issues and dilemmas in the CBA process were pointed out, explained, and reflected upon with the help of planning literature. That the CBA and its process are perceived as problematic and characterized as frustrating when applied to assess complex infrastructure plans must not come as a surprise. It appears to be a logical result of clashing values and approaches. This dichotomy helps to explain and understand many of the issues explored: deficient communication among plan owners and evaluators ; the fear that hard effects dominate over soft effects in decision making; the characterization of CBA as a black box; the occurrence of strategic behavior in relation to CBA input and output; the experience of time pressure in the CBA process because of the underestimation of the length of discussions on assumptions and starting points; the use of the CBA as final assessment, too late in the planning process, instead of for learning; and having too little room for uncertainties and nuance in the decision-making process. Moreover, the dichotomy between plan owners and evaluators explains the two problems connecting most of the fundamental issues that were expressed: the mistrust between plan owners and evaluators towards each other and the plan or instrument which they represent and, furthermore, how this leads to a communication deficit and inferior cooperation. Furthermore, the research findings show that awareness of process-related issues and dilemmas is crucial if we want to improve the use of CBAs to assess integrated spatial transport plans. The main challenge from a process perspective is then to narrow the exposed gap between plan owners and economists 80 81

44 and their related frames of thinking and assessment. Moreover, the levels of trust and communication should be increased so that cooperation between participants in the process could be improved, and possibly a use of the CBA for learning instead of solely as a final assessment. The in depth understanding of the CBA process issues discussed in this chapter forms a stepping stone for future research, which should be directed at identifying theoretical solutions for the problems as described and finding out how these theoretical solutions may work in practice

45 3 84

46 CHAPTER 3: Theoretical improvements of CBA process issues Using cost benefit analysis as a learning process: identifying interventions for improving communication and trust 87

47 ABSTRACT Integrated transportation plans require assessment approaches that can adequately support their multidimensional, context-specific needs. The suitability of cost benefit analysis (CBA) for answering this need has been studied in recent research: an analysis of participant perceptions in the Netherlands showed several problematic process issues when assessing integrated transportation plans with CBA (Beukers et al., 2012a). CBA was perceived by the participants as a final test, in contrast to the desired outcome of using CBA as a learning tool to optimize the plans. Furthermore, the two main groups of participants (plan owners and evaluators) appeared to hold different and sometimes clashing rationales. This clash was expressed through lack of communication and mutual trust. Using a literature review of the fields of deliberative planning and organizational learning to explore how to improve communication and build trust, this chapter provides a deeper understanding of the process issues at hand and contends that strong communication and trust between plan owners and evaluators are crucial conditions for employing CBA as a learning tool. Finally, based on these theoretical insights, this chapter proposes an approach for supporting the practical use of CBA as a learning tool

48 3.1 Introduction: Unsuitable assessment Transport planning has a strong and persistent focus on technical and quantitative information (Willson, 2001), which is a problematic characteristic for at least two reasons. First, issues like quality of life and sustainability have become core subjects of public debates about transport plans. Second, and relatedly, transport planning is no longer an isolated profession. Integrated approaches that combine various aspects (e.g., local economic development, social equity, environmental preservation) are becoming increasingly relevant (Bertolini et al., 2005; Banister, 2008; Straatemeier & Bertolini, 2008). This shift has altered the demands posed on assessing transport plans (Handy, 2008; Willson, 2001), requiring multiple dimensions and context specificities to be taken into account (Curtis, 2008; Hull et al., 2011). Commonly used, conventional assessment tools appear ill-suited for the task. The Netherlands is a case in point. In the Netherlands, cost benefit analysis (CBA) is a mandatory tool for assessing transport plans that request funding from the national government (Ministry of transport and water management and Ministry of housing spatial planning and the environment, 2009). However, it has been questioned whether CBA, with its origins in welfare economics, can actually be used to assess the multi-dimensional content of integrated plans (Annema et al., 2007; Jong & Geerlings, 2003; Beukers et al., 2012a; Mackie, 2010; Naess, 2006; Mackie & Preston, 1998). In addition to these content issues, the process of applying CBA for such plans is often hampered by several additional issues rooted in the changing planning context (Sager & Ravlum, 2005; Martinsen et al., 2010; Eliasson & Lundberg, 2010). As an analysis of practitioner perceptions revealed, the CBA was perceived mainly as a final examination of the plan s fulfillment of CBA criteria, as a formality, leading some plan owners to see it as just another obstacle to overcome (Page et al., 2009; Beukers et al., 2012a). This, however, limits the opportunity to utilize CBA assessments to optimize plans. In the process of developing integrated transport plans, it is important to be able to continuously adapt the plan s goals and means based on emerging insights. This learning process is desired by planning practitioners, but not well supported by instruments such as the CBA (ECMT, 2004). Furthermore, the analysis revealed that CBA processes are often characterized by two opposing groups: plan owners on one side and evaluators on the other. A lack of communication and mutual trust between these groups leads to strongly opposing rationales (Beukers et al., 2012a). This article seeks to understand how the aforementioned problems can be overcome, through a literature review addressing the question: What are promising interventions and mechanisms for improving CBA process issues, specifically improving communication and trust between plan owners and evaluators? In Paragraph 3.2, a conceptual framework is first formed to provide an understanding of the relationships between the different process issues. This is the starting point for a review of theories in Paragraph 3.3. In Paragraph 3.4, these theoretical lessons are then synthesized into an approach for improving the CBA process. The chapter closes with some concluding remarks in Paragraph Communication, trust, and the use of CBA as a learning process This Paragraph addresses the importance of communication and mutual trust between plan owners and evaluators for the use of an assessment device as a learning tool in a changing planning context, and how these variables are interrelated. This gives a better understanding of why the lack of communication and trust is a significant impediment to the desired use of CBA in a learning process Communication and trust as conditions for learning from assessments In response to failures of the technical-rational paradigms of planning, more communicative-rational approaches (like communicative or deliberative planning) are being introduced in planning theory and practice (Allmendinger, 2002). Within this development, the central issues of evaluation have also changed. According to Khakee (2003, p. 345), the aim of evaluation has been expanded from measuring all performances of a plan or program to facilitating learning. Evaluation may be seen as a cognitive process in which social realities are constructed and actors develop self-reflective learning abilities, find unexpected meanings in their actions, and build up networks of people, actions, and thoughts (Selicato & Maggio, 2011, p. 173)): Research shows [ ] the need to consider evaluation not as the final outcome of administrative actions, with the aim of approving or rejecting the contents of a given planning tool, but as a work methodology, a process of gradual learning [ ] to increase the awareness of the choices taken, within the context of the decision-making process. Expanding on this point, an increase of stakeholder engagement and openness in decision making is seen as promoting demands 90 91

49 for the intensification of the participatory dimension in assessment processes (Kidd & Fisher, 2007). Likewise, Saarikosi explored the integration of participation into environmental impact assessments (EIAs), not just as a supplement, but as: [ ] A collective process where different actors affected citizens, interest groups, authorities, and experts can deliberate and exchange their views of the goals and their knowledge on the impacts of the proposed developments. (2000, p. 5) In order for EIA to support a learning process, participants should be allowed to discuss the conclusions and collectively seek out mutually agreeable solutions. This articulation of assessment tools as supporting instruments for learning processes showed the need for organizing an inclusive discourse in which those who are involved can explain and share their values, problems, and concerns in an open decision-making process. This relates to how actors communicate, how individual knowledge (tacit and explicit) can be shared and integrated, as well as the importance of mutual trust and a trustworthy environment. The changing planning context therefore calls for reassessment of the application of CBA (as with other assessment methods) with a high level of communication and trust between participants as preconditions for supporting learning. In the remainder of this Paragraph, we will further explore these concepts and the relationships between communication and trust Interrelatedness of communication and trust Communication is a muddy and dynamic concept with many definitions (Littlejohn & Foss, 2008). In this chapter, we will follow Lievrouw & Finn s definition of communication as a: human behavior that facilitates the sharing of meaning and takes place in a particular social context. Any interacting set of social and technical structure which facilitates the sharing of meaning among people is a communication system. (1990, p. 49) Within the CBA process, communication takes place mainly between plan owners and evaluators as a form of interpersonal communication or small group communication. Interpersonal communication relates to interpersonal behavior and relationships (Miller, 1978): at least two actors are involved, there is close physical proximity, many communication channels (modes) are available, and there are optimal conditions for immediate feedback. During this type of dialogue, i.e., face-to-face communication, concepts are built in cooperation with others, providing the opportunity for one s assumptions to be tested (Nonaka, 1994). As with communication, the literature on trust is diverse and entails different perspectives: Economists tend to view trust either as calculative or institutional. Psychologists commonly frame their assessments of trust in terms of attributes of trustors and trustees and focus upon a host of internal cognitions that personal attributes yield. Sociologists often find trust in socially embedded properties among relationships among people and institutions. (Rousseau et al., 1998, p. 393) Nevertheless, the literature agrees that trust is important in several ways: it enables cooperative behavior; it promotes adaptive organizational forms, like network relations; it reduces harmful conflicts; it decreases transaction costs (e.g., a business transaction); it facilitates rapid formation of working groups; and it promotes effective responses to crises (Rousseau et al., 1998). Furthermore, trust is in its essence relational because the interests of one party cannot be achieved without reliance upon another (Rousseau et al., 1998). Gambetta clarified this relationship: A (the trustor) trusts B (the trustee) with regard to X. Trust is A s subjective assessment of the probability that B will act as agreed when B s actions significantly affect A, independently of A s capacity to monitor B s actions. (as cited in Laurian, 2009, p. 371) Whereas the communication literature emphasizes the importance of forming relationships, the trust literature describes how trust is a necessary component for establishing such relationships. The trust literature thereby sees communication as necessary for building trust: through communication, people can get to know each other better and relationships can take shape and become more personal, subsequently increasing the degree of interpersonal trust (Miller, 1978). Interpersonal communication and interpersonal trust are thus interrelated concepts, as recognized by several planning scholars. For Stein & Harper (2003), trust is essential for community, social, political, family, and even linguistic relations; it is a necessary precondition for any kind of communication, cooperation, understanding, knowledge, or learning. Thus, trust is also essential for the work of planners, promoting communicative performance and mutual understanding (Healey, 1999). Laurian also emphasized the interrelatedness of trust and communication: While trust is necessary for open communication and collaboration, open communication and collaboration are also preconditions of trust. [ ] When participants trust each other (even if they hold different values or goals) and trust the fairness of the process, they are more likely to communicate actively, listen empathically, and work toward consensual solutions. (2009, p. 382) 92 93

50 3.2.3 Communication and trust as conditions for the use of CBA as a learning process The literature on evaluation emphasize that assessments should facilitate a learning process in which planning ideas, means, and ends are progressively constructed by participants. Such an approach entails active and engaged participants who communicate with and trust each other. Moreover, the literature explained that communication and trust are intertwined. Combining these insights, we can state that the levels of interpersonal communication and trust need to be increased in order to achieve the participants desired outcome of using CBA as a tool to support learning processes. This forms the basis for the conceptual framework illustrated in figure 3.1. The conceptual framework focuses on communication and trust between plan owners and evaluators, the interrelatedness of these variables, and their relation to the use of CBA as a learning tool. There are, however, also other plausible influences, for instance the power balance between plan owners and evaluators in the CBA process, or the specific timing of the CBA in the planning process (if it happens when planning legislation or policy rules still permit changes of the plan due to a learning process, for example). These alternative explanations, though, are not within the scope of this article. Figure 3.1. Conceptual model on communication and trust conditions for increasing the use of CBA as a learning tool 3.3 Improving communication and building trust between experts Paragraph 3.2 asserted that communication and trust are crucial conditions for using CBA as a learning process. Additional insight is needed on how to fulfill these conditions. The aim of Paragraph 3.3 is to use a literature review to identify the most promising interventions and mechanisms for improving communication and trust between plan owners and evaluators. The literature review thereby concentrates on the research fields of deliberative planning and organizational learning, two fields with rich ideas on improving communication and trust. Although deliberative planning theorists mainly focus on improving communication processes between planners and the community (see Innes & Booher, 2010; Bickerstaff & Walker, 2005; Huxley & Yiftachel, 2000; MacCallum, 2008; McAlister, 2010), their work provides valuable insights into improving communication and trust between multiple experts. The research field of organizational learning also provides valuable insights, as it has developed an understanding of how interactive and shared knowledge within organizations can be created (Nonaka, 1994; Argyris, 2006). Planning, evaluation, and CBA processes can be considered as temporary organizations, which makes the literature on organizational learning relevant (Te Brömmelstroet & Bertolini, 2008; Suárez-Herrera et al., 2009). The literature selection followed two criteria: most frequently cited publications and publications that provided innovative ideas on communication and trust building. In processing these publications, all lessons on how to improve communication and trust between different kinds of experts were first identified and then grouped based on the similarities in their reasoning. In order to preserve the richness of the initial selection, all the directions on communication and trust identified prior to this grouping are presented in Appendix I Communication and trust in deliberative planning Open conversation Much effort has been made by deliberative planning researchers to understand and effectuate Habermas ideas on open conversation among diverse people (through which shared truths and values can be established). Using Habermas social theories for shaping places and forming policy, Healey (1999) argued that a precondition for conversation is the acceptance of a degree of collaboration and reciprocity. This means being open to the opinions of others and accepting that there is no absolute truth, that truth and values are the outcome of social interaction within specific contexts

51 According to Innes (1998), a fruitful communicative process asks participants to not only accept and address formal knowledge, but also make an effort to include other types of knowledge, like their own experiences, personal stories, and intuition. Furthermore, she stated that: [ ] we need appropriate rules, parallel to those of the scientific method, to ensure that the products of these discussions are acceptable and socially worthwhile, as well as properly informed. (Innes, 1998, p. 60) These rules prescribe that the individuals representing all important interests must be at the table; that all must be equally empowered in the discussion; that power differences from other contexts must not influence who can speak or who is listened to; that the discussion must allow all claims and assumptions to be questioned and all constraints to be tested; and, finally, that the group should seek consensus (Innes, 1998). THE PLANNERS COMMUNICATION EXPERIENCES Another seminal description of communication strategies can be found in the work of Forester (1987). He elaborated that planners have to deal with distrust between stakeholders and planners and among the stakeholders themselves. Building trust therefore appeared to be an important element of the planners tasks. They needed to listen carefully and make assurances that the thoughts and feelings of all stakeholders were acknowledged and respected. According to Forester (1987), the planner should not behave arrogantly (as a know-it-all) nor as a neutral party, but rather act as a diplomat. Furthermore, the planner had a role in preparing the stakeholders to face each other and prevent unpleasant surprises by helping the stakeholders formulate their objectives and arguments in preparation for the discussion. In a later study, Forester (1999) stressed that deliberative planning is about acting and learning together, building relationships and finding win-win situations. This entails several behavioral challenges: to be close and keep a distance at the same time; to show empathy and critical judgment; and to recognize and respect rather than dismiss human emotions like anger, fear, impatience, and suspicion. Deliberative action thereby requires a dialogue space, i.e., meetings, negotiations, discussions, project reviews, hearings, and informal meetings that bring affected citizens, regulators, developers, and public officials face to face. Furthermore, communication modes and attributes can also function as a dialogue space, as illustrated by Forester (1999) by the role of using a sketch of a plan: the provision of some boundaries and a clear topic helped the participants to clarify what they meant, to share and sharpen their arguments, and to focus the discussion. Thereby, the essential precondition for using a sketch, plan, evaluation, or other mode to foster a deliberative process is not to present it as the ultimate sketch or plan, but as a work in progress. COLLABORATIVE DIALOGUE Innes & Booher (2003) shared insights for many communication and (to some extent) trust building lessons that can achieve a collaborative dialogue. Basic preconditions include listening to others, treating others with respect, looking for common interests rather than differences, and challenging assumptions. Thereby, parties must begin with sharing their interests rather than their positions, learn about each other, seek win-win solutions, and accept that there is tension between cooperation and competition as well as between advocacy and inquiry in collaborative public policy processes. When stakeholders explain their own situation and needs, they may learn about their interdependence and the need to negotiate. Furthermore, participants also need to interact with one another, for example in brainstorming or scenario building. Scientists and agency staff need to be engaged with lay people who can challenge assumptions and analyses by using their local knowledge. To achieve this collaborative dialogue, a professional facilitator plays a critical role in ensuring that the group members make a shared analysis of interests and conflicts, do joint factfinding, address issues deeply, and feel comfortable and safe in sharing their thoughts. Moreover, participants have to be stimulated to think outside of the box as well as to be willing to put forward half-baked ideas (Innes & Booher, 2003, p. 46). A facilitator can stimulate this by asking for clarifications or examples when needed, or by challenging assumptions and the status quo. SPATIAL STRATEGY MAKING Healey has developed several ideas on communication and trust building between multiple experts related to ideas of spatial strategy making. Spatial strategy making happens in social construction sites or arenas, where what is considered significant and possible is explored, conceptualized, symbolized, and tested in various ways (Healey, 2007, p. 236). This asks for a specific approach to the production of knowledge and understandings that should be shaped by situations, trajectories, activities, and values of particular social groups (Healey, 2007, p. 243). Spatial strategic ideas need to be validated and legitimized in an interactive way through social encounters, discussion, debate, and exchanges of thoughts (Healey, 2009, p. 452). This is espe

52 cially relevant when different stakeholders with multiple frames of reference, rationales, and values are involved. Besides recognizing the value of multiple sources of knowledge, it is important to maintain an open-minded stance, i.e., actively seek out multiple perspectives, challenge established assumptions, and cultivate debate among different viewpoints. Healey (2009, p. 453) formulated guiding dimensions that characterize the process steps in transformative spatial strategy making: mobilizing attention, scoping the situation, enlarging intelligence, and creating frames and selecting actions. In mobilizing attention, the aim is to reorient attention to issues behind immediate agendas, highlighting neglected opportunities and challenges. When scoping the situation, the aim is to identify where the energy for change exists and to build coalitions for change to expand this energy. In enlarging intelligence, the focus is to access multiple sources of knowledge to explore and recast agendas of problems, issues, and potential actions and stakes. Creating frames and selecting actions imply articulating strategic ideas, within which specific issues and actions can be prioritized and given justification and coherence. PLANNING AND TRUST Despite the rising interest in trust issues, this topic remains underrepresented in the field of planning (Stein & Harper, 2003). This is somewhat paradoxical, as the importance of trust has increased with the communicative and collaborative turn in planning. As noted by Ehrman & Stinson (1999), while consensus does not require common values, it does require the existence of trust among stakeholders, agencies, and mediators. However, Laurian (2009) gave several directions for building trust, for example through face-to-face interactions and sharing decision-making power. Through sharing decisionmaking power, an agency (like a planner) displays trustworthy behavior and can reduce the distance between experts and non-experts. Information sharing, deliberation, and democratic governance geared towards enabling shared decision-making thus provide support to the emergence of trust, and planners can play an integral role in this process (Laurian, 2009, p. 375; Switzer et al., 2013). Another way to facilitate trust (or to mitigate distrust) is to use mediators to support open and effective communication. Moreover, Laurian emphasized that Habermas ideal speech conditions for undistorted communication also support trust, namely through comprehensibility, truthfulness, sincerity, legitimacy, equal standing, and respect for all forms of knowledge, inclusiveness, openness and transparency, mutual disclosure, and information sharing. DIRECTIONS FOR IMPROVING COMMUNICATION IN DELIBERATIVE PLANNING The cited deliberative planning literature shows that there are plenty of ideas on how to overcome problems related to communication and trust. These lessons on building communication and trust are grouped in Appendix I and summarized in Table 3.1 and Table 3.2. Table 3.1 Directions for improving communication in deliberative planning 1. Have an open approach to all knowledge: a. Exchange different kinds of knowledge, feelings, and interpretations b. Allow all knowledge in the discussion and allow it to be questioned c. Give attention to issues behind immediate agendas 2. Behave appropriately: a. Have an open perspective towards each other b. Have an open attitude to changing one s own assumptions and understandings c. Behave actively by asking questions and listening carefully 3. Have interaction: a. Create shared and embedded knowledge and understanding b. Have all stakeholders involved, empowered, and engaged in interaction c. Act and learn together to develop close relationships d. Focus on shared interests and win-win solutions 4. Have a supportive, communicative context: a. Create a safe environment for participants to speak their mind b. Prepare stakeholders for a constructive discussion and formulate arguments beforehand c. Create a dialogue space d. Use familiar dialogue modes e. Use a mediator 98 99

53 Table 3.2 Directions for building trust in deliberative planning 1. Show trustworthy behavior: a. Do not behave arrogantly b. Fulfill promises c. Listen carefully and focus attention first on the speaker and then on the words d. Share information and decision-making power e. Communicate comprehensively, truthfully, sincerely, supportively, and legitimately 2. Have interaction: a. Acknowledge and respect all knowledge, thoughts, and feelings b. Build relationships c. Use a dialogue space d. Use a mediator 3. Prepare participants to take part in the discussion: a. Prevent confrontation and unpleasant surprises b. Formulate arguments beforehand c. Mention objectives from all stakeholders Communication and trust in organizational learning Organizational learning focuses on finding ways to facilitate the fragile transmission of knowledge between individuals within an organization. According to Akgün et al. (2003), knowledge develops from and is manifested by a complex web of relationships and social activities among people for which culture, communication, and group activities in organizations are of utmost importance. KNOWLEDGE CREATION Nonaka & Takeuchi (1995, in: Nonaka et al., 2006) formulated four stages for knowledge creation, i.e., the transformation of knowledge from being to becoming in an interactive process. In such knowledge conversion, personal knowledge is validated, connected to, and synthesized with the knowledge of others (Nonaka et al., 2006; Nonaka et al., 2000). This process evolves in four stages socialization, externalization, combination, and internalization (SECI) which link and transform tacit knowledge (i.e., personal, subjective, experiential) and explicit knowledge (i.e., universal, objective, codified). Socialization aims to share tacit knowledge among individuals; externalization aims to articulate tacit knowledge into explicit concepts; combination aims to combine different entities of explicit knowledge; and internalization aims to embody explicit knowledge into tacit knowledge. This process of SECI is related to the concept of ba: a shared space for emerging relationships as the environment for knowledge creation, requiring the sharing of individual experiences as well as reflections on the experiences of others (Nonaka & Konno, 1998). The concept of ba knows different stages that follow the different steps in the SECI process. The first stage of ba is meeting face-to-face and sharing emotions, feelings, experiences, and mental models. This represents the socialization of individuals. This interactive ba supports externalization and combination, and through dialogue the mental models and skills are probed, analyzed, and converted into common terms and concepts. Then, in the next stage of ba, the individual s internalization of explicit knowledge is supported, for example through training with instructors or colleagues or with repetitive exercises to engrain specific behavior (Nonaka et al., 2006; Nonaka et al., 2000). DOUBLE-LOOP LEARNING Also central to organizational learning are insights on doubleloop learning, which explain why it is difficult for organizations to change internal structures and truly solve problems (and cover them up instead). To detect and correct errors in an organization, double-loop learning publicly questions underlying policies and goals, assumptions, norms, and objectives (Argyris, 1977, 1991). A critical factor is creating awareness of existing positions. To change individual assumptions, people need to become aware of their internal maps and see that some assumptions are counterproductive to learning. Moreover, it is necessary to produce new assumptions by combining articulated views with questions posed by others. All participants must acknowledge their own mistakes, recognize defensive behavior, and engage in open dialogue. Workshops and seminars are useful for stimulating this change, i.e., moving from old assumptions to new and shared assumptions. In such learning processes, people should advocate their views in a way that invites confrontation and challenges to existing positions as well as public assessment of these positions (Argyris, 1977). REFLECTION-IN-ACTION According to Schön (1983), it is essential for professionals to reflect on their thoughts during a learning process, because thoughts influence actions and vice versa. Past experiences from different and similar situations are used to guide actions, in addition to learned knowledge (see also Flyvbjerg [2001] on how professionals reflect in action). This is phrased by Schön as tacit knowing or reflecting-in-action (1983, p. 54). However,

54 the actor may not be aware of this reflecting-in-action and may even ignore it. Schön gave several directions for achieving this reflecting-in-action (1983, pp ), stating that it is necessary to give and get valid information and speak in directly observable categories, thus providing information, data, reports, and speech that are open to disconfirmation. Furthermore, Schön underlined the urgency in the reflective process to create awareness of the values at stake and of the limits of one s capacities. There is also the need to make designing and managing a multilateral task and involve several parties to work towards freedom of choice and internal commitment. Also, a protected environment is needed to ensure that no one withholds negative information or prevents testing this information and assumptions publicly. Practitioners use various modes to reflect on their action, such as media, languages, conducting experiments, and repertoires (theories, role playing, storytelling) to describe reality. Independent of the modes used in reflection-in-action, it is important that different practitioners are familiar with the modes they use when working together. Furthermore, reflection-in-action requires a social context that supports reflective actions as well as cooperation with other reflective practitioners or clients. For example, the classical professional-client relationship, where the professional has all the authority and the client submits to it, is not seen as productive. In reflecting-in-action, the client participates in a reflective conversation with the professional, thus providing valuable input. This asks the expert to not act arrogantly and to encourage the clients to share their thoughts and feelings. It asks for a different attitude by the client, not to passively expect the expert to solve the client s problem but rather to engage in a dialogue, ask questions, and question the expert s knowledge, all without hostility (Schön, 1983, p. 300). ORGANIZATIONAL LEARNING AND TRUST Many studies in organizational learning focus specifically on trust: how it relates to innovations, contracts, competitiveness, institutions, teamwork, group performances, or cooperation (Peters & Karren, 2009; Jones & George, 1998; Bachmann & Inkpen, 2011; Erhardt, 2011), and how to build trust in an organizational context (Six & Sorge, 2008; Mayer et al., 1995; Abrams et al., 2003; Lander et al., 2004; Edelenbos & Klijn, 2007). Edelenbos & Klijn (2007) explained that trust is generated in social interaction and is influenced by existing codes and institutional rules, and vice versa. Trust develops especially in embedded relationships, through stable, frequent, and tight interactions. They emphasized that trust is fragile and needs to be nurtured and maintained following three key guidelines: create reciprocity in the relationships and repeated interaction; stabilize and manage interactions, for example through mediators and facilitators; and design process rules to frame risk and opportunistic behavior, to decide on what to do when in conflict, how benefits are distributed, and what to do if one of the involved actors wants to change the relationship. Abrams et al. (2003) espoused a more abstract view on building trust. When engaged in a network for sharing information and creating knowledge, they noticed benevolence and competence as two important dimensions of trust. Benevolence is related to the experiences of caring and being interested in the well-being of others and their goals. Competence relates to the notion that one has relevant expertise. Abrams et al. highlighted the following factors that promote trustworthiness (2003, p. 68): act with discretion so that people feel safe in sharing confidential information; be consistent in words and deeds; ensure frequent and rich communication on a personal as well as on a professional level; engage in collaborative communication in which both sides feel free to share and really listen to each other s thoughts and ideas; ensure that decisions are fair and transparent; establish and ensure a shared vision and language; and disclose your expertise and limitations by being open about strengths and weaknesses. Another variation of trust building factors is given by Lander et al. (2004, p. 512), asserting that trust building was influenced by repeated and positive interactions, integrity (being forthright and truthful in interactions and fulfilling promises), and perceived reputation. Moreover, communication was to them the most relevant factor in building trust. It enabled the sharing of relevant information and knowledge, the provision of timely feedback, the creation of a common language, the creation of a shared vision, and explanations for decisions. Besides communication, Lander et al. (2004) noticed that sharing control and responsibility was perceived as an act of trust. DIRECTIONS FOR IMPROVING COMMUNICATION AND TRUST IN ORGANIZATIONAL LEARNING The cited literature from organizational learning showed a wide range of (partly overlapping) views on and approaches to communication and trust. Table 3.3 and Table 3.4 summarize these views. A more detailed overview can be found in Appendix I

55 1. Interpersonal behavior: Table 3.3 Directions for improving communication in organizational learning a. Have an open attitude towards testing assumptions b. Have an open attitude towards learning, including from risky ideas c. Be explicit and accurate 2. Interaction: a. Share and combine tacit and explicit knowledge b. Define means and ends together and interactively c. Ensure that all actors meet, discuss, and become engaged in a knowledge creation process d. Use various conversation modes that are familiar to all practitioners, such as sketches 3. Organizational context: a. Adopt a leadership style open to transformation and sharing of decision-making b. Establish a supportive structure through stability of team members, learning procedures, training c. Create a safe environment and cultivate emerging relationships Table 3.4 Directions for building trust in organizational learning 1. Personal awareness of others and trustworthy behavior: a. Be aware of preconceived positions and be open to discussing them b. Have trustworthy behavior: act sincerely, act discretely, really listen, be honest about expertise and limitations, give compliments, address troublemakers directly c. Fulfill promises 2. Interaction and forming of relationships: a. Collaboratively create knowledge through communication b. Have stable, frequent, and informal interactions to form relationships c. Form a shared language or vision 3. Trustworthy environment: a. Share control b. Make fair and transparent decisions c. Make rules to frame risks, opportunistic behavior, and methods of dealing with conflicts 3.4 An applicable approach for CBA processes The literature review addressed comprehensive ideas on how to improve communication and build up trust between different types of experts. The review found that deliberative planning and organizational learning share many insights and confirmed that communication and trust are interrelated concepts asking for similar actions and attitudes. Although these communication and trust building insights are mostly quite applicable, it remains a challenge to implement them in CBA practice and to explore how they can improve the levels of communication and trust in CBA processes. With this objective in mind, we have structured and translated the identified insights into an applicable approach for CBA processes with the help of the CIMO framework (context, interventions, mechanisms, and outcome), introduced by Denyer et al. (2008). The CIMO framework states that within a specific context (C), interventions (I) can be used to trigger the necessary mechanisms (M) in order to get a desired outcome (O). The interventions thereby inform which concrete actions are needed to trigger the mechanisms, whereas the mechanisms reveal how something works and how the different elements function in relation to each other. Following the reasoning of the CIMO framework, it is expected that within the context of a CBA process, applying the right communication and trust interventions will trigger the necessary communicative and trust mechanisms that will lead to the desired outcome of improved communication and trust between plan owners and evaluators. This reasoning resulted in a synthesis of the insights from the literature review into five communication-improving and trust-building interventions with related mechanisms Interventions and mechanisms for improving communication and building trust The first intervention advises the creation of a dialogue space where plan owners (those whose plan is going to be assessed in a CBA) and evaluators (conducting the CBA) meet face to face. This triggers the mechanisms by which plan owners and evaluators have the opportunity to get to know each other and build relationships, which enables them to better assess how the other will react in future interactions, thus building interpersonal trust. They may feel safer and may share more detailed information and listen more carefully. For example, plan owners may share doubts about assumptions used in the CBA and the evaluators might pay more attention to difficult-to-measure effects that are nonetheless important to plan owners. Then they may pay more attention to issues behind immediate agendas, ask and receive

56 critical questions, and collectively change and build assumptions. This, for example, could help to crystalize the underlying problem analysis of the plan at stake: What is the actual problem (or opportunity) and how does the plan resolve (or fulfill) this? The second intervention is to share and discuss the plan and CBA simultaneously and together. This triggers the mechanisms by which plan owners and evaluators can better understand each other s standpoints and see their shared interests, increasing opportunities for finding win-win solutions. They will know more information about each other (the plan and the CBA), which increases the predictability of how they might respond and therefore increases the interpersonal trust. This helps to create a more open and critical interaction, sharing even more knowledge and possibly changing one s own perspective based on the perspectives of others. This could help to decide if the plan is, for example, the most cost-effective, or if other solutions (such as a different phasing in time of the benefits and expenses) are possibly better. It could also help to understand if the CBA addressed the right effects and provides the information required to make a decision. The third intervention is to be prepared to share and discuss knowledge together. This triggers the mechanisms by which plan owners and evaluators do not feel unpleasantly surprised or attacked by new arguments, are better able to make their own standpoints and reasoning explicit, and know how to incorporate or internalize outside critiques without acting defensively. This preparation helps to make explicit what the plan is actually about and how different effects could be assessed in a CBA. Furthermore, plan owners and evaluators may feel supported by the management or political environment to act in a spirit of open communication and trust and will be more likely to act along these lines if they receive training. The fourth intervention is to have the interaction guided by a moderator. This triggers the mechanisms by which all can feel safe to speak freely, all types of knowledge are considered and included in the discussion, and the focus of the discussion is on finding shared interests, win-win solutions, and issues behind immediate agendas. As such, the different types of knowledge possessed by plan owners (such as planning concepts and qualitative visions) and evaluators (such as the reasoning behind the CBA and used assumptions) alike get equal attention in the discussion. Furthermore, when plan owners and evaluators feel guided and protected by the moderator, they may feel encouraged to share more detailed and diverse knowledge, to be more open towards each other, and to give and receive constructive criticism without responding defensively. The fifth intervention is to use dialogue modes (e.g., sketches, simulations, storyboards, role play) presented as work in progress. This triggers the mechanisms by which plan owners and evaluators may feel encouraged to share their standpoints, to illustrate their arguments, and to make them context-specific and explicit. This sharing will give the others the opportunity to respond, illustrate, contextualize, and explicate their points of views. Moreover, it may help to overcome the knowledge differences between plan owners and evaluators. To sum up and fill in the CIMO structure as illustrated in table 3.5, the following interventions trigger the mechanisms by which a CBA can possibly be used as a tool to support learning in the context of a CBA process in which plan owners and evaluators participate: (1) creating a dialogue space where plan owners and evaluators meet face-to-face, (2) sharing and discussing the plan and CBA together, (3) being prepared to share and discuss knowledge together, (4) having the interaction guided by a moderator, and (5) using dialogue modes (sketches, simulations, storyboards, role play) presented as work in progress. So, the CBA process should be organized as a dialogue between plan owners and evaluators, the plan and CBA should be discussed when both are still open for input, plan owners and evaluators should prepare themselves for the discussion, and the discussion should be guided by a moderator and tools such as sketches or maps. 3.5 Conclusion This article presented an investigation into deliberative planning and organizational learning theory in order to address some of the process issues of CBA (i.e., low levels of interpersonal trust and communication and the use of CBA as a final assessment) and to increase its suitability for assessing integrated transport plans. First, a conceptual model was formed to provide an understanding of the relationships between the process issues. The model showed that communication and trust between the plan owner and evaluator are crucial conditions for the use of CBA as a learning tool. Secondly, the literature review provided more insight into how to improve these conditions. The research fields of deliberative planning and organizational learning were explored for ideas on this. These insights were synthesized in five concrete interventions that are expected to trigger the right mechanisms for improving communication and building up trust between participating plan owners and evaluators in CBA processes in order to use the CBA to support learning

57 Table 3.5 Interventions and related mechanisms for improving communication and building trust in a CBA process Context: In CBA processes in which plan owners and evaluators participate Intervention 1. creating a dialogue space where plan owners and evaluators meet face-to-face Triggered mechanisms: Plan owners and evaluators have the opportunity to get to know each other and build relationships, which enables them to better assess how the other will react in future interactions, thus building interpersonal trust. They may feel safer and may share more detailed information and listen more carefully. Then they may pay more attention to issues behind immediate agendas, ask and receive critical questions, and collectively change and build assumptions. Intervention 2. sharing and discussing the plan and CBA together Triggered mechanisms: Plan owners and evaluators better understand each other s standpoints and may see shared interests, increasing opportunities for finding win-win solutions. They will know more information about each other (the plan and the CBA), which increases the predictability of how they might respond and therefore increases the interpersonal trust. The development continues towards more open and critical interaction, sharing even more knowledge, and possibly changing one s own perspective based on the perspectives of others. Intervention 3. being prepared to share and discuss knowledge together Triggered mechanisms: Plan owners and evaluators do not feel unpleasantly surprised or attacked by new arguments, are better able to make their own standpoints and reasoning explicit, and know how to incorporate or internalize outside critique without acting defensively. Furthermore, plan owners and evaluators may feel supported by the management or political environment to act in a spirit of open communication and trust and will be more likely to act along these lines especially if they receive training. Intervention 4. having the interaction guided by a moderator Triggered mechanisms: The moderator makes sure that all feel safe to speak freely, all types of knowledge are considered and included in the discussion, and the focus of the discussion is on finding shared interests, win-win solutions, and issues behind immediate agendas. When plan owners and evaluators feel guided and protected by the moderator, they may feel encouraged to share more detailed and diverse knowledge, to be more open towards each other, and to give and receive constructive criticism without responding defensively. Intervention 5. using dialogue modes (sketches, simulation, story board, role play): presented as work in progress Triggered mechanisms: Plan owners and evaluators may feel encouraged to give their standpoints, to illustrate their arguments, and to make them context specific and explicit. This sharing will give the others the opportunity to respond, illustrate, contextualize, and explicate their points of views. Outcome leads to CBA being used as a tool to support learning Through this review, we answered our research question: What are promising interventions and mechanisms for improving CBA process issues, specifically improving communication and trust between plan owners and evaluators? It turned out that scholars from deliberative planning and organizational learning agree largely on desired actions and behavior for both improving communication and building trust. As framed in the five interventions, it is, in short, necessary for plan owners and evaluators to meet, to discuss the plan and CBA together and simultaneously when these are still open for external input, to be prepared for such a meeting, and have it guided by a moderator and a discussion tool like a sketch. It is our expectation that applying these interventions in a CBA process will stimulate the participating plan owners and evaluators to improve their interpersonal communication and trust so that they can use CBA more as a learning tool. Moreover, we expect that this will increase the suitability of CBA when assessing integrated transport plans. However, although these interventions might seem selfevident, they are not common practice in CBA processes (ECMT, 2004; Beukers et al., 2012a). Furthermore, and related to this, while they are theoretically grounded, they have not yet been tested in the context of their intended use, which is an essential component of research aiming at improving an existing practice (Straatemeier et al. 2010). It is thus still unclear if they give the desired outcome of improving communication and building trust when applied in a CBA process, if some interventions and mechanisms are more relevant than others, or if there are other influencing conditions. A next research step would therefore be to test and analyze these expectations and uncertainties in applied CBA processes, in order to develop a richer understanding of which interventions and mechanisms are successful under which conditions. Such testing and analysis should be done in both context poor, control rich environments (i.e., controlled experiments) and in context rich, control poor environments (i.e., in-depth case studies). This dual approach would help both isolate the impact of the identified interventions and mechanisms (i.e., ensure internal validity of the findings) and verify their relevance in actual practice (i.e., ensuring ecological validity) (Te Brömmelstroet, 2010c)

58 4 110

59 CHAPTER 4: Testing in an experiential case study An assessment of interventions for improving communication and trust in Cost Benefit Analysis processes 113

60 ABSTRACT Evaluation literature suggests that assessments of integrated transport plans should be an inclusive dialogue, for which it is crucial that participants communicate with and trust each other. However, Cost Benefit Analysis (CBAs) of integrated transport plans is often characterized by communication deficits and distrust among plan owners and evaluators. A literature review suggested five communication and trust building interventions and related mechanisms that might improve this. In this chapter, we have tested the efficacy of these five communication and trust building interventions by applying them in an experiential study with two sequential cases, representing close to real situations. The research aimed to develop field-tested knowledge to address the aforementioned class of CBA process problems. The research demonstrated how the five interventions could facilitate an exchange of information, knowledge, and experiences, which according to the participants will increase the effectiveness of the CBA. Furthermore, it illustrated that a communication and trust building strategy such as the one tested might be a useful complement to CBA practices, if adapted to the characteristics of the specific assessment process and planning context

61 4.1 Introduction A problematic use of CBAs Cost Benefit Analysis (CBA), grounded in welfare economics, is a frequently used method for assessing transport plans (Annema et al., 2007; Odgaar, 2005; Vickerman 2007; ECTM, 2004). However, significant doubts have been raised whether CBA is usable in integrated transport planning contexts (Annema et al., 2007; Jong & Geerlings, 2003; Beukers et al., 2012a; Mackie, 2010; Naess, 2006; Mackie & Preston, 1998). Especially complex transport land use plans that aim to influence fundamental but hard to grasp aspects of societal challenges like sustainability or quality of life are difficult to fit into a CBA logic (Ziller & Phibbs, 2003; Vaughan et al., 2000; van Wee & Molin, 2012). There are a number of content-related problems that hamper assessment in CBA of such plans: disputable calculation methods for translating soft variables like quality of nature into money (Mackie & Preston, 1998; Ziller & Phibbs, 2003; Vaughan et al., 2000), missing information about winners and losers and equity issues (Ackerman and Heinzerling, 2002), and misrepresentation of long-term and irreversible effects (van Wee, 2011), to name a few (see also Mouter et al., 2013 for an overview of CBA related problems). In addition, the process of applying CBA itself is often highly problematic ( Beukers et al., 2012a; Damart & Roy, 2009; Sager & Ravlum, 2005; Martinsen et al., 2010; Eliasson & Lundberg, 2010; Veisten et al., 2010). Several studies on planning practitioners perceptions indicate that CBAs of integrated transport plans are mainly perceived as a final judgment of a plan s fulfillment of CBA criteria, leading some planners to see it as just another obstacle to overcome instead of an opportunity for learning and for improving the plan as desired by many practitioners (Page et al., 2009; Beukers et al., 2012a; ECTM, 2004). Furthermore, these processes are often characterized by severe communication deficits and distrust between the involved plan owners and evaluators, who tend to act in opposition to each other ( Beukers et al., 2012a). This study, for example, found that plan owners fear a dominance of easier-to-calculate hard effects over difficult-to-calculate soft effects in the decision making, and perceive the CBA as not understandable. On the other hand, evaluators perceived that plan owners use CBAs strategically by embellishing their plans. (See also Morrison-Saunders et al., 2014 on communication difficulties across disciplines with integrated impact assessments) Assessing integrated transport plans This problematic use of CBA for assessing integrated transport plans seems rooted in the changing transport planning context. Transport plans have become ever more intertwined with economic, social, environmental, and spatial issues. Integrated approaches to transport planning therefore have become increasingly relevant (Banister, 2008; Straatemeier & Bertolini, 2008). This has an effect on how transport plans should be assessed (Handy, 2008; Willson, 2001; Curtis, 2008; Hull et al., 2011). First, issues like quality of life and sustainability have become core subjects of transport planning (Banister, 2008). However, the impact of plans on these issues is hard to measure or to translate into monetary terms (Mackie and Preston, 1998). Moreover, these issues are context- and location-specific (Allmendinger, 2002). This means that local knowledge and experience, often brought in by local experts and lay people, is essential for an accepted assessment (Runhaar and Driessen, 2007; Stoeglehner et al., 2009). Second, and relatedly, transport planning is no longer an isolated profession; it continuously interacts with planning processes and professionals from other domains (e.g., land use, environment, health) (Bertolini et al., 2008). Third, interaction among stakeholders often takes place at early, more strategic, phases of the planning process, which indicates that the supporting instruments used need to be sensitive to different dimensions, perspectives, goals, and professional languages (Owens et al. 2004; Handy, 2008; Healey, 2009). Because of this new context, assessments of integrated transport plans should be an inclusive dialogue in which social realities are constructed and networks of people, actions, and thoughts are built, rather than just providing objective and technical rational analyses (Khakee, 2003; Selicato & Maggio, 2011). These assessments should ideally take place in an ongoing creative process of gradual learning between plan owners and evaluators, and between plans and analyses. Thus, it is crucial that actors communicate with and trust each other, actively engage and appreciate each other, recognize entrenched positions and struggles, and gain insight into their strong and weak points (Kidd & Fisher, 2007; Saarikosi, 2000; van Buuren & Nooteboom, 2009). At the same time, the limits of this deliberative planning due to, for instance, the influence of power imbalances should be also recognized (Huxley 2000). However, CBA processes are not characterized as such. On the contrary, as mentioned, practitioners often perceive CBAs as a final judgment rather than a learning process, and plan owners and evaluators experience communication deficits and distrust

62 4.1.3 Communication and trust as conditions for using the CBA for learning The preceding discussion suggests that a different approach to CBAs when assessing integrated transport plans might be necessary, whereby high levels of communication and trust between plan owners and evaluators could form crucial conditions for using CBA for learning, see Figure 4.1. A review of communicative planning and organizational learning literature followed this line of reasoning and structured existing insights on how to improve the levels of communication and trust between different types of experts, like plan owners and evaluators in CBA processes (Beukers et al., 2014). This review concluded with five interventions and related mechanisms to increase the levels of both trust and communication (see Table 4.1). The interventions thereby represent the action and the mechanisms explain how and why related communication and trust effects are expected to happen. Figure 4.1. Conceptual model on communication and trust conditions for a learning use of CBA (Beukers et al. 2014) testing them in the context of their intended use is fundamental (Van Aken, 2004). Furthermore, by focusing on both the if (does it work?) as well as the how (how does it work, or not work?), testing also allows for improvements to be made in the fit of the interventions to the specific characteristics of CBA processes (Straatemeier et al., 2010). Therefore, we aim to answer the following questions: Do the interventions derived from the theoretical literature improve communication and trust between plan owners and evaluators in CBA processes, and through which mechanisms? The research, thus, aims to develop field-tested abstract knowledge to solve a class of CBA process problems (communication and trust issues between plan owners and evaluators). Such knowledge should not be interpreted as a prescription or recipe, but as a design example of grounded technological rules to address these problems (Van Aken, 2004). Table 1 gives an overview of the tested interventions and related mechanisms. In short, the interventions ask for plan owners and evaluators to (1) meet, (2) share and discuss the plan and CBA, (3) be prepared for the interaction, (4) have the interaction guided by a moderator, and (5) use dialogue modes, presenting the work as work in progress. The fifth intervention thereby calls for interaction early in the planning process, when the plan and assessment are still open for discussion (see also Runhaar and Driessen, 2007) Aim and setup of this article The literature review suggested an array of interventions and related mechanisms that are argued to improve communication and trust between experts, like plan owners and evaluators. However, their actual effectiveness in practice has not been tested, let alone in the context of CBAs of integrated transport plans. To improve the academic rigor and practical relevance of such theory-derived interventions and related mechanisms,

63 Table 4.1. Interventions and related mechanisms for improving communication and building trust between plan owners and evaluators in CBA processes (Beukers et al., 2014). Theory-derived communication and trust building interventions 1. Organizing a dialogue space where plan owners and evaluators meet face-to-face (Argyris, 1977; Forester, 1999; Innes, 1998; Nonaka & Konno, 1998; Akgün et al., 2003; Lander et al., 2004; Healey, 2007; Laurian, 2009) 2. Sharing and discussing [the plan and CBA] together and simultaneously (Schön, 1983; Healey, 1999, 2007, 2009; Argyris, 1977; Argyris, 2006; Nonaka et al., 2006; Nonaka et al., 2000; Innes & Booher, 2003; Laurian, 2009) Related communication and trust building mechanisms 1. Sharpening assumptions based on the input of others 2. Receiving critical questions 3. Asking critical questions 4. Giving attention to issues behind immediate agendas 5. Listening carefully 6. Sharing detailed and diverse information 7. Being open in the discussion 8. Feeling safe 9. Estimating how others will respond in the discussion 10. Strengthening relationships with others 11. Getting to know others better 12. Knowing more about the plan as well as the CBA 13. Seeing opportunities for win-win solutions 14. Recognizing shared interests 15. Understanding the viewpoints of others and vice-versa 4. Having the interaction guided by a moderator (Innes and Booher, 2003; Laurian, 2009; Edelenbos & Klijn, 2007) 5. Using dialogue modes (e.g., sketches, simulations, storyboards, role play); presenting the work as work in progress (Schön, 1983; Forester, 1999) 20. The moderator focuses the discussion on finding issues behind immediate agendas 21. The moderator focuses the discussion on finding win-win solutions 22. The moderator focuses the discussion on finding shared interests 23. The moderator stimulates the inclusion of different types of knowledge: expert and lay, implicit and explicit, etc. 24. The moderator ensures that everybody feels safe 25. The moderator ensures that everybody can speak freely 26. Feeling encouraged to react to others 27. Feeling encouraged to make viewpoints explicit and context-specific 28. Feeling encouraged to give viewpoints 29. Feeling encouraged to illustrate viewpoints 30. Seeing the discussed subject as work in progress The article is organized as follows. Paragraph 4.2 discusses the methodological setup. Then the research findings of the two cases are presented in Paragraphs 4.3 and 4.4. In Paragraph 4.5, conclusions are drawn and further research steps are discussed. 3. Being prepared to share and discuss knowledge together (Forester, 1987; Argyris, 1977; Edelenbos & Klijn, 2007) 16. Not seeing unknown information as a threat 17. Being able to handle new information 18. Articulating one s own ideas and reasoning 19. Not feeling unpleasantly surprised when new arguments are raised 4.2 Research methodology To understand the effectiveness of the theory-derived communication and trust building interventions, we follow the logic of the experiential case study research design (Straatemeier et al., 2010). This approach allows practitioners and researchers to experience and reflect upon the interventions by applying them in an academically constructed case representing a close to real situation. An important element of such an approach is finding a balance between a situation close to practice (to guarantee ecological validity) while maintaining enough distance for all participants to enable critical questioning and analysis (to ensure internal validity; see Van Aken, 2004). Based on the experiences and reflections of the participants with the application

64 of the interventions in the case, the interventions can be refined and tested in a subsequent case. Such reflective cycles result in insights into if and how underlying mechanisms work, as in realistic evaluation (Pawson and Tilly, 1997). However, although this approach is crucial for gaining in-depth insight for answering our research question, a consequence (due to its time intensiveness) is that it can only cover a small sample. This limits the ability to generalize our findings. Here, two such experiential cases were set up. The situation close to practice was found in two early staged CBAs of integrated transport plans in Amsterdam (the capital and largest city of the Netherlands) and Utrecht (the fourth largest city of the Netherlands). The participating practitioners were the actual plan owners who had been involved, whereas the evaluators were invited by the research team and were not previously involved. Furthermore, the cases were clearly set up as an academic exercise; the participants were informed beforehand of the scientific, rather than practical, aim of the meeting. Both cases followed the same procedure (see Figure 4.2): performing the five interventions (dashed box in Figure 4.2), measuring the impact on the related communication and trust mechanisms, and analyzing the acquired data. Thereby, the application of the interventions in Case 2 was improved through the experiences and reflections from Case 1 through an experiential cycle. The precise operationalization of the applied interventions is further explained in Paragraphs 4.3 and 4.4 (see also Table 4.2 in Paragraph 4.3). Figure 4.2. Setup experiential case study How was the information analyzed Five complementary quantitative and qualitative measurement instruments were applied to analyze the effects of the interventions on communication and trust between the participants. These mixed methods focused on different aspects of the communication and trust mechanisms (individual and group perceptions, interaction dynamics) and enabled triangulation (Bryman, 2008, p. 611). First, the interactions were observed by a member of the research team who was present in person, but not participating (i.e., structured observation, Bryman, 2008, p. 257). This observer ranked statements about the specific communication and trust mechanisms on a 5-point Likert scale for the participating plan owners and evaluators. For example, Mechanism 1 in Table 1, Sharpening assumptions based on the input of others, was operationalized in the observation report as The plan owners sharpened their assumptions based on the input of the evaluators (this statement was repeated with the evaluators in place of the plan owners ). Second, individual participants filled out a questionnaire (Bryman, 2008, p. 216) containing statements on a 5-point Likert scale describing their personal experiences of communication and trust from the meeting. These statements were similar as those ranked by the observer. For example, Mechanism 1 was operationalized in the questionnaire as I sharpened my assumptions based on the input of the others. Third, the interactions were recorded on video and analyzed by a different member of the research team. By using video, the analysis could focus on the role of the different participants plan owners, evaluators, and the moderator during all stages of the case (Bryman, 2008, p. 476). Fourth, participants were invited to discuss in a focus group setting if, how, and why they thought the interventions influenced their communication and trust experiences (Bryman, 2008, p. 473). Five statements were used to structure this discussion: We had enough room (both literally, in terms of time, and figuratively) to perform a dialogue ; We could discuss the plan and CBA simultaneously. Now they both feel more as a product of our own (a sense of ownership) ; The dialogue would have been less fluent without having prepared ourselves for the dialogue ; The moderator was crucial in the dialogue ; and The plan and CBA were clearly works in progress and were open for our input. Fifth, and finally, two participants per case one plan owner and one evaluator were interviewed some weeks after the experience. These reflective interviews functioned as a control for the results from the other research methods. Moreover, they gave some insight on if, how, and why the interventions in

65 fluenced real-world planning processes and daily routines after the meeting. The experiential case study thus resulted in rich quantitative and qualitative information about if and how the five applied interventions affected communication and trust between the participants in the two cases. Therefore, although the case study consists of a small sample, the variety of measurements enabled triangulation of our findings and increased their reliability (Bryman, 2008, p. 611). tion as the dialogue mode (Intervention 5): What are the expected effects of the plan? How to measure them? And how to include these effects in the CBA? Figure 4.3. Train stations in and around Amsterdam. The potential development of the station areas of Sloterdijk and Lelylaan, in red circles, were discussed in Case 1 (Municipality of Amsterdam, Adapted) 4.3 Experiential Case 1 A CBA of station area development in Amsterdam Applying the interventions in Case 1 In Case 1, the five theoretical interventions were operationalized as follows (see also Table 2). Six plan owners, four evaluators, and a moderator met for approximately one hour (excluding preparation and follow-up see Figure 2) to jointly discuss an actual CBA assignment of an integrated transport plan (Intervention 1). The plan owners who participated had been involved in the actual planning process and represented different municipal authorities from the local districts and the central city. The four evaluators were asked by the research team to participate based on their knowledge of the CBA method. The discussed plan proposed investments in train station areas in Amsterdam (the capital and largest city of the Netherlands) to facilitate transit-oriented development (Municipality of Amsterdam, 2008). However, Amsterdam has at least ten important train stations, and the municipality intended to use the CBA to prioritize investments for the different stations. In the case study, the CBA setup for two station areas was discussed: Sloterdijk and Lelylaan (Intervention 2), see Figure 4.3. The participants were prepared at the start of the meeting by first discussing behavioral guidelines, underscoring the importance of being open, critical, non-defensive, sincere, and aware of differences; getting to know the other s interests; and actively asking questions (Intervention 3). The discussion was guided by a moderator, a senior consultant who had experience facilitating similar discussions. He was asked by the research team to focus on: making everybody feel safe; letting participants speak freely; including all types of knowledge; focusing on finding shared interests and win-win solutions; and finding issues behind immediate agendas (Intervention 4). The discussion itself was guided by three substantive questions written on a memo board placed in the middle of the group that was to func Observing and reflecting on Case 1: Influencing interpersonal communication and trust? This Paragraph presents the prominent findings of Case 1 for each intervention based on the five different measurement methods, describing if and how communication and trust between the participating plan owners and evaluators increased. The findings from the questionnaire are presented in Figure 4.4. Furthermore, all quotes are translated from Dutch. INTERVENTION 1: ROOM FOR DIALOGUE The participants in Case 1 noted in the questionnaire that they felt safe, open, and able to listen to each other, as these mechanisms had a Likert score above average (see Mechanisms 5, 7, and 8 in Figure 4.4). In the focus group discussion, the evaluators mentioned that their openness could have been influenced by not being actually involved in the case and not having any vested interests. Nevertheless, the evaluators were perceived by the plan

66 owners as rather critical, which they mentioned in the group discussion. For example, after the plan was introduced by one of the plan owners, a participating evaluator asked, So, what are you actually aiming for, and what are the goals? I missed that in your introduction. In the reflective interview, this evaluator explained that he felt the plan should have been more problem-solution structured. Instead, the plan owners mainly presented general normative statements like, The station should function as a gate to the city of Amsterdam, and Our aim is to better include this part of the city [being at the edge of the municipality] by extending the spatial qualities of the city center. The participants seemed to experience difficulties with understanding concepts and reasoning outside of their own domains. When the participants were asked to reflect on Intervention 1 in the focus group discussion, they pointed out that the time spent together was too little to build up a relationship and share a lot of information, especially because the discussed plan was perceived as complex and premature. INTERVENTION 2: SHARING AND DISCUSSING THE PLAN AND CBA TOGETHER AND SIMULTANEOUSLY Relating to Intervention 2, and in contrast with the findings of Intervention 1, the evaluators felt that they learned more about the plan and the participants perceived to understand each other s viewpoints and reasoning (Mechanisms 12 and 15 scored above average on the questionnaire). In the focus group discussion, the participants stated that they appreciated being able to discuss together, with plan owners and evaluators, in such an early planning stage as in the experiential case, which is not common in the average Dutch CBA process (Beukers et al., 2012a). One of the evaluators mentioned, It is better to find out together what the assignment is exactly about than alone behind your desk. And a plan owner felt the discussion helped him to realize that their plan remained rather vague in CBA terms: What are the indicators that matter [in CBAs]? It is clear to me now that we need to talk about that further. INTERVENTION 3: BEING PREPARED In the group discussion, the participants thought that although Intervention 3 was helpful in creating an open and fair discussion in a comfortable atmosphere, the behavior guideline used would have been more effective and necessary if participants were holding strong opposing viewpoints. Since the participating evaluators weren t involved with the actual planning and assessment, they mentioned feeling neutral towards the discussed topic. Furthermore, the participants thought the preparation fell short on the content side, which limited the breadth and depth of the shared information. For example, one evaluator mentioned, If we had had some homework [documentation about the plan], we could have been more critical. Also, one plan owner mentioned in the reflective interviews, I didn t feel comfortable with the CBA terms that were discussed, and would have liked to know more about it in advance. Such information, however, was not provided in this case. INTERVENTION 4: FACILITATING DISCUSSION USING A MODERATOR The participants perceived that the moderator helped them to be open in the discussion, speak freely, and focus on finding issues behind immediate agendas (Mechanisms 20, 24, and 25). However, the moderator didn t spur the participants to explain jargon or planning and CBA concepts, as mentioned earlier, so the discussion remained somewhat superficial. So, although the moderator played an important role in creating a comfortable atmosphere for discussion, he didn t pay enough attention to explicitly sharing different types of knowledge technical and non-technical, implicit and explicit, and across disciplines (Mechanism 23). INTERVENTION 5: USE OF DIALOGUE MODES, PRESENTED AS WORK IN PROGRESS The fifth intervention for this case corresponded to the evaluators feeling encouraged to share their viewpoints and discussing the plan and CBA as work in progress (Mechanisms 28 and 30). However, in the video analysis and reflective interview with the participating plan owners, it seemed that the three substantive questions that steered the dialogue (What are the expected effects of the plan? How to measure them? And how to include these effects in the CBA?) were too CBA-focused and too precise, considering the prematurity of the plan. Thus, the dialogue mode used mainly related to the evaluators reasoning, and as such wasn t very effective in stimulating communication and trust in the interaction. It hardly stimulated the participants to share more and richer knowledge, as the discussion remained somewhat superficial. Nevertheless, the participants mentioned in the group discussion that both the plan and CBA were perceived as work in progress and this helped to encourage their input. The evaluators, for example, felt they could influence the plan and the setup of its assessment: The plan owners adopted what was being said, so that made me understand the plan was still in development

67 Figure communication and trust mechanisms perceived by plan owners and evaluators (5-point Likert scale), questionnaire Case Reflecting on Case 1: Refining the interventions The findings from Case 1 show diverse results in whether the five interventions helped to improve communication and trust between plan owners and evaluators, and how they may have done so. Most positively influenced seemed to be the mechanisms of feeling safe, being open towards each other, listening, asking critical questions, and learning more about the plan and/or CBA. Less positively influenced mechanisms seemed to be sharing detailed information, being explicit about the reasoning behind one s viewpoints, forming new insights together, and seeing shared interests (see Figure 4.4). Overall, the participants in Case 1 seemed to have minimally increased their mutual understanding and the discussion remained somewhat superficial. However, Case 1 also illustrated that such confusion and difficulties understanding each other was still valuable for the plan owners, as it made them realize their plan remained rather vague for outsiders. It also became more clear to them what elements would be important when confronted with CBA reasoning. Furthermore, the position of the evaluators as not being actually involved showed different effects. On the one hand, the evaluators mentioned they felt less critical because they didn t have any vested interest. On the other hand, this neutrality might have increased the ability of the participants to be open in the discussion. LESSONS FROM CASE 1 Related to the sequential setup of the experiential case study, four possible reasons were identified for the limited effectiveness of the interventions in Case 1. First, the time dedicated to the dialogue didn t seem to be commensurate with the complexity of the discussed topic. Second, the dialogue seemed to require better preparation of the participants for the discussion s content, not just process rules about attitudes towards the discussion. Third, the moderator could have paid more attention to urging the participants to be explicit and explain any jargon used. And fourth, the three questions that steered the dialogue didn t invite all participants to share their viewpoints or react to the views of others, as they mainly related to the evaluators reasoning being too CBA oriented. Although other reasons could be given for the limited effectiveness of the interventions in Case 1 (for example the personal characteristics of the participants or the political context of the planning task), the four aforementioned explanations were used to refine Interventions 1, 3, 4, and 5 as follows (leaving Intervention 2 unchanged see also Table 4.2): - Improving Intervention 1 by devoting more time to the discussion

68 - Improving Intervention 3 by also preparing the participants for the content of the discussion in advance - Improving Intervention 4 by asking the moderator to encourage participants to be explicit and avoid jargon - Improving Intervention 5 by using a dialogue mode that is related to planning and CBA rationale Table 4.2. Overview of the operationalization of the interventions in Case 1, the lessons learned, and the refined operationalization in Case 2. Theory derived Interventions I. Meet face-toface in an open, critical, nondefensive, and sincere way II. Interact and discuss the plan and CBA together and simultaneously III. Be prepared and trained to interact IV. Have the interaction guided by a moderator V. Use discussion modes presenting the plan and CBA as work in progress Operationalization Case 1 A dialogue was organized with 6 plan owners (representing the involved departments of the local government), 4 CBA evaluators, and a moderator. The dialogue lasted about one hour. The plan (station area development of two Amsterdam train stations) and setup of a CBA were discussed together and simultaneously. The participants were prepared for this interaction by reading a pamphlet of guidelines that discussed how to approach one another. The interaction was guided by an experienced moderator who was prepared for the task by the research team. Three questions were used to steer the discussion: What are the expected effects of the plan? How to measure them? And how to include these effects in the CBA? Lessons from Case 1 Devote more time to the discussion No fundamental changes Also prepare the participants for the content of the discussion in advance Ask the moderator to encourage participants to be explicit and avoid jargon Use a dialogue mode that relates to both planning and CBA rationale Refined operationalization Case 2 A dialogue was organized with 7 plan owners (representing the involved regional and local governments), 2 evaluators, and a moderator. The dialogue lasted two hours. The plan (two routes of a tramway in the Utrecht region) and setup of a CBA were discussed together and simultaneously. The participants were prepared for this interaction with a guide discussing how to approach one another, and information about the discussed plan was shared in advance. The interaction was guided by an experienced moderator who was prepared for the task by the research team, and was instructed to pay attention to be explicit and avoid jargon. The Effects Arena was used to steer the discussion (see Figure 6). 4.4 Experiential Case 2 A CBA of tramway routes in the Utrecht region The experiential case study continued with applying the refined interventions in a second case following the same procedure as illustrated in Figure 4.2 (see also Table 4.2 for an overview of the operationalization of the interventions in Case 2) Applying the refined interventions in Case 2 In Case 2, seven plan owners, two evaluators, and a moderator met for approximately two hours (refined Intervention 1) to discuss a CBA assignment of an integrated transport plan. The plan dealt with the development of a second tramway in the region of Utrecht in the Netherlands, initiated by the regional planning authority of Utrecht. The plan proposed two different tramway routes connecting the eastern and western parts of the region (see Figure 4.5). Both routes were expected to increase the livability of the city center by replacing the buses that ran through it by a tram (Intervention 2). As in Case 1, the plan owners had been involved in the actual planning process and represented different governmental parties: the regional planning authority of Utrecht, the spatial planning and economic development departments of the city of Utrecht, and the smaller neighboring municipality of Zeist. The two evaluators and the moderator were asked by the research team to participate and were not previously involved. The moderator was a different person than in Case 1, but had similar experience with guiding CBA processes. One of the two evaluators also participated in Case 1. Both evaluators had extensive experience in conducting complex CBAs. Figure 4.5. Discussed tramway routes in the Utrecht region (gray lines). (Bestuursregio Utrecht, 2009, adapted)

69 Performing the communication and trust building interventions, a member of the research team had a preliminary meeting with the coordinating plan owner to review the information needed during the discussion, which the participants received beforehand (refined Intervention 3). Furthermore, a member of the research team had a preliminary meeting with the moderator to talk about her role in the discussion and instruct her to encourage participants to be explicit and explain any jargon used (refined Intervention 4). The discussion itself was steered by a dialogue tool called the Effects Arena (Figure 4.6) (refined Intervention 5). Effects Arena was inspired by a tool developed to support public housing companies when assessing plans with CBA (Stichting Experiment Volkshuisvesting, 2010) and entails four main questions: Who are the initiators?; What is the actual initiative?; What are the expected effects?; and What is the expected impact (including where and on whom)? Furthermore, the Effects Arena explores how the initiative could lead to the expected effects, how the effects could lead to the expected impacts, and pays attention to both the local and regional contexts. Effects Arena is a visual, interactive tool that aims to invite all participants to pick up a pencil and fill in their thoughts, to respond to each other, and to jump from one question to another and back. The Effects Arena was printed on an A1-sized paper and placed in the middle of the table to invite all participants to write or draw on it Figure 4.6. The Effects Arena. (Stichting Experiment Volkshuisvesting, Adapted) Observing Case 2: Influencing interpersonal communication and trust? In line with Case 1, this paragraph presents the prominent findings of Case 2 for each intervention based on the five different measurement methods, describing if and how communication and trust between the participating plan owners and evaluators increased. The findings from the questionnaire are presented in Figure 4.7. Furthermore, all quotes are translated from Dutch. INTERVENTION 1: ROOM FOR DIALOGUE Case 2 scored relatively high on Mechanisms 1 to 8, and 11 (see Figure 4.7): sharpening assumptions together, asking and receiving critical questions, giving attention to issues behind immediate agendas, listening carefully, sharing diverse information, being open in the discussion, feeling safe, and getting to know each other. These mechanisms were apparent in the following example: The group discussed the assumption that replacing buses by a tramway would increase the quality of the public space of the city center. Some participants were skeptical about this, which encouraged the plan owners to explain how they thought this could happen. In response, one of the evaluators mentioned, Well, solely replacing buses by trams would not be enough if the quality of the public space forms such an important element of the plan. Then you need to invest in that, too. This was perceived by the plan owners as a valuable change of scope, which they mentioned in the group discussion, since the plan was mainly concerned with investing in the tram infrastructure. Another example of the mentioned mechanisms at work was when one of the local plan owners wondered in the middle of a critical discussion, Why are we actually aiming for a tramway, isn t that old-fashioned? This question provoked besides laughter the plan owners of the regional authority to further explain their reasoning: expecting that a tram could be more reliable, robust, and provide a more pleasant traveling experience. So, although the tone in the discussion was critical, this didn t irritate the plan owners, or make them feel being attacked. Instead, they took it as an opportunity to explain and to refine their ideas. Relatively low scores were measured for the mechanisms of building up a relationship and estimating how others would respond in the discussion (Mechanisms 9 and 10). Nevertheless, the participants mentioned in the group discussion that the two hours dedicated to the dialogue was enough to discuss the plan thoroughly, and if more time was available their concentration may have waned

70 INTERVENTION 2: SHARING AND DISCUSSING THE PLAN AND CBA TOGETHER AND SIMULTANEOUSLY The participants perceived to understand the viewpoints of others and vice-versa (Mechanism 15). The evaluators felt they had a good grasp of the plan and its relevant issues. One evaluator mentioned, Discussing the plan and CBA in such a setting is far more vivid and dynamic than normal, when the main topic of discussion is negotiating over a sole assumption. The other evaluator added, This [discussion] was also much more informative then reading about the plan in a memo, which normally happens. Furthermore, the plan owners also thought they got a better understanding of the CBA technique, reasoning, possibilities, and limitations, as they mentioned in the group discussion and reflective interview. According to the participants, this happened because the evaluators explained throughout the discussion how certain effects would be approached in a CBA. For example, that attracting more businesses would be considered as an indirect effect, and that atmosphere of the public space could be included in the CBA qualitatively. Furthermore, the group discussed whether the plan could be made more effective if some parts of the plan would be phased in differently. Through this discussion, new alternatives (i.e., a rearrangement of the original plan) emerged. Figure 4.7. Communication and trust mechanisms perceived by plan owners and evaluators (5-point Likert scale), questionnaire Case 2 INTERVENTION 3: BEING PREPARED Intervention 3 scored relatively high on Mechanisms 16 to 19 of not regarding unknown information as a threat, articulating one s own ideas and reasoning, and not feeling unpleasantly surprised when new arguments were raised (see the examples already given). The participants thus expressed feeling prepared for the discussion, and that this helped them to be critical in the discussion (the evaluators), without feeling attacked when critical issues were raised (the plan owners). However, as in Case 1, the participants during the group discussion stated that the behavior guide wasn t that necessary in this particular instance, but would have been if conflicting parties interacted If it was about getting funding it is a different story, one plan owner said. Nevertheless, they thought the right persons with adequate knowledge were present at the table, and that this was an important element for what they considered to be a fruitful discussion

71 INTERVENTION 4: FACILITATING DISCUSSION WITH A MODERATOR Mechanisms 20 to 25 of Intervention 4 scored well overall. According to both the questionnaire and the observation, the moderator focused the discussion on finding issues behind the immediate problem and ensured that everybody felt safe and could speak freely. For example, the moderator encouraged one of the plan owners from the economic department of the municipality to share his thoughts about the business areas that might be affected by the initiated tramway. Without this encouragement, a valuable insight could have been missed, since this plan owner appeared to be a rather silent person. Furthermore, it was mentioned in the group discussion that the moderator helped the group be time efficient, inclusive, and explicit. She achieved this by using simple questions, like What do you mean by that? ; Can you explain that viewpoint further? ; or Do we all agree with that or are there different viewpoints? before proceeding to the next box in the Effects Arena. INTERVENTION 5: USE OF DIALOGUE MODES, PRESENTED AS WORK IN PROGRESS The scores of the fifth intervention s mechanisms were relatively high for feeling encouraged to react to others, giving viewpoints, and seeing the discussed subject as work in progress. (Mechanisms 26, 29, and 30). This last mechanism was apparent, for example, when alternatives outside of the initial scope were discussed (see example discussed under Intervention 2). Furthermore, the participants mentioned in the group discussion and reflective interviews that the Effects Arena was of great support to the dialogue for sharing a lot of knowledge without becoming too broad or vague. A plan owner shared his experience saying, Often discussions like these end up in ineffective chit chat without clear goals or agenda. An advantage of this meeting was having a structure that guided us how to approach the issues at stake. However, the participants missed some indicators in the Effects Arena, like interests, uncertainties, and preconditions Reflecting on Case 2 Compared to Case 1, Case 2 resulted in higher measured communication and trust effects of the interventions for both plan owners and evaluators, as illustrated in Figure 4.8. The refined interventions thus seemed to better fit the assignment in Case 2: the devoted time of two hours was considered enough, the preparation of both the behavior and the content was seen as effective, the moderator was judged to be prepared for her role, and the discussion tool was perceived as suited for its task. Furthermore, the participants in Case 2 concluded during the group discussion and reflective interviews that they thought the participants had the appropriate knowledge and experience to perform the CBA assignment of the integrated transport plan at stake. Figure 4.8. Average scores of the five Interventions in the questionnaires in the Cases 1 and 2 (5-point Likert scale) These factors led to what the participants of Case 2 described as a fruitful and rich discussion in which diverse knowledge and information was shared and formed, and in contrast with their previous experiences. In the group discussion and reflective interviews, both plan owners and evaluators expressed their perception the dialogue added value. The plan owners thought it was valuable that the evaluators brought in knowledge the plan owners did not have, which helped them in understanding how the CBA arrives at its conclusions, and determining if a CBA was the appropriate assessment instrument for the plan at stake. As such, it became less of an insider s tool, which they expected to increase the overall support for and acceptance of the assessment. The plan owners also thought that the dialogue supported their critical view towards the plan. However, the plan owners also noted that having all stakeholders at the table would be necessary to get a complete picture

72 From the evaluators point of view, the dialogue was valuable for sharpening the problem analysis, and vividly understanding the issues behind the plan. They expected this last point to support these issues in receiving more and adequate attention in a CBA report, even if it would not change the final CBA ratio. Furthermore, the evaluators expected that the dialogue could help them to work more effectively if all effects were discussed and explained by both sides. Through this, they thought, the analysis could fit better with the experiences of plan owners, and be better accepted and supported. Moreover, when its methods were widely shared, the evaluators thought the CBA could become less threatening to non-cba experts. 4.5 Conclusion and reflection Conclusion This chapter started by discussing the problematic use of the CBA when assessing integrated infrastructure plans, and the process characteristics of communication deficits and distrust between plan owners and evaluators. Furthermore, rather than only used as a final judgment, a learning use is desired, for which improving the communication and trust between plan owners and evaluators seem crucial. We introduced five potential communication and trust improvements (interventions and related mechanisms) derived from the academic literature, which were tested in an experiential case study design with two sequential cases. Our main research question was: Do the interventions, derived from the theoretical literature, improve communication and trust conditions between plan owners and evaluators in CBA processes in practice, and through which mechanisms? Notwithstanding the limitations of the research setup, which will be discussed below, our findings indicate that the five interventions have some influence on the conditions for interpersonal communication and trust in CBA processes, if they are adapted to fit the assessment context, as demonstrated in the case study. However, some mechanisms appeared easier to measure than other. The observer had, for example, trouble estimating Strengthening relationships with others. Also, the mechanisms of perceiving win-win solutions and shared interests seemed rather abstract to the participants upon reflection in the questionnaire, which they mentioned in the group discussion. Nevertheless, the interventions seemed to positively influence communication and trust enhancing mechanisms through the following ways: - The impact of deliberatively organizing a meeting between the plan owners and evaluators, since such meetings do not always happen spontaneously in CBA processes. - The impact of discussing the plan and CBA simultaneously early on; including participants with knowledge and experience with the plan or CBA. - The impact of being prepared to have an open attitude towards each other, the plan, and the CBA as well as on the content of the discussion; understanding one s own reasoning, viewpoints, and role in the discussion. - The impact of having such a discussion guided by a moderator who is able to create an open and safe atmosphere, and who also creates an explicit and jargon-free discussion. - The impact of using an effective dialogue mode that invites different views and encourages others to react, and which structures the discussion and facilitates interaction Reflection: Practice and theory In addition to testing and discussing how communication and trust levels between plan owners and evaluators were improved, our research also underscored that an application of theoretical innovations in practice hardly works without some fine-tuning. The experiential case study, with two sequential cases, was supportive for this transition and narrowed the gap between theory and practice by making the interventions more context-specific. Furthermore, the experiential case study increased the chances that participants will actually adopt the planning innovation by allowing both researchers and practitioners to gain experience with it (Straatemeier et al., 2010). After Case 2, there were indeed signs that the participants were interested in adopting the performed interventions in their daily practices. These experiences form arguments for embracing research approaches like the experiential case study. The findings of case study also resonated with insights from planning theory, and at the same time indicated where further research could focus. For example, that it is crucial for participants in an assessment process to actively engage with each other, appreciate each other, recognize commitments and struggles, and gain insight into strengths and weaknesses (Kidd & Fisher, 2007). Moreover, the research indicated that a meeting as such is not enough. Specific effort is necessary to collectively create meaning by not only addressing formal knowledge, but also including other types of knowledge, like participants own experiences, or personal stories (Innes, 1998; Healey, 2008; Non

73 aka et al., 2000). The Effects Arena dialogue tool that was used in Case 2 seemed important for this, as it invited the participants to share a lot of knowledge without becoming too broad or vague, as advocated by Forester (1999) and Schön (1983). However, the assessment and planning context of such a CBA dialogue seemed important in deciding its characteristics: how much time is needed, which questions need to be raised, what instructions are necessary for the participant roles, what should be the level of preparation, who needs to participate, and what discussion tool needs to be used. More research could reveal how a communication and trust building strategy should be applied in other phases of CBA assessment processes and planning contexts. This additional research could also explore the role of the different personal characteristics of participants. The setup of the experiential case study, though, also has some disadvantages, such as limited validity and reliability of the research results. Besides the limitation of the small sample (two cases with 19 participants in total), as mentioned in Paragraph 2, the case study was largely an academic setting. How the communication and trust building interventions could function in real planning contexts, and how they support the use of CBA as a learning tool, warrants more investigation. Furthermore, the participants selected themselves (they were willing to cooperate) and knew they were participating in an academic experiment. As such, the possible impact of, for example, their personalities was not taken into account. Moreover, the interventions were performed only once in both cases. Participants suggested that they should be performed several times during a CBA process and possibly with various participants, such as citizens or politicians. Again, further investigation is needed to understand how a communication and trust building strategy might function throughout the whole CBA process, and when other factors like power imbalances between participants are at play. All these questions point to two possible research strategies that go beyond the academic setting. One could be to integrate the interventions into real-world planning processes and monitor their workings. This approach would require acceptance and commitment on the part of plan owners and evaluators, and enough research time and capacity for the planning process to be monitored as it unfolds. Another research strategy could be to assess ex post facto planning processes that, while not formally applying the five interventions, show clear differences in the way elements of the interventions have been applied, or not applied (e.g., planning processes where there were face-to-face meetings between plan owners and evaluators and where there were not, planning processes where a mediator was employed and where one was not, etc.). Such a context-rich research strategy could form a valuable addition to the context-poor but control-rich research that has been conducted

74

75 Chapter 5: Testing in an ex-post case study Communication and trust as conditions for using Cost Benefit Analysis for learning: Comparing the CBA processes of Schiphol Airport and the IJmuiden sea lock

76 ABSTRACT Cost Benefit Analysis (CBA) is a frequently used but controversial instrument for assessing complex spatial infrastructure plans. A different approach to CBA might be necessary. Instead of using CBA solely to give a final assessment, evaluation literature suggests that assessments (such as CBAs) of complex plans should be used for learning. High levels of communication and trust between plan owners and evaluators could form crucial conditions for this. This chapter advances a study that synthesized five interventions, with the purpose of increasing communication and trust between plan owners and evaluators in CBA processes in order to support its use as a learning tool. The chapter aims to understand how these interventions influence communication and trust in CBA processes, and use of the assessment for learning. For this, two contrasting cases were compared: one in which these five interventions were likely present (the CBA process of Schiphol Airport), and one in which they probably were not (the CBA process of the IJmuiden sea lock). The case study resulted in a context-rich analysis, allowing for the understanding of the complexity of CBA processes that are interwoven with the planning and decision-making processes. The findings indicated that the interventions seemed to stimulate communication and trust between plan owners and evaluators, and how this facilitated the sharing of meaning, as demonstrated in the Schiphol case. Furthermore, the case study showed how achieving communication and trust could also be reached through factors outside of the CBA process, like a shared sense of urgency, as in the IJmuiden sea lock case. However, both cases point at the necessity of achieving communication and trust in order for learning to occur

77 5.1 Introduction Cost benefit analysis (CBA) would seem to be an attractive assessment tool for planning and evaluating infrastructure (or other kind of) plans. It promises to structurally expose all expected effects on society, translate these effects in monetary terms, and assess whether an investment might be worthwhile by giving a benefit cost (B/C) ratio. Based on these qualities, the CBA has become a widely used instrument for the appraisal and evaluation of large infrastructure projects, and is the norm in most OECD countries (Haezendonck, 2007; Mackie, 2010; May et al., 2008; Odgaard et al., 2005; Rotaris et al., 2010; Vickerman, 2000). However, the CBA often can t live up to its high expectations, and its use can be problematic (Annema et al., 2007; Jong & Geerlings, 2003; Beukers et al., 2012a; Mackie, 2010; Naess, 2006; Mackie & Preston, 1998; Mouter et al., 2013). Especially complex spatial infrastructure plans that aim to influence fundamental but hard to grasp aspects of societal challenges like sustainability or quality of life that fail to be cut down into monetizable pieces (Ziller & Phibbs, 2003; Vaughan et al., 2000; van Wee and Molin, 2012). Aside from content challenges, assessing spatial infrastructure plans in a CBA seems to come with severe process challenges (Reference to self; Damart & Roy, 2009; Sager & Ravlum, 2005; Martinsen et al., 2010; Eliasson & Lundberg, 2010; Veisten et al., 2010). Planning practitioners often perceive CBAs as a final judgment of a plan s fulfillment of CBA criteria, leading some planners to see it as just another obstacle to overcome instead of an opportunity for learning and improving the plan (Page et al., 2009; Reference to self; ECTM, 2004). Furthermore, CBA processes often appear to be characterized by communication deficits and distrust between the involved plan owners and evaluators, who act in opposition to each other (Reference to self). These process challenges relate to a changing assessment context in which issues like quality of life and sustainability have become core subjects of public debates, and content challenges relate to the increasingly multi-actor, political nature of CBA processes. Because of this changed context, many scholars have argued that the assessments of complex plans should be much more part of an inclusive dialogue in which social realities are constructed and networks of people, actions, and thoughts are built, rather than just providing objective and technical rational analyses (Khakee, 2003; Selicato & Maggio, 2011; Handy, 2008; Owens et al., 2004; Willson, 2001). In this view, assessments should ideally take place in an ongoing creative process of gradual learning between plan owners and evaluators, and between plans and analyses, and require multiple dimensions and context specificities to be taken into account (Curtis, 2008; Hull et al., 2011). Furthermore, literature has been put forward indicating that communication and trust are conditions for using assessments for learning (Selicato & Maggio, 2011; Kidd & Fischer, 2007; Khakee, 2003; Saarikosi, 2000) CBA s unfulfilled potential for learning However, CBA processes are not regularly characterized as learning processes. On the contrary, as mentioned above, plan owners tend to perceive CBAs as a final examination, and communication deficits and distrust are experienced among plan owners and evaluators. Considering the altered demands imposed on assessing plans, this suggests that a different approach to CBAs might be necessary, whereby high levels of communication and trust between plan owners and evaluators could form crucial conditions for using the CBA for learning. Through such a learning process, the CBA and the plan would develop in interaction with each other, leading ideally to a better plan (i.e., a better solution, or problem definition) and a better assessment (i.e., more accepted by the participants and representing the plan s degree of achievement of the pursued goals). Figure 5.1. Conceptual framework of interventions for improving communication and trust between plan owners and evaluators as conditions for using the CBA for learning A recent review of communicative planning and organizational learning literature supports the importance of communication and trust (Reference to self). That study uncovered five interventions that could potentially improve the communication and trust between plan owners and evaluators by using a CIMO logic (introduced by Denyer et al., 2008). CIMO refers to how, in a CBA context (C), the five interventions (I) are expected to trigger

78 communication and trust building mechanisms (M) in order to get the desired outcome (O) of communication and trust between plan owners and evaluators. These five interventions are: organize a dialogue space, discuss together and simultaneously, be prepared, be guided by a moderator, use dialogue modes (chapter three). As illustrated in Figure 5.1 and further elaborated in Paragraph 5.2, applying these five communication and trust building interventions in a CBA process is expected to increase the communication and trust between plan owners and evaluators, which is expected to support the learning use of the CBA Aim and setup of the article This introduced framework of Interventions, Mechanisms, and Outcomes in the specific Context allows for a more realistic evaluation (Pawson & Tilley, 1997) and can be tested and further refined by ongoing inquiry (Dewey 1960; 1964; Straatemeier et al., 2010). As such, this chapter aims first to better understand if and through which mechanisms these interventions indeed help to improve communication and trust between plan owners and evaluators. The main research question is therefore: Do the interventions, derived from the theoretical literature, improve communication and trust between plan owners and evaluators in a CBA process, and through which mechanisms? The second aim of this chapter is to explore the learning potential of CBAs, and how this might be influenced by communication and trust. For these two aims, a comparative case study was conducted of two finalized CBA processes of complex infrastructure plans in the Netherlands: the CBA of Schiphol Airport in 2008 (Case 1), and the multiple CBAs and counter-analyses of the sea lock in IJmuiden from 1998 to 2012 (Case 2). The chapter is organized as follows. In Paragraph 5.2, the theoretical framework of communication and trust building, and learning is further elaborated. Paragraph 5.3 then discusses the research design used. Next, the two cases are discussed and analyzed using the theoretical framework: Schiphol Airport in Paragraph 5.4 and the IJmuiden sea lock in Paragraph 5.5. In the concluding Paragraph 5.6, the cases are compared and the research findings are summarized and reflected upon. 5.2 Theoretical framework: Communication, trust, and learning This Paragraph discusses the framework of communication and trust interventions and mechanisms, and the definitions of communication, trust, and learning that were used in analyzing the cases Communication and trust In this chapter, the importance of communication and trust in complex assessment processes is emphasized. Although communication is a muddy and dynamic concept (Littlejohn & Foss, 2008), this chapter uses the following definition (reference to self): Communication is human behavior that facilitates the sharing of meaning and takes place in a particular social context. (Lievrouw & Finn, 1990, p. 49) Within a CBA process, communication takes place mainly between plan owners and evaluators as a form of interpersonal or small group communication. Interpersonal communication relates to interpersonal behavior and relationships (Miller, 1978): at least two actors are involved, there is close physical proximity, many communication channels (modes) are available, and there are optimal conditions for immediate feedback. During this type of dialogue, i.e., face-to-face communication, concepts are built in cooperation with others, providing the opportunity for one s assumptions to be tested (Nonaka, 1994). As with communication, the literature on trust is diverse (Rousseau et al., 1998) and describes its importance in several ways: it enables cooperative behavior; it promotes adaptive organizational forms, like network relations; it reduces harmful conflicts; it decreases transaction costs (e.g., a business transaction); it facilitates rapid formation of working groups; and it promotes effective responses to crises (Rousseau et al., 1998). Furthermore, trust is in essence relational because the interests of one party cannot be achieved without reliance upon another (Rousseau et al., 1998). Gambetta clarified this relationship (as cited in Laurian, 2009, p. 371): A (the trustor) trusts B (the trustee) with regard to X. Trust is A s subjective assessment of the probability that B will act as agreed when B s actions significantly affect A, independently of A s capacity to monitor B s actions. Whereas communication stands for sharing of meaning, trust could be interpreted as (willing to, feeling able to, or) being able to share meaning in interaction with each other. As such, communication and trust are strongly interrelated (Reference to self; Laurian, 2009) and are treated as one concept for the remainder of this chapter. Following the introduced CIMO logic, and building on the literature, it is expected that within the context (C) of a CBA process, applying the five aforementioned communication and trust interventions (I) will trigger communicative and trust mechanisms (M) leading to the desired outcome (O) of communicationand trust between plan owners and evaluators, i.e., the sharing of meaning about the assessed plan and CBA. These interventions and related mechanisms are summarized in Table

79 Table 5.1. Interventions and related mechanisms for improving communication and building trust between plan owners and evaluators in CBA processes (Beukers et al., 2014). Theory-derived communication and trust building interventions 1. Organizing a dialogue space where plan owners and evaluators meet face-to-face (Argyris, 1977; Forester, 1999; Innes, 1998; Nonaka & Konno, 1998; Akgün et al., 2003; Lander et al., 2004; Healey, 2007; Laurian, 2009) 2. Sharing and discussing [the plan and CBA] together and simultaneously (Schön, 1983; Healey, 1999, 2007, 2009; Argyris, 1977; Argyris, 2006; Nonaka et al., 2006; Nonaka et al., 2000; Innes & Booher, 2003; Laurian, 2009) 3. Being prepared to share and discuss knowledge together (Forester, 1987; Argyris, 1977; Edelenbos & Klijn, 2007) Related communication and trust building mechanisms 1. Sharpening assumptions based on the input of others 2. Receiving critical questions 3. Asking critical questions 4. Giving attention to issues behind immediate agendas 5. Listening carefully 6. Sharing detailed and diverse information 7. Being open in the discussion 8. Feeling safe 9. Estimating how others will respond in the discussion 10. Strengthening relationships with others 11. Getting to know others better 12. Knowing more about the plan as well as the CBA 13. Seeing opportunities for win-win solutions 14. Recognizing shared interests 15. Understanding the viewpoints of others and vice-versa 16. Not seeing unknown information as a threat 17. Being able to handle new information 18. Articulating one s own ideas and reasoning 19. Not feeling unpleasantly surprised when new arguments are raised 4. Having the interaction guided by a moderator (Innes and Booher, 2003; Laurian, 2009; Edelenbos & Klijn, 2007) 5. Using dialogue modes (e.g., sketches, simulations, storyboards, role play); presenting the work as work in progress (Schön, 1983; Forester, 1999) 20. The moderator focuses the discussion on finding issues behind immediate agendas 21. The moderator focuses the discussion on finding win-win solutions 22. The moderator focuses the discussion on finding shared interests 23. The moderator stimulates the inclusion of different types of knowledge: expert and lay, implicit and explicit, etc. 24. The moderator ensures that everybody feels safe 25. The moderator ensures that everybody can speak freely 26. Feeling encouraged to react to others 27. Feeling encouraged to make viewpoints explicit and context-specific 28. Feeling encouraged to give viewpoints 29. Feeling encouraged to illustrate viewpoints 30. Seeing the discussed subject as work in progress Learning The second focus of this study is on learning. As many have stated, assessments of integrated transport plans should become much more part of an inclusive discourse in which social realities are constructed and networks of people, actions, and thoughts are built (Khakee, 2003; Selicato & Maggio, 2011; Handy, 2008; Owens et al., 2004; Willson, 2001). According to Fischer et al. (2009), assessment should not only provide decision makers with better information, but it should also enable attitudes and perceptions to change through participation and involvement, and change established routines. Assessments are thus seen as institutional spaces for challenging the status quo (Cowell and Owens, 2006, p. 405). Assessments could do this both directly, in the form of making immediate changes to plans and assessments concerned, and more indirectly in longer-term and less instrumental ways by assimilation of environmental, social, or economic understanding into individual and organizational norms, practices, and skills. Jha-Thakur et al. (2009, p.136) defined this as learning about and learning through assessments. A similar distinction can be found in the seminal work of Argyris and Schön (1978) theorizing learning in organizations through single and double loop learning. Single loop learning oc

80 curs when small and incremental changes are made to repair errors in organizations. However, if errors are persistent, a more fundamental or radical understanding is needed double loop learning to change internal structures and truly solve the problems at hand. Thereby, a critical factor is creating awareness of existing positions and assumptions, and to produce new assumptions by combining articulated views with questions posed by others in an open dialogue (reference to self). However, there are also moments when there is an opportunity or pressure to learn, but nothing changes, as in zero learning or inertia (Shrivastava, 1983; Giezen et al., 2012). In Table 5.2, these types of learning are articulated for the context of a CBA process. Learning type Direct or single loop learning Indirect or double loop learning Zero learning Table 5.2. Learning types Learning in the context of a CBA Learning about a CBA and the plan, which results in small and immediate changes of the plan and CBA. Learning through a CBA and the plan, which emphasizes the CBA as a cognitive process in which social realities are (re)constructed resulting in more fundamental changes of underlying strategies, norms, or goals of the plan and CBA. There is a pressure to learn, but nothing changes 5.3 Research design and methodology As mentioned above, this chapter first aims to better understand if and through which mechanisms the theoretical interventions help to improve communication and trust between plan owners and evaluators. And second, to explore the learning potential of CBAs and how this might be influenced by communication and trust. This chapter thus aims to make sense of complex human action in a CBA process without clear boundaries between the assessment process and its wider planning and decision-making context, taking an interpretive stand (Bryman, 2008, p. 384). The study is therefore designed as a comparative case study, allowing for the retention of the holistic and meaningful characteristics of a real-life event, like a CBA process (Yin, 2008), and making the findings context-rich and ecologically valid (Bryman, 2008, p. 33). However, given the specificity of the cases, this study is much more about better understanding the complexity of these specific contexts and giving a robust description than constructing generalizable claims. Table 5.3. Characteristics of cases and CBA processes Characteristics of cases CBA process, Schiphol Airport CBA process, IJmuiden sea lock INDEPENDENT VARIABLE: Interventions present Likely Not likely OTHER VARIABLES: Type of plan Stakeholders/Plan owners Stakeholders/CBA evaluators Planning process characteristics Development of the Schiphol airport to stimulate the regional and national economy Residents, regional government, City of Amsterdam, national government, Schiphol Airport, and KLM Airlines (representing all airlines at Schiphol) CBA consultants The CBA process was part of a wider and long running multiactor dialogue named the Alders table Increase of the capacity of the IJmuiden sea lock to stimulate growth of the port of Amsterdam and the regional and national economy Ministry of Infrastructure and Environment (I&E), the regional department of the Ministry of I&E, and the Port of Amsterdam CBA consultants, Dutch Bureau for Economic Policy Analysis (DBEPA) The CBA was a formal requirement in the process for obtaining national funding. Location Amsterdam metropolitan area Amsterdam metropolitan area Number of CBAs and reviews made Range of investment ( million) 1 CBA 4 CBAs and 4 second opinions Case selection For the purposes of this research, it was necessary to select cases that would vary on the interventions as independent variables in order to assess their effect; preferably, one case with and one without the interventions (see Table 5.3). However, the aforementioned five interventions are not common in Dutch CBA processes (Eijgenraam et al., 2000, Annema et al., 2007). Nevertheless, a case was found of a CBA of the development of Schiphol Airport near Amsterdam, which was conducted in the setting of an intensive, multi-actor dialogue, known as the Alders table after the chairman (Decisio, 2008; De Jong, 2012; van Buuren et al., 2012). The presence of a moderator was a key

81 indicator of the likelihood that the five interventions were present in this CBA process. The contrasting case, which would preferably also be located in the Amsterdam metropolitan area as to increase its comparability, was found in the CBA process of the sea lock of IJmuiden, part of the Amsterdam metropolitan region, that took place from 1998 to 2012, and in which four CBAs and counter-analyses were made. This project was examined in Annema et al. (2007) as one of 13 Dutch infrastructure projects assessed in a CBA after new assessment guidelines were introduced by the Dutch Ministry of Infrastructure and Environment in These guidelines do not recommend a communication and trust building strategy, which made it likely that the interventions were not present in this case Data gathering and analysis To map the communication, trust, and learning that did or did not take place between participating plan owners and evaluators, their perceptions of this were gathered in narrative interviews. These perceptions are complemented with secondary data and policy documents. However, the related CBA reports did not really discuss process characteristics, which made them less useful for the analysis. In narrative interviews, the focus is on the stories and recollections of past experiences (Giezen, 2012). In this case, the interviewees were asked to reconstruct the CBA process, using open-ended questions that did not prime them on communication, trust, or learning, in order to prevent the interviews from being pushed into a hypothesis confirming direction. For Case 1, 11 interviews were conducted. For Case 2, 13 interviews were conducted. The interview selection was stratified to ensure that both evaluators and plan owners were represented. Furthermore, the interviewees were asked to reflect on whether the list of interviewees represented the breadth of standpoints in the case. This supported the inclusion of all relevant perspectives and attainment of theoretical saturation (Bryman, 2008, p. 416). For each case, the interviews were analyzed deductively by coding citations that reflected, or explicitly did not reflect, the theoretical framework in Paragraph 5.2 (Tables 5.1 and 5.2), and then synthesized. For example, all citations that reflected sharing the plan and CBA together (or not) were coded as relating to Intervention 2, before synthesizing comparable and contrasting perspectives. Or, all citations that referred to how participants were listening to each other s reasoning (or not) were coded as relating to Mechanism 2, before synthesizing the comparable and contrasting perspectives. Furthermore, because the assessment of the IJmuiden sea lock was conducted over a longer pe- riod of time, the citations of this case were first divided into four periods: , , , and (see Paragraph 5.5 for a discussion of these periods) before being coded and synthesized. As such, the case analysis gave an overview of similar and contradicting perspectives on communication, trust, and learning Interventions and Mechanisms. Thereby, the explicit aim was to also look for other factors pointing at alternative explanations and theories. The following two Paragraphs report the findings of the two cases. 5.4 Case 1: CBA process of Schiphol Airport development Case one was analyzed using the framework of communication, trust, and learning elaborated in Paragraph 2. This Paragraph first gives a brief description of the case s context. Then the identified communication and trust Interventions, Mechanisms, and Outcomes are discussed and highlighted in brackets corresponding with Table 5.1 (I = intervention; M = mechanism). This is followed by a discussion of the moments of learning and possible other factors Context of CBA process Schiphol is the main airport of the Netherlands and considered one of Europe s four main airports due to its dense network of intercontinental and European flight connections (van Wijk, 2007; Schiphol group, 2012). The airport is located close to the city of Amsterdam and has a limited and politically determined number of takeoff and landing rights to control the noise the aircrafts produce. At the start of the second millennium, the airport reached the limits of these rights and negotiated for more flights. However, residents of the Amsterdam region do not want more noise and disturbed nights. They had protested in past decades, and knew how to reach national media and political parties. Notwithstanding several attempts to reach agreement, the development of the airport reached a deadlock, which was considered a severe threat to the Dutch economy (De Jong, 2012). In 2006, a dialogue platform (the Alders table) was established to break this impasse (De Jong, 2012; van Buuren et al., 2012). The political aim of this dialogue was to find an agreed upon and accepted solution for all involved actors that would facilitate future economic development without an increase of the environmental costs (nuisance) (De Jong, 2012). From 2007 to 2008 a quick scan CBA was made (as well as other studies, like an environmental impact analysis) in the context of this dialogue table at its administrative subdivision. The participating parties

82 at this subdivision were: resident groups, the regional government, the City of Amsterdam, the national government, Schiphol Airport, and KLM Airlines (representing all airlines at Schiphol). These parties are considered as the plan owners, although they had different visions of the preferred development (or lack thereof) at Schiphol. The consultants who conducted the CBA are considered as the CBA evaluators. Table 5.4 gives an overview of the interviewees and their involvement in the CBA process. Table 5.4 Classification of interviewees in Case 1: CBA process Schiphol Airport development Name alternative Table 5.5 Alternatives assessed in the quick scan CBA Development Schiphol Airport 2020 Schiphol Airport Eindhoven Airport 0. Zero alternative Growth at Schiphol Relocation option a Lelystad Airport 3. Relocation option b Relocation option c Residents alternative Type of actor and involvement in the CBA process Consultants conducting the CBA 3 Number of interviewees Moderators (process manager and assistant): Guided the process of developing a shared plan for Schiphol Airport, including the CBA Local resident representative: Intensively involved in the Schiphol case since the 1970s. Permanently participating in the discussion table KLM airline representatives: Represented the aviation sector Schiphol Airport representatives 2 Representative of the City of Amsterdam: Represented local government Total Plan alternatives The CBA analyzed the monetized effects for the Netherlands of the autonomous development of the Schiphol airport up to 2020, compared with alternatives that partly accommodated flights at the regional airports of Eindhoven and Lelystad (Decisio, 2008). At the dialogue table, the parties agreed to assess alternatives 1 to 4 in Table 5.5. These alternatives, though, all considered more flights at Schiphol compared to the zero alternative. At the start of the CBA process, the residents representative therefore proposed another alternative (number 5) that relocated more flights to the regional airports and diminished the flights at Schiphol to Eventually, this alternative was also included in the CBA. The main disputes were thus about the effects of noise and the wider regional economic impact of relocating flights outside of Schiphol Interventions, Mechanisms, and Outcomes of communication and trust PRESENCE OF THE FIVE INTERVENTIONS As expected, all five interventions from the theoretical framework were to a certain level represented in Case 1. An ongoing dialogue was organized at the Alders table and its administrative subdivision between all relevant parties [I 1 in Table 5.1]. The evaluators were also asked to participate at the dialogue table. The assistant moderator explained that, for this purpose, they deliberately choose a consultancy firm that was willing to participate in the table. Second, the CBA and the alternatives were discussed at the table while being developed [I 2]. As mentioned above, the residents brought in another alternative early in the CBA process because they didn t see their interests (less noise around Schiphol through less flights there) reflected in the other alternatives. Although the participants already agreed on the set of alternatives for the assessment, the group was willing to listen to the plan of the residents and to include it in the assessment. For this, the moderator and representatives of KLM, Schiphol, and the City of Amsterdam recalled it was important that the residents alternative didn t bluntly say no to a flourishing Airport Schiphol, but perceived it as a serious alternative. Third, the moderators invested a lot of time in preparing the participants for the CBA [I 3]. They pre-discussed the CBA methodology with the governmental parties and the Dutch Bureau for Economic Policy Analysis (DBEPA) and estimated that a strict use of the welfare theory could become problematic. Instead, a quick scan CBA was chosen as the methodological

83 framework, giving explicit attention to difficult to monetize effects. This basically meant framing the CBA differently, it did not result in severe methodological changes since the evaluators still followed the regular Dutch CBA guidelines with the welfare theory as the line of reasoning (Decisio, 2008). Furthermore, the moderators discussed separately with all participants what the important issues were to them and what could be problematic in the assessment, namely the difficult to monetize effects on health and the indirect effects on the regional economy. Also, if difficulties were raised along the process, the moderators would discuss this in bilateral sessions with the concerned participants before continuing plenary. For example, the CBA evaluators were told to have an extra meeting solely with the resident representatives to explain how the health effects would be approached in the CBA. This gave the residents the opportunity to really understand the calculations, to include their own experiences and knowledge of the issue, and prepared them for the plenary dialogue. Moreover, the evaluators started the actual CBA by explaining the methodology and how effects were going to be calculated. Fourth, as mentioned above, the complete CBA process was guided by a senior moderator and assistant [I 4]. The moderator recalled that their biggest challenge was to not let the CBA become an obstacle in forming an agreement. Fifth, although there weren t any deliberately used discussion tools, a sensitivity analysis and the writing the final report functioned as such [I 5]. In the sensitivity analysis, the evaluators showed, for example, what happened with the B/C ratio if extremely high values on health effects where used; they did not make a big difference. Furthermore, the final report was written collectively to find the right tone. For example, the residents weren t allowed by their constituency to discuss the growth of Schiphol. Therefore, the process assistant recalled that in the final report the word growth was systematically replaced by the more neutral word development. PERCEIVED MECHANISMS According to the analysis of the interviews, it was indicated that participants got to know each other better, strengthened their relationships during the meetings, and listened to each other [M 10, 11, and 7 in Table 5.1]. This seemed especially important for the position of the residents. According to the moderator, the biggest opponents the residents and representatives of Schiphol and the aviation sector had not met before. During the dialogue, it became clear that the residents were quite reasonable in their approach to Schiphol by acknowledging its economic importance. This positively surprised all other participants [M 5 and M 15] and helped the residents to gain their respect. The resident representative, evaluators, and process leaders remembered the atmosphere during the discussion as open and constructive [M 7]. The evaluators were surprised nobody got angry, even though the CBA did not please everybody s interests. According to the moderator, this was achieved by speaking to each other in a direct way without power plays [M 24 and M 25] and by asking and receiving critical questions [M 2, 3]. According to the assistant moderator, all participants got answers to the questions they had and understood how the calculations were made [M 12]. The moderators and evaluators thought that knowing in advance which effects could be difficult to assess enabled them to give these the right attention and to include all types of knowledge [M 19, M 23]. The resident representative also felt that being well informed and understanding how the effects were calculated helped him feel safe and enabled him to have discussions with the professional experts [M 7, M 18, and O 2]. Furthermore, attention was given to issues behind immediate agendas [M 20]. For example, the assistant moderator recalled that the terminology of environmental effects was confusing: the real issue for the residents was people being awake at night due to airplane noise. It was necessary to name the problem as such instead of calling it an environmental issue. The representative of the residents recalled that the presence of the moderator was important for him to feel safe and speak freely in the discussion [M 24 and M 25]. He felt his position at the table was less evident compared to the representative of Schiphol. The evaluators and representatives of Schiphol remembered that the moderators gave importance to focusing the discussion on the real issues at stake [M 20]. Furthermore, ideas and viewpoints were made explicit in the final report, and the evaluators were encouraged to avoid jargon so as to make the CBA understandable for all [M 27]. Therefore, it took a lot of time finding the right nuance and making it acceptable to all. However, not all Mechanisms in this case were positive. The representatives of Schiphol, the City of Amsterdam, and the aviation sector were not happy about the economic scenario from DBEPA that was used. They had to make a strong statement that, according to them, the scenario didn t represent the aviation economy [M 1, M 6, and M 15 negative]. Although the scenario was adjusted in the end [M 1, M 6], these representatives were disappointed about not being involved beforehand so as to include their knowledge. Furthermore, the representative of the City of Amsterdam was disappointed in the handling of the

84 indirect economic effects of Schiphol for the Amsterdam region. This was described qualitatively, which didn t do justice to the importance of Schiphol, the Amsterdam representative thought [M 15 - negatively]. The resident representative felt dissatisfied about the weight given to the health effects (too low, in his opinion), but accepted the way the calculation was made [M 15]. As mentioned above, the evaluators gave much attention to explaining how these health effects were approached, and showed in a sensitivity analysis what happened if extreme values were used [M 27, M 29]. COMMUNICATION AND TRUST OUTCOMES The analysis of Case 1 pointed at several outcomes of communication and trust: sharing ideas about the development of the airport, developing agreed-upon facts during the process, creating a shared language, exploring together the different interests, and discussing the assessment methodology used. Furthermore, according to the participants, the communicative approach that allowed such sharing of meaning supported the acceptance of the CBA by everyone. representative of the residents felt that it was important for this inclusion in that he got the opportunity to explain the alternative and its line of reasoning, convincing the others that it was a serious option. Furthermore, the evaluators also thought that the original set of alternatives was rather one-sided and a real opposing alternative was missing. Also of influence was that the residents presented their alternative early in the CBA process and including it was still possible as a practical matter. Two main moments of zero learning were noticed. First, the aviation sector and regional government argued that the assumption that airlines could be relocated from Schiphol to a regional airport was not realistic, because airlines are independent companies that cannot easily be forced to move elsewhere. Although the participants acknowledged that this assumption could be rather unrealistic, it was left unchanged and used in the CBA, anyway. Second, when participants realized that the B/C ratios would be positive and the outcomes of the alternatives did not differ that much, the alternatives and CBA were accepted as they were without any attempts to find more improvements Learning Several moments of learning in the CBA process were collected (see Table 5.6). First, a moment of learning about the CBA and plan occurred when the aviation representatives protested the economic scenario used. They thought this scenario did not reflect how the aviation economy worked. The dispute was mainly about the complexity of airport and airline networks that function only as a whole and cannot be separated into parts without huge consequences. After this explanation, the scenario was adjusted accordingly. This was an important moment for the aviation representatives and determined their acceptance of the assessment. Second, the residents used the CBA to argue that their plan would do no harm to the economy while also mitigating the local environmental effects (less noise). Third, the CBA outlined the expected regional economic effects of the airport, elucidating this generic concept. Although this information was not new to the representatives of the aviation sector or the Amsterdam region, it was useful information to explain to outsiders, they thought. Including the residents alternative in the assessment, even though the parties already agreed on another set of alternatives, could be interpreted as a moment of learning through the CBA and the plan. Several communicative aspects influenced this inclusion (besides other factors as described below). The Learning types Learning about CBA and/or plan Learning through CBA and/or plan Zero learning Table 5.6. Learning in Case 1, CBA process Schiphol Airport Identified in the case Adjusting the used scenarios CBA used as argument for residents alternative CBA confirms regional economic value of Schiphol Airport Including the residents alternative in CBA Unrealistic assumption about relocating airlines still used in CBA Assumed positive B/C ratio did not stimulate looking for improvements Other factors The research also focused on possible influences of other factors on communication, trust, and learning, with respect to two main aspects in Case 1: the strong position of the residents and the high quality of their alternative. The participating residents were explicitly supported by political parties in the Dutch parliament (De Jong, 2012), which helped them to be heard in the dialogue and their alternative to be included in the assessment. The position of the residents was also supported by the personal characteristics of their representative; being a well-educated, well-informed, extremely

85 dedicated (never missed a single meeting), and long-participating person. Furthermore, the residents alternative was a well-considered plan, which envisioned growth of the aviation sector while also diminishing the nuisances from Schiphol. This helped in its consideration as a serious option. However, the representatives of Schiphol Airport perceived that the parties had already agreed on the preferred alternative that being the residents alternative before the CBA was conducted. In general, the main merit of the CBA was perceived as showing that the residents alternative would not do any harm to the Dutch economy and development of the aviation economy Summing up Case 1 The analysis of Case 1 showed that the CBA process of Schiphol was characterized by the communication and trust building strategy of the Alders table. It contained all theoretical interventions, a wide variety of mostly positive and some negative communication and trust mechanisms, and reflected a setting in which participants were able to share meaning with each other. These communication and trust conditions seemed to have supported an accepted CBA without many disputes afterwards over its methodology or inputs and outputs. This was an unexpected result according to the participating evaluators and moderators, considering the tense political context and history. Furthermore, the communication and trust between the participants seemed to have supported some learning in the process. The case also pointed at the influence of the strong political position of the residents (and their alternative), which was mainly established outside the CBA process, on establishing communication, trust, and learning Context CBA process The sea lock of IJmuiden is located at the North Sea and forms the entrance to the port of Amsterdam. The sea lock is important for Amsterdam s regional economy, being the fourth largest European harbor in transshipment, Europe s largest petrol port, the world s biggest cacao port, and the second-largest European coal port (Port of Amsterdam, 2014). Currently, only the North Lock accommodates large cargo ships, with a maximum width of 47 meters. This lock was built in According to the Port of Amsterdam, the lock is unreliable and is going to be too small for the newest cargo ships in the near future. The Port of Amsterdam believed that developing a second and up-to-date lock of 65 to 70 meters in width was necessary for maintaining the competitiveness of the port. However, that would require investment by the national government, which meant that its national relevance needed to be proven, as well as in relation to the largest port of the Netherlands in Rotterdam. Furthermore, the second lock s benefit for society should be demonstrated, since the Port of Amsterdam is a private company, although the municipality of Amsterdam is the main shareholder (Port of Amsterdam, 2014). The possible investment in the sea lock was investigated over the period from 1998 to 2012 in four CBAs, four second opinions, and several related researches. Figure 5.2 gives a timeline and (partial) overview of the main research documents. The CBA process was divided in four periods. Table 5.7 describes the main events. Figure 5.2. Timeline of IJmuiden sea lock CBA process indicating the four periods from 1998 to Case 2: CBA process of the IJmuiden sea lock In the second case, the CBA process of the sea lock of IJmuiden was analyzed, using the framework of communication, trust, and learning elaborated in Paragraph 5.2. After a brief description of the CBA context, the identified communication and trust Interventions, Mechanisms, and Outcomes are discussed and highlighted in brackets corresponding with Table 5.1. This is followed by discussing the moments of learning corresponding with Table 5.2, and possible other factors

86 Period Table 5.7 Main events in the CBA process of the IJmuiden sea lock from 1998 to 2012 Main events CBA process I > A second sea lock is investigated in two CBAs (NEI, 2001; SEO, 2004). II > The Dutch Bureau for Economic Policy Analysis (DBEPA) is very critical about the assessments in their second opinions, arguing that the CBAs are too optimistic about the future growth (DBEPA, 2001). > The minister formally decides to reject the project from the national funding program. > The port of Amsterdam flourishes. Due to this real economic growth, the DBEPA writes an intermediate analysis advising that continuing the research process of a second lock would be worthwhile (DBEPA, 2007). A fourth important party was the Dutch Bureau for Economic Policy Analysis (DBEPA). The DBEPA is the national governmental accounting center for budgetary and economic policy and a research center that prepares independent forecasts and analyses (Van de Riet, 2003). They gave the second opinions on all CBAs and wrote the intermediate analysis in 2007 that investigated if a continuation of the research process was necessary. Furthermore, several consultants were involved as CBA evaluators. Notwithstanding different points of view, the representatives of the ministry, the regional department, and the Port of Amsterdam are considered as the plan owners in this case. The consultants conducting the CBAs and the DBEPA representative are considered as the CBA evaluators. Table 5.8 gives an overview of the interviewees and the parties they represented. III > The sea lock project is formally re-included in the national funding program. > A second sea lock is again investigated in a CBA, but still doesn t show a convincing B/C ratio (Ecorys, 2008). Table 5.8. Interviewees Case 2: CBA process IJmuiden sea lock IV > Political consensus was reached in 2009 by reframing the project. The focus changed from developing a second lock to replacing the existing lock with a new one at a new location and closing the old North Lock. > The regional parties agree to finance an earlier and bigger development of the new lock, whereas the ministry agrees to finance the replacement lock. > The national funding program was extended to 2025 in order to include the replacement lock. > Another CBA was made, using the reframed alternatives. This CBA showed a convincing B/C ratio (Rijkswaterstaat Noord- Holland, 2012). The CBA process was dominated by four parties. First, the Port of Amsterdam, which stimulated the investigation of the sea lock project in the first place. Second, the Ministry of Infrastructure and the Environment (formerly known as the Ministry of Transport and Water Management), which was responsible for the maintenance of the sea lock and the potential investor. The ministry commissioned three of the four CBAs; in 2001, 2008, and Third, the regional department of the ministry (known as Rijkswaterstaat Noord-Holland ) was strongly involved. Involvement interviewees in the CBA process CBA evaluators: Involved with the making of the CBA in CBA evaluators: Involved with the making of the CBA in CBA evaluator: Involved with the making of the CBA in Representative of regional department of the Ministry of Infrastructure and the Environment (the regional department): involved from 2004 to 2009 Representatives of regional department of the Ministry of Infrastructure and the Environment (the regional department): involved from 2010 to present Consultant representing the Port of Amsterdam: 2005 to Representative of the Port of Amsterdam: 2009 to present 1 Representative of the Ministry of Infrastructure and the Environment: involved from 2008 to present Representative of the Ministry of Infrastructure and the Environment: involved from 2008 to present Representative of the Dutch Bureau for Economic Policy Analysis (DBEPA): involved from 2000 to present To tal 13 Number Plan alternatives The CBAs included different plan alternatives (see Table 5.9). Building a second lock was the main alternative in the first three CBAs. In the 2012 assessment, this alternative was reframed as replacing the old North Lock, instead of a second lock

87 Table 5.9 CBA alternatives of IJmuiden sea lock CBA Zero alternative Project alternatives 2001 (NEI, 2001) 2004 (SEO, 2004) 2008 (Rijkswaterstaat Noord-Holland, 2008) 2012 (Rijkswaterstaat Noord-Holland, 2012) No change of current situation No change of current situation Short-term measures for improving the management of ships Replacing the North Lock by a similar lock (length 500 meter; width 50 meter; depth -15 meter) at a new location in and closing the old North Lock. - Improvement of sea access - Improvement of sea access and a second North Lock - Improvement of sea access and a second and bigger North Lock 13 alternatives comprising a variety of ideas ranging from a second big lock to a different system of unloading - Selectivity alternative: unloading sand ships at the sea side - Facilitation alternative: extension of capacity by a second big lock Sooner replacement of the North Lock by a bigger lock with dimensions: - Length 500 meter; width 65 meter; depth -18 meter - Length 500 meter; width 70 meter; depth -17 meter - Length 500 meter; width 60 meter; depth -18 meter Interventions, Mechanisms, and Outcomes of communication and trust The following paragraphs focus on the perceived communication and trust Interventions, Mechanisms, and Outcomes per period. Period I: PRESENCE OF THE FIVE INTERVENTIONS The CBA of 2001 was discussed in a working group in which the main parties were represented [element of Intervention 1 in Table 5.1]. However, the CBA evaluator recalled to regret that the DBEPA didn t participate in this group. The DBEPA preferred keep its distance from the project in order to maintain objectivity in its evaluation of the assessment, as the DBEPA representative reported. Nevertheless, the DBEPA was consulted about the CBA methodology used in this phase. Several economic researchers, in addition to the CBA evaluators, cooperated in a project team that produced the CBA input, without the plan owners participating [contrary to Intervention 2]. The CBA evaluators recalled depending on input from other researches, which resulted in time pressure. PERCEIVED MECHANISMS The CBA evaluators and representative of the DBEPA recalled this period as a turbulent one dominated by anger. Emotions ran high, for example, when discussing the assumptions used and economic scenarios. The DBEPA made a strong negative statement about the assessment, both orally and in the second opinion, and argued that the information used was too optimistic [M 3 in Table 5.1]. The representatives of the Port of Amsterdam, the regional department of the ministry, and the involved evaluators recalled that participating plan owners got very angry at the DBEPA, perceiving them as being dogmatic and having an unrealistic idea of the port economy [M 2 negative; M 5 negatively; M 7 negatively; M 8 negatively]. However, in retrospect, the involved evaluator and representative of the Port of Amsterdam thought that the estimations used were indeed rather optimistic, and that they should have been more critical themselves [M 1 negatively]. COMMUNICATION AND TRUST OUTCOMES The sequential organization of the assessment process seemed not to have facilitated as outcomes of communication and trust the sharing of meaning between the plan owners and the evaluators (including the DBEPA) early in the process. The assessment was criticized when it was already finalized, which frustrated the CBA evaluators and was perceived as harming their image. Period II: PRESENCE OF THE FIVE INTERVENTIONS In this period, the Port of Amsterdam tried to get the project back into the national funding program, from which it was excluded in 2005 (see Figure 5.2). To do so, the Port hired a consultant who was involved in making the CBA of 2001, understood the CBA method, and was familiar with the DBEPA. This consultant deliberately discussed with the DBEPA representative how to address the critique expressed in the DBEPA

88 second opinions [element of intervention 1]. Furthermore, he made a graph showing that the real economic growth of the port of Amsterdam until 2005 was higher than estimated in the CBA of This impressed the representatives of the DBEPA and the ministry, as they remembered this well in the interviews, and might have been functioned as a dialogue mode [element of intervention 5]. PERCEIVED MECHANISMS The hired consultant and representative of the DBEPA recalled that employees of the Port of Amsterdam were still angry when the assessment was discussed [M2 and 7 negatively]. The consultant, however, felt he had enough distance from the project and was therefore able to ask and receive critical questions, listen carefully, be open in the discussion, feel safe, and estimate how the representative of the DBEPA would respond [M 2, 3, 5, 7, 8, and 9]. Furthermore, the graph showing the real growth illustrated the harbor s viewpoints, made them explicit, and was easily understandable by the representative of the DBEPA [M 27, M 29]. COMMUNICATION AND TRUST OUTCOMES The interaction between the Port of Amsterdam and the DBEPA, initiated by the hired consultant, resulted in a sharing of thoughts about how the harsh critiques on the earlier two CBAs could be addressed, as an outcome of communication and trust. Period III: PRESENCE OF THE FIVE INTERVENTIONS The sea lock project got back into the national funding program, and participating parties recalled that they were now more convinced of the necessity of the sea lock to facilitate the growth of the port of Amsterdam. The third CBA was again discussed in a steering group in which the main parties were represented [element of Intervention 1], but without the CBA evaluators. The evaluators perceived that the CBA was strongly controlled by the steering group, as they determined its setup (like the alternatives and assumptions to be used). This was confirmed by the representative of the regional department of the ministry, who mentioned that the CBA was experienced as a potential risk, considering the past assessments. The participants in the steering group wanted to prevent the CBA from delaying the project again. The participants in the steering group also cooperated to prepare the political agreement that was signed in The CBA evaluators recalled investing time to explain the method to all participants and how the effects were going to be assessed. They deemed this as a necessary step, because not all participants were familiar with the CBA method [elements of Interventions 2 and 3]. Furthermore, the consultant hired by the Port of Amsterdam was still involved and emphasized to discuss the economic scenarios used with experts of the Port so as to include their experiences. Although there was no moderator involved, the consultant hired by the Port seemed to function as such, negotiating between the Port of Amsterdam on the one hand and the DBEPA and CBA evaluators on the other hand [element of Intervention 4]. PERCEIVED MECHANISMS The representatives of the ministry, the regional department, and the DBEPA recalled that the atmosphere during the discussions was less tense compared to earlier phases. Some of the representatives of the ministry and the regional department were replaced by new employees, which the CBA evaluators thought was beneficial, perceiving the newly involved participants as more open in the discussion [M 7]. Furthermore, the representative of the ministry felt the DBEPA was now more sensitive and used a more subtle tone. The DBEPA representative felt the plan owners were now more rational, reasonable, and listened more carefully [M5, 7]. The representatives of the Port of Amsterdam, the ministry, and the regional department recalled that they cooperated intensively in this period and had a shared goal in mind: enlarging and improving the capacity of the sea lock [M 13, M 14, O 2]. The participants mentioned an understanding of each other s viewpoints: the representatives of the regional department understood why this enlarged capacity was necessary for the port, and the representative of the port understood that simple a big, second lock would be too expensive and too much to ask [M 15, O 1]. However, the CBA setup was in large part determined before the evaluators became involved. The CBA evaluators mentioned that they saw opportunities to improve the alternatives, but they weren t allowed to do so [ M 30 negatively]. The B/C ratio was still not showing convincing numbers. COMMUNICATION AND TRUST OUTCOMES The participants in the steering group shared their experiences with each other with the assessment of the sea lock and were driven by a shared sense of urgency. This enabled them to find

89 creative solutions for the sea lock, in light of the complex political context. However, the actual CBA evaluators got involved late in the process, and could not influence the CBA setup in any meaningful way. Period IV: PRESENCE OF THE FIVE INTERVENTIONS In the fourth CBA, the politically agreed upon replacement alternative was assessed. This CBA was again guided by a steering group that also prepared the CBA by deciding which assumptions, alternatives, parameters, scenarios, and time frame to be used [element of Intervention 1]. Also, experts from the Port of Amsterdam were consulted on the economic scenarios used. However, although the CBA was intensively discussed, the CBA evaluator again felt neither involved nor having any influence in its setup [element of Intervention 2 negatively]. PERCEIVED MECHANISMS The representatives of the ministry, the regional department, and the DBEPA described the CBA process in this phase as harmonious and open [M 7]. Nevertheless, the representatives of the regional department needed to persuade the participants of the Port of Amsterdam to share the business information that was used as input for the CBA. The fear of a possible critical response from the DBEPA was used as the stick to make the Port open up [M 6 negatively]. Furthermore, the evaluator felt that the participants were insecure and too focused on the CBA outcome, namely if the B/C ratio would be positive or not [M 8 negatively]. The representatives of the ministry and the regional department also recalled that only after the first positive CBA results were presented did these representatives become less tense about the assessment. However, the participants felt that they were able to handle critical questions [M 2], as they didn t get angry about the DBEPA s fourth second opinion, which raised some criticisms. Furthermore, the representatives of the ministry consulted the DBEPA representatives about the assessment along the process, which was helpful for them to better understand the CBA and made the second option more nuanced [M 12]. Due to the range of CBAs and second opinions that were already made, the representatives of the Port of Amsterdam, the ministry, and the regional department felt they knew better what to aim for in this phase, and which potential bottlenecks would need extra attention [M 12, M 18]. However, the representative of the ministry reported that he thought knowing more about the CBA methodology in advance could have helped him in estimating the CBA outcome [M 17 negatively]. COMMUNICATION AND TRUST OUTCOMES The participants discussed the CBA throughout the assessment process in the steering group and bilateral meetings, to make sure the final calculations would show an acceptable B/C ratio. These moments of dialogue, though, seemed mainly intended as damage control, and did not seem to have influenced the plan or the CBA Learning The analysis pointed at two main moments of learning about the CBA (single loop learning), one moment of more fundamental learning (double loop learning), and two moments of zero learning (see Table 5.10). According to the representatives of the regional department, learning about the CBA occurred when each CBA raised new questions that were answered in the next analysis. The range of CBAs aided the determination of what kind of development at the sea lock could be desirable. Furthermore, having the project removed from the national funding program and then included again created a delay. This turned out to be perceived as a good thing by the representatives of the ministry, the DBE- PA, the evaluators in 2008 and 2012, and the Port of Amsterdam, because the real economic growth of the port demonstrated the necessity of increasing the capacity of the sea lock. The CBAs helped the participants to understand this. Deeper learning occurred when the very negative CBA ratios of the second lock alternative in the CBAs of 2001, 2004, and 2008 made it clear that something fundamentally had to change to reach agreement, and that sticking to the original plan would be unrealistic. However, the thinking to reframe the alternatives as a replacement of the North Lock was stimulated by other factors (see below). Zero learning also occurred. First, the evaluators of the CBA of 2008 claimed that they saw opportunities to optimize the alternatives. However, they weren t allowed by their client (formally the ministry) to investigate this, and didn t want to be accused of seeking for more work. Second, the evaluator of the CBA of 2012 mentioned that the alternatives were too similar to show any difference in some of the calculations. The CBA therefore didn t give any extra decision-making information

90 Learning types Learning about CBA Learning through CBA Table Moments of learning, Case 2 Identified in the case - Postponing investments until economic necessity was proven in CBA - Sharpening the research questions and design after every CBA for the next assessment - Understanding that a fundamental change was necessary for an acceptable project Zero learning - No optimization of the alternatives in the CBA in Alternatives were too similar in the CBA in Other factors The reframing of the project as a replacement of the old North Lock instead of developing a second lock was influenced by an interplay of several aspects outside of the CBA process. First of all, the port of Amsterdam flourished in the period and lobbied the ministry to have this growth acknowledged. A member of parliament also plead to reopen the investigation of the sea lock. The ministry then requested the DBEPA to analyze this real economic development in an intermediate review. The DBEPA advised that that the investigation should indeed be reopened. Furthermore, a technical study in 2008 concluded that the North Lock should be replaced in 2030, instead of in 2050, as was previously assumed. The real economic growth, the fiat of the DBEPA, the political pressure, and the technical necessity of replacing the lock in 2030 resulted in a shared sense of urgency and interest between the ministry, the regional department, and the Port of Amsterdam. As such, the representatives of these parties recalled that their cooperation changed; from then on they were actively looking for solutions together. An important next step was made by the representatives of the ministry. They lobbied internally to extend the time frame of the national funding program to 2030, which enabled the possibility for financing. Furthermore, a political commission plead to parliament for the possibility for regional parties to finance an accelerated development of national infrastructure projects. In light of these developments, all the elements were there to reach a political agreement in 2009 on the accelerated replacement of the North Lock by a bigger lock at a new location, whereby the regional parties would finance the accelerated and enlarged development and the ministry the replacement. The final CBA in 2012 confirmed that this already agreed upon alternative was indeed a worthwhile investment, although the B/C ra- tio was only minimally positive (Rijkswaterstaat Noord-Holland, 2012). According to the representative of the ministry, the CBA on its own would not have led to a positive decision about the investment. Finally, the representative of the DBEPA recalled that early in this case the Port of Amsterdam was too ambitious and asked for too much. To him, the second opinions helped to temper the eagerness to build more capacity. This was confirmed by the representatives of the ministry and the regional department, who mentioned that the second opinions of the DBEPA were very influential in the decision making. Furthermore, CBA evaluators tended to work more precisely and critically if they knew the assessment was going to be reviewed by the DBEPA. The evaluators in this case admitted that they sometimes used a potential critique of the DBEPA as an argument in the assessment process to convince the clients to be more sincere. For example, convincing the client to not use only very optimistic economic scenarios, but also a scenario with marginal or zero growth. Also, the consultant hired by the Port of Amsterdam used the DBEPA critique to further improve the CBA input Summing up Case 2 There was no deliberate communication strategy used in the IJmuiden sea lock case, other than guidance through a steering group in which the CBA evaluators only became involved late in the process and then did not participate very much. It seemed that sharing and discussing the plan and CBA together mainly took place through bilateral meetings. Furthermore, the analysis showed that the assessment process of the IJmuiden sea lock was characterized by differences in the perceived communication and trust mechanisms in the four assessment phases. Whereas distrust and anger dominated in the early phases, more cooperation was perceived in the third and fourth phases. Also, the DBEPA seemed more open for discussion and consultation in the latter two phases. According to the plan owners, this improvement of communication and trust over time was influenced by knowing more about the CBA methodology and the optimal design for the project through former assessments, and by reaching a shared sense of urgency in the third assessment phase. Furthermore, the sharing of ideas in the steering group between the Port of Amsterdam, the ministry, and the regional department in the third phase seemed important for reframing the project as a replacement, resulting in a political agreement. The communication and trust conditions were thus strongly influenced by the wider planning and decision-making processes

91 The reframing of the project as a replacement in the third phase was indicated as an important moment of deeper learning (learning through a CBA). This was partly influenced by the first three negative CBAs that made it clear that the project as proposed was not realistic. However, most important was reaching a shared sense of urgency to increase of the capacity of the sea lock. The representative of the Port of Amsterdam, for example, vividly recalled that from then on the cooperation was effective, resulting in a creative rearrangement of the project. This shared sense of urgency, though, was mainly caused by other factors, like the real economic growth, the political pressure, the approval of the DBEPA to continue the investigation of extending the capacity of the sea lock, and the proven limited technical lifespan of the North Lock. 5.6 Conclusion and reflection The aim of this chapter was to better understand if the communication and trust building interventions derived from theory improved communication and trust between plan owners and evaluators in the context of a CBA process, and through which mechanisms. Furthermore, the chapter aimed to explore how communication and trust influenced the use of the CBA for learning. For this, two contrasting cases were analyzed: one in which the interventions were likely present (Case 1, the CBA process of the Schiphol airport) and one in which they were likely not (Case 2, the CBA process of the IJmuiden sea lock). In this concluding Paragraph, the findings of the cases are compared and reflected upon Comparing the two cases The selection of the cases was based on the probable presence of the independent variable (the five interventions), for which guidance by a moderator was the key indicator. Case 1 indeed represented a situation with the communication and trust building interventions. In Case 2, the interventions were mostly absent, although elements of the interventions were detected. There was, however, no deliberative communication and trust building strategy. Case 2 therefore by and large represented a CBA process without the interventions. Upon analysis, a further difference between the cases emerged with respect to the involvement of local or residential actors. Whereas in the Schiphol case the residential actor was important in the assessment process, in the sea lock case a comparable party was absent. The analysis of the cases showed contrasting findings for their communication and trust conditions, that were largely corresponding with the expected influence of the five interventions (see Figure 5.1). The analysis of the Schiphol case signaled perceptions of a large variety of mostly positive communication and trust mechanisms, and outcomes of sharing of meaning. The IJmuiden sea lock showed more negative communication and trust mechanisms, and a lack of sharing of meaning in the early phases. Nevertheless, participants in the sea lock case appeared better able to communicate with and trust each other in the last two phases as the plan and assessment matured, and by reaching a shared sense of urgency in the third assessment phase. Furthermore, both cases showed several moments of learning large and small about the plan and the CBAs, through which the alternatives and assessments evolved during the process. Whereas the learning moments in the sea lock case seemed influenced by the succession of different CBAs, the participants in the Schiphol case seemed able to learn within the dialogue process of the same CBA. However, for the moments of deeper learning (through a CBA) in both cases, communication and trust seemed important conditions, notwithstanding how this communication and trust was achieved. The cases also showed that these moments of learning were strongly influenced by the wider planning and decision making processes, that is, by other factors. In the Schiphol case, the strong and politically supported position of the residents influenced the establishment of communication, trust, and learning. In the sea lock case, the shared sense of urgency supported the establishment of communication and trust, and a creative reframing of the assessed alternatives. The assessment, planning, and decision making processes thus seem strongly interwoven, especially when large changes in the alternatives and assessments were discussed Reflection This case study aimed for a context-rich understanding of the complexity of CBA processes and the influence of conditions for communication and trust (i.e., the five interventions) on learning between plan owners and evaluators. However, and in light of this complexity, the findings from these specific cases cannot be generalized as such, and are not free from bias. The interviewees were, for example, asked to look back at a CBA process that in actuality formed a small part of the planning process. Some had a hard time remembering what exactly was going on. The variety of interviewees, though, helped to reconstruct the perceived communication, trust, and learning in the CBA processes. Furthermore, the framework used (Table 5.1) ap

92 peared well-suited to capture personal experiences of communication and trust in retrospect, as the coded perceptions could easily be related to the abstract interventions and mechanisms. This analytical step was made transparent by giving a robust description of the perceived interventions and mechanisms in this chapter. The analysis of moments of learning, though, was more explorative and less detailed. A more in-depth understanding of learning in CBA processes could be valuable. Both cases, for example, point at the strong influence of other factors on deeper learning. A better understanding of how the wider planning and decision making processes influence deeper learning (or its absence) in CBA processes might therefore be necessary

93

94 Chapter 6: Overall conclusions and reflections