Christian Roesener Adrian Zlocki A Comprehensive Evaluation Methodology for Automated Driving Final Event Aachen, Germany 29 June 2017
parameter x // Bottleneck for the Introduction of Automated Driving? ACC (SAE Level 1) Urban Robot Taxi (SAE Level 4) Source: Audi (2013) Motorway Automation (SAE Level 3) parameter z Source: Google (2015) [ ]. If testing and assessment methods cannot keep pace with this functional growth, they will become the bottleneck of the introduction of advanced DAS to the market. ( Three Decades of Driver Assistance Systems, UNI-DAS, IEEE ITS Magazine, 2014). Source: BMW (2015) 2 // 29 June 2017 AdaptIVe Final Event, Aachen
// Evaluation of AdaptIVe functions // Real-traffic Impact Assessment // User-Related Assessment // // Test track Technical Assessment // In-Traffic Behaviour Assessment // // Simulations Obstacle 3 // 29 June 2017 AdaptIVe Final Event, Aachen
// Evaluation Approach in AdaptIVe Function / System // Classification // Operation time Level of automation Focus of Evaluation Evaluation // What should be assessed? (depending on classification) Research Questions Hypotheses Indicators Test Methods // How should it be assessed? (depending on classification) Test environment Test tools Test amount Evaluation // User-related Technical In traffic Impact Assessment // Safety Environment 4 // 29 June 2017 AdaptIVe Final Event, Aachen
// Definitions for Evaluation Traffic Scenario: A traffic scenario describes a larger traffic context, which includes different (not predefined) driving scenarios. Driving Scenario: A driving scenario is the abstraction and the general description of a driving situation without any specification of the parameters of the driving situation. Ego vehicle speed Driving Situation: A driving situation is a specific driving manoeuvre (e.g. a concrete lane change with defined parameters). 5 // 29 June 2017 AdaptIVe Final Event, Aachen
// Classification of Automated Driving Functions Classification by operation time: Event based operating Function that is only active for a short period in time (typically vehicle stands still at the end or the automated driving ends) Examples: Parking, Minimum Risk Manoeuvres Continuously operating Function that is active for a longer period in time (typically vehicle is still moving at the end of an manoeuvre respectively automated driving is continued) Example: Highway Pilot 6 // 29 June 2017 AdaptIVe Final Event, Aachen
//How to limit the test amount? Different approaches for eventbased and continuously operation functions: Event-based functions: similar approaches as in previous research project e.g. interactive Continuously operating functions: small field test on public road in order to assess the function in many different situations Number of variations per situation Event-based functions Resources for testing Ideal Solution Continuous functions Number of investigated situations 7 // 29 June 2017 AdaptIVe Final Event, Aachen
// Evaluation Tools Which tool should be applied for which type of assessment? Tool Field Operational Test Application Impact assessment in reality Assessment of behaviour/components/systems R R R Assessment of components and systems Controlled Field R R R \ V Assessment of driver behaviour Dynamic Driving Simulator Assessment of driver behaviour Human machine interaction R V V Virtual layout and assessment Simulation V V V Potential impact assessment R: real, V: virtual 8 // 29 June 2017 AdaptIVe Final Event, Aachen
// Evaluation Tools in AdaptIVe Identification of an appropriate evaluation methodology for the technical, user-related, in-traffic behaviour and impact assessment Tool Technical User-related In-traffic Impact Field Operational Test Yes Continously Yes (Yes) No R R R Yes Event-based Yes No No Controlled Field R R R \ V Dynamic Driving Simulator No Yes No No R V V No No Yes Yes Simulation V V V 9 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment Event-based// 1. Defining evaluation scope Definition of research questions, hypotheses & indicators 2. Planning of assessment Analyse system description and adaption of hypotheses Planning of test cases (Risk assessment) 3. Tests in controlled field Number of test variations Logging of test data 4. Assessment of tests Analysis of hypotheses based on test data & indicators Continuous// 1. Defining evaluation scope Definition of research questions, hypotheses & indicators 2. Planning of assessment Analyse system description and adaption of hypotheses Planning of test cases and test route Definition evaluation criteria (distributions & boundaries) Risk assessment 3. Pre-/component tests in controlled field Basic tests of functionality Sensor tests 4. Tests in real traffic Test route and test amount to be determined 5. Assessment of tests Analysis of hypotheses based on test data & indicators 10 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment Event-based 11 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment // Parking For assessment of event-based automated driving functions, classical use-case based testing was conducted. Five repetitions per test case were conducted. Tests executed in a closed environment. 12 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment Close distance functions can be evaluated by classical use-case based testing. It turned out that close distance functions are providing accurate positioning in parking applications. // Parking Standard deviations Long: 0.13 m Lat: 0.02 m Angle: 1.80 Small variations in automated parking positioning mean: 3.31 m std: 0.90 m Safe distances to pedestrians were kept in all test cases. Distances to pedestrians in mean more than 3 m 13 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment Continuous operating 14 // 29 June 2017 AdaptIVe Final Event, Aachen
parameter x parameter x // Scenario Classification of Real-world data S 1 e.g. cut-in of other vehicle S 3 e.g. approaching FOT data v S 2 e.g. lane change v 15 // 29 June 2017 AdaptIVe Final Event, Aachen
// Scenario Classification of Real-world data Benmimoun (2011) Offline classification Uses decision trees parameterized by hand No easy adaptation, no consideration of time series Reichel (2010), Roesener (2016) Proficient using of Machine Learning Techniques Partial automated Choice of classifier based on expert knowledge Machine learning techniques provide an efficent & automated data clustering (a) Training label (b) Prediction Characteristic Extraction Characteristic Extraction As far as results not sufficient: Learning manual adaptation of classifier Characteristic Characteristic Machine Classifier Result Label Label Reichel (2010), Benmimoun Roesener (2016) (2011) 16 // 29 June 2017 AdaptIVe Final Event, Aachen
// Baseline for Assessment of Automated Driving Description of the baseline for the evaluation Objectives of automated driving functions Objective is a collision free traffic Operation in mixed traffic conditions ( not disturbing normal traffic) The functions have to be operated within range of normal driver behaviour What is normal driving behaviour? 0.016 0.014 0.012 Proportion [-] 0.01 0.008 0.006 0.004 0.002 0-50 0 50 100 150 200 250 long. Distance [m] 17 // 29 June 2017 AdaptIVe Final Event, Aachen
Lateral Acceleration [m/s²] // Baseline for Assessment of Automated Driving Analysis of eurofot data Plot displayed: Acceleration during normal driving Motorway Frequency [km -1 ] 100 10 1 0,1 Data from 98 vehicles 0,01 0,001 Motorway, rural roads and urban roads 0,0001 0,00001 0,000001 Longitudinal Acceleration [m/s²] 18 // 29 June 2017 AdaptIVe Final Event, Aachen
// Scenario-based Assessment of Automated Driving Data Source// Classification of Scenarios// Scenario-based Assessment// AdaptIVe demonstrator Reference: eurofot Human driving as a baseline Classification of Scenarios// Classifier Scenario 1 Classifier Scenario 2 Classifier Scenario x Classification of scenarios by using time series classification algorithms (Hidden Markov Models) PIs Assessment// Calculation of: Derived Measures Performance Indicators Assessment of frequency and effect induced by system in scenario Frequency (Scenario) Effect (Scenario) 19 // 29 June 2017 AdaptIVe Final Event, Aachen
//Scenario Classification Scenario Lane Change 20 // 29 June 2017 AdaptIVe Final Event, Aachen
// Technical Assessment - Results // Highway Human driving as a baseline Overlap The AdaptIVe Highway-Chauffeur is showing a control capability similar to human driving from eurofot. Two results stand out: Top figure: duration of lane change is much more uniform with automation Bottom figure: time headway in vehicle following shows much less variability with automation 22 // 29 June 2017 AdaptIVe Final Event, Aachen
Frequnecy of occurence [km-1] // Application of Method Frequencies // Highway 0,5 0,4 0,3 0,2 0,1 0 Lane change Small increase of lane change scenarios Cut-in of other vehicle More cut-in of other vehicle scenarios with automation eurofot AdaptIVe Highway Automation 23 // 29 June 2017 AdaptIVe Final Event, Aachen
//Summary The baseline for assessment of automated driving should be human driving behaviour AD Ref Automated driving functions are showing less variability in driving behaviour (headway keeping, lane changing) compared to human driving. Automated driving is leading to a change in frequency of occurence of relevant scenarios due a different driving behaviour compared to humans. 24 // 29 June 2017 AdaptIVe Final Event, Aachen
// Deliverable D7.2 Methodology and Results are provided in Deliverable D7.2 Application of AdaptIVe Evaluation Methodology Many thanks to all, who have contributed to the assessments: András Várhélyi, Erwin de Gelder, Jan Sauerbier, Felix Fahrenkrog and Pablo Mejuto 25 // 29 June 2017 AdaptIVe Final Event, Aachen
Christian Roesener Institut für Kraftfahrzeuge, RWTH Aachen University Email: roesener@ika.rwth-aachen.de Thank you.