Predicting events from physiological data while driving COMP8715 Computing Project Lei Wang Supervisor: Tom Gedeon, Leana Copeland, Christopher Chow
Contents Introduction Motivation & Objectives Methodology Results and Discussion Conclusion Limitations and future work
Introduction The collection of valuable information on drivers behaviors contributes to the development of intelligent vehicular systems which is able to interpret and deal with different situations in traffic[1]. There are a large number of studies focusing on use of physiological data to monitor drivers behaviors during virtual-world and real-world driving situations under different conditions[1][2][3][4][5][6][7][8]. With the increasing use of wearable smart electronics, the application of collecting and using physiological data to estimate driving behavior becomes practical[9]. In previous study, it suggests that there are two kinds of physiological data, Electrocardiogram(ECG) and galvanic skin response (GSR), which can be used as objective measures in monitoring the reaction of non-phobic participants to virtual-world fear driving and flying, that is set by different situations, such as driving in different traffic congestion level and flying in different weather conditions[2].
Motivation & Objectives As the different event driving meet while driver meets on the road would cause different reaction on physiological data, this project presents a method for pre-processing and analyzing physiological data during virtual-world driving tasks to predicting the road event driver meets.
Methodology Collect data by conducting an experiment Video data, ECG,GSR data Pre-processing X = (ECG, GSR) Y= labels Data processing of event labels( human evaluation of video), ECG, and GSR X=(ECG,GSR) Y = labels Training and Testing Classifier Train classifier X =(ECG,GSR ) Classifier Y = labels Compare predictions of new data(ecg & GSR) with human evaluation of new data(video) Video data Y =labels Observing Comparing results
Methodology: Experiment Object: Collect the data for training and testing a road-event-prediction classifier. Previous Work: Hardware and software setup based on Lor and Le s work[10][11] Subjects: 11 young students in Australian National University Virtual world Setting: A city in normal traffic condition Task for each subject: Fill in the pre-experiment survey questionnaire to collect personal information Drive in virtual world for 10 minutes with physiological(gsr and ECG) sensors Fill in the post-experiment survey questionnaire to collect the feedback about this experiment. Data collected: Video data: 1) participants facial expression recording; 2) virtual world driving simulation; 3) real time GSR and ECG graph; 4)digital clock. Numerical data: 1) GSR datasets; 2) ECG datasets.
Methodology: Data Preprocessing Select useful subjects data Some subjects GSR data are keeping at maximum as right figure showing, since the temperature of skin is too low, or subjects have excessive swearing. Thus, we drop such subjects data. Finally, there are 8 subjects data for training and testing the classifier. Arbitrary[Arb] ECG value Arbitrary[Arb] GSR value 3.5 3 2.5 2 1.5 1 0.5 4 x 104 Subject 7: The Arbitrary[Arb] GSR data 0 0 500 1000 1500 2000 2500 3000 Sampling Index Subject 7: The Arbitrary[Arb] ECG data 5000 4000 3000 2000 1000 0 1000 0 500 1000 1500 2000 2500 3000 Sampling Index
Methodology: Data Preprocessing Clean GSR and ECG data The temporary loss of GSR data is happened to some subjects as right up figure showing, since GSR sensor lost contact with the electrodes due to sensors cables tangled and stuck into electronic steering wheel. Replace missing GSR data with mean of neighbors Arbitrary[Arb] GSR value Arbitrary[Arb] GSR value 3.5 3 2.5 2 1.5 1 0.5 0 0 500 1000 1500 2000 2500 3000 Sampling Index 3.5 3 2.5 2 1.5 1 0.5 4 x 104 Subject 4: The Arbitrary[Arb] GSR data 4 x 104 Subject 4: The Arbitrary[Arb] GSR data 0 0 500 1000 1500 2000 2500 3000 Sampling Index
Methodology: Data Preprocessing Filter GSR and ECG data with low-pass filter Eliminate the high frequency noises created by sensors and cables. A simple moving average low-pass filter with setting window size(w) equal to four is applied in this project. Normalize GSR and ECG data for each subject to the range of [-1, 1] eliminate the individual differences for subjects
Methodology: Data Preprocessing Generate features from GSR and ECG data Examples: They are generated by segmentation of GSR and ECG data in 4 seconds interval with 50% overlap. For each Example, it is a 4 seconds interval and the corresponding features are generated from this interval. Features: There are 27 features[13] generated in total. After forming a feature matrix with all examples, normalization is applied to this matrix in order to eliminate the individual differences for features. Quantity Features Data 8 Minimum, Maximum, Median, Interquartile range, Mean, Standard deviation, Variance, Root mean square GSR, Gradient GSR, ECG 3 Energy Ratio, Low Frequency(LF 0.05HZ 0.15 HZ), High Frequency(HF 0.16HZ 0.4HZ) ECG
Methodology: Data Preprocessing Label examples with event tags Define Event tags as following figure showing Data synchronization: Matching start time of video data with GSR and ECG data with considering the delay of GSR and ECG sensor. Identify event category and label with corresponding event tags Tag Sub-category Event category 0 Normal driving Normal driving 1 Other objects Near hit stationary object 2 related Hit stationary object 3 Near hit car Car related 4 Hit car 5 Pedestrian Near hit pedestrian 6 related Hit pedestrian
Methodology: Classifier Train classifier with Extreme Learning Machine(ELM) algorithm[14] ELM works for single-hidden-layer feedforward networks There are three layers: Input layer, hidden layer, and output layer Input layer: The number of input neurons is equal to the number of features Hidden layer: The number of hidden neurons is selected with suggested number for ELM, such as 400[12] Output layer: The number of output neurons is equal to the number of event category Validation of classifier K-fold cross validation with setting K = 10 Select the best performance classifier according to the correct rate
Results and Discussion For seven categories Average correct rage = 67.2% Average Recall = 91.5% Average Precise = 75.8% F1-score = 82.9% Tag Event category 0 Normal driving 1 Near hit stationary object 2 Hit stationary object 3 Near hit car 4 Hit car 5 Near hit pedestrian 6 Hit pedestrian Counting Table Classify results True tag 0 1 2 3 4 5 6 0 1456 124 110 178 48 4 2 1 30 22 18 23 4 4 0 2 36 24 32 18 6 0 0 3 63 30 26 87 8 2 0 4 0 0 0 1 0 0 0 5 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0
Results and Discussion For six categories Average correct rage = 39.3% Average Recall = 39.0% Average Precise = 37.0% F1-score = 38.0% Tag Event category 0 Normal driving 1 Near hit stationary object 2 Hit stationary object 3 Near hit car 4 Hit car 5 Near hit pedestrian 6 Hit pedestrian Counting Table Classify results 0 True tag 0 1 2 3 4 5 6 1 78 73 4 42 14 4 2 64 143 4 62 16 0 3 2 3 1 0 2 0 4 45 65 0 71 23 2 5 11 29 2 23 22 0 6 0 0 0 0 0 0
Results and Discussion For four categories Average correct rage = 70.7% Average Recall = 90.7% Average Precise = 78.5% F1-score = 83.7% Tag Event category Sub-Event category 0 Normal driving Normal driving 1 2 3 Other objects related Car related Pedestrian related Near hit stationary object Hit stationary object Near hit car Hit car Near hit pedestrian Hit pedestrian Counting Table Classify results True tag 0 1 2 3 0 1443 188 203 5 1 90 136 74 3 2 58 73 113 5 3 0 1 0 0
Results and Discussion For three categories Average correct rage = 76.1% Average Recall = 91.8% Average Precise = 80.8% F1-score = 85.9% Tag Event category Sub-Event category 0 Nothing Normal driving 1 2 Incident Accident Near hit stationary object Hit stationary object Near hit car Near hit pedestrian Hit car Hit pedestrian Counting Table Classify results True tag 0 1 2 0 1644 338 53 1 139 169 19 2 8 15 7
Results and Discussion For two categories Average correct rage = 82.6.1% Average Recall = 93.9% Average Precise = 86.4% F1-score = 90.0% Compare predictions of new data(ecg & GSR) with human evaluation of new data(video) It has been found that when predicted serious incident happens most subjects have reaction on their face Tag Event category Sub-Event category 0 Minor incident 1 Serious incident Normal driving Near hit stationary object Hit stationary object Near hit car Hit car Near hit pedestrian Hit pedestrian Counting Table Classify results True tag 0 1 0 1867 295 1 122 108
Results and Discussion In summary The GSR and ECG datasets can only provide limited features to describe a unique event leads to that it is hard to distinguish every event precisely and that the classifier performance is not good enough, but it is practical to distinguish general classes where some events are sharing similar features and are considered belonging to same class There are many examples that represent normal driving period, but there are a bit of examples that represent hit pedestrian. Thus, the classifier trained with imbalanced datasets can well distinguish the class who has amount of examples but cannot well distinguish the class who has a bit of examples. Even the classifier can have high average correct rage or F1 score for all classes but it cannot reach high correct rage for every class.
Conclusion The aim of this report is to build a road-events-prediction classifier by using physiological data collected from young participants while they are using driving simulator. A driving simulator experiment has been designed and conducted to collect 11 young participants physiological datasets and video datasets while they undertake a task of using driving simulator to mock driving in the city. The classifier performance not only suffers from inappropriate division of events into several categories, but also suffers from imbalanced class distributions in datasets. Besides, most subjects have reaction on their face when predicted serious incident happens.
Limitations and future work It is hard to control the speed via driving simulator Improve the control performance Less features to describe a unique event Generate features by combination data of other physiological data like brain activity and physical data like eye gaze, and blink rates. Incorrect event annotation which results from personal subjective judgment and lack of an integrated event definition investigate traffic related research to get an integrated event definition Imbalanced datasets Exploring specific approaches to cope with imbalanced
Q&A
Reference [1] Malta, Lucas, et al. Multi-modal real-world driving data collection, transcription, and integration using Bayesian network. Intelligent Vehicles Symposium, 2008 IEEE. IEEE, 2008. [2] Jang, Dong P., et al. Analysis of physiological response to two virtual environments: driving and flying simulation. CyberPsychology & Behavior 5.1 (2002): 11-18. [3] Lisetti, Christine Lætitia, and Fatma Nasoz. 279.Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP journal on applied signal processing 2004 (2004): 1672-1687. [4] Healey, Jennifer A., and Rosalind W. Picard. 481.Detecting stress during real-world driving tasks using physiological sensors. Intelligent Transportation Systems, IEEE Transactions on 6.2 (2005): 156-166. [5] Anstey, Kaarin J., et al. 323.Cognitive, sensory and physical factors enabling driving safety in older adults. Clinical psychology review 25.1 (2005): 45-65. [6] Collet, C., et al. Physiological and behavioural changes associated to the management of secondary tasks while driving. Applied ergonomics 40.6 (2009): 1041-1046. [7] Wang, Jeen-Shin, et al. Driving Conditions Recognition Using Heart Rate Variability Indexes. Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), 2010 Sixth International Conference on. IEEE, 2010. [8] Singh, Rajiv Ranjan, Sailesh Conjeti, and Rahul Banerjee. An approach for real-time stress-trend detection using physiological signals in wearable computing systems for automotive drivers. Intelligent Transportation Systems (ITSC), 2011 14th International IEEE Conference on. IEEE, 2011. [9] Eren, Haluk, et al. Estimating driving behavior by a smartphone. Intelligent Vehicles Symposium (IV), 2012 IEEE. IEEE, 2012. [10] Lor, W.). Operator observation infrastructure: hardware and management. Research School of Computer Science, Australian National University, 2013. [11] Le, P.. Software and data infrastructure. Research School of Computer Science, Australian National University,2013 [12] Zhu.X.Using physiological feedback to reduce observer stress. Research School of Computer Science, Australian National University,2014 [13] Sharma, N. & Gedeon, T. Artificial neural network classification models for stress in reading. Proceeding of 19th International Conference, ICONIP 2012, 407 414. [14] Huang G B, Zhu Q Y, Siew C K. Extreme learning machine: theory and applications[j]. Neurocomputing, 2006, 70(1): 489-501.