Sharing | EEG-Based Assessment of Driver Trust in Autonomous Vehicles

Autonomous vehicles (AVs) have the potential to reduce road traffic accidents, improve traffic efficiency and reduce fuel consumption. In addition, they promise to provide greater mobility for those who are unable to drive due to age or disability. However, a key issue faced is establishing and calibrating driver trust in this technology, as trust plays an important role in determining whether or how a driver makes an AV.

Low trust is reported to be a major factor in discouraging drivers from embracing AVs. The study found that mistrust was the main factor contributing to the low willingness to use AVs. In addition, excessive trust may lead to misuse and abuse of the technology, and may also compromise driver readiness to regain manual control, leading to longer reaction times and increased accident risk. To ensure the safe introduction of AVs into road traffic, it is crucial to develop AVs that can dynamically adapt to the driver's task level in real time, which requires the use of reliable human trust assessment models. The aim of this study is to develop an EEG-based trust assessment model through the use of machine learning algorithms that utilize EEG signals captured by the driver while interacting with a simulated autopilot system that may be malfunctioning.

Related work

Preliminary work has revealed various neural correlates of human trust in automation. Compared to the distrust situation, the trust situation is characterized by stronger Alpha and Beta waves and weaker Gamma waves; the power of Theta waves (6-10 Hz) in frontal regions decreases with decreasing levels of trust; and the power of Delta and Gamma waves correlates significantly with the level of trust, with frontal and occipital lobes being the two main brain regions that are sensitive to changes in trust. In addition, recent evidence suggests that patterns of connectivity between different brain regions and event-related potentials (ERPs) are also sensitive to changes in trust. The above studies reveal significant correlations between behavioral and physiological characteristics and trust levels, providing a theoretical basis for the development of trust assessment models.

The following figure summarizes the proposed research framework. Specifically, a simulated driving task was conducted during which different types of AV failures were introduced to set the driver's trust level. Both subjective trust ratings and objective EEG signals were recorded. The raw EEG signals were preprocessed, and then time-domain and frequency-domain EEG features were extracted for each of the four regions: frontal, temporal, occipital and parietal lobes.

Driving Experiment

A total of 71 college students (36 males), with a mean age of 22.6 years (SD = 1.6), participated in the driving experiment . All subjects had a valid driver's license and normal or corrected-to-normal vision. Subjects were randomly assigned to one of three experimental conditions: normal, miss, and false alarms, with 24 (12 men and 12 women) in each condition.

The figure below outlines the driving simulator and EEG recording equipment used in the experiment. The driving simulator consists of three 27-inch LED display flats, a high-performance computer, and other components. Driving data such as speed, acceleration, and lane position were recorded at 60 Hz. The simulator can be run in manual or automatic mode. The EEG consists of 32 channels and records signals at a frequency of 500 Hz, and the contact impedance between the electrodes and the subject is kept below 20 kΩ.

The experiment utilized a 2 (gender) x 3 (experimental condition) between-subjects design. The three experimental conditions were: normal, miss, and false alarm.

The driving scenario was a 95-kilometer-long road in both directions with three lanes in each direction, simulating city driving. The subject is required to drive in the center lane unless a TOR (Take Over Request) is handled. The TOR is triggered when the autopilot system is unable to handle the danger and the subject needs to be put in control of the vehicle. Eight TOR scenarios (4 hazards × 2 levels of road curvature) were developed. the 4 hazards were: broken cars, work zones, crosswalks, and landslides; and the 2 road curvatures were a straight road surface and a sharp turn with a radius of 200 m. The TOR scenarios were developed to simulate the driving conditions in the city.

The subjects initially drove in autopilot mode, with the system maintaining 80 km/h.constant speed at the time. Subjects could perform the NDRT (non-driving related task) at their own pace. The TOR was triggered when the subject's vehicle was approximately 10 seconds away from the hazard (~278 m).The TOR consisted of a 75 dB sinusoidal sound (2010 Hz, duration 0.47 s) and a red text cue displayed in the lower area of the windshield. At the onset of the TOR, subjects were required to press the X button on the steering wheel to switch to manual driving and deal with the hazard. After passing the hazard and continuing to drive for 1 km, subjects were reminded to switch back to automatic driving, and then, after 1 km of driving, task ratings (on a scale of 0 to 100) were provided verbally.

NDRT:The NDRT used in this paper was a typing task that required subjects to accurately type Chinese characters displayed on a tablet computer. The tablet was located on the right side of the steering wheel, simulating the location of the center control panel in an actual vehicle. Therefore, subjects were required to divert their visual attention completely away from the road while performing the NDRT.

Upon arrival at the laboratory, the subjects were informed about the purpose and procedures of the experiment. Then, an informed consent form was signed. Next, the NDRT was practiced and the operation of the driving simulator was familiarized. Finally, the subjects were given an EEG device to wear and the formal driving phase began. Throughout the experiment, the subjects were asked to comply with road traffic regulations and duty codes. The whole duration event of the experiment was about 60 minutes.

data processing

1. Use FIR bandpass filtering (1-30Hz) to remove noise;

2. The signal will be re-referenced to an average reference;

3. Use the FASTER toolbox to identify and mark bad guides, then use spherical interpolation to remove them;

4. Independent principal component analysis (ICA) was performed to decompose the EEG signal, and the ADJUST toolbox was applied to identify and remove artifacts.

The EEG signals recorded at a distance of 1Km before the trust rating, corresponding to a duration of about 45s, are used for feature extraction, and model development. Each scene is extracted for about 5 cycles.

Verbal scores in the range of [0,60] are labeled as "low trust", [60,80] as "medium trust", and [80,100] as "high trust". ".

Six time-domain features were calculated for the EEG signal, including mean, maximum, minimum, standard deviation, skewness and kurtosis. To calculate the frequency domain features, the EEG signal was divided into four bands: delta (1-4Hz), theta (4-8Hz), alpha (8-13Hz) and beta (13-30hz). The amplitude and power spectral density (PSD) of each band were then calculated using FFT and Welch algorithms, respectively.

The samples were allocated as training and test sets in a ratio of 70-30. To construct the trust assessment model, nine classifiers were used: dt, KNN, SVM, RF, adaptive boosting (AdaBoost), gradient boosting decision tree (GBDT), XGBoost, LightGBM, multilayer perceptron (MLP).

In order to fully evaluate the performance of the model under unbalanced multiclassification problems, four metrics derived from the confusion matrix species, i.e., accuracy, precision, recall, and F1 score, are calculated. The four metrics expressions are given below:

Results

Whole-brain based trust assessment model

Using the FEF method, a total of 300 features were selected from the whole-brain regional feature set species to build a trust assessment model.

A model for trust assessment based on individual brain regions

The 32 electrodes were divided into 4 brain regions as shown in Figure 4 below.The REF method selected 74, 72, 40 and 86 features from frontal, temporal, occipital and parietal regions respectively.

These features were used to develop a trust model with 9 classifiers.

Comparing brain region performance in trust models

The figure below shows the average performance metrics for the 9 classifiers for the entire brain region. The whole brain overall showed better performance compared to individual brain regions. Frontal and parietal regions performed significantly better than occipital regions for all 4 performance metrics. Frontal regions outperformed temporal regions in accuracy (p = 0.044).

Confusion Matrix and Characteristic Importance Analysis

Based on the above results, the LightGBM model using whole-brain features was used. First, the confusion matrix of the model was pushed to. The results showed that the model effectively learned the categorization boundaries between low and medium trust and between ground trust and high confidence.

The SHAP method was used to calculate the feature importance values. The top 10 features that contribute the most to the model are given in the following figures respectively. Five of the eight PSD features are from the beta wave, which suggests that PSD of the beta wave has a key role in trust assessment. Most of the 10 belong to the parietal lobe from the brain region perspective.