Part.1
introductory
Industry 5.0 is centered on a new engineering approach that prioritizes solutions for human needs, emphasizing the critical role of humans in the system. In this context, human cybernetic systems require a direct and reliable understanding of the human state, both physically and psychologically. This is particularly important for assessing operator performance under conditions of intense thermal stress, high mental load, and heavy fatigue.
There have been significant developments in neurophysiological monitoring, such as electroencephalography (EEG), functional near infrared spectroscopy (fNIRS), electrocardiography (ECG), and electrodermal (EDA). These biosignals reflect fundamental aspects of the human state, including brain activity, energy metabolism, arousal levels, muscle fatigue, and more. Emerging and effective technologies are available for real-time assessment of human state in operational environments.
With the rapid advancement of machine learning and the progress of computer memory and processing speed, large-parameter deep learning networks built on artificial neural networks (ANNs) have demonstrated higher performance in feature learning and classification tasks. In this paper, a performance evaluation model (ECG-fNIRS) based on multi-physiological data is developed and a deep learning architecture is explored to extract effective discriminative features. Time-frequency spectra and prefrontal cortex (PFC) networks are acquired by processing ECG and fNIRS signals, respectively, to obtain better evaluation results.
Part.2
Research Methodology
Figure 1 shows the performance assessment model, which consists of five components: data acquisition and processing, ECG manual feature extraction to obtain ECG vectors, fNIRS manual feature extraction to obtain PFC brain networks, deep discriminative feature extraction, and model fusion and classification. Finally, various methods are applied for visualization of results and performance evaluation. For ECG data, filtering was performed to analyze the time and frequency domains, and ECG manual feature vectors were extracted, including the amplitude and width of typical waves and the power spectral density (PSD) in specific frequency bands. In addition, ECG signals were subjected to short-time Fourier transform (STFT) to obtain the time-frequency spectrum. Finally, spectral learning vectors were extracted from each time-frequency spectrum by a 4-layer CNN (convolutional neural network). For fNIRS data, general linear model (GLM) and functional connectivity (FC) analyses were performed after filtering to obtain the beta value (brain region activation) and FC strength (synchronization between channels). fNIRS PFC brain networks consisted of beta value and FC strength as node features and edge features, respectively. fNIRS PFC brain network learning vectors were extracted from each fNIRS PFC brain network by 2-layer GAT (Fig. 2). extracted through a 2-layer GAT (Graph Attention Network). Finally, performance categories are predicted by a 3-categorization dense layer based on the connection of ECG manual feature vectors, spectral learning vectors and PFC brain network learning vectors.
Part 2.1
data acquisition
The chapter focuses on summarizing and synthesizing the role of EEG in assessing indoor light environments, and its highlights the validity of EEG in assessing indoor light environment conditions, particularly with regard to how spectrum, illuminance, color temperature, and lighting patterns affect brain activity.
Experiments were conducted with nuclear power plant (NPP) operators (trained volunteers) in four humid heat exposure environments (25°C 60% RH, 30°C 70% RH, 35°C 80% RH, and 40°C 90% RH) to examine their performance and the effects of typical cognitive and executive functions. In order to assess operators' overall performance under heat stress, the performance metrics selected in this experiment included primary NPP task performance and basic cognitive function:
Major task performance: According to the NPP Operator Cognitive Model, detection and attention, understanding and perception, decision making, action, and teamwork are the five basic cognitive and executive functions required by operators. A fault detection task and a reactor core shutdown operation task were conducted to cover these basic functions.
Situational Awareness (SA): SA reflects the operator's perception, understanding, and prediction of the current state and future projections of critical systems, primarily through the Situational Awareness Global Assessment Technology (SAGAT).
Workload (WL): WL includes both mental and physical workloads, which are usually U-shaped correlations with performance and are primarily assessed through the NASA Task Load Index Scale (NASA-TLX).
Working Memory (WM): operators have to deal with huge streams of information and retain important information in stressful situations, which is an important cognitive function called working memory. It is assessed primarily through N-back tasks.
Each scenario is shown in Figure 2 below, and duration, accuracy, reaction time, and error counts were recorded for the experiment. The performance of the subjects was categorized into three categories based on their performance in different scenarios:
Category 1: Good performance in a suitable workplace (25°C 60% RH and 30°C 70% RH)
Category 2: Reduced performance in a sleepy state in a higher temperature and humidity environment (35°C 80% RH).
Category 3: At the highest temperatures and humidity (40°C 90% RH), there is a temporary slight improvement in simple response tasks, but higher cognitive functions are rapidly impaired.
Thirty subjects (16 females and 14 males), aged between 20-39 years, participated in the experiment. Data from 4 of the subjects were excluded due to incomplete recording. A total of 515 ECG data and fNIRS data were finally obtained.
Part 2.2
ECG manual features
ECG signals were recorded by a 3-lead ECG module.
Time domain features: signal filtering and QRS wave group detection to extract RR intervals of the sequence. Various features are extracted based on typical wave amplitudes, time widths, and corresponding statistical amplitudes.
Frequency domain characterization: Fast Fourier Transform (FFT) of the filtered data to compute PSDs for specific frequency bands
Part 2.3
ECG time spectrum
ECG is a non-stationary signal. Therefore, STFT can be applied to ECG data to observe the change of frequency components and energy distribution over time. In order to obtain the input of CNN, the spectrum is adjusted to 64*64 pixels as shown in Fig. 3.
Part 2.4
fNIRS characteristics
The fNIRS data were recorded by 8 channels placed in the prefrontal lobes.
Beta activation: after data preprocessing, the beta value of oxygenated hemoglobin (HBO) was calculated by GLM. For the input, the beta value is converted to a 1*24 vector (3 tasks * 8 channels).
FC: For resting-state data, the correlation coefficients between channels, correlation coefficients after fisher-Z transform, coherence and phase-locked values are calculated to obtain the FC matrix (8*8 symmetric matrix) and flatten it into 1*112 vectors (4 FC matrices * 28 channel combinations).
Part 2.5
PFC brain network
Based on the small-world property of the brain functional network, the fNIRS signal is constructed as the fNIRS PFC brain network, as shown in Figure 4. In the PFC network, the fNIRS channels are nodes, and the connectivity between the channels is the edges, the beta values activated in the fNIRS channels are the node features, and the functional connections between the channels are the edge features.
Part3
Deep learning models and methods
In order to classify performance in different scenarios using physiological data, a CNN (Model A) is employed on the time-frequency spectrum. In order to improve the accuracy of the model, four models are developed in this paper to explore effective architectures. The architectures of the four models are shown in detail in Fig. 5 and Fig. 6.
Part 3.1
Graph Attention Network (GAT)
The fNIRS can illustrate the executive function network of a target brain region, with channels as nodes and connections between channels as edges forming an undirected graph. Typically, GLM is used for task-state data, calculating each channel activation as a node feature. Functional connectivity from resting-state data is obtained and synchronization between channels is computed as edges.
Part 3.2
Architecture of the proposed model
Four models were constructed, Model A, Model B, Model C and Model D. The specific input parameters, as shown in Table 1 below:
Part 4
Results
The performance of the four different models is shown in Table 2 below.In the test dataset, the accuracy of Model D is 81.821 TP3T, while the accuracy of Models A, B, and C are 69.321 TP3T, 70.451 TP3T, and 68.181 TP3T, respectively.In addition, the loss curves and accuracy curves of the four models for the 5-cross-validation are plotted as shown in Figure 7 below. Models A and B converge to the local optimal solution faster than models C and D in both accuracy and loss curves. However, their loss in the validation dataset reaches a relatively low level that increases after a few epochs, indicating overfitting. In contrast, Models C and D show better generalization ability when their remains decrease in the validation dataset and increase after one epoch.
Part 5
Conclusion
In this paper, we propose a CNN-GAT fusion model for performance evaluation driven by ECG temporal spectra and fNIRS prefrontal brain network to identify abnormal work states under extreme temperature and humidity environments. A multi-physiological fusion model for performance classification under heat stress based on ECG signals and fNIRS signals demonstrated high ecological validity. The method overcomes the subjectivity of expert observation and the hysteresis of post-testing, providing potential neuro-ergonomic applications for human state assessment in important human cybernetic systems in Industry 5.0 scenarios.
Part 6
literatures
Cortivision Wireless Portable Near Infrared Optical Brain Imaging System
Company Profile
English 
Chinese 
