\section{Motivation}
The main motivation of conducting this experiment at Eurecat's Augmented Workplace is to see how emotions affect students' learning outcomes. The hypothesis is that positive emotions promote good learning outcomes, while negative emotions could be a reason for bad results. From the common sense perspective, this hypothesis, sounds reasonable. The outcome of this study is to formally prove the hypothesis. Also, to the best of our knowledge there is no suitable dataset available in the emotion research domain which is related to emotional state analysis in the real life E-learning or education context.

\section{Experimental Protocol}
\subsection{Experimental environment}
The experiment is conducted at a classroom of Eurecat Academy, Sabadell, Barcelona. The classroom was a regular classroom with fully equipped machineries and other instruments. The picture (Fig.~\ref{fig:classroom}) is the actual view of the classroom where the experiment was conducted.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\textwidth]{class-room2.png}
    \caption{The classroom where the experiment was conducted (Eurecat Academy, Sabadell, Barcelona).}\label{fig:classroom}
\end{figure}

\subsection{Learning exercise}
The stimuli used in this experiment are the exams for the course taught by the teacher at Eurecat Academy, Sabadell, Barcelona in order to elicit the emotional reactions to the students in a class room scenario and record Electrodermal Activity (EDA) and Photoplethysmography (PPG) data. The exam was designed by the tutor of that course and it was divided into smaller exercises (see Annex~\ref{app:exam} for the examination formulation). 

Each exercise duration was 5 minute slots and there were a total of 12 sets of exercises, so in total the whole exam session was 84 minutes. In between each of the 12 sets we gave 2 minutes gap to for the students to “cool down”.  Also, students were advised that their annotations would not affect their evaluations to avoid any possible bias. In each of the 12 sets, from exercise Set 1 to Set 7; and 11 and 12 had 4 questions in each. Exercise Set 8 to 10 had one subjective question in each. The exercises sets are added in the additional materials for further reference. 

Before starting the exam we instructed the students the whole exam pattern. In which students were asked to give their current emotional states that they had felt after answering each questions. For the emotion states input from students we used 6 basic discrete emotion model~\cite{ekman1992}\footnote{\url{https://www.paulekman.com/universal-emotions/}} developed by Paul Ekman, widely accepted theory of basic emotions and their expressions. These emotion labels are as follows:
\begin{enumerate}
    \item {Sadness}
    \item {Happiness}
    \item {Fear}
    \item {Anger}
    \item {Surprise}
    \item {Disgust}
\end{enumerate}

\subsection{Students}
In the exam there were 10 students. The students were all male. Their age varies from 23 to 57 years. 

\subsection{Data Acquisition}

The sensors used to record the EDA and PPG data is Consensys Bundle Development kit from Shimmer Sensing. The pictures of the sensor is in Fig~\ref{fig:shimmer-sensor}. 

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.4\linewidth]{shimmer-sensor.png}
    \caption{Shimmer Sensors}
    \label{fig:shimmer-sensor}
\end{figure}

Before starting the exam we put the sensors on the students shown in the following picture (Fig.~\ref{fig:studenst-with-sensors}) for collecting the EDA and PPG data. The sensors were put on the non active hand of each student (such as if he/she is right handed we put the sensor on the left hand and so on). The whole experiment was carried out in 2 sessions, in each session there were 5 students.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.4\linewidth]{shimmer-wearable.png}
    \caption{Students wearing Shimmer sensors in the classroom while taking the exams.}
    \label{fig:studenst-with-sensors}
\end{figure}


The following picture (Fig.~\ref{fig:exam-scenario}) is from the class while doing the experiment. 
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\linewidth ]{class-room1.png}
    \caption{Students taking the exams with the senors wearing.} \label{fig:exam-scenario}
\end{figure}

The data collection from multiple Shimmer sensors were done using the Shimmer's Consensys software, in which all the data streams are recorded and saved. In the picture (Fig.~\ref{fig:software-interface}), the software interface while doing the experiment is presented. 
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\linewidth]{software-interface.png}
    \caption{Shimmer's Consensys software interface while collecting the data.}
    \label{fig:software-interface}
\end{figure}

\subsection{Data Formatting}
The data recorded in the Shimmer's Consensys software is exported into two format, such as \textit{.csv} and \textit{.mat}. In each file the data is saved in the following format (Fig.~\ref{fig:file-format}):
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.5\linewidth]{file-format.png}
    \caption{Default file format in Shimmer's Consensys software.}
    \label{fig:file-format}
\end{figure}

Each \textit{.mat} or \textit{.csv} file contains these channels which are available in the Shimmer sensor. From the filename shown in Fig.~\ref{fig:file-format}, we can see that Consensys software saves the file in the following format: \[ [Trial\; name/number, Session number, Sensor Unique ID, Calibrated/not calibrated, PC] \]

The sensor recordings can be accessed by providing the channel names as dot(.) operator, e.g. in Fig.~\ref{fig:data-read}.
\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\linewidth]{data-read.png}
    \caption{Reading data from the \textit{.mat} file.}
    \label{fig:data-read}
\end{figure}

The emotion labels were saved in .csv file with the file name which contains [session name + student ID + type of data (EDA/PPG)]. 

\subsection{Data Set}

For easy accessibility and better understandability we have extracted only the EDA and PPG data from the data recordings saved in Shimmer's Consensys software. The final datatset with the EDA and PPG recordings with the emotion labels are saved in .csv file. The folder structure of the MDEAW dataset is shown in Fig.~\ref{fig:MDEAW-folder-structure}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\linewidth]{folder-structure-MDEAW.png}
    \caption{MDEAW dataset folder structure and files inside it.}
    \label{fig:MDEAW-folder-structure}
\end{figure}
Inside each csv file the EDA and PPG recordings are organised as follows:
\[ [ Question no, Session No, Student ID, EDA/PPG data, Emotion labels ] \]

\noindent The MDEAW dataset will be made publicly available.

\noindent In Fig~\ref{fig:individual-student-emotion-class-labels}, the emotion class frequencies per student is presented and in Fig.~\ref{fig:emotion-class-all} the overall emotion class frequencies for the MDEAW dataset is presented. This figures shows the class distributions of the MDEAW dataset. 

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\linewidth]{emotion-frequency-per-students.png}
    \caption{Number of emotion class labels for individual student in MDEAW dataset.}
    \label{fig:individual-student-emotion-class-labels}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{emotion-frequency-all.png}
    \caption{Emotion class distribution in MDEAW dataset}
    \label{fig:emotion-class-all}
\end{figure}

\section{Analysis of the Data Set}
\subsection{Feature Extraction}
It's important to extract features from different physiological signals (such as EDA, PPG etc.) for retrieving relevant information, which effectively represents emotion states. Based on the extracted features the emotion classifier is tested and trained.

The most popular and widely used time-frequency analysis of various signals (especially EDA, PPG, etc.) for feature extraction is Wavelet Decomposition (WD)~\cite{islam2019}. Its popularity and wide use are due to its localized analysis approach(i,e. time-frequency), multi-rate filtering, and multi-scale zooming. It is better suited for non-stationary signals (such as EDA, PPG etc.)~\cite{ZHANG2020}. Most frequently used wavelet base functions are Meyer WD, Morlet Mother WD, Haar Mother WD and Daubechies WD~\cite{SUBASI2007}. The most frequently used features extracted from each sub-bands of EDA and PPG are entropy, median, mean, standard deviation, variance, 5th percentile value, 25th percentile value, 75th percentile value, 95th percentile value,  root means square value, zero crossing rate, mean crossing rate~\cite{bota2020,Ayata2020,Ayata2016}. 

In our experiment we have extracted and used these features for emotion classification. The wavelet feature extraction technique is used to extract features from multi-modal signal streams (EDA, and PPG signals from the dataset) in this experiment. The wavelet Daubechies 4 (Db4) is the base function for feature extraction. Our experiment decomposes EDA and RB, into three levels, respectively.

\subsubsection{Fusion of EDA and PPG-based features}
The use of features based on multiple modalities has been shown to provide increased classification accuracy compared to approaches based on a single modality. In order to evaluate the performance of the combined EDA and PPG-based features, the two feature vectors $F_{EDA}$ and $F_{PPG}$ are fused as follows: First, the values of each feature vector are normalised in the range [0, 1] in order to compensate for the differences in numerical range. Then, the two normalised feature vectors $F_{EDA}$ and $F_{PPG}$ are concatenated in the final feature vector $F_{fused}$ = [ $F_{EDA}$ $F_{PPG}$].


\section{Emotion Recognition Results for the Data Set}

For the emotion recognition results, we have used our previously developed Real-time Multimodal Emotion Classification System (ReMECS~\cite{remecs}) based on Feed-Forward Neural Network, trained in an online fashion using the Incremental Stochastic Gradient Descent algorithm. The avg. accuracy and F1-score along with the standard deviation (STD) are presented in TABLE~\ref{tab:result}. More results can be found in Appendix~\ref{app:further-results}.

\begin{table}[htbp]
\centering
\caption{Avg. accuracy and F1-score of real-time emotion classification from MDEAW multimodal dataset}\label{tab:result}
\begin{tabular}{|c|c|c|}
\hline
\textbf{\begin{tabular}[c]{@{}c@{}}Dataset\\ Name\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Avg. \\ Accuracy\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Avg.\\ F1-score\end{tabular}} \\ \hline
\textbf{MDEAW} & 0.9583 ($\pm$ 0.1998) & 0.9583 ($\pm$ 0.1998) \\\hline
\end{tabular}
\end{table}

Also, the confusion matrix is presented in Fig.~\ref{fig:confusion-matrix}. 
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.5\linewidth]{confusion-matrix-results-remecs.png}
    \caption{Confusion matrix showing the individual class accuracy for MDEAW using ReMECS}
    \label{fig:confusion-matrix}
\end{figure}

\section{A New Round of the Experiment}

We have planned a new round of the experiment, to extend the dataset. The reason for conducting another round of experiment to make the MDEAW dataset more diverse and also to increase number data samples by increasing number participants. The diversity can help each procedure to guarantee a total good machine learning: diversity of the training data ensures that the training data can provide more discriminative information for the model, diversity of the learned model (diversity in parameters of each model or diversity among different base models) makes each parameter/model capture unique or complement information and the diversity in inference can provide multiple choices each of which corresponds to a specific plausible local optimal result~\cite{diversity-ml-dl}.  Having more number of diverse data samples in the MDEAW dataset will help ML/DL models to generalize more so that the accuracy and efficiency of those intelligent emotion models increase.

In the new round of experiment we will follow the same design pattern followed for the first experiment for the EDA and PPG data collection from the students but the course and the subject may be different or same based on the availability. The reason for adding the term "based on availability" is because we do the experiment on real students with the real courses studied at our Eurecat Academy.

Right now, the dataset has 10 students data, which is rather small and if we train (offline mode training) any ML/DL model, there might be a chance that the model will be over-fitted easily. So, by increasing the data samples in the MDEAW dataset we can avoid the issue so that ML/DL models can learn and generalize properly and those trained models can be used for emotion classification from new EDA and PPG signals. Also, making the MDEAW dataset diverse, balanced so that the ML/DL model's accuracy can be good for emotion classification.

\section*{Acknowledgement}
\noindent The authors would like to thank the tutors and students at Eurecat - Centro Tecnológico de Cataluña for their participation and collaboration in the experimental study.

This study is carried out as part of the Ph.D work of the student Arijit Nandi and has been funded by ACCIÓ, Catalunya, Spain (Pla d’Actuació de Centres Tecnològics 2021) under the project TutorIA for three years, from December 2019 to December 2022.

\section*{Disclaimer}
\noindent While every care has been taken to ensure the accuracy of the data included in the MDEAW dataset,the authors and the Eurecat - Centro Tecnológico de Cataluña, Barcelona, Spain do not provide any guaranties and disclaim all responsibility and all liability (including without limitation, liability in negligence) for all expenses, losses, damages (including indirect or consequential damage) and costs which you might incur as a result of the provided data being inaccurate or incomplete in any way and for any reason 2022, Eurecat - Centro Tecnológico de Cataluña, Barcelona, Spain.

\section*{Contact}
\noindent For any questions regarding the MDEAW database please contact: mail@eurecat.org
Eurecat - Centro Tecnológico de Cataluña, Barcelona, Spain.

\bibliographystyle{IEEEtran}
