
\section{Introduction}
\label{sec:introduction}

\IEEEPARstart{C}{ardiac} auscultation, i.\,e.,\  listening and interpreting the heart sound, is an indispensable and critical part for the clinical examination of the patient~\cite{alam2010cardiac}. As a low-cost and non-invasive examination, cardiac auscultation is invaluable for 
the presence of a heart disease
and providing an estimate of its severity, evolution, and prognosis~\cite{hanna2002history}. Accurate cardiac auscultation may determine whether more expensive examination should be planed~\cite{hanna2002history}. 
Nevertheless, due to difficulties in diagnosing diastolic murmurs, the overall 
sensitivity of cardiac auscultation is poor (i.\,e.,\  ranging from 0.21 to 1.00)~\cite{alam2010cardiac}. Moreover, cardiac auscultation depends on the physician's skills which have been shown decreased over time~\cite{noor2013heart}. Poor cardiac auscultation skills may miss significant pathology, causing worsening condition, and may over diagnose pathology, leading to inappropriate referral for expensive echocardiography~\cite{alam2010cardiac}.




To solve the above problem of cardiac auscultation, classic machine learning (ML) has been widely used for automated heart sound analysis, including denoising, segmentation, and classification. For instance, noisy audio clips were detected by support vector machines (SVMs)~\cite{tang2021quality}, hidden markov models (HMMs) were used for heart sound segmentation~\cite{rabiner1989tutorial}, and classifiers like SVMs and decision trees were applied to heart sound classification~\cite{whitaker2017combining,langley2017heart}. Classic machine learning often takes acoustic features as the input, while selecting useful features needs lots of human efforts. Additionally, classic machine learning usually performs very well on small-scale data, nevertheless, model performance on big data has been a bottleneck of classic machine learning.

More recently, deep learning (DL) has demonstrated its more powerful capability of analysing heart sounds than classic machine learning~\cite{ren2018heartsound}. DL models are usually fed with either raw audio signals or time-frequency representations extracted from heart sounds as the inputs~\cite{chowdhury2020time,xiao2020heart}, therefore improving the efficiency by skipping selecting hand-crafted acoustic features. More complex model structures in deep learning models also enhance the models' capability of learning abstract representations from large-scale datasets.



\begin{table*}[t]
  \centering
 \caption{A comparison between existing surveys on heart sound analysis.}
  \label{tab:surveys}
   \footnotesize
  \begin{tabular}{lccccc}
    \toprule
   Surveys & Deep Learning & Heart sound denoising & Heart sound segmentation & Heart sound classification & Heart sound interpretation  \\
\midrule
Bhoi et al.~\cite{bhoi2015multidimensional} & \ding{55} & mentioned only & \ding{51} & \ding{51} & \ding{55} \\
Chakrabarti et al.~\cite{chakrabarti2015phonocardiogram} & \ding{55} & \ding{51} & \ding{51}&\ding{51} &\ding{55} \\
Nabih-Ali et al.~\cite{nabih2017review} & \ding{55} &\ding{51} &\ding{51} &\ding{51} &\ding{55} \\
Clifford et al.~\cite{clifford2017recent} & \ding{51} & \ding{51} & \ding{51} & \ding{51} & \ding{55} \\
Ghosh et al.~\cite{ghosh2020heart} & mentioned only & \ding{51} & \ding{51} & \ding{51} & \ding{55} \\
Majhi et al.~\cite{majhi2020application} &--& -- &--&\ding{51}&--\\
Dwivedi et al.~\cite{dwivedi2018algorithms} &\ding{55}&\ding{55}&\ding{51}&\ding{51}&\ding{55} \\
\textbf{This survey} & \ding{51} &\ding{51} &\ding{51} &\ding{51} &\ding{51}\\
    \bottomrule
  \end{tabular}
\end{table*}

\subsection{Differences between this survey and the former ones}
There are several review research studies on heart sound analysis. Feature extraction and basic ML models (e.\,g.,\  SVMs) were introduced in~\cite{bhoi2015multidimensional}. Waveform features of heart sounds and shallow artificial neural networks were overviewed in~\cite{chakrabarti2015phonocardiogram}. Heart sound denoising and segmentation were discussed in~\cite{nabih2017review}. In~\cite{clifford2017recent}, approaches for heart sound classification submitted to the PhysioNet challenge 2016 were summarised. The study in~\cite{ghosh2020heart} summarised approaches used for heart sound analysis, from heart sound segmentation and feature extraction to heart sound classification. Approaches for heart sound classification were summarised and analysed in~\cite{majhi2020application}. 
Different from these above studies, we will review all DL technologies on heart sound segmentation and classification. Furthermore, the state-of-the-art approaches about interpretability of DL models will be summarised and discussed (see~\autoref{tab:surveys}). We will also discuss the potential research problems and future research directions in this survey, helping promote research studies in heart sound analysis.  


\subsection{Challenges in Heart Sound Analysis}
Many machine learning and deep learning methods have been applied to heart sound analysis. However, this research field is still facing many technical challenges. 

The first challenge is denoising, which aims to remove the noise from heart sounds. As the recording environments can be noisy with environmental noise and speech, denoising is an essential pre-processing procedure to improve the audio quality for better performance of segmentation and classification. Other pre-processing procedures such as separation (from lung sounds)~\cite{munoz2021parallel,tsai2020blind,grooby2021new} are not introduced here, as they were mostly 
dealing 
with signal processing methods.

The second challenge is segmentation that targets on splitting a heart sound signal into multiple parts, i.\,e.,\  cardiac cycles or smaller segments (S1, systole, S2, and diastole). Heart sound segmentation is often a pre-processing process of heart sound classification. 

The third is classification which predicts the severe level of cardiovascular diseases or abnormality of a heart from heart sounds. Heart sound classification is helpful for early screening of heart diseases in primary care.

The final one is explaining DL models for heart sound analysis. DL models are black boxes for human due to their complex structures, although they are promising in performance improvement for heart sound analysis. As digital health is a sensitive domain, explainable DL models are crucial for clinicians to give in-time and suitable therapies for patients. Correspondingly, trust from clinicians and patients can promote applications of explainable DL models in real life.




\subsection{Contributions of this survey}
The survey is expected to have the following contributions.

\emph{The first comprehensive survey in heart sound analysis with deep learning.} Apart from summarising machine learning techniques for heart sound denoising, segmentation, and classification, we review the state-of-the-art deep learning topologies for heart sound analysis, especially segmentation and classification. 

\emph{Summarisation of resources.} We summarise publicly available datasets for heart sound analysis, particularly classification. We also provide the collection of open-sourced deep learning algorithms for heart sound classification, and discuss possible evaluation metrics.

\emph{Future research directions.} We discuss the limitation of current deep learning methods for heart sound classification, and point out potential future research topics in this area. We also discuss the importance of explainable DL models for heart sound classification, current advances, and future directions in explainable AI.  






\section{Background}
\label{sec:background}

\begin{figure*}[t]
    \centering
    \includegraphics[ width=.85\linewidth]{figures/heart_sound-framework_3.pdf}
    \caption{The framework of heart sound analysis. After denoising and segmentation, a classifier is trained to produce the prediction and the corresponding interpretation for users (clinicians and patients). The \ding{51} is a normal prediction, and the \ding{55} means the prediction is abnormal. The dashes `-\,-\,-' denote optional procedures.}
    \label{fig:framework}
\end{figure*}

\subsection{Heart Sounds}


In a human's cardiac system, a normal cardiac cycle contains two heart sounds: the first heart sound S1 and the second heart sound S2. Additional sounds indicate diseases: a presented third heart sound S3 could be a sign of heart failure; a murmur could indicate defective valves or an orifice in the septal wall~\cite{ahlstrom2006processing}.
The frequency range of S1 and S2 is 20--200\,Hz, while the frequency of S3 and S4 ranges in 15--65\,Hz~\cite{naseri2013detection}.

As an example, murmurs, caused by the turbulent blood flow in the heart system, are identified as abnormal sounds. They are very important for verifying the timing and pitch for diagnosing cardiovascular diseases~\cite{noor2013heart}. Murmurs often constitute the only basis for diagnosing valvular heart disease~\cite{Gerbarg1963ComputerAO}. Clinically, murmurs consists of two types: systolic murmurs and diastolic murmurs. Aortic stenosis, mitral regurgitation, and tricuspid regurgitation happen during systole; mitral stenosis and tricuspid stenosis occur during diastole~\cite{noor2013heart}.


\subsection{Diagnosis of Cardiovascular Diseases}

Nowadays, there are several non-invasive diagnostic tools for cardiovascular diseases. Electrocardiogram (ECG), 
sensing 
the P-QRS-T wave depicting the electrical activity of the heart~\cite{martis2014current}, is an inexpensive and commonly-used tool for the screening of heart diseases. Yet, it has difficulty in detecting structural abnormalities in heart valves and defects characterised by heart murmurs~\cite{molcer2010examination}. 
Additionally, several medical imaging tools are able to visualise the cardiovascular system. For instance, 
the 
echocardiogram (echo) is an ultra sound scan to create a moving picture of the heart. It can provide information about the heart's size, shape, structure, and function~\cite{leng2015electronic}. Cardiac computed tomography (CT) uses x-rays to create detailed pictures of the heart and its blood vessels~\cite{leng2015electronic}. For assessment of the cardiovascular system's function and structure, cardiac magnetic resonance imaging (CMRI) creates both still and moving pictures of the heart and major blood vessels~\cite{leng2015electronic}. However, these above imaging instruments are expensive and require medical professionals for operation, thereby limiting their application in clinics, and small and medium hospitals. 

Compared to the above diagnostic instruments, cardiac auscultation is low-cost and essential in preliminary physical examinations. Phonocardiogram (PCG) signals recorded with a phonocardiograph have proven to be important in pediatric cardiology, cardiology, and internal diseases~\cite{delgado2009digital}. Recent advances of electronic stethoscopes facilitated computer-aided auscultation by integrating sensor design, signal processing, and machine learning techniques~\cite{delgado2009digital}. The low-cost and portable advantage of electronic stethoscopes make it possible to apply computer-aided auscultation to primary care and remote/home health care. Fig.\ref{fig:framework} depicts a pipeline of heart sound analysis. Heart sounds are processed by de-noising, segmentation, and classification, and then clinicians and patients receive the predictions and interpretations in primary care. In real-life, patients with heart sounds predicted as abnormal will be suggested further professional medical examinations for accurate diagnosis.





\section{Problem Description}
\section{Heart Sound Analysis Tasks}
\label{sec:problems}





\begin{figure*}
    \centering
    \includegraphics[trim={0cm 2cm 0cm 0cm}, clip, width={.9\linewidth}]{figures/heart-sound-categorization-3.pdf}
    \caption{Categorisation of methods for heart sound analysis. Bold texts are deep learning approaches.}
    \label{fig:categorise}
\end{figure*}



The goal of this section is to describe the tasks and summarise classic machine learning techniques for each problem. Specifically, denoising and segmentation are two pre-processing steps for heart sound classification in many research studies.  In~Fig.\ref{fig:categorise}, we summarise approaches used for the four tasks during 2017--2022. 
We can see that, classic machine learning was still used in the past years, and deep learning was mainly applied to segmentation and classification, as well as interpretation methods. For comparison of machine learning and deep learning used for heart sound analysis, this section introduce the tasks and classic machine learning methods, and deep learning will be discussed in \autoref{sec:method}.
As there are only a few works for interpretation, we will discuss heart sound interpretation 
in the discussion part of this 
survey~(see~\autoref{sec:discussion}). Therefore, we describe the three tasks (de-noising, segmentation, and classification) as independent sections in the following for detailed discussion.

\subsection{Denoising}
Generally, recorded heart sounds consist of many kinds of noises~\cite{tsao2019robust}, including white noise and other sounds presented in the recording environments, e.\,g.,\  
humans' 
speech. Noise may degrade the segmentation and classification performance of heart sounds\cite{tsao2019robust}. In this regard, many studies have investigated denoising methods for better performance on the task of heart sound segmentation and classification. 

\sstitle{Filters} 
As a preprocessing procedure of heart sound classification, many denoising approaches employ signal filters for removing noise from noisy heart sounds~\cite{chowdhury2020time,deperlioglu2019classification,deperliuglu2018classification}. Highpass filters~\cite{alaskar2019implementation,khan2018automated} have been used to eliminate low-frequency noise. With the capability of mitigating both high- and low-frequency noises, bandpass filters are more often used for heart sounds denoising~\cite{yadav2018automatic,hu2020automatic,ghosh2019automated}. 
A 
Butterworth bandpass filter has been successfully employed in many studies~\cite{ahmad2021automatic,singh2020short,noman2019markov,noman2019short,dastagir2021computer,bourouhou2020heart,meintjes2018fundamental,abduh2020classification}. The cutoff frequency of a Butterworth bandpass filter is set with a low frequency for filtering out noise with very low frequencies and a high frequency for filtering out high-frequency noises. A range of Butterworth bandpass filters with various orders have been applied with different cutoff frequency settings. For instance, a 4-th order Butterworth filter was set with a cutoff frequency of 25-400\,Hz in~\cite{singh2019classification}, and a 5-th order Butterworth filter was designed to have a cutoff frequency of 25-500\,Hz in~\cite{li2021heart} and 25-250\,Hz in~\cite{ibrahim2021comparative}. A 6-th order Butterworth filter was designed with a cutoff frequency of 50–950\,Hz~\cite{zhang2017heart,zhang2017heart1} and 30-900\,Hz in~\cite{banerjee2020multi}. 
Additionally, several other filters were also used for denoising heart sound, such as a Savitzky–Golay filter~\cite{wu2019applying,krishnan2020automated}, Chebyshev low-pass filter~\cite{zeng2021new,shervegar2018heart}, and Notch filter~\cite{al2020framework}.


\sstitle{Spectrum-based denoising}
To remove noise, 
the 
spectrogram was simply selected with a threshold of -30, -45, -60, or -75\,dB in~\cite{humayun2018ensemble}. However, it is time-consuming to search a suitable threshold
among different heart sounds. A more flexible method, spectral subtraction~\cite{boll1979suppression}, was used to estimate the noise and remove it from the heart sounds~\cite{abduh2020classification}. 





















\sstitle{Spike removal}
Frictional spike is a redundant part of the amplitude of a heart sound. In several studies~\cite{singh2019classification,noman2019markov}, 
frictional spikes were 
detected and removed (i.\,e.,\  replaced by zeros) during pre-processing of heart sounds. 



\sstitle{Selection of noise-free segments}
Apart from removing noise from heart sounds, the usage of noise-free heart sound segments has been shown helpful for heart sound analysis.
Wavelet entropy was used to 
evaluate 
noise in heart sound segments~\cite{langley2017heart}, as clean heart sounds have relatively higher wavelet entropy than noisy heart sounds.
Empirical wavelet transform 
was used to separate heart sounds, murmurs, low-frequency artefacts, and high-frequency noises in another study~\cite{varghees2017effective}. 
Additionally, classic machine learning was also used for detecting noise-free heart sound segments. In~\cite{tang2021quality}, SVMs were applied to classify the quality of heart sound signals into binary classes (i.\,e.,\  `unacceptable' and `acceptable') or three classes (i.\,e.,\  `unacceptable', `good', `excellent') without segmentation based on ten types of multi-domain features.


\subsection{Segmentation}

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/segmentation.pdf}
    \caption{The PCG recording of a \emph{normal} heart sound recording from the PhysioNet/CinC Database~\cite{liu2016open}. Frames in the middle with four states labelled (i.\,e.,\  S1, systole, S2, and diastole) are depicted.}
    \label{fig:segmentation}
\end{figure}

Heart sound segmentation was used to split an audio sample into a set of smaller audio segments, which could be equal to or shorter than a complete cardiac cycle~\cite{noman2019short,baydoun2020analysis,upretee2019accurate}. The segments shorter than a cardiac cycle could include S1, systole, S2, and diastole, as indicated in Fig.~\ref{fig:segmentation}.  

\sstitle{Energy-based segmentation}
As heart sounds at different states have various energies, signal energy has been used for localising S1 and S2 peaks~\cite{low2018classification,deperlioglu2019classification,deperliuglu2018classification,deperlioglu2021heart}. Based on frequency information (e.\,g.,\  Wavelet Transform (WT)) of heart sounds, energy peaks of wavelet coefficients were detected for localising S1 and S2 in~\cite{eslamizadeh2017heart}. 

\sstitle{Envelope-based segmentation}
Apart from energy, heart sound segmentation can be achieved based on envelopes~\cite{akram2018analysis,ibrahim2021comparative}. For instance, heart sound segmentation was implemented based on Shannon energy envelope and zero crossings of heart sounds in~\cite{chowdhury2020time}.
In another two studies~\cite{zhang2017heart,zhang2017heart1}, S1 of the first heart cycle was detected based on Shannon energy envelopes, and the next S1 heart sounds were detected by a sliding heart cycle window. 




\sstitle{Loudness-based segmentation}
Loudness has proven its potential to segment heart sounds~\cite{shervegar2018heart,shervegar2017automatic}.
Specifically, spectrograms extracted from heart sounds are firstly converted into the Bark scale and smoothed with a Hanning window. At each time frame, the sensation of loudness is then calculated by the mean of the amplitudes at all frequency bands: $L(t)=\frac{\sum_{t=1}^TA(t)}{T}$, where $A(t)$ is the amplitude at the $t$-th time frame, and $T$ is the total number of time frames. Furthermore, the derivation of the loudness function is computed for obtaining peaks. Therefore, systoles and diastoles can be localised as they have different time lengths.






\sstitle{Classic Machine Learning for Segmentation}
ML models have been proposed for more precise heart sound segmentation than the above mentioned rule-based segmentation methods. ML models for segmentation are mainly trained in supervised learning frameworks. There are also a few works using unsupervised learning. Both unsupervised and supervised learning approaches are introduced in the following.

\emph{Unsupervised Learning}
Considering the limited availability of the heart sound datasets, 
the authors of~\cite{DAS2020103990} adopted an unsupervised spectral clustering technique based on Gaussian kernel similarity to get the frame labels (e.\,g.,\  S1 and S2), which are further utilised to segment the heart sound. 

\emph{Supervised Learning}
Particularly, a hidden Markov model (HMM) has been widely used for this purpose~\cite{rabiner1989tutorial}. Let us assume the heart states as $S=\{s_1, s_2, s_3, s_4\}=\{S1, \mbox{systole}, S2, \mbox{diastole}\}$, and the observations $O=\{o_1, o_2, ..., o_T\}$ as the raw heart sounds or acoustic features. A transmission matrix $A=\{a_{ij}\}$ denotes the probablity of a state $s_i$ at the $t$-th time frame moving to $s_j$ at the $(t+1)$-th time frame. The probability density distribution of an observation $o_t$ to be generated by a state $s_i$ is $B={b_i(o_t)}=P[o_t|s_i]$, 
where $P$ means probability. The initial state distribution is $\pi=\{\pi_i\}$, representing the probability of state $s_i$ at the start time frame. Given $A$, $B$, $\pi$, and $O$, an HMM model aims to optimise the state sequence. The Viterbi algorithm is often used for this purpose~\cite{schmidt2010segmentation} and more details can be found in~\cite{rabiner1989tutorial}.  



In order to better capture the abrupt changes in the PCG signal, the envelope of the signal was obtained in~\cite{SHUKLA2020101762}, and the kurtosis of the envelope was computed to get the impulse-like characteristics, which were further passed through a zero frequency filter for pure impulse information. Along with the heart sound labels, the extracted features were fed into a hidden semi-Markov model (HSMM).
To better adapt the variability of the heart cycle duration (HCD) in the PCG recordings, a multi-centroid-duration-based HSMM was introduced in~\cite{KAMSON2019265}, where HCDs were estimated at various instances of a PCG to get maximum possible duration values and those nearest values were clubbed into clusters to refer each centroid. With more accurate state duration information, the HSMM achieved better segmentation performance. Similarly, considering the inter-patient variability, the emission probability distributions to each patient were estimated through a Gaussian mixture model (GMM) in an unsupervised and adaptive way in the improved HSMM in another study~\cite{8678744}. Moreover, the expectation maximisation algorithm developed in~\cite{8367820} searched for sojourn time distribution parameters of an HSMM for each subject.
Many studies~\cite{whitaker2017combining,rubin2017recognizing,maknickas2017recognition,liu2022abnormal,khan2020automatic,dastagir2021computer,bourouhou2020heart,chen2020classification,abduh2020classification,alaskar2019implementation,han2018supervised,han2019heart,chen2019heart} have employed logistic regression-based HSMM (LR-HSMM) proposed in~\cite{springer2015logistic} for heart sound segmentation. LR was incorporated to predict the probability of $P[s_j|o_t]$, and $B$ is then computed with the Bayes rule. 
There are also other improved HMM methods, such as the \emph{duration-dependent HMM}~\cite{schmidt2010segmentation,duggento2021classification}, which considers the probability density function of the duration at each state. Another study~\cite{noman2019markov} proposed a Markov-switching model for heart sound segmentation.





\subsection{Classification}
The target task of automated auscultation is heart sound classification, including i) abnormal heart sound detection (e.\,g.,\  murmurs, mitral stenosis, etc.) and ii) severity of cardiovascular diseases (normal/mild/moderate). 

\sstitle{Feature Engineering}
After denoising and segmentation, feature extraction is often performed before training a classifier. Low-level descriptors (LLDs) and functionals are often extracted as acoustic features. LLDs are segmental features on short-time segment analysis (see Table~\ref{tab:classification_feature}), and functionals are supra-segmental feature vectors projected from LLDs. Functionals are generally statistical features, such as mean, max, standard deviation, and many others. 

We mainly list the LLDs used for heart sound classification in \autoref{tab:classification_feature}. 
Apart from time-domain LLDs, LLDs in the frequency-domain are widely used for heart sound classification. Specifically, the frequency-domain features consist of spectral features based on Fourier Transform, Mel-scaled spectral features, and wavelet features. There are also pre-existing feature sets used for heart sound classification, including the \textsc{ComParE} feature set~\cite{schuller2018compare} and the eGeMAPS feature set~\cite{eyben2015geneva}. Both feature sets can be extracted with openSMILE which is an open-source toolkit~\cite{eyben2010opensmile}. 

In addition to the aforementioned hand-crafted features, deep representations have been studied more recently (see~\autoref{tab:classification_feature}). Due to the strong capability of deep learning models for extracting abstract features, deep representations bear potential to improve the performance of hand-crafted features.



\begin{table*}[]
    \centering
    \caption{Hand-crafted features and deep representations for heart sound classification. We present various low-level descriptors (LLDs) in hand-crafted features.}
    \label{tab:classification_feature}  
    \begin{minipage}{\textwidth}
    \renewcommand*\footnoterule{}
    \begin{tabular}{llp{5cm}p{3cm}}
    \toprule
         Group& Features & Description & Reference \\
\hline
       Time-domain &Envolope& Envelope of a signal & \cite{singh2019classification1,varghees2017effective}\\
         & Amplitude &Amplitude of a signal &\cite{eslamizadeh2017heart}\\
         &Energy &Energy of a signal &\cite{ibrahim2021comparative,khan2018automated,deperlioglu2021heart}\\
         &Entropy &Signal entropy&\cite{soares2020autonomous,khan2018automated}\\
        
         &Loudness&Perception of sound magnitude&\cite{shervegar2018heart}\\
        
         &Peak amplitude&Amplitude of peaks &\cite{dastagir2021computer}\\      
\hline 
         Spectral &Spectral amplitude & Fourier transform  &\cite{langley2017heart,khaled2022analysis,ibrahim2021comparative,arora2019heart,soares2020autonomous,yadav2018automatic,khan2018automated,sotaquira2018phonocardiogram,noman2019markov}\\
         &Dominant frequency value &Frequency which leads to the maximum spectrum &\cite{khaled2022analysis,arora2019heart} \\
         &Dominant frequency ratio & Ratio of the maximun energy to the total energy&\cite{khaled2022analysis,arora2019heart}\\
         & Energy &Spectral energy&\cite{noman2019markov}\\
         &Spectral roll-off& Frequency below a specific percentage of the total spectral energy&\cite{khan2018automated}\\
         &Spectral centroid &Average of magnitude spectrogram at each frame &\cite{khan2018automated,sotaquira2018phonocardiogram}\\
         &Specrtal flux & Changing speed of the power spectrum &\cite{khan2018automated}\\
         &Power spectral density (PSD) &Distribution of power in spectral components &\cite{ibrahim2021comparative,singh2019classification1,sotaquira2018phonocardiogram}\\
         &Spectral entropy & Shannon entropy of PSD&\cite{khaled2022analysis,arora2019heart,soares2020autonomous,al2020framework}\\
         &Instantaneous frequency &Frequency for non-stationary signals  &\cite{alqudah2019towards}\\
        &Fractional Fourier transform entropy&Spectral entropy of the fractional Fourier transform&\cite{tan2022heart}\\
        &Spectrogram & Short-Time Fourier Transform (STFT)  &\cite{zhang2017heart,zhang2017heart1}\\
        &Cepstrum& Inverse Fourier transform on the logarithm of the signal spectrum&\cite{yadav2018automatic}\\         
\hline
         Mel frequency
        &Mel-frequency & Mel-scaled frequency &\cite{dong2019heart,maknickas2017recognition} \\
         &Mel-Frequency Cepstral Coefficients (MFCCs) & Discrete cosine transform of Mel-scaled spectrogram & \cite{noman2019markov,chen2022heart,khaled2022analysis,soares2020autonomous,chen2019heart,rahmandani2018cardiac,khan2020automatic,adiban2019statistical} \\
         &Fractional Fourier transform-based Mel-frequency& Mel-frequency from the  fractional Fourier transform&\cite{abduh2020classification}\\
\hline
         Wavelet &Wavelet transform & Frequency analysis of a signal at various scales &\cite{singh2019classification1,baydoun2020analysis,sotaquira2018phonocardiogram}\\
         &Wavelet scattering transform &``Wavelet convolution with nonlinear modulus and averaging scaling function''\footnote{https://de.mathworks.com/help/wavelet/ug/wavelet-scattering.html} (translation invariance and elastic deformation stability~\cite{li2019heart}) & \cite{mei2021classification,li2019heart} \\
         &Wavelet synchrosqueezing transform & Reassignment of wavelet coefficients &\cite{ghosh2019automated}\\
         &Tunable quality wavelet transform & ``Wavelet multiresolution analysis with a user-specified Q-factor, which is the ratio of the centre frequency to the bandwidth of the filters''\footnote{https://de.mathworks.com/help/wavelet/ug/tunable-q-factor-wavelet-transform.html}&\cite{sawant2021automated,zeng2021new}\\
         &Wavelet entropy&Temporal energy distribution based on wavelet coefficients & \cite{langley2017heart}\\
\hline
         Feature set&\textsc{ComParE}& Computational Paralinguistics ChallengE feature set &\cite{dong2019heart}\\
         &eGeMAPS&The extended Geneva Minimalistic Acoustic Parameter Set &\cite{dong2019heart}\\
\hline
         Deep representation&Graph-based features &Petersen graph pattern &\cite{tuncer2021application}\\
        
        
        
         &Sparse coefficient&Result of sparse coding&\cite{whitaker2017combining}\\
        
         &Autoencoder-based features& Features extracted by an autoencoder from hand-crafted features &\cite{amiriparian2018deep,humayun2018ensemble}\\
    \bottomrule
    \end{tabular}
    \end{minipage}
    \vspace{-20pt}
\end{table*}


\sstitle{Classic Machine Learning for Classification}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/fig_machine_learning.pdf}
    \caption{Statistics of the literature using discriminative machine learning models for heart sound classification during 2017--2022. FNN: feed-forward neural network; KNN: $k$-nearest neighbour; LDC: linear discriminant classifier.}
    \label{fig:statistic_ml}
\end{figure}

Rule-based classification was proposed for heart sound classification in~\cite{varghees2017effective,karar2017automated}. For better performance, ML was used for heart sound classification in most research studies. We briefly introduce the classifiers in the two groups of generative models and discriminative models in the following.

\emph{Generative models} 
aim to generate the joint probability distribution $P(X,y)$, given the features $X$ and the labels $y$. The posterior probability $P(y|X)$ is computed via the Bayes rule $P(y|X)=\frac{P(X,y)}{P(X)}=\frac{P(X|y)P(y)}{P(X)}$, where $P(X|y)$ is the likelihood probability distribution. 
The 
Na\"ive 
Bayes Classifier~\cite{plesinger2017heart,tan2022heart,bourouhou2020heart,ibarra2018benchmark} was widely used for heart sound classification due to its advantage of being easy-to-use.
Gaussian Mixture Models (GMMs)~\cite{adiban2019statistical,shervegar2018heart} were used to estimate the data distribution by optimising the weights of Gaussian mixture components and mean and variance in each component.
A Gaussian mixture-based HMM~\cite{noman2019markov} was employed for heart sound classification considering the four sequential heart states, i.\,e.,\  S1, systole, S2, and diastole.

\emph{Discriminative models} are designed to directly predict the posterior
probability $P(y|X)$ given $X$. Fig.~\ref{fig:statistic_ml} presents a statistic of recent works from 2017 to 2022 that employ classic machine learning models for heart sound classification. 
SVMs have been very widely used for heart sound classification by learning a 
supporting hyperplane between classes~\cite{whitaker2017combining,zhang2017heart,zhang2017heart1,tan2022heart,chen2022heart,mei2021classification,ibrahim2021comparative,tuncer2021application,dastagir2021computer,khan2020automatic,bourouhou2020heart,adiban2019statistical,upretee2019accurate,abduh2020classification,dong2019heart,amiriparian2018deep,humayun2018ensemble,ibarra2018benchmark,yadav2018automatic,chen2019heart,akram2018analysis,singh2019classification1,li2019heart}. Apart from linear projection between data samples and labels, SVMs can learn separating hyperplanes on non-linear data via non-linear kernels, such as radial basis function. 


Furthermore, $k$-nearest neighbours (KNNs) has shown good performance on heart sound classification with the idea of classifying a data sample according to the classes of its $k$-nearest neighbours~\cite{tan2022heart,chen2022heart,ibrahim2021comparative,dastagir2021computer,khan2020automatic,singh2019classification1,alqudah2019towards,bourouhou2020heart,upretee2019accurate,abduh2020classification,ibarra2018benchmark,sofwan2019normal}.
Also, decision trees were successfully used for heart sound classification~\cite{langley2017heart,tuncer2021application,dastagir2021computer,arora2019heart,khan2020automatic,bourouhou2020heart,ibarra2018benchmark}. One reason is that limiting the number of decision nodes can help avoid overfitting~\cite{langley2017heart}, and another reason is that the structure of a decision tree can show the internal logic for classification.
Bagged trees~\cite{tuncer2021application,dastagir2021computer} assemble multiple decision trees for more complex model architectures, therefore it is possible to achieve better performance.
Random forests~\cite{chen2022heart,arora2019heart,alqudah2019towards,ibarra2018benchmark,chen2019heart,ghosh2019automated} further improve bagged trees with less features to be used when splitting each node. 

In recent years, feed-forward Neural Networks (FNNs) have started to be applied to heart sound classification~\cite{eslamizadeh2017heart,zeng2021new,khaled2022analysis,khan2020automatic,humayun2018ensemble,khan2018automated,rahmandani2018cardiac,deperliuglu2018classification,sotaquira2018phonocardiogram}. FNNs can automatically learn a non-linear projection between acoustic features and labels. Although FNNs lack of explainability compared to other classifiers such as SVMs and decision trees, they are promising to produce good performance. 

There are also several other machine learning models, such as, linear discriminant classifiers~\cite{tuncer2021application,humayun2018ensemble,ibarra2018benchmark}, logistic regression~\cite{ibarra2018benchmark}, quadratic discriminant analysis~\cite{ibrahim2021comparative}, boosting methods~\cite{arora2019heart,baydoun2020analysis,chen2019heart,sawant2021automated}, and others~\cite{soares2020autonomous,al2020framework}. 
Finally, multiple classifiers can be further assembled for better performance compared to that of a single model~\cite{chen2022heart,khan2020automatic,baydoun2020analysis,abduh2020classification,singh2019classification1}.


\section{State-of-the-art Studies}
\label{sec:method}




Due to the strong capability of extracting effective representations, deep learning has been successfully applied to  a range of acoustic tasks, e.\,g.,\  speech emotion recognition~\cite{ren2020enhancing}, respiratory sound classification~\cite{ren2021prototype}, snore sound classification~\cite{qian2020can}, and many more. 
In recent advances, deep learning has been also employed in processing heart sound signals and achieved good performance~\cite{ren2018heartsound}. In this regard, the applications of deep learning methods for heart sound analysis tasks are introduced and discussed in the following. Notably, as there are few works focusing on denoising 
in this context 
with deep learning, we introduce deep learning topologies for segmentation and classification in this section.





\input{deep_segmentation}

\input{deep_classification}



\sstitle{Deep learning on time-frequency representations}
As it is challenging to extract effective representations from raw heart sound signals, 2D time-frequency representations have been widely used as the input of 2D CNNs for heart sound classification, respectively~\cite{rubin2017recognizing,maknickas2017recognition,wu2019applying,noman2019short,takezaki2021construction,duggento2021classification,chen2020classification,cheng2019design,alqudah2020classification,banerjee2020multi,han2018supervised,meintjes2018fundamental,wibawa2018abnormal,low2018classification,deperlioglu2019classification,ranipa2021multimodal}. Spectrograms, extracted by STFT from heart sounds, were fed into ResNet for abormal heart sounds detection in~\cite{balamurugan2019reshnet}.
Multi-domain features were considered to be more comprehensive in reflecting the characteristics of all heart sound classes.
In~\cite{wu2019applying}, spectrograms, Mel spectrograms and MFCCs were extracted as the inputs of VGG models, and the final predictions were obtained by ensemble learning. 

Apart from CNNs that are good at extracting spatial features, RNNs are more capable of extracting temporal features from sequential signals. 
LSTM nets were applied to process discrete wavelet transforms and MFCCs for heart sound classification~\cite{ahmad2021automatic,khan2020automatic}. Deng et al.\ employed convolutional recurrent neural networks which combined the CNNs and RNNs~\cite{deng2020heart}. 

Additionally, there are also other classifiers used for heart sound classification, such as a stacked sparse autoencoder deep neural network~\cite{abduh2019classification} and a semi-non-negative matrix factorisation classifier~\cite{han2019heart}. More details can be found in the listed references.



\sstitle{Deep learning on other features}
Apart from the above excellent works about DL on time-frequency representations, other features extracted from heart sounds were also used as the input of DL models.

i) \emph{Time-domain features}.
Similar to features in classic machine learning, 1D time-domain features can be also fed into DL models for heart sound classification. For instance, instant energy of the heart sound was extracted and used as the input of stacked auto-encoder networks~\cite{deperlioglu2021heart}. Multiple statistical features (such as mean, median, variance, and many more) were extracted from all 75\,ms segments in each complete heart sound clip, and fed into a bidirectional LSTM (BiLSTM) net for classification in~\cite{fakhry2022comparison}.

ii) \emph{1D frequency-based features}. 
Either 1D CNNs or feed-forward DNN models can be used to process 1D features. General frequency features and Mel domain features were fed into 1D CNNs and then the multiple CNNs were assembled for the final prediction in~\cite{ranipa2021multimodal}. 
Moreover, Mel spectrograms and MFCC were used to further extract features as the input of a 5-layer feed-forward DNN model in~\cite{chowdhury2020time}. 

iii) \emph{2D frequency-based features}.
Different from the above time-frequency representations, herein, we list 2D frequency-based features to include (a) multiple 1D frequency features directly extracted from audio segments rather than window functions in the STFT domain and (b) features computed from time-frequency features.
Qian et al.\ utilised wavelets to calculate wavelet energy features from a set of short acoustic segments and further used GRU-RNNs as the classifier~\cite{qian2019deep}.
Dong et al.\ extracted log Mel features and corresponding functionals from heart sound segments and implemented classification LSTM-RNNs and GRU-RNNs~\cite{dong2019heart}. In their experiments, log Mel features performed better than MFCCs and other LLDs~\cite{dong2019heart}.
Zhang et al.\ extracted temporal quasi-periodic features computed by an average magnitude difference function from spectrograms and applied LSTM-RNNs for exploring the dependency relation within the features~\cite{zhang2019abnormal}. 
A denoising auto-encoder was employed to extract deep representations from spectrograms as the input of the classifier of 1D CNNs in~\cite{li2019feature}. 




\sstitle{End-to-end learning}
In recent years, as time-frequency representations and other features still need human efforts to select features, it has been becoming increasingly popular to use end-to-end networks to learn representations from heart sounds.
Based on raw heart sound signals, various 1D CNN architectures have been proposed and applied to the task of heart sound  classification~\cite{xiao2020heart,noman2019short,krishnan2020automated,hu2020automatic,avanzato2020heart}.
Furthermore, Liu et al.\ introduced a temporal convolutional network (TCN) that performed a high sensitivity for heart sound classification~\cite{liu2022abnormal}, as a TCN benefiting from dilated and casual convolutions is more suitable for sequential data than typical CNNs are.
A 1D CNN model consisting of residual blocks was developed for classifying heart sounds~\cite{oh2020classification}.
Apart from CNNs, GRU-RNNs were also used to process raw heart sound signals for the screening of heart failure~\cite{gao2020gated}.

Several studies also mentioned the capability of CNNs and RNNs for learning frequency-domain and time-domain characteristics of heart sounds. For instance, Shuvo et al.\ proposed a CardioXNet model that employed representation learning followed by sequence residual learning without any preprocessing~\cite{shuvo2021cardioxnet}. In the representation learning phase, three parallel 1D CNN pathways were constructed to extract time-invariant features from heart sound signals; in the sequence residual learning phase, BiLSTM nets were employed to learn sequential representation.
The study in~\cite{li2021heart} attempted to automatically learn time-frequency features, i.\,e.,\  frequency-domain features were extracted by 1D CNNs and the time-domain characteristics were extracted by GRU-RNNs. A self-attention mechanism was further used to fuse the two types of features for the final classification.
In~\cite{humayun2020towards,humayun2018learning}, time-convolution (tConv) layers were implemented at the front end of the network for learning finite impulse response filters. 







\sstitle{Transfer learning}
As data collection in the healthcare domain has to be executed according to extremely strict regulations, heart sound datasets are usually not as large as datasets in other areas of Computer Audition. Transfer learning attempts to employ pre-trained DL models which are optimised on large-scale datasets. In recent studies, pre-trained models are mainly learnt on either an image dataset (i.\,e.,\  ImageNet~\cite{deng2009imagenet}) and an audio dataset (i.\,e.,\  AudioSet~\cite{gemmeke2017audio}). Although heart sounds are presented as audio signals which are a different data type from ImageNet, DL models trained on ImageNet have shown good performance on time-frequency representations extracted from heart sounds for heart sound classification. Typical DL models on ImageNet, such as AlexNet~\cite{krizhevsky2017imagenet} and VGG~\cite{simonyan2014very}, have been successfully used for heart sound classification~\cite{singh2019classification,alaskar2019implementation,demir2019towards,ren2018heartsound}. Compared to ImageNet, AudioSet includes multiple types of acoustic signals and therefore is more close to heart sounds with the consideration of data type. In~\cite{koike2020audio}, pre-trained Audio Neural Networks (PANNs) trained on AudioSet were used for classifying heart sounds with inputs of time-frequency representations. PANNs outperformed ImageNet-based models~\cite{koike2020audio}, including VGG, MobileNet V2~\cite{howard2017mobilenets}, ResNet~\cite{he2016deep}, and ResNeXt~\cite{xie2017aggregated}.

After extracting representations by pre-trained models, transfer learning uses various types of classifiers for classification, mainly including classic machine learning classifiers and feed-forward neural networks. For instance, SVMs were applied to representations extracted by AlexNet, VGG16, and VGG19 in~\cite{singh2019classification,demir2019towards,alaskar2019implementation}. Other classifiers such as KNNs were used in~\cite{singh2019classification}. Pre-trained VGG model was frozen and followed by fully connected layers in~\cite{ren2018heartsound}. 

Additionally, fine-tuning pre-trained models in transfer learning has also shown good performance for heart sound classification. A pre-trained AlexNet after fine-tuning provided effective representations for heart sound classification~\cite{singh2019classification,alaskar2019implementation}. Similarly, PANNs was also fine-tuned in~\cite{koike2020audio}. Fine-tuned models have even outperformed pre-trained models as they adapted to the data distribution of heart sound datasets. In~\cite{ren2018heartsound}, fine-tuned VGG performed better than pre-trained VGG on heart sound classification when SVMs were the classifiers.




\section{Published Resources on Heart Sound Analysis}

\subsection{Published Datasets}
In the past years, several heart sound databases have been collected. Herein, we briefly introduce the following access-available databases listed in~\autoref{tab:datasets}.

The PASCAL challenge Database~\cite{pascal-chsc-2011} was split into two sets A and B. In the dataset A, 176 heart sounds (0.393\,hour) were recorded with the iStethoscope Pro iPhone app and annotated into S1 and S2 sounds for heart sound segmentation. Each heart sound in the dataset A was also labelled into one of the four classes: \emph{normal}, \emph{murmur}, \emph{extra heart sound}, and \emph{artifact}. The dataset B with 656 recordings (1.194\,hours) was annotated into three classes: \emph{normal}, \emph{murmur}, and \emph{extrasystole}. 

The PhysioNet/CinC Database~\cite{liu2016open}, used in the PhysioNet/CinC Challenge 2016~\cite{goldberger2000physionet}, consists of multiple databases that were recorded from different data collectors. Especially, the publicly available training set has five databases collected from both healthy individuals and patients. The training set consists of 3,240 recordings (20.216\,hours in total) from more than 764 subjects. The task was set to classifying each data sample into three classes: \emph{normal} and \emph{abnormal}, and \emph{noisy}. 

The Heart Sounds Shenzhen (HSS) corpus~\cite{dong2019heart}, employed in the INTERSPEECH Computaional Paralinguistic challengE (ComParE) 2018, was collected by the Shenzhen University General Hospital. In total, 170 subjects (f: 55, m: 115) participated in the data collection with an electronic stethoscope, leading to 845 heart sound recordings of 7.047\,hours. Each audio sample was annotated into one of the three classes: \emph{normal}, \emph{mild}, \emph{moderate/severe}.

An open heart sound database on GitHub~\cite{son2018classification} was collected for 1,000 audio files (0.679\,hour in total). The audio recordings are balanced in five classes: \emph{normal}, \emph{aortic stenosis}, \emph{mitral regurgitation}, \emph{mitral stenosis}, \emph{mitral valve prolapse}. 

The Michigan Heart sound database\footnote{https://open.umich.edu/find/open-educational-resources/medical/heart-sound-murmur-library}\footnote{https://www.med.umich.edu/lrc/psb\_open/html/repo/primer\_heartsound/\\primer\_heartsound.html} provides heart sound samples recorded from different areas and poses: the apex area when a subject is supine, the apex area 
for left decubitus, 
the aortic area when sitting, and the pulmonic area 
for supine. In total, 23 heart sounds were recorded and the total duration is 0.413\,hour.
The heart sounds were annotated into \emph{normal} and multiple \emph{pathological} states.

The CirCor DigiScope Database~\cite{oliveira2021circor}, used in the George B. Moody PhysioNet Challenge 2022~\cite{Reyna2022}, was collected from a pediatric population at or under 21 years old. The heart sounds were recorded from one or multiple locations: pulmonary valve, aortic valve, mitral valve, tricuspid valve, and others. The publicly available training set consist of 3,163 audio samples with 20.094\,hours in total from 942 participants. Two classification tasks were targeted: i) \emph{normal} and \emph{abnormal}, and ii) \emph{presence of murmurs}, \emph{absence of murmurs}, and \emph{unclear cases of murmurs}.


\begin{table*}[!h]
  \centering
 \caption{Published datasets for heart sound classification. AS: aortic stenosis, MR: mitral regurgitation, MS: mitral stenosis, MVP: mitral valve prolapse. Notably, the statistics in this table considered access available data sets only.}
  \label{tab:datasets}
   \footnotesize
\scalebox{0.75}{
  \begin{tabular}{llrrrl}
    \toprule
   Dataset & Challenge & \#Samples & Duration (h) & \#Subjects & Task \\
\midrule
\multirow{2}*{PASCAL Database~\cite{pascal-chsc-2011}} & \multirow{2}*{PASCAL Challenge~\cite{pascal-chsc-2011}}& 176 & 0.393 &unknown &Dataset A: Normal, Murmur, Extra Heart Sound, Artifact \\
&&656  &1.194 &unknown&Dataset B: Normal, Murmur, Extrasystole \\
\hline
PhysioNet/CinC Database~\cite{liu2016open} & PhysioNet/CinC Challenge 2016~\cite{goldberger2000physionet} & 3,240  & 20.216 &764+& Normal, Abnormal, Too noisy or ambiguous  \\
\hline
HSS~\cite{dong2019heart} & ComParE Challenge 2018~\cite{schuller2018compare} & 845  & 7.047&170& Normal, Mild, Moderate/Severe \\
\hline
Data on GitHub\cite{son2018classification}&--& 1,000& 0.679& unknown& Normal, AS, MR, MS, MVP \\
\hline
Michigan Heart sound database\footnote{\url{https://open.umich.edu/find/open-educational-resources/medical/heart-sound-murmur-library}\url{https://www.med.umich.edu/lrc/psb_open/html/repo/primer_heartsound/primer_heartsound.html}} &--&23& 0.413 & unknown&Normal, Pathological \\%Apex, Aortic, and Pulm\\
\hline
 CirCor DigiScope Database~\cite{oliveira2021circor} & George B. Moody PhysioNet Challenge 2022~\cite{Reyna2022} &3,163& 20.094& 942& Normal, abnormal; presence, absence, or unclear cases of murmurs\\
    \bottomrule
  \end{tabular}
}
\end{table*}

\begin{table*}[!h]
  \centering
 \caption{Evaluation metrics. $k$ is the $k$-th class of all $K$ classes; $N_k$, $\hat{N_k}$, and $\tilde{N_k}$ are the total number of samples, the total number of the predicted samples, and the number of correctly predicted samples for the $k$-th class, respectively. $N$ is the total number of all samples. TPR: true positive rate, FPR: false positive rate, AUC: area under curve, ROC: receiver operating characteristic, PR: precision-recall. }
  \label{tab:metrics}
   \footnotesize
\scalebox{0.9}{
  \begin{tabular}{lll}
    \toprule
   Evaluation Metric & Description & Formula \\
\midrule
Sensitivity & Proportion of the actual positive cases which are correctly classified, i.\,e.,\  true positive rate (TPR)& 
$\frac{\mbox{True Positives}}{\mbox{True Positives} + \mbox{False Negatives}}$\\
Specificity & Proportion of the actual negative cases which are correctly classified, i.\,e.,\  true negative rate (TNR) &
$\frac{\mbox{True Negatives}}{\mbox{True Negatives} + \mbox{False Positives}}$\\
Mean Accuracy & Arithmetic mean of sensitivity and specificity & $\frac{\mbox{sensitivity} + \mbox{specificity}}{2}$
\\
Youden's index & A measure of the ability to balance sensitivity and specificity &  $\mbox{sensitivity} + \mbox{specificity} - 1$\\
Precision & Proportion of correctly classified cases based on all cases predicted into $k$ &
$\frac{\tilde{N_k}}{\hat{N_k}}$\\
Recall & Proportion of correctly classified cases based on all cases in the class $k$.
&  $\frac{\tilde{N_k}}{N_k}$\\
UAR & Unweighted average recall & $\frac{\sum_{k=1}^{K}\mbox{recall}_k}{K}$\\
WAR & Weighted average recall, i.\,e.,\  accuracy & $\sum_{k=1}^{K}\frac{N_k}{N} \mbox{recall}_k$\\
F1-Score & Harmonic mean of precision and recall & $ 2 \times \frac{\mbox{precision} \times \mbox{recall}}{\mbox{precision} + \mbox{recall}}$ \\
AUC-ROC & The area under the probability curve plotting TPR against FPR at various probability thresholds  &  ------ \\
AUC-PR & The area under the probability curve plotting precision against recall at various probability thresholds & ------ \\
Average precision & Summarisation of a precision-recall curve as the weighted mean of precisions at each threshold & $\sum_{n} (\mbox{recall}_{n} - \mbox{recall}_{n-1}) \times\mbox{precision}_{n}$ \\
    \bottomrule
  \end{tabular}
  }
\end{table*}


\subsection{Evaluation Metrics}
\autoref{tab:metrics} summarises evaluation metrics that can be used for evaluating model performance on heart sound classification.

\sstitle{Class-wise evaluation metrics} In binary classification tasks, sensitivity and specificity focus on calculating the proportion of correctly classified positive samples and negative samples, respectively. In order to balance sensitivity and specificity, mean accuracy and Youden's index are utilised. As for evaluating a model's performance on a specific class in a multi-class classification task, precision and recall are frequently used.
For a specific class, precision calculates the proportion of correctly classified cases based on all cases predicted to this class, whereas recall quantifies the proportion of correctly classified cases based on all cases in the class. 
Usually, there is an inverse relation between precision and recall. Therefore, their combinations (e.\,g.,\  F1-Score, average precision) are referred to 
to better evaluate the performance of a model. Notably, in multi-class classification, recall is an extension of sensitivity.

\sstitle{Multi-class evaluation metrics} Besides the class-wise evaluation, there are other metrics across all classes, such as weighted average recall (WAR) and unweighted average recall (UAR). WAR, also known as accuracy and as the name implies, tends to overestimate a model if it performs better on the majority classes. Therefore, with an imbalanced dataset, UAR is preferred. UAR is the mean recall across classes w/o instance-based weighting. With various probability decision thresholds, we can also plot AUC-ROC and precision-recall curves. In addition, we would like to mention the cost-based metric devised in the PhysioNet Challenge 2022~\cite{Reyna2022}, where cost calculation involves model pre-screening, expert screening, treatment, and diagnostic errors. In this way, more clinically practical models can be developed, especially in resource-constrained scenarios.


\subsection{Published Algorithms}
We discuss publicly available codes in the papers discussed in Section~\ref{sec:dep_class} as well as other papers on open-access repositories during 2017--2022. There are only a few publicly available codes in the past years, while there are many more released codes in 2016 benefiting from the PhysioNet challenge 2016~\cite{liu2016open}. Notably, codes in 2016 are not listed, as we want to list the most state-of-the-art studies in the past six years. 

In~\cite{son2018classification}, the authors released a Matlab code\footnote{https://github.com/yaseen21khan/Classification-of-Heart-Sound-Signal-Using-Multiple-Features-} of training machine learning deep neural networks with inputs of multiple features, including MFCCs and discrete wavelets transform. In~\cite{humayun2020towards}, a CNN model with time-convolutional units which simulate finite impulse response filters was implemented with Python in the code\footnote{https://github.com/mhealthbuet/heartnet}. ResNets on linear and logarithmic spectrogram-image features were implemented in the Python code\footnote{https://github.com/mHealthBuet/CepsNET} of the study~\cite{azam2021heart}. A study~\cite{waaler2022algorithm} released a Matlab code\footnote{https://github.com/uit-hdl/heart-sound-classification} for detecting valvular heart disease from heart sounds and echocardiograms. 









\section{Future Research Directions and Open Issues}
\label{sec:discussion}


\subsection{Findings}
In classification tasks which mainly focus on the screening of heart diseases, segmentation was often considered as a preprocessing procedure before classifying heart sounds. It is still an open question whether segmentation is helpful for classification or not.

\sstitle{Segmentation + Classification}
Many studies applied segmentation techniques or given segmentation information before the classification procedure. For instance, segmented cardiac cycles were used as the input of DL models in~\cite{humayun2020towards}. Clips starting from the S1 heart sound with a fixed 1.6\,s length were used for classification in~\cite{gao2020gated}. In~\cite{dissanayake2020robust}, the importance of segmentation was demonstrated for abnormal heart sound detection. Compared to models without segmentation information, segmentation did not significantly improve the model performance in the experiments~\cite{dissanayake2020robust}. The reason can be that models have already been powerful and robust, therefore, segmentation can be automatically done by the intermediate models layers. The authors in~\cite{dissanayake2020robust} proved this by explaining models with SHapley Additive exPlanations (SHAP)~\cite{lundberg2017unified}. The contribution of S1 and S2 sounds was found to be larger than other clips in a heart sound segment. Therefore, segmentation is required either as an additional procedure for classifiers that are not very powerful, or as an internal procedure in robust classifiers.

\sstitle{No-segment} There are also many approaches which proposed to use non-segmented heart sounds, therefore, the complexity of automated auscultation can be reduced~\cite{xiao2020heart,zeng2021new,singh2019classification,alqudah2019towards,takezaki2021construction}.
Apart from feeding a complete heart sound sample into neural networks, heart sounds are segmented into shorter clips with an equal length for model training~\cite{wu2019applying}. For instance, the fist 5\,s of each audio sample were selected as the model input in~\cite{langley2017heart,singh2019classification1,chen2022heart,singh2020short}, and segmented 5\,s clips were also used in~\cite{tan2022heart,arora2019heart,cheng2019design}. Most studies employ audio clips with a time length from 2\,s to 6\,s~\cite{li2021heart,hu2020automatic,banerjee2020multi,krishnan2020automated}. 




\subsection{Limitations and Outlook}
\sstitle{Hardware development}
In clinics, echocardiography obtains ultrasound scan with a small probe which emits high-frequency sound waves, therefore, physicians can diagnose via observing the heart and blood vessels, as well as blood flows\footnote{https://www.nhs.uk/conditions/echocardiogram/}. However, echocardiography requires well-trained skills for professionals, limiting its usage in primary care. Classic acoustic stethoscopes used in primary care need physicians and nurses to be trained as well. To this end, there is a high demand of electronic stethoscopes in primary care. In recent years, electronic stethoscopes have been developed and produced to record heart sounds and transmit heart sounds to computers or mobile phones for further analysis~\cite{leng2015electronic}. Most electronic stethoscopes can only achieve basic functions such as amplifying and visualising heart sounds without diagnosis. There are a few studies and hardware working on automated diagnosis more recently. For instance, a field-programmable gate array (FPGA) was designed to classify heart sounds via an LSTM-RNN model in~\cite{jhong2020deep}. ``HD Steth with ECG''\footnote{https://www.stethoscope.com/hd-steth-with-ecg/} embedded artificial intelligence (AI) into the electronic stethoscope to detect multiple cardiac abnormalities. 
As outlined throughout this overview, 
AI is promising to diagnose heart sound abnormalities, therefore reducing the requirement of well-trained professionals. Devices which diagnose cardiac diseases with high accuracies will be very helpful for promoting the early screening of cardiac diseases to primary care and home care.

\sstitle{Performance improvement}
Although automated auscultation is ideally expected to replace human analysis, model performance can be a bottleneck for applying automated auscultation to clinical usages. False negative predictions can result in delayed or missed therapies and aggravated condition. In future efforts, i) automated auscultation will be required to achieve very good performance and consider the difference between individuals in the context of personalised healthcare. The current research studies are mostly based on heart sounds only, while many types of individual information such as age can affect model performance~\cite{thompson2019artificial}. Therefore, how to integrate individual information will be a question for performance improvement. 
ii) In terms of ML and DL, we are currently witnessing the advent and adoption of foundation models \cite{bommasani2021opportunities} (pre-)trained on large to big data. We have already seen and listed above several approaches using pre-trained often self-supervised learnt models. However, one can expect even bigger models to appear with the potential emergence of abilities directly related to heart sound analysis as a `downstream' task. On the other hand, the upcoming era of foundation models is expected to be marked by homogenisation, and it remains to be seen if the diversity of heart sound analysis approaches reported herein will indeed converge to a few large data trained models over the next years \cite{bommasani2021opportunities}.
iii) Furthermore, human-machine collaboration will be very promising to improve the system performance and provide more accurate diagnosis and in-time treatments for patients. Human-machine classification can be a solution to combine both machines' predictions and human's (crowd workers' and experts') predictions for more precise diagnosis~\cite{callaghan2018mechanicalheart}. In~\cite{callaghan2018mechanicalheart}, data samples predicted with high uncertainties were sent to crowd workers for majority voting; similarly, samples will be sent to an expert according to a certainty threshold of predictions from the crowd. 




\subsection{Interpretable, Dependable, and Actionable Deep Heart Sound Analysis}
Explainable deep learning has been emerging as one of the key topics in many applications, especially in the healthcare area. Building explainable deep learning models can help physicians and patients to trust the predictions. In a past study~\cite{ren2022deep}, an attention mechanism was used to visualise the contribution of each feature unit to the final predictions of heart sounds. Similarly, SHAP was also used to compute the contribution of each feature unit in~\cite{lundberg2017unified}. The above explanation methods are both local explanations that interpret DL models case-by-case, lacking of a global explanation that can reveal the hidden classification rule in the models or  summarise the characteristics of each heart sound class. Explainable DL models, such as deep neural decision trees~\cite{yang2018deep}, are promising to explain the models themselves from the perspectives of structure. Learnt or searched data samples of prototypes, criticisms, and counterfactuals~\cite{ren2021prototype,chang2022example} can present the typical characteristics of each class, therefore, physicians can compare new samples with these heart sounds for better understanding and analysis. Specifically, by analysing the patterns of criticisms, physicians can potentially decrease the number of false negatives, which is crucial in the healthcare field.
More recently, sonification was proposed to explain DL models for better human-computer interaction in~\cite{schuller2021towards}. Apart from visualisation, sonification can provide a new perspective to explain models by listening.



In addition, given the health implications, it appears crucial that heart sound analysis by AI is utmost dependable \cite{pelillo2021machines}. There are mechanisms available, but more adaptation to the field of application if not novel algorithms will need to be designed. Ultimately, dependability will be a major driving factor for the trustability of according heart monitoring solutions -- applied in everyday situations, trustability is a key factor to win users. 

Moreover, to enable DL models being actionable in real-life, data privacy has been another emerging research topic in protecting users' data from leakage or external attacks. 
Machine unlearning~\cite{NEURIPS2021_87f7ee4f} and federated learning methods~\cite{9871319} can help healthcare institutions better organise patients' private data in a secure way without losing diagnosis accuracy. 
Further, AI attacks on AI for heart sound analysis could be thought off, such as by adversarial attacks, and needs to dealt with.
 In summary, DL models are promising to guide healthcare providers' actions in their daily practices for providing better care for patients. It will be essential to improve DL models from not only the performance, but also the human-centred perspectives in future.
\section{Conclusion}
In this work, we summarised both classic machine learning and deep learning technologies applied to heart sound analysis during 2017--2022, including denoising, segmentation, and classification. Databases available for these above tasks were introduced with evaluation metrics in our study. We also listed publicly released codes for implementation of heart sound classification. Additionally, several findings and limitations of heart sound classification were analysed and possible future works were discussed. Finally, we discussed the importance of heart sound interpretation in the context of deep learning. This work is expected to present a summary of the advances of heart sound analysis, provide helpful discussions, and point out promising research directions that are helpful for the community.

\section{History}
\label{sec:history}
Before the 19th century, immediate auscultation conducted by a physician putting his/her ear directly to the patient's chest was disliked because of its implications from a social perspective and in technique~\cite{hanna2002history}. Immediate auscultation was even partially unacceptable due to the patient's age and gender~\cite{hanna2002history}. Therefore, physical examination was mostly limited to inspection and palpation, which were not available in some cases, e.\,g.,\  the great degree of 
body fat~\cite{hanna2002history}. Until 1816, the invention of the earliest stethoscope (Greek: stethos = chest, skopein = to view or to see) by a clinician, Ren\'e  Th\'eoophile Hyacinthe Laennec, started a new era of mediate auscultation~\cite{hanna2002history}. After Laennec's cylinder-like stethoscope, cardiac auscultation was widely applied in physical examination and the structure of stethoscope was also improved. Charles J.\ B.\ Williams designed a flexible monaural stethoscope with a trumpet-shaped head~\cite{hanna2002history}. Arthur Leared claimed  the invention of a binaural stethoscope for more flexibility in 1851~\cite{williams1907lecture}, and George Cammann further invented a binaural stethoscope for commercial production in 1852~\cite{silverman2019digital}. Afterwards, the stethoscope was successfully and widely used as a critical diagnostic tool. However, there are several difficulties to employ such traditional acoustic stethoscopes. Firstly, acoustic stethoscopes cannot record, play back, and process the heart sounds, thus limiting the teaching of auscultation~\cite{tavel2006cardiac}. Secondly, acoustic stethoscopes demand physicians' substantial clinical experience, but auscultation is a difficult skill that take years to acquire and refine~\cite{delgado2009digital,leng2015electronic}. Thirdly, the performance of the human ear is limited to its physical limitations~\cite{delgado2009digital}. Although the apparent sound amplification has been seen by many acoustic stethoscopes, the increasing insensitivity of the human ear at frequencies below 100\,Hz is still a problem for auscultation~\cite{abella1992comparison}. 
Finally, the sound level of acoustic stethoscopes is very low, so they are not very suitable in noisy environment~\cite{bakshi2014wireless}. To this end, developing equipment and technologies for computer-aided auscultation is necessary to overcome the limitations of acoustic stethoscopes.

As early as in the late 1950s and 1960s, the computational capability of digital electronic computers promoted their applications in heart disease diagnosis~\cite{10.1001/jama.1960.03020190008002,Gerbarg1963ComputerAO}. One direction was to deal with clinical data, such as diagnosis of congenital heart disease based on the incidence of clinical signs, symptoms, and electrocardiographic findings~\cite{10.1001/jama.1961.03040290005002}. Compared to processing human `predigested' clinical data, analysing raw physiologic records (e.\,g.,\  electrocardiogram) is a more automated direction to save the time and efforts of physicians~\cite{Gerbarg1963ComputerAO,5007946,10.1001/archinte.1963.03620260056010}. Additionally, another automated way similar to physicians' cardiac auscultation is to analyse magnetic tape recordings of heart sounds and murmurs~\cite{Gerbarg1963ComputerAO}. Automatically analysing recorded heart sounds and murmurs is promising to solve the aforementioned problems of acoustic stethoscopes.

Phonocardiograph equipments were developed in the 1930s and 1940s~\cite{sprague1957history}. The recorded PCG signals can be objectively analysed and explained. An early study~\cite{sprague1957history} reported the phonocardiograph is useful in timing and explaining whether an individual's heart sounds contain abnormal sounds: atrial sounds, opening snaps, and gallop rhythms. A phonocardiograph was also described to be valuable in studying murmurs, particularly in mitral disease and congenital heart disease~\cite{evans1948use,sprague1957history}. In the past decades, due to the development of digital technologies, computer-aided auscultation from PCG signals played an important role in pediatric cardiology, internal diseases, and evaluating congenital cardiac defects~\cite{quiceno2010selection}. Particularly, the electronic stethoscope, as a light phonocardiograph equipment, gains an edge over acoustic stethoscopes for automatically percepting, processing, and analysing heart sounds. Heart sounds are electronically amplified in electronic stethoscopes to overcome the low sound levels of acoustic stethoscopes~\cite{nowak2018sound}. 
Generally, the recorded PCG signals with an electronic stethoscope are transferred to a computer for visualisation and further analysis~\cite{leng2015electronic}. More recently, portable electronic stethoscopes were developed to be either connected to other mobile devices, e.\,g.,\  mobile phones, or to wirelessly transmit to a remote processing unit via a Bluetooth/wireless interface~\cite{leng2015electronic}. Consequently, electronic stethoscopes are able to be applied to several applications, such as real-time/remote monitoring and diagnosis~\cite{shin2013development,hadiyoso2020implementation}. In 2021, an ear-contactless electronic stethoscope was designed for auscultation of patients with coronavirus disease 2019~\cite{yang2021lowcost}. 



        





\subsection{Deep Learning for Classification}
\label{sec:dep_class}

\begin{figure*}
    \centering
    \includegraphics[width=.9\linewidth]{figures/dl-classification.pdf}
    \caption{Pipeline of DL models working on heart sounds. ``1'' indicates transfer learning; ``2'' means deep learning on the time-frequency representation; ``3'' depicts end-to-end learning. The three branches can be in parallel or assembled at the feature-/decision-level. DL can be also utilised for processing features apart from raw audio signals and time-frequency representations. The processing of DL on other features can be found in \autoref{sec:dep_class}.}
    \label{fig:dl_classification}
\end{figure*}

Different from classic machine learning, DL learns effective representations from either raw heart sounds or simple time-frequency representations using models with more parameters. DL often performs very well on many acoustic tasks benefiting from its strong capability. Apart from the pipeline shown in Fig.~\ref{fig:dl_classification}, we summarise the advances of deep learning methods for heart sound classification in the following.






\subsection{Deep Learning for Denoising}
\subsection{Deep Learning for Segmentation}







Various DL models have been proposed for heart sound segmentation. We group them into CNNs and RNNs which extract spatial and sequential representations, respectively.

\sstitle{Convolutional neural networks}
Inspired by successful applications of deep convolutional neural networks (CNNs) in image segmentation, deep CNNs have been applied to heart sound segmentation in recent studies~\cite{8620278}. For instance, several CNN-based segmentation algorithms were proposed and compared in~\cite{8620278}, including CNNs with sequential max temporal modelling, CNNs with HMMs or HSMMs to model the probability density distribution of observations.

\sstitle{Recurrent neural networks}
Recurrent neural networks (RNNs) have demonstrated their ability to exploit temporal information in sequential data. Therefore, RNNs can also help in locating the states of heart sounds.
In~\cite{8370727}, the authors regarded segmentation as an event detection task and developed bi-directional 
Gated Recurrent Unit (GRU)-RNNs based on spectrogram and envelop features.
Since envelope features fail to effectively model the intrinsic duration information of the heart cycles, a duration-LSTM was proposed in~\cite{CHEN2020106540} to integrate the duration vector into the standard LSTM cells with envelope features to obtain better segmentation performance. Specifically, duration parameters consist of heart cycle duration and systole duration estimated from the envelope autocorrelation.
Without envelopes and time-frequency based features, the authors of~\cite{8749010} utilised bi-directional GRU-RNNs to segment the heart sound directly.
Considering the possible noisy and irregular sequences in heart sound signals, an attention-based RNN framework was introduced in~\cite{8883031}. Specifically, before the final classification layer, with the hidden representation returned by bi-directional LSTMs, a single linear layer was applied to learn the weight score of each hidden state. Such weight score values are multiplied with the hidden representation for the final classification.

\sstitle{CNNs + RNNs}
An end-to-end model was proposed in~\cite{chen2021cnn}, where CNNs and LSTM recurrent neural networks (RNNs) are integrated to learn rich and efficient features from the audio directly. The gate structures of each LSTM unit are optimised in~\cite{8903349} for efficiency. 




\section{Applications}




\input{6-discussion}

\input{7-conclusion}







\bibliographystyle{ieeetr}
