
\section{Conclusions}
In this paper, we investigated a DL-enabled semantic communication system for speech recognition and speech synthesis tasks, named DeepSC-ST, which recovers the text transcription by utilizing the text-related semantic features and reconstructs the speech sample sequence at the receiver. Particularly, we design a joint semantic-channel coding scheme to learn and extract semantic features and mitigate the channel effects to achieve the speech recognition. Simulation results verified that the DeepSC-ST outperforms the conventional communication systems, especially in the low SNR regime. Moreover, we build a software demonstration to allow the real-time human speech input for the proof-of-concept. Our proposed DeepSC-ST is envisioned to be a promising candidate for semantic communication systems for speech recognition and speech synthesis tasks.

\section{Numerical Results}
In this section, we compare the proposed DeepSC-ST and the conventional communication systems under different channels, where the accurate CSI is assumed at the receiver. The experiment is conducted on the LJSpeech dataset~\cite{ljspeech17} for both speech recognition and speech synthesis tasks, which is a corpus of English speech with the sampling rate of 22,050 Hz. In the speech recognition task, the speech is down-sampled into 16,000 Hz. The simulation environment adopted in this work is Tensorflow 2.4.


\subsection{Simulation Setting and Benchmarks}
In the proposed DeepSC-ST, the numbers of CNN modules and BRNN modules in the~\emph{semantic encoder} are two and six, respectively. The number of~\emph{filters} for each CNN module is 32 and the number of GRU~\emph{units} for each BRNN module is 800. Moreover, two dense layers are utilized in the~\emph{channel encoder} with 40~\emph{units} and three dense layers are utilized in the~\emph{channel decoder} with 40, 40, and 29~\emph{units}, respectively. The batch size is $B=24$ and the learning rate is $\eta=0.0001$. The parameter settings of the proposed DeepSC-ST for speech recognition task are summarized in Table~\ref{DeepSC-ST NN parameters}. For performance comparison, we provide the following two benchmarks:


\subsubsection{\textbf{Benchmark 1}}
The first benchmark is a conventional system that transmits speech signals, named speech transceiver. The input of the system is the speech signals, which is restored at the receiver. Moreover, the text transcription is obtained from the recovered speech signals after passing through a speech recognition model. The adaptive multi-rate wideband (AMR-WB) system~\cite{1175533} is used for speech source coding and 64-QAM is utilized for modulation. Polar code with successive cancellation list (SCL) decoding algorithm~\cite{7114328} is employed for channel coding with the block length of 512 bits and the list size of four. Moreover, the speech recognition task aims to recover the text transcription accurately, which is realized by the Deep Speech 2 model~\cite{pmlr-v48-amodei16}.


\subsubsection{\textbf{Benchmark 2}}
The second benchmark is a conventional system that transmits text, named text transceiver. Particularly, the speech signals are converted into the text sequence before feeding into the conventional system and the text sequence is recovered at the receiver. The Huffman coding~\cite{4051119} is employed for text source coding in the system, the settings of channel coding and modulation are same as that in~\emph{benchmark 1}. In addition, The Deep Speech 2 model is utilized to implement speech recognition at the transmitter. Furthermore, the recovered text sequence is passed through the Tacotron 2 model to reconstruct the speech signals in the speech synthesis task.

As aforementioned, the semantic communication system transmits much less amount of data than the source information. We compute the average encoded symbols to transmit 16,000 original speech samples in the proposed DeepSC-ST, the speech transceiver, and the text transceiver, which is summarized in Table~\ref{symbols comparison}. From the table, the proposed DeepSC-ST reduces the transmission data by nearly ten times of the speech transceiver while the text transceiver needs average 57 encoded symbols because it only comprises few characters in the long speech sample sequence.


\subsection{Experiments for Speech Recognition Task}
The relationship between the CTC loss and the number of epochs is shown in Fig.~\ref{Figure: CTC loss VS epoch}. From the figure, when learning rate is 0.0005, the CTC loss converges slowest and has the highest value amongst the three learning rates, which experiences some fluctuations when epoch$>$30. When learning rate is 0.0001 or 0.00005, the CTC loss decreases to around 10 after 20 epochs and converges after about 40 epochs.
\renewcommand\arraystretch{1.15}
\begin{table}[tbp]
\footnotesize
\caption{Parameter settings for speech recognition task.}
\label{DeepSC-ST NN parameters}
\centering
\begin{tabular}{|c|c|c|c|}
\hline
               & \textbf{Layer Name}  & \textbf{Filters/Units}  & \textbf{Activation}    \\
\hline
            \textbf{Semantic}        &   2$\times$CNN modules   & 2$\times$32   & ReLU \\
\cline{2-4}
            \textbf{Encoder}         &  7$\times$BRNN modules  & 7$\times$800  & Tanh \\
\hline
            \textbf{Channel}        &        Dense layer        &    40         & ReLU \\
\cline{2-4}
            \textbf{Encoder}        &        Dense layer        &    40         & None \\
\hline            
   \multirow{3}{3.5em}{\textbf{\centering Channel\\Decoder}}  &  Dense layer  &  40  &  ReLU \\
\cline{2-4}
                                                              &  Dense layer  &  40  &  ReLU \\
\cline{2-4}
                                                              &  Dense layer  &  29  &  None \\
\hline
\end{tabular}
\end{table}
\renewcommand\arraystretch{1.15}
\begin{table}[tbp]
\footnotesize
\caption{Comparison of encoded symbols in different systems.}
\label{symbols comparison}
\centering
\begin{tabular}{|c|c|c|}
\hline
                       &   Original Speech Samples   &       Encoded Symbols \\
\hline
    DeepSC-ST          &           16,000            &              520          \\
\hline
    Speech Transceiver &           16,000            &             6,600         \\
\hline
    Text Transceiver   &           16,000            &              57           \\
\hline
\end{tabular}
\end{table}
\begin{figure}[tbp]
\includegraphics[width=0.58\textwidth]{Figures/simulation/recognition/CTC_different_lr.pdf}
\centering
\caption{The training CTC loss versus epoch.} 
\label{Figure: CTC loss VS epoch}  
\end{figure}

Fig.~\ref{CER result} compares the CER of the DeepSC-ST and two benchmarks for the AWGN channels, the Rayleigh channels, and the Rician channels, where the ground truth is the result tested by feeding the speech sample sequence into the Deep Speech 2 model directly without considering communication problems. From the figure, the DeepSC-ST has lower CER scores than the speech transceiver and the text transceiver under all tested channel environments. Particularly, the recognized text transcription in the proposed DeepSC-ST is readable when SNR is higher than -2 dB under the AWGN channels while the requited SNR is 4 dB in the speech transceiver and the text transceiver. In addition, the DeepSC-ST performs steadily when coping with dynamic channels and SNR$>$0 dB. However, the performance of two benchmarks is quite poor under different channel conditions. Moreover, the DeepSC-ST significantly outperforms the benchmarks when the SNR ranges from -12 dB to 4 dB for the AWGN channels, and -12 dB to 8 dB for the Rayleigh channels and the Rician channels. The fluctuation in the speech transceiver and the text transceiver for SNR$>$14 dB is because the Deep Speech 2 model is trained under the clear speech signals, which results in the slight uncertainty to process the speech signals at the similar noise level.

Fig.~\ref{WER result} compares the WER of different approaches. From the figure, the proposed DeepSC-ST can provide lower WER and outperform the speech transceiver under various channel conditions, as well as the text transceiver when SNR$<$8 dB. Moreover, similar to the results of CER, the DeepSC-ST has low WER on average when coping with channel variations while the conventional systems provide poor scores when SNR is low. According to the simulation results, the DeepSC-ST is able to achieve better performance to recover the text transcription at the receiver from the input speech signals at the transmitter when coping with the complicated communication scenarios than the conventional systems, especially in the low SNR regime.
\begin{figure*}[tbp]
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/cer_AWGN.pdf} 
\subcaption{AWGN channels}
\label{CER AWGN}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/cer_Rayleigh.pdf}
\subcaption{Rayleigh channels}
\label{CER Rayleigh}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/cer_Rician.pdf}
\subcaption{Rician channels}
\label{CER Rician}
\end{minipage}
\caption{CER score versus SNR for the speech transceiver, the text transceiver, the proposed DeepSC-ST.}
\label{CER result}
\end{figure*}
\begin{figure*}[tbp]
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/wer_AWGN.pdf} 
\subcaption{AWGN channels}
\label{WER AWGN}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/wer_Rayleigh.pdf}
\subcaption{Rayleigh channels}
\label{WER Rayleigh}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/recognition/wer_Rician.pdf}
\subcaption{Rician channels}
\label{WER Rician}
\end{minipage}
\caption{WER score versus SNR for the speech transceiver, the text transceiver, the proposed DeepSC-ST.}
\label{WER result}
\end{figure*}

In the speech transceiver and the text transceiver, the employed Deep Speech 2 model includes 85,788,733 trainable parameters while it is 85,796,042 in the proposed DeepSC-ST, which nearly increases no training cost than the adopted Deep Speech 2 model. Besides, the extra computational burden is required in the speech transceiver and the text transceiver due to the conventional encoding/decoding mechanism. According to this, the proposed DeepSC-ST is at the same level as the two benchmarks in terms of the system complexity.


\subsection{Experiments for Speech Synthesis Task}
In this experiment, we test FDSD and KDSD values of the received speech in the speech transceiver, the synthesized speech in the text transceiver, and the synthesized speech in the proposed DeepSC-ST. In practice, the audio is generally treated as unacceptable when it is unable to understand or full of background noise. Therefore, FDSD and KDSD thresholds are necessary to determine the validity of the audio. Inspired by this, a survey is developed to collect the opinions of 100 evaluators by selecting the satisfaction degree of the speech signals corresponding to different FDSD and KDSD values. Particularly, the Deep Speech 2 model is utilized to extract the aforementioned features, $\boldsymbol D$ and $\widehat{\boldsymbol D}$, to compute FDSD and KDSD values. The survey result is summarized in Table~\ref{FDSD/KDSD survey result}\footnote{Note that KDSD and FDSD values are computed under the adopted Deep Speech 2 model in our experiment, these values may different when the model parameters are varied or other models are employed.}. From the table, the speech signals can be considered as acceptable when their FDSD or KDSD values are less than 10 or 0.012, respectively.
\renewcommand\arraystretch{1.15}
\begin{table}[tbp]
\footnotesize
\caption{FDSD and KDSD survey result.}
\label{FDSD/KDSD survey result}
\centering
\begin{tabular}{|c|c|c|}
\hline
                &    FDSD    &    KDSD         \\
\hline
    Quite Clear &  $\leq2$   & $\leq0.001$     \\
\hline
    Clear       & $2\sim10$  & $0.001\sim0.012$ \\
\hline
    Acceptable  & $10\sim18$ & $0.012\sim0.025$ \\
\hline
    Noisy       & $18\sim30$ & $0.025\sim0.045$ \\
\hline
    Quite Noisy & $\geq30$   & $\geq0.045$ \\
\hline
\end{tabular}
\end{table}

The FDSD and KDSD results of the DeepSC-ST and two benchmarks are shown in Fig.~\ref{FDSD result} and Fig.~\ref{KDSD result}, respectively, where the ground truth is the KDSD and FDSD values computed by passing the plain text sequence through the Tacotron 2 model directly. From the figure, the DeepSC-ST achieves better speech recovery, i.e., lower FDSD and KDSD values, than the speech transceiver and the text transceiver from -8 dB to 2 dB under the AWGN channels, as well nearly -10 dB to 10 dB under the Rayleigh channels and the Rician channels. Particularly, when SNR is lower than around 4 dB under the AWGN channels, the received speech signals in the speech transceiver is invalid while the synthesized waveform in the proposed DeepSC-ST is acceptable for SNR$>$-10 dB, which proves the adaptability of DeepSC-ST in the low SNR regime. In addition, the proposed DeepSC-ST outperforms the speech transceiver and the text transceiver under the Rayleigh channels and the Rician channels amongst all the tested SNRs. Furthermore, the DeepSC-ST is robust to cope with the diverse channel environments because the SNR thresholds yielding the valid speech waveform are nearly same in the adopted channels.
\begin{figure*}[tbp]
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/AWGN/DS_distance_plot_FDSD.pdf} 
\subcaption{AWGN channels}
\label{FDSD AWGN}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/Rayleigh/DS_distance_plot_FDSD.pdf}
\subcaption{Rayleigh channels}
\label{FDSD Rayleigh}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/Rician/DS_distance_plot_FDSD.pdf}
\subcaption{Rician channels}
\label{FDSD Rician}
\end{minipage}
\caption{FDSD value versus SNR for the speech transceiver, the text transceiver, the proposed DeepSC-ST.}
\label{FDSD result}
\end{figure*}
\begin{figure*}[tbp]
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/AWGN/DS_distance_plot_KDSD.pdf} 
\subcaption{AWGN channels}
\label{KDSD AWGN}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/Rayleigh/DS_distance_plot_KDSD.pdf}
\subcaption{Rayleigh channels}
\label{KDSD Rayleigh}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1\textwidth]{Figures/simulation/synthesis/Rician/DS_distance_plot_KDSD.pdf}
\subcaption{Rician channels}
\label{KDSD Rician}
\end{minipage}
\caption{FDSD value versus SNR for the speech transceiver, the text transceiver, the proposed DeepSC-ST.}
\label{KDSD result}
\end{figure*}


\subsection{DeepSC-ST Demonstration}
According to the proposed DeepSC-ST, we design a software demonstration with user interface, as shown in Fig.~\ref{Figure: DeepSC-ST demo}\footnote{This software demonstration can be found at https://github.com/Zhenzi-Weng/DeepSC-ST$\_$demonstration.}. In the figure, users could choose a~\emph{.wav} file from the local testset as the source, or record the real-time speech sequence by a microphone device. In the ``Channel Condition'', the software demonstration allows users to select a channel fading amongst three different channels, i.e, the AWGN channel, the Rayleigh channel, and the Rician channel, besides, users can input an arbitrary value as the channel SNR. Afterwards, by clicking the ``Run'' button, the software demonstration will run a well-trained DeepSC-ST model under the user-defined transmitter input and channel condition. After running successfully, the user interface will display the recognized text and the local file directory of the synthesized speech, as shown in Fig.~\ref{Figure: DeepSC-ST demo}. By clicking the ``Play audio'' button, the software demonstration will play the synthesized speech by a speaker device. The representative results of the recognized text and the synthesized speech are shown in Table~\ref{comparison of sentences} and Fig.~\ref{spectrogram result}, respectively.
\begin{figure}[tbp]
\includegraphics[width=0.75\textwidth]{Figures/demo/DeepSC-ST_demo.pdf}
\centering
\caption{Software demonstration of the developed DeepSC-ST.} 
\label{Figure: DeepSC-ST demo}
\end{figure}
\renewcommand\arraystretch{1.15}
\begin{table*}[tbp]
\footnotesize
\caption{Recognized sentences in different systems over Rayleigh channel when SNR is 4 dB.}
\label{comparison of sentences}
\centering
\begin{tabular}{|c|c|}
\hline
    Original Sentence  & he concluded that school had nothing to offer him   \\
\hline
    DeepSC-ST          & he concluded that school had noghing to offer him    \\
\hline
    Speech Transceiver & it cood sili ite bebou a pims t lup ar of ig mote terigytit w \\
\hline
    Text Transceiver   & h  ea aahesourhhtntchchen ehaoeitdcofo offer him              \\
\hline
\end{tabular}
\end{table*}
\begin{figure*}[tbp]
\begin{minipage}[t]{0.5\linewidth}
\flushright
\subcaption{Original}
\includegraphics[width=0.85\textwidth]{Figures/spectrogram/original.pdf} 
\label{spectrogram orginal}
\end{minipage}
\begin{minipage}[t]{0.5\linewidth}
\flushleft
\subcaption{DeepSC-ST}
\includegraphics[width=0.85\textwidth]{Figures/spectrogram/DeepSCST.pdf}
\label{spectrogram DeepSC-ST}
\end{minipage}
\begin{minipage}[t]{0.5\linewidth}
\flushright
\subcaption{Speech transceiver}
\includegraphics[width=0.85\textwidth]{Figures/spectrogram/speech_transceiver.pdf}
\label{spectrogram speech transceiver}
\end{minipage}
\begin{minipage}[t]{0.5\linewidth}
\flushleft
\subcaption{Text transceiver}
\includegraphics[width=0.85\textwidth]{Figures/spectrogram/text_transceiver.pdf}
\label{spectrogram text transceiver}
\end{minipage}
\caption{Spectrograms of speech sample sequences reconstructed by different systems over Rayleigh channel when SNR is 4 dB.}
\label{spectrogram result}
\end{figure*}




\section{Introduction}
With the booming of artificial intelligence (AI) in the recent years, the unprecedented demands on intelligent applications require extremely high transmission efficiency and impose enormous challenges on conventional communication systems. According to Shannon and Weaver~\cite{weaver1953recent}, the ultimate goal of communications is to exchange semantic information, named semantic communications. The transmission efficiency can be significantly improved with semantic communications.

Semantic information represents the meaning and veracity of source information~\cite{carnap1952outline}. However, it is challenge to quantify the semantic information due to the lack of a mathematical model. The breakthroughs of deep learning (DL) make it possible to tackle many challenges without requiring a mathematical model. Recently, DL-enabled semantic communications have shown great potentials to break the bottlenecks in conventional communication systems~\cite{qin2019deep} and facilitate semantic information exchange. Moreover, in some applications, the receiver only requires some critical semantic information from the source, which motivates us to transmit the application-related semantic information. Therefore, task-oriented semantic communications have been regarded as a promising solution for the 6G and beyond~\cite{qin2022semantic}, especially when communication resources are limited.

According to the state-of-the-art of DL-enabled semantic communications, the transmission goal could be categorized into two types: source data reconstruction and intelligent task execution. To achieve the data reconstruction, the global semantic information is extracted for transmission. But to serve the intelligent tasks, the extracted semantic information only consists of the task-related semantic features and the other irrelative features can be ignored to minimize the data to be transmitted. Besides, the semantic information can be significantly compressed by employing a lossless compression method~\cite{basu2014preserving}, which guarantees the feasibility to transmit the task-related semantic information instead of the global semantic information to achieve very high bandwidth efficiency.

Semantic communications have attracted intensive research for text~\cite{8461983,9398576,jiang2021deep,9700645,liang2022life}, speech/audio~~\cite{Weng2101:Semantic,tong2021federated,shi2021new}, and image/video transmission~\cite{9464731,9685667,tung2021deepwive,jiang2022wireless}. For semantic-aware speech/audio transmission, Weng~\emph{et al.}~\cite{Weng2101:Semantic} first proposed a DL-enabled semantic communication system, named DeepSC-S, to extract the global semantic information by leveraging an attention mechanism-powered module and reconstruct the speech signals at the receiver. Tong~\emph{et al.}~\cite{tong2021federated} developed a multi-user audio semantic communication system to collaboratively train the convolution neural network (CNN)-based autoencoder by implementing federated learning over multiple devices and a server. Moreover, Shi~\emph{et al.}~\cite{shi2021new} designed an understanding and transmission architecture for semantic communications and verified its effectiveness by deploying the architecture into the speech transmission system, which converts speech signals into semantic symbols to ensure high semantic fidelity and decodes the received semantic symbols into a reconstructed speech waveform. Although semantic commutations have been designed for intelligent speech transmission, investigation on semantic communications to execute speech-centric intelligent tasks at the receiver is still missing.

In this paper, we propose a DL-enabled semantic communication system, named DeepSC-ST, for tasks like speech recognition and speech synthesis. Particularly, DeepSC-ST first learns and extracts the text-related semantic features from the input speech signals for transmission, and then recovers the text sequence at the receiver based on the received semantic features. By doing so, the characteristics of speech signals, e.g., the voice of speaker, speech delay, and background noise, etc., are omitted to be transmitted, which lowers the network traffic significantly. Besides, to reconstruct the speech signals efficiently, the recognized text is converted into the speech waveform through a speech synthesis module. The main contributions of this paper are summarized as follows:
\begin{itemize}
\item A novel semantic communication system, named DeepSC-ST, is first proposed, which integrates semantic and channel coding into a neural network.

\item The text-related semantic features are extracted for transmission by leveraging CNN and recurrent neural network (RNN) based semantic-channel coding scheme, which serves the speech recognition task at the receiver.

\item To reconstruct the input speech signals, the recognized text is fed into an independent speech synthesis module, which extracts the linguistic features by a CNN and RNN based neural network, and produces the speech waveform by a spectrogram prediction network.

\item A demonstration of the DeepSC-ST is built to produce the recognized text and the synthesized speech based on the real-time human speech input.
\end{itemize}

The rest of this article is structured as follows. Section \uppercase\expandafter{\romannumeral2} presents the related work. The model of the semantic communication system for speech recognition and speech synthesis, as well as the performance metrics are introduced in Section \uppercase\expandafter{\romannumeral3}. In Section \uppercase\expandafter{\romannumeral4}, the proposed DeepSC-ST is detailed. Simulation results are discussed in Section \uppercase\expandafter{\romannumeral5} and Section \uppercase\expandafter{\romannumeral6} draws conclusions.

\emph{Notation}: The single boldface letters are used to represent vectors or matrices and single plain capital letters denote integers. $x_i$ indicates the $i$-th component of vector $\boldsymbol x$, $\left\|\boldsymbol x\right\|$ denotes the Euclidean norm of $\boldsymbol x$. $\boldsymbol Y\in\mathfrak R^{M\times N}$ indicates that $\boldsymbol Y$ is a $M\times N$ real matrix. Superscript swash letters refer the blocks in the system, e.g., $\mathcal T$ in $\boldsymbol\theta^{\mathcal T}$ represents the parameter at the transmitter. $\mathcal{CN}(\boldsymbol m,\;\boldsymbol V)$ denotes multivariate circular complex Gaussian distribution with mean vector $\boldsymbol m$ and co-variance matrix $\boldsymbol V$. Moreover, $\boldsymbol a\ast\boldsymbol b$ represents the convolution operation of vectors $\boldsymbol a$ and $\boldsymbol b$.


\section{Semantic Communications for Speech Recognition and Synthesis}
In this section, we present the details of the DeepSC-ST. Specifically, in the speech recognition task, CNN and RNN are adopted for the semantic encoding, dense layers are employed for the channel encoding and decoding. In the speech synthesis task, the Tacotron 2~\cite{8461368} is utilized to reconstruct the speech waveform from the recognized text.


\subsection{Model Description}
The proposed DeepSC-ST is shown in Fig.~\ref{proposed sys}. In the figure, $\boldsymbol M$ is the set of speech sample sequences drawn from the speech dataset and is converted into a set of spectra, $\boldsymbol S=\left[{\boldsymbol S}_1,\;{\boldsymbol S}_2,\;...,\;{\boldsymbol S}_B\right]$, where $B$ is the batch size. In addition, $\boldsymbol T\in\mathfrak R^{B\times\widetilde K}$ is the set of text sequences, $\boldsymbol t$, corresponding to $\boldsymbol M\in\mathfrak R^{B\times Q}$, where $Q$ is the number of samples in a sample sequence and $\widetilde K$ indicates the numbers of tokens, $K$, in different batches of text sequences may be different. The input of the proposed DeepSC-ST is spectra, $\boldsymbol S\in\mathfrak R^{B\times N\times F}$, where $N$ denotes the number of frames and $F$ indicates the number of frequency bins corresponding to each speech frame. The spectra, $\boldsymbol S$, are fed into the~\emph{semantic encoder} to learn and extract the text-related semantic features and to output the features, $\boldsymbol P\in\mathfrak R^{B\times L\times29}$. The details of the~\emph{semantic encoder} are presented in part B of this section. Afterwards, the~\emph{channel encoder}, implemented by two dense layers, converts $\boldsymbol P$ into $\boldsymbol U\in\mathfrak R^{B\times L\times2Z}$. In order to transmit $\boldsymbol U$ into a physical channel, it is reshaped into symbols, $\boldsymbol X\in\mathfrak R^{B\times LZ\times2}$, via a reshape layer.

The received symbols, $\boldsymbol Y$, are reshaped into $\boldsymbol V\in\mathfrak R^{B\times F\times2N}$ before feeding into the~\emph{channel decoder}, represented by three dense layers. The output of the~\emph{channel decoder} is the recovered text-related semantic features, $\widehat{\boldsymbol P}\in\mathfrak R^{B\times L\times29}$. Next, the~\emph{greedy decoder}, i.e., the~\emph{feature decoder}, decodes $\widehat{\boldsymbol P}$ into the text transcriptions, $\widehat{\boldsymbol T}\in\mathfrak R^{B\times\widetilde{K^{'}}}$. Note that the numbers of tokens in different text transcriptions are typically different. The details of the~\emph{greedy decoder} are presented in part C of this section. The CTC loss is calculated at the end of the~\emph{channel decoder} and backpropagated to the transmitter. Thus, the trainable parameters, $\boldsymbol\theta=(\boldsymbol\theta^{\mathcal T},\;\boldsymbol\theta^{\mathcal R})$, can be updated simultaneously.
\begin{figure*}[tbp]
\includegraphics[width=1.0\textwidth]{Figures/system_figures/proposed_sys.pdf}
\centering
\caption{The proposed system architecture for semantic communication system for speech recognition and speech synthesis.} 
\label{proposed sys}
\end{figure*}

Based on the text transcriptions, $\widehat{\boldsymbol T}$, the speech sample sequences, $\widehat{\boldsymbol M}$, can be reconstructed by the~\emph{speech synthesis module}, i.e., Tacotron 2, by combing the corresponding speaker identity (ID). It is worth mentioning that Tacotron 2 is trained separately from the speech recognition task and can be omitted in the communication scenarios, where users only desire the text information. Some details of Tacotron 2 are introduced in part D of this section.


\subsection{Semantic Encoder}
The~\emph{semantic encoder} is constructed by the CNN and the gated recurrent unit (GRU)-based bidirectional recurrent neural network (BRNN)~\cite{650093} modules. As shown in Fig.~\ref{proposed sys}, the input spectra, $\boldsymbol S$, are first converted into the intermediate features via several CNN modules. Particularly, the number of~\emph{filters} in each CNN module is $E_p$, $p\in\left[1,\;2,\;\dots,\;P\right]$, and the output of the last CNN module is $\boldsymbol b\in\mathfrak R^{B\times C_P\times D_P\times E_P}$. Then, $\boldsymbol b$ is fed into $Q$ BRNN modules, successively, and produces $\boldsymbol d\in\mathfrak R^{B\times G_Q\times H_Q}$, where the number of GRU~\emph{units} in each BRNN module, $H_q$, $q\in\left[1,\;2,\;\dots,\;Q\right]$, is consistent. Finally, the text-related semantic features, $\boldsymbol P$, are obtained from $\boldsymbol d$ by passing through multiple cascaded dense layers and a softmax layer.


\subsection{Greedy Decoder}
As aforementioned that the recovered text features, $\widehat{\boldsymbol P}$, are decoded into the text transcriptions, $\widehat{\boldsymbol T}$, via the~\emph{greedy decoder}. An example to obtain the text transcription by the~\emph{greedy decoder} is shown in Fig.~\ref{greedy decoder}. During the decoding process, in each step $l$, the token corresponding to the maximum probability in the vector, ${\widehat{\boldsymbol p}}_l$, is used to construct the final text transcription, $\widehat{\boldsymbol t}$.
\begin{figure}[tbp]
\includegraphics[width=0.6\textwidth]{Figures/system_figures/greedy_decoder.pdf}
\centering
\caption{An example of the~\emph{greedy decoder}.} 
\label{greedy decoder}
\end{figure}


\subsection{Training and Testing for Speech Recognition Task}
According to the prior knowledge of CSI, the training and testing algorithms for speech recognition task are described in Algorithm~\ref{training algorithm} and Algorithm~\ref{testing algorithm}, respectively. During the training stage, the~\emph{greedy decoder} and the~\emph{speech synthesis module} are omitted.


\subsection{Speech Synthesis Module}\label{speech synthesis details}
As shown in Fig.~\ref{proposed sys}, Tacotron 2 is adopted to reconstruct the speech sample sequences, $\widehat{\boldsymbol M}$, as close to the input sample sequences, $\boldsymbol M$, as possible. Tacotron 2 is composed of the spectrogram prediction network and a variant of WaveNet vocoder~\cite{tamamori2017speaker}, which converts the input character sequence into the mel-frequency spectrogram and produces time-domain waveform according to the predicted spectrogram, respectively. The mel-frequency spectrogram is obtained by implementing a nonlinear process to the frequency of the short-time Fourier transform (STFT) magnitude, which is straightforward for the WaveNet model to generate audio owing to the simple and low-level acoustic characteristic. Particularly, an encoder maps the input character sequence into the internal feature representation after a 512-dimensional character embedding, which is achieved by a stack of three convolutional layers with 512~\emph{filters} followed by batch normalization and ReLU activation, as well as a bidirectional long short-term memory (LSTM) layer containing 512~\emph{units}. 
\begin{algorithm}[tbp]
\caption{Training algorithm for speech recognition task.}
\label{training algorithm}
\textbf{Initialization:} initialize parameters $\boldsymbol\theta^{(0)}$, $i=0$.

\begin{algorithmic}[1]
    \State \textbf{Input:} Speech sample sequences $\boldsymbol M$ and transcriptions $\boldsymbol T$ from trainset $\mathfrak S$, fading channel $\boldsymbol H$, noise $\boldsymbol W$.
    \State Generate spectra $\boldsymbol S$ from sample sequences $\boldsymbol M$.
        \While{CTC loss is not converged}
            \State $\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\boldsymbol S))\rightarrow\boldsymbol X$.
            \State Transmit $\boldsymbol X$ and receive $\boldsymbol Y$ via (\ref{channel}).
            \State $\mathbf R_{\boldsymbol\delta}^{\mathcal S}(\boldsymbol Y)\rightarrow\widehat{\boldsymbol P}$.
            \State Compute loss ${\mathcal L}_{CTC}(\boldsymbol\theta)$ via (\ref{CTC loss}).
            \State Update parameters $\boldsymbol\theta$ via SGD according to (\ref{SGD}).
            \State $i\leftarrow i+1$.
        \EndWhile
    \State \textbf{end while}
    \State \textbf{Output:} Trained networks $\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\cdot)$, $\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\cdot)$, and $\mathbf R_{\boldsymbol\delta}^{\mathcal C}(\cdot)$.
\end{algorithmic}

\end{algorithm}
\begin{algorithm}[tbp]
\caption{Testing algorithm for speech recognition task.}
\label{testing algorithm}

\begin{algorithmic}[1]  
    \State \textbf{Input:} Speech sample sequences $\boldsymbol M$ from testset, trained networks $\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\cdot)$, $\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\cdot)$, and $\mathbf R_{\boldsymbol\delta}^{\mathcal C}(\cdot)$, testing channel set $\mathcal H$, a wide range of SNR regime.
    \State Generate spectra $\boldsymbol S$ from sample sequences $\boldsymbol M$.
    	\For{channel condition $\boldsymbol H$ drawn from $\mathcal H$}
    	    \For{each SNR value}
    	        \State Generate Gaussian noise $\boldsymbol W$ under the SNR value.
    	        \State $\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\boldsymbol S))\rightarrow\boldsymbol X$.
                \State Transmit $\boldsymbol X$ and receive $\boldsymbol Y$ via (\ref{channel}).
                \State $\mathbf R_{\boldsymbol\delta}^{\mathcal S}(\boldsymbol Y)\rightarrow\widehat{\boldsymbol P}$.
                \State Decoding $\widehat{\boldsymbol P}$ into $\widehat{\boldsymbol T}$ via (\ref{feature decoder}).
                \EndFor
            \State \textbf{end for}
        \EndFor
    \State \textbf{end for}
	\State \textbf{Output:} Recovered text transcriptions, $\widehat{\boldsymbol S}$.
\end{algorithmic}

\end{algorithm}

As a consequence, the encoded features are processed by a decoder enabled by an attention-based neural network to predict the mel-frequency spectrogram frame by frame. In details, the previous synthesized frame is fed into a~\emph{pre-net} with two connected layers of 256 ReLU~\emph{units}, then the output is combined with the encoded features and the speaker ID to predict the current frame after passing through a stack of two LSTM layers containing 1024~\emph{units} and a~\emph{post-net} with a stack of five convolutional layers with 512~\emph{filters} followed by batch normalization and ReLU activation. The spectrogram prediction network is trained independently by minimizing the sum of the mean-squared error (MSE) before and after the~\emph{post-net}.

Furthermore, the mel-frequency spectrogram frames are fed into the WaveNet vocoder to predict the parameters, e.g., mean and log scale, of the synthesized waveform after a ReLU activation followed by a linear projection, which encourages to train the WaveNet vocoder by maximizing the log-likelihood of the speech waveform w.r.t. the trainable parameters due to the tractability of log-likelihoods.

\section{Related Work}
In this section, we introduce the related work on DL-enabled semantic communication systems and present the state-of-the-art models for speech recognition and speech synthesis.


\subsection{Semantic Communication Systems}
Semantic communications have attracted extensive research interest very recently. Particularly, the transformer-powered system for text transmission, named DeepSC, has been proposed in~\cite{9398576} to measure the semantic error at the word level instead of the sentence level. Besides, a variant of DeepSC has been developed in~\cite{jiang2021deep} to further reduce the transmission error by leveraging hybrid automatic repeat request (HARQ) to improve the reliability of semantic transmission. Furthermore, the reasoning-based semantic communicator (R-SC) architecture in~\cite{liang2022life} automatically infers the hidden information by an inference function-based approach and introduces a life-long method to learn from previously received messages. In addition to perform text transmission, semantic communications for serving the text-based intelligent tasks have been investigated. Particularly, the transformer-enabled model, named DeepSC-MT in~\cite{xie2021task} can achieve the machine translation task by learning the word distribution of target language to map the meaning of source sentences to the target language. Besides, a visual question-answering (VQA) task has been investigated in DeepSC-MT by designing a multi-modal system to compress the correlated text-image semantic features, and then to perform the information query at the receiver before fusing the text-image information to infer an accurate answer.

In semantic communications for image and video transmission, the DL-enabled semantic communication system for image transmission in~\cite{9685667} utilizes a generative adversarial network (GAN)-based semantic coding scheme to interpret the meaning of images and reconstruct images with high semantic fidelity. In~\cite{jiang2022wireless}, a basal semantic video conference network has been established, which considers the impact of channel feedback and designs a semantic detector to detect semantic error by leveraging the identity classifier and fluency detection. Inspired by the booming of computer vision, semantic communications for serving intelligent vision tasks have shown great potentials to tackle many challenges beyond human limits. Particularly, the image classification task at the edge server has been investigated in~\cite{jankowski2020wireless}, which imposes IoT devices to compress images to the low computational complexity and to reduce the required transmission bandwidth. The semantic communication system for image classification in~\cite{9606667} adopts information bottleneck~\cite{tishby2000information} framework to identify the optimal tradeoff between compression rate and classification accuracy. In~\cite{kang2021task}, the deep reinforcement learning-enabled semantic communication system has been developed for joint image transmission and scene classification. Furthermore, a robust semantic communication system to combat semantic error has been proposed in~\cite{hu2022robust}, which incorporates the image with the generated semantic noise and utilizes the masked autoencoder to mitigate the effect of semantic noise with the aid of the discrete codebook shared by the transmitter and the receiver.


\subsection{Speech Recognition}
The exploration of speech recognition can be tracked back to several decades ago by exploiting the hidden Markov model~\cite{18626}. Afterwards, the neural network-powered speech recognition has experienced remarkable improvements owing to the thriving of natural language processing (NLP). About a decade ago, deep neural network (DNN) has been utilized to construct speech recognition system~\cite{6296526}. The speech recognition systems leveraging CNN can achieve high recognition accuracy~\cite{6288864,6857341}. Besides, the RNN-based speech recognition systems in~\cite{6638947,sak14_interspeech} analyse the temporal dependencies of excessively long speech sequences and greatly increase the recognition accuracy. Moreover, they can be further improved by combing convolutional layers for functional feature extraction~\cite{pmlr-v48-amodei16}. Furthermore, the attention-enabled speech recognition systems have been widely utilized due to their capabilities on interaction amongst long sentences and superior training efficiency~\cite{9053896,gulati2020conformer}.


\subsection{Speech Synthesis}
Converting text into natural human speech is a challenging task. Despite various breakthrough in this filed~\cite{541110,6639215,6495700}, the produced speech still sounds mechanical and unnatural. While the synthesized speech quality can be comparable to real human voice by generating waveform in the time domain from the input linguistic features due to the advent of WaveNet~\cite{oord2016wavenet}. Moreover, WaveNet is also employed in some speech synthesis systems, e.g., Deep Voice 3~\cite{ping2017deep}. The sequence-to-sequence architecture, named Tacotron~\cite{Wang2017TacotronTE}, maps the text into magnitude spectrogram instead of linguistic and acoustic features, and simplifies the conventional synthesis procedure by a single neural network trained separately. The improved system, Tacotron 2, in~\cite{8461368} produces the speech waveform from normalized character sequences, which can synthesize the realistic human voice. More recently, some non-autoregressive speech synthesis pipelines have been developed~\cite{9054484,donahue2020end,ren2020fastspeech} to eliminate the time dependency of the produced waveform.
\section{System Model}
The semantic communication system for speech recognition and speech synthesis first extracts and transmits the text-related semantic features, then recognizes the text sequence based on the received semantic features. Finally, the speech waveform is reconstructed at the receiver according to the recognized text. In this section, we introduce the considered system model and performance metrics for speech recognition and speech synthesis tasks.


\subsection{Input Spectrum and Text Information}
The input speech sample sequence is converted into the spectrum before feeding into the transmitter. First, the input speech sample sequence, $\boldsymbol m=\left[m_1,\boldsymbol\;m_2,\;\dots,\;m_Q\right]$, is divided into $N$ frames, then these frames are converted into the spectrum through the Hamming window, fast Fourier transform (FFT), logarithm operation, and normalization. By doing so, the spectrum, $\boldsymbol s=\left[{\boldsymbol s}_{\mathbf1},\boldsymbol\;{\boldsymbol s}_{\mathbf2},\;\dots,\;{\boldsymbol s}_{\mathbf N}\right]$, contains the characteristics of the sample sequence, $\boldsymbol m$.

Moreover, denote $\boldsymbol t$ as the corresponding text of the single speech sample sequence, $\boldsymbol m$. The ultimate goal of the speech recognition task is to recover the final text transcription, $\widehat{\boldsymbol t}$, as close to $\boldsymbol t$ as possible. Denote $\boldsymbol t=\left[t_1,\boldsymbol\;t_2,\;\dots,\;t_K\right]$, where $t_k$ is a token from the token set, $\boldsymbol t$, that could be a character in the alphabet or a word boundary. For English, there are 26 characters in the alphabet. Then there are 29 tokens if including $apostrophe$, $space$, and $blank$ as word boundaries, that is, $\overline{\boldsymbol t}=\left[\mathrm a,\;\mathrm b,\;\mathrm c,\;\dots,\;\mathrm z,\;apostrophe,\;space,\;blank\right]$.


\subsection{Transmitter}
Based on the input spectrum and text sequence of the speech sample sequence, the proposed system model is shown in Fig.~\ref{sys model}. In the figure, the transmitter consists of the~\emph{semantic encoder} and the~\emph{channel encoder}, implemented by two neural networks. At the transmitter, the input spectrum, $\boldsymbol s$, is converted into the text-related semantic features, $\boldsymbol p$, by the~\emph{semantic encoder}, and these features are mapped into symbols, $\boldsymbol x$, by the~\emph{channel encoder} to be transmitted over physical channels. Denote the neural network parameters of the~\emph{semantic encoder} and the~\emph{channel encoder} as $\boldsymbol\alpha$ and $\boldsymbol\beta$, respectively, then the neural network parameters at the transmitter can be expressed as $\boldsymbol\theta^{\mathcal T}=(\boldsymbol\alpha,\boldsymbol\;\boldsymbol\beta)$. Hence, the encoded symbols, $\boldsymbol x$, can be expressed as
\begin{equation}
\boldsymbol x=\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\boldsymbol s)),
\label{auto-encoder}
\end{equation}
where $\mathbf T_{\boldsymbol\alpha}^{\mathcal S}(\cdot)$ and $\mathbf T_{\boldsymbol\beta}^{\mathcal C}(\cdot)$ indicate the~\emph{semantic encoder} and the~\emph{channel encoder} with respect to (w.r.t.) parameters $\boldsymbol\alpha$ and $\boldsymbol\beta$, respectively.

The encoded symbols, $\boldsymbol x$, are transmitted over a physical channel. $\boldsymbol x$ is assumed to be normalized, i.e., $\mathbb{E}\left\|\boldsymbol x\right\|^2=1$. 

In Fig.~\ref{sys model}, the wireless channel, represented by $p_h\left(\left.\boldsymbol y\right|\boldsymbol x\right)$, takes $\boldsymbol x$ as the input and produces the output, $\boldsymbol y$, as the received symbols. Denote the coefficients of a linear channel as $\boldsymbol h$, then the transmission process from the transmitter to the receiver can be modeled as
\begin{equation}
\boldsymbol y=\boldsymbol h\ast\boldsymbol x+\boldsymbol w,
\label{channel}
\end{equation}
where $\boldsymbol w\sim\mathcal{CN}(0,\;\sigma^2\mathbf I)$ denotes independent and identically distributed (i.i.d.) Gaussian noise vector with variance $\sigma^2$ for each channel.
\begin{figure*}[tbp]
\centering
\includegraphics[width=0.9\textwidth]{Figures/system_figures/sys_model.pdf} 
\caption{Structure of DL-enabled semantic communication system for speech recognition and speech synthesis.}
\label{sys model}
\end{figure*}

\subsection{Receiver}
As in Fig.~\ref{sys model}, the receiver includes the~\emph{channel decoder} and the~\emph{feature decoder} to recover the text-related semantic features and recognize the final text transcription as close to the raw text sequence as possible. First, the received symbols, $\boldsymbol y$, is mapped into the text-related semantic features, $\widehat{\boldsymbol p}$, by the~\emph{channel decoder}, where $\widehat{\boldsymbol p}=\left[{\widehat{\boldsymbol p}}_1,\;{\widehat{\boldsymbol p}}_2,\;\dots,\;{\widehat{\boldsymbol p}}_L\right]$ denotes a probability matrix and probability vector ${\widehat{\boldsymbol p}}_l=\left[\widehat p_l^1,\;\widehat p_l^2,\;\dots,\;\widehat p_l^{29}\right]$ comprises 29 probabilities corresponding to 29 tokens in $\overline{\boldsymbol t}$. Denote the neural network parameters of the~\emph{channel decoder} as $\boldsymbol\theta^{\mathcal R}$, then the recovered features, $\widehat{\boldsymbol p}$, can be obtained from the received symbols, $\boldsymbol y$, by
\begin{equation}
\widehat{\boldsymbol p}=\mathbf R_{\boldsymbol\theta^{\mathcal R}}^{\mathcal S}(\boldsymbol y),
\label{auto-decoder}
\end{equation}
where $\mathbf R_{\boldsymbol\theta^{\mathcal R}}^{\mathcal S}(\cdot)$ indicates the~\emph{channel decoder} w.r.t. parameters $\boldsymbol\theta^{\mathcal R}$.

Then, the text-related semantic features, $\widehat{\boldsymbol p}$, are decoded into the text transcription, $\widehat{\boldsymbol t}$, by the~\emph{feature decoder}, denoted as
\begin{equation}
\widehat{\boldsymbol t}=\mathbf R^{\mathcal F}(\widehat{\boldsymbol p}),
\label{feature decoder}
\end{equation}
where $\mathbf R^{\mathcal F}(\cdot)$ represents the~\emph{feature decoder}.

It is worth mentioning that the process to decode the probabilities into the text transcription by the~\emph{feature decoder} is unable to be implemented by the neural network due to the non-differentiability. Moreover, the text transcription, $\widehat{\boldsymbol t}=\left[{\widehat t}_1,\boldsymbol\;{\widehat t}_2,\;\dots,\;{\widehat t}_{K'}\right]$, includes $K'$ tokens, where $K'$ may be different from the number of tokens, $K$, in the original text sequence, $\boldsymbol t$.

The objective of the semantic communication system for speech recognition task is to recover the text information of the input speech signals, which is equivalent to maximize the posterior probability $p\left(\left.\boldsymbol t\right|\boldsymbol s\right)$. By introducing connectionist temporal classification (CTC)~\cite{graves2006connectionist}, the posterior probability $p\left(\left.\boldsymbol t\right|\boldsymbol s\right)$ can be expressed as
\begin{equation}
p\left(\left.\boldsymbol t\right|\boldsymbol s\right)=\sum_{A\in\mathfrak A(\boldsymbol s,\boldsymbol\;\boldsymbol t)}\left(\prod_{l=1}^L{\widehat p}_l\left(\left.a_l\right|\boldsymbol s\right)\right),
\label{posterior probability CTC}
\end{equation}
where $\mathfrak A(\boldsymbol s,\boldsymbol\;\boldsymbol t)$ represents the set of all possible valid alignments of text sequence $\boldsymbol t$ to spectrum $\boldsymbol s$, and $a_l$ is the token under the valid alignments. For example, if text sequence $\boldsymbol t=\left[\mathrm t,\;\mathrm a,\;\mathrm s,\;\mathrm t,\;\mathrm e\right]$, the valid alignments could be $\left[blank,\;\mathrm t,\;blank,\;\mathrm a,\;\mathrm s,\;blank,\;\mathrm t,\;\mathrm e\right]$, or $\left[\mathrm t,\;blank,\;\mathrm a,\;\mathrm s,\;blank,\;blank,\;\mathrm t,\;\mathrm e\right]$, etc., because the $blank$ token is removed when obtaining the final text transcription, $\widehat{\boldsymbol t}$. Note that the number of tokens in every valid alignment is $L$. If the valid alignment is $\left[blank,\;\mathrm t,\;blank,\;\mathrm a,\;\mathrm s,\;blank,\;\mathrm t,\;\mathrm e\right]$, the first token is $blank$, i.e., $a_1=blank$, then we have
\begin{equation}
{\widehat p}_l\left(\left.a_l\right|\boldsymbol s\right)={\widehat p}_l\left(\left.blank\right|\boldsymbol s\right)=\widehat p_l^{29},\;\;\;\;\;l=1,
\label{probability pl}
\end{equation}
where $\widehat p_l^{29}$ is one of the probabilities in probability vector ${\widehat{\boldsymbol p}}_l$ and number 29 represents the $blank$ token is the 29th token in $\overline{\boldsymbol t}$.

To maximize the posterior probability $p\left(\left.\boldsymbol t\right|\boldsymbol s\right)$, the CTC loss is adopted as the loss function for speech recognition task in our system, denoted as
\begin{equation}
{\mathcal L}_{CTC}(\boldsymbol\theta)=-\ln\left(\sum_{A\in\mathfrak A(\boldsymbol s,\boldsymbol\;\boldsymbol t)}\left(\prod_{l=1}^L{\widehat p}_l\left(\left.a_l\right|\boldsymbol s,\boldsymbol\theta\right)\right)\right),
\label{CTC loss}
\end{equation}
where $\boldsymbol\theta$ denotes the neural network parameters of the transmitter and the receiver, $\boldsymbol\theta=(\boldsymbol\theta^{\mathcal T},\;\boldsymbol\theta^{\mathcal R})$. 

Moreover, for given prior channel state information (CSI), the neural network parameters, $\boldsymbol\theta$, can be updated by the stochastic gradient descent (SGD) algorithm as follows,
\begin{equation}
\boldsymbol\theta^{(i+1)}\leftarrow\boldsymbol\theta^{(i)}-\eta\nabla_{\boldsymbol\theta^{(i)}}{\mathcal L}_{CTC}(\boldsymbol\theta),
\label{SGD}
\end{equation}
where $\eta>0$ is a learning rate and $\nabla$ indicates the nabla operator.


\subsection{Speech Synthesis Module}
By introducing a flexible task mechanism, the produced information at the receiver not only limited to the text transcription, but also the speech information, which grants users the privilege to check the speech characteristics and expands the system diversity. As shown in Fig.~\ref{sys model}, the input to the~\emph{speech synthesis module} refers to the recognized text transcription, $\widehat{\boldsymbol t}$, and the~\emph{speech synthesis module}, represented by a neural network, processes and converts $\widehat{\boldsymbol t}$ into the speech sample sequence, $\widehat{\boldsymbol m}$, which dedicates to reconstruct the speech waveform as close to the original sample sequence, $\boldsymbol m$, as possible. Denote the neural network parameters of the~\emph{speech synthesis module} as ${\boldsymbol\chi}$, then the reconstructed speech sample sequence, $\widehat{\boldsymbol m}$, can be denoted as
\begin{equation}
\widehat{\boldsymbol m}=\mathbf R_{\boldsymbol\chi}^{\mathcal{SS}}\left(\widehat{\boldsymbol t}\right),
\label{speech synthesis module}
\end{equation}
where $\mathbf R_{\boldsymbol\chi}^{\mathcal{SS}}\left(\cdot\right)$ indicates the~\emph{speech synthesis module} w.r.t. parameters $\boldsymbol\chi$.


\subsection{Performance Metrics}
\subsubsection{Speech Recognition Task}
As aforementioned that text information is one type of semantic information required by the receiver in speech semantic communications. In this task, the semantic similarity between the recovered text and the original text is equivalent to measuring the accuracy of the recognized text transcription. Thanks to the advanced developments in speech recognition field, character error-rate (CER) and word error-rate (WER) are effective metrics to indicate the accuracy of the recognized text transcription. Therefore, we adopt CER and WER as two performance metrics for the speech recognition task. According to the text transcription, $\widehat{\boldsymbol t}$, the substitution, deletion, and insertion operations in character are utilized to restore the raw text sequence, $\boldsymbol t$. The calculation of CER can be denoted as
\begin{equation}
\mathrm{CER}=\frac{S_C+D_C+I_C}{N_C},
\label{CER}
\end{equation}
where $S_C$, $D_C$, and $I_C$ represent the numbers of character substations, deletions, and insertions, respectively, and $N_C$ is the number of characters in $\boldsymbol t$.

Similarly, the substitution, deletion, and insertion operations in word are employed to calculate WER, which can be expressed as
\begin{equation}
\mathrm{WER}=\frac{S_W+D_W+I_W}{N_W},
\label{WER}
\end{equation}
where $S_W$, $D_W$, and $I_W$ denote the numbers of word substations, deletions, and insertions, respectively, and $N_W$ is the number of words in $\boldsymbol t$.

Note that CER and WER may exceed one owing to a large number of deletions. Moreover, for the same sentence, CER is typically lower than WER. In reality, the recognized sentence is usually readable when CER is lower than around 0.15.


\subsubsection{Speech Synthesis Task}
Fréchet distance and maximum mean discrepancy (MMD) are usually utilized to evaluate the similarity of two distributions. In this task, we adopt unconditional Fréchet deep speech distance (FDSD) and unconditional kernel deep speech distance (KDSD) in~\cite{binkowski2019high} as two quantitative metrics to assess the quality of synthesized speech sample sequence. The lower the FDSD or KDSD values, the higher similarity between real and synthesized speech sample sequences.

Given the real speech sample sequence, $\boldsymbol m$, the synthesized speech sample sequence, $\widehat{\boldsymbol m}$, and a publicly available deep speech recognition model, the features of real and synthesized speech sample sequences, denoted as $\boldsymbol D\in\mathfrak R^{U\times V}$ and $\widehat{\boldsymbol D}\in\mathfrak R^{\widehat U\times V}$ respectively, can be extracted by passing $\boldsymbol m$ and $\widehat{\boldsymbol m}$ through the speech recognition model, respectively. Therefore, their FDSD can be calculated as
\begin{equation}
\begin{split}
    \mathrm{FDSD}^2=\left\|{\boldsymbol\mu}_{\boldsymbol D}-{\boldsymbol\mu}_{\widehat{\boldsymbol D}}\right\|^2+\mathrm{Tr}\left[{\boldsymbol\Sigma}_{\boldsymbol D}+{\boldsymbol\Sigma}_{\widehat{\boldsymbol D}}-\left({\boldsymbol\Sigma}_{\boldsymbol D}{\boldsymbol\Sigma}_{\widehat{\boldsymbol D}}\right)^\frac12\right],
\end{split}
\label{cFDSD}
\end{equation}
where ${\boldsymbol\mu}_{\boldsymbol D}$ and ${\boldsymbol\mu}_{\widehat{\boldsymbol D}}$ represent the means of $\boldsymbol D$ and $\widehat{\boldsymbol D}$, respectively, ${\boldsymbol\Sigma}_{\boldsymbol D}$ and ${\boldsymbol\Sigma}_{\widehat{\boldsymbol D}}$ denote their covariance matrices, and $\mathrm{Tr}(\cdot)$ indicates the trace of a square matrix.

On the other hand, KDSD can be obtained by~(\ref{cKDSD}),
\begin{equation}
\begin{split}
    \mathrm{KDSD}^2&=\frac1{U\left(U-1\right)}\sum_{\underset{i\neq j}{1\leq i,j\leq U}}kf\left({\boldsymbol D}_{i,}{\widehat{\boldsymbol D}}_j\right)+\frac1{\widehat U\left(\widehat U-1\right)}\sum_{\underset{i\neq j}{1\leq i,j\leq\widehat U}}kf\left({\boldsymbol D}_{i,}{\widehat{\boldsymbol D}}_j\right) \\
    &+\sum_{i=1}^U\sum_{j=1}^{\widehat U}kf\left({\boldsymbol D}_{i,}{\widehat{\boldsymbol D}}_j\right),
\end{split}
\label{cKDSD}
\end{equation}
where $kf(\cdot)$ is a kernel function, defined as
\begin{equation}
kf\left({\boldsymbol D}_{i,}{\widehat{\boldsymbol D}}_j\right)=\left(\frac1V{\boldsymbol D}_i{\widehat{\boldsymbol D}_j}^T+1\right)^3.
\label{kernel function}
\end{equation}


