


\subsection{Automatic Speech Recognition}

\input{experiments_librispeech}






\subsection{Keyword Detection}

\input{experiment_speechcmd}



\subsection{Controllable Speech Generative Model}
\vspace{-0.1cm}
Controllable speech generative models can generate speech in a given style, where the content is specified by an input text. 
As shown in Figure~\ref{fig:motivation}, for effective learning with synthetic data, we need a generative model that has \emph{as small gap as possible} with the true distribution. 
Therefore, the generative model should be able to synthesize any text in a wide range of speaking styles in various environmental conditions such as background noise, microphone response, etc. without introducing artifacts to the signal. 
To this end, we utilize a recently introduced auto regressive controllable generative model called Style Equalization~\cite{style_eq_chang_2021} that enables learning a controllable generative model in the wild (thousands of speakers recorded in various conditions) without requiring style labels. 
As shown in Figure~\ref{fig:overview}(a), the model takes a text and a reference style speech as input and utters the text in the style of the given reference. 
In this model, the style is broadly defined, not only the speaker’s voice characteristics but also all aspects of the
reference speech sample, including background noise, echo, microphone response, etc., which are necessary to have a smaller distribution gap (particularly missing samples). 
This model also allows sampling styles from a prior distribution that is important for sampling new styles not contained in the dataset. 
We refer readers to~\cite{style_eq_chang_2021} for a detailed description of this model.






\vspace{-0.3cm}
\subsection{Rejection Sampling}


\label{method:rej_sampling}
Let $\boldsymbol x$ be a speech signal and $y$ be the corresponding token label, and $p_d(\boldsymbol x, y)$, $p_g(\boldsymbol x, y)$ denote the joint true and synthetic data distributions, respectively.
We use the synthesizer as proposal distribution and use rejection sampling to make it closer to the true distribution.
Following standard rejection sampling~\cite{prml_book}, a synthetic data $(\boldsymbol x, y)$ is accepted with probability $\frac{p_d(\boldsymbol x, y)}{M p_g(\boldsymbol x, y)}$, where scalar $M > 0$ is selected such that $M p_g(\boldsymbol x, y) >p_d(\boldsymbol x, y), \forall (\boldsymbol x, y)$ in domain of $p_d$, i.e., $M = \max_{\boldsymbol x, y} \frac{p_d(\boldsymbol x, y)}{p_g(\boldsymbol x, y)}$.
Azadi et. al.~\cite{azadi2018discriminator}  proposed a rejection sampling method to match the distribution of the images from a generative adversarial network to the true distribution; however, their algorithm does not consider the joint distribution of $(\boldsymbol x, y)$ that is needed to train downstream recognizers.

To approximate the ratio $\frac{p_d(\boldsymbol x, y)}{p_g(\boldsymbol x, y)}$, we train a discriminator, $D(\boldsymbol x, y)$,  that takes a speech signal and the corresponding text and produces probability of $(\boldsymbol x, y)$ coming from true distribution, i.e., $D(\boldsymbol x, y) \in [0, 1]$. 
When $D$ is trained to its optimal value $D^*$, it satisfies  (\cite{gan_goodfellow_2014,azadi2018discriminator,pmlr_grover_2018})
\vspace{-0.3cm}
\begin{align}
    D^*(\boldsymbol x, y) = \frac{p_d(\boldsymbol x, y)}{p_d(\boldsymbol x, y) + p_g(\boldsymbol x, y)},
    \label{eq:D_star_equals_pd_div_pd_plus_pg}
\end{align}
which can be re-arranged to yield
\begin{align}
r(\boldsymbol x, y) := \frac{p_d(\boldsymbol x, y)}{p_g(\boldsymbol x, y)} =  \frac{D^*(\boldsymbol x, y)}{1-D^*(\boldsymbol x, y)}.
\label{eq:pd_div_pg}
\end{align}
To train $D$, we minimize
\vspace{-0.3cm}
\begin{align}
\mathcal L_D = \mathbb E_{(\boldsymbol x, y) \sim p_d} (\log (D(\boldsymbol x, y))) + \mathbb E_{(\boldsymbol x, y) \sim p_g} (1 - \log (D(\boldsymbol{x}, y))), \nonumber
\end{align}
which is equivalent to training a two class classification model, where the positive class (label $1$) corresponds to the real samples and the negative class (label $0$) corresponds to the synthetic samples. 
Note that, we model the joint speech and label distribution, therefore $D$ should also utilize whether the speech signal matches the token labels. 
To this end, we use a pre-trained ASR model, $A$ to predict the token labels, $\hat{y}$, of the speech signal, i.e., $\hat{y} = A(\boldsymbol x)$.
Then, we characterize the discrepancy between $\hat{y}$ and $y$ by a 5-dimensional feature vector $\Phi(\boldsymbol x,y)$, whose elements include the cross-entropy (CE) loss, connectionist temporal classification loss, word error rate (WER), and number of tokens in $y$ and $\hat{y}$.
Finally, we train a two-layer DNN, $D_{\boldsymbol \theta}$, that takes $\Phi(\boldsymbol x,y)$ as input and produces the probability of the sample being real
\begin{align}
    D(\boldsymbol x, y):=  D_{\boldsymbol \theta}(\Phi(\boldsymbol x, y)),
    \label{eq:D_score}
\end{align}
where $\boldsymbol \theta$ denotes the parameters of the DNN.


Practically, we need to compute $M$ and address the fact that the rejection sampling can be slow to select a fixed number of samples.
First, we estimate an initial value of $M$ by computing the maximum value of $p_d / p_g$  (using $D^*$ and Eq.~\ref{eq:pd_div_pg}) over a few (approximately $200$) generated samples. 
Following~\cite{azadi2018discriminator}, to compute the acceptance probability for each generated sample, $(\boldsymbol x, y)$, we first compute the discriminator score $D^*(\boldsymbol x, y)$ and then set $M \leftarrow \max(M, {p_d(\boldsymbol x, y) / p_g(\boldsymbol x, y)}) = \max(M, \frac{D^*(\boldsymbol x, y)}{1-D^*(\boldsymbol x, y)})$).
We stop the sampling process after accepting $N$ synthetic samples. Note that, the proposed rejection sampling technique applies only to the domain where the generative model has support and ignores the missing samples region shown in Figure~\ref{fig:motivation}.


\subsection{Double Batch Normalization Statistics}


\label{method:dbl_bn}
A BN layer~\cite{batchnorm_ioffe15} computes the mean and the standard deviation of the intermediate features and normalizes these features by subtracting the mean and dividing by the standard deviation within a batch. 
During training, the model uses means and variances computed over the samples in a mini-batch.
A running average of the BN statistics, which we call running statistics, is computed to be used during inference.

Naively using BN will update the running statistics with samples from both the real and the synthetic distributions and, due to the distribution gap, the estimates will be biased (i.e. the estimated BN statistics will not match the true data distribution).
Instead, we form the mini-batches consisting of only the real samples or only the synthetic samples, and compute two sets of running statistics -- one from the real mini-batches and the other from the synthetic mini-batches. 
Since the test data (during inference) consists of only real samples, we discard the synthetic statistics and use only the real statistics. Note that, we share the affine parameters of the BN between synthetic and real batches. 

The proposed technique is similar to Xie et al.~\cite{Xie_2020_CVPR} that used auxiliary BN layers in an adversarial learning framework where images augmented with adversarial perturbations were used to improve model accuracy. In contrast, we use separate BN statistics to improve model training with synthetic data. 




\section{Introduction}
\label{sec: intro}

\input{intro.tex}

\vspace{-0.2cm}
\section{Related Work}
\input{related_works.tex}

\vspace{-0.3cm}
\section{Method}
\input{method.tex}
\vspace{-0.3cm}
\section{Experiments}
\label{sec:experiment}
 \input{experiments.tex}
 
\vspace{-0.3cm}
\section{Conclusions}
\input{conclusion.tex}

\vspace{0.2cm}
\setlength{\parindent}{0pt}
\textbf{Acknowledgement:}
We are grateful to our colleagues Russ Webb and Barry Theobald for their valuable inputs.

\footnotesize
