
\section{Introduction}

This template can be found on the conference website. Templates are provided for Microsoft Word\textregistered, and \LaTeX. However, we highly recommend using \LaTeX when preparing your submission. Information for full paper submission is available on the conference website.

\section{Page layout and style}

Authors should observe the following rules for page layout. A highly recommended way to meet these requirements is to use a given template (Microsoft Word\textregistered\ or \LaTeX) and check details against the corresponding example PDF file. Given templates, Microsoft Word\textregistered\ or \LaTeX, can be adapted/imported easily in other software such as LibreOffice, Apple Pages, Lua\LaTeX, and Xe\LaTeX, but please be careful to match the layout of the provided PDF example.

\subsection{Basic layout features}

\begin{itemize}
\item Proceedings will be printed in DIN A4 format. Authors must submit their papers in DIN A4 format.
\item Two columns are used except for the title section and for large figures that may need a full page width.
\item Left and right margin are 20 mm each. 
\item Column width is 80 mm. 
\item Spacing between columns is 10 mm.
\item Top margin is 25 mm (except for the first page which is 30 mm to the title top).
\item Bottom margin is 35 mm.
\item Text height (without headers and footers) is maximum 235 mm.
\item Headers and footers must be left empty.
\item Check indentations and spacings by comparing to this example file (in PDF).
\end{itemize}

\subsubsection{Headings}

Section headings are centered in boldface with the first word capitalized and the rest of the heading in lower case. Sub- headings appear like major headings, except they start at the left margin in the column. Sub-sub-headings appear like sub-headings, except they are in italics and not boldface. See the examples in this file. No more than 3 levels of headings should be used.

\subsection{Text font}

Times or Times Roman font is used for the main text. Font size in the main text must be 9 points, and in the References section 8 points. Other font types may be used if needed for special purposes. It is VERY IMPORTANT that while making the final PDF file, you embed all used fonts! To embed the fonts, you may use the following instructions:
\begin{enumerate}
\item For Windows users, the bullzip printer can convert any PDF to have embedded and subsetted fonts.
\item For Linux/Mac users, you may use \\
   pdftops file.pdf\\
   pstopdf -dPDFSETTINGS=/prepress file.pdf
\end{enumerate}

\LaTeX users: users should use Adobe Type 1 fonts such as Times or Times Roman. These are used automatically by the INTERSPEECH2021.sty style file. Authors must not use Type 3 (bitmap) fonts.

\subsection{Figures}

All figures must be centered on the column (or page, if the figure spans both columns). Figure captions should follow each figure and have the format given in Figure~\ref{fig:speech_production}.

Figures should be preferably line drawings. If they contain gray levels or colors, they should be checked to print well on a high-quality non-color laser printer.

Graphics (i.\,e., illustrations, figures) must not use stipple fill patterns because they will not reproduce properly in Adobe PDF. Please use only SOLID FILL COLORS.

Figures which span 2 columns (i.\,e., occupy full page width) must be placed at the top or bottom of the page.

\subsection{Tables}

An example of a table is shown in Table~\ref{tab:example}. The caption text must be above the table.

\begin{table}[th]
  \caption{This is an example of a table}
  \label{tab:example}
  \centering
  \begin{tabular}{ r@{}l  r }
    \toprule
    \multicolumn{2}{c}{\textbf{Ratio}} & 
                                         \multicolumn{1}{c}{\textbf{Decibels}} \\
    \midrule
    $1$                       & $/10$ & $-20$~~~             \\
    $1$                       & $/1$  & $0$~~~               \\
    $2$                       & $/1$  & $\approx 6$~~~       \\
    $3.16$                    & $/1$  & $10$~~~              \\
    $10$                      & $/1$  & $20$~~~              \\
    $100$                     & $/1$  & $40$~~~              \\
    $1000$                    & $/1$  & $60$~~~              \\
    \bottomrule
  \end{tabular}
  
\end{table}

\subsection{Equations}

Equations should be placed on separate lines and numbered. Examples of equations are given below. Particularly,
\begin{equation}
  x(t) = s(f_\omega(t))
  \label{eq1}
\end{equation}
where \(f_\omega(t)\) is a special warping function
\begin{equation}
  f_\omega(t) = \frac{1}{2 \pi j} \oint_C 
  \frac{\nu^{-1k} \mathrm{d} \nu}
  {(1-\beta\nu^{-1})(\nu^{-1}-\beta)}
  \label{eq2}
\end{equation}
A residue theorem states that
\begin{equation}
  \oint_C F(z)\,\mathrm{d}z = 2 \pi j \sum_k \mathrm{Res}[F(z),p_k]
  \label{eq3}
\end{equation}
Applying (\ref{eq3}) to (\ref{eq1}), it is straightforward to see that
\begin{equation}
  1 + 1 = \pi
  \label{eq4}
\end{equation}

Finally we have proven the secret theorem of all speech sciences. No more math is needed to show how useful the result is!

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figure.pdf}
  \caption{Schematic diagram of speech production.}
  \label{fig:speech_production}
\end{figure}

\subsection{Information for Word users only}

For ease of formatting, please use the styles listed in Table 2. The styles are defined in this template file and are shown in the order in which they would be used when writing a paper. When the heading styles in Table 2 are used, section numbers are no longer required to be typed in because they will be automatically numbered by Word. Similarly, reference items will be automatically numbered by Word when the ``Reference'' style is used.

\begin{table}[t]
  \caption{Main predefined styles in Word}
  \label{tab:word_styles}
  \centering
  \begin{tabular}{ll}
    \toprule
    \textbf{Style Name}      & \textbf{Entities in a Paper}                \\
    \midrule
    Title                    & Title                                       \\
    Author                   & Author name                                 \\
    Affiliation              & Author affiliation                          \\
    Email                    & Email address                               \\
    AbstractHeading          & Abstract section heading                    \\
    Body Text                & First paragraph in abstract                 \\
    Body Text Next           & Following paragraphs in abstract            \\
    Index                    & Index terms                                 \\
    1. Heading 1             & 1\textsuperscript{st} level section heading \\
    1.1 Heading 2            & 2\textsuperscript{nd} level section heading \\
    1.1.1 Heading 3          & 3\textsuperscript{rd} level section heading \\
    Body Text                & First paragraph in section                  \\
    Body Text Next           & Following paragraphs in section             \\
    Figure Caption           & Figure caption                              \\
    Table Caption            & Table caption                               \\
    Equation                 & Equations                                   \\
    \textbullet\ List Bullet & Bulleted lists                              \\\relax
    [1] Reference            & References                                  \\
    \bottomrule
  \end{tabular}
\end{table}

If your Word document contains equations, you must not save your Word document from ``.docx'' to ``.doc'' because when doing so, Word will convert all equations to images of unacceptably low resolution.

\subsection{Hyperlinks}

For technical reasons, the proceedings editor will strip all active links from the papers during processing. Hyperlinks can be included in your paper, if written in full, e.\,g.\ ``http://www.foo.com/index.html''. The link text must be all black. 
Please make sure that they present no problems in printing to paper.

\subsection{Multimedia files}

The INTERSPEECH organizing committee offers the possibility to submit multimedia files. These files are meant for audio-visual illustrations that cannot be conveyed in text, tables and graphs. Just like you would when including graphics, make sure that you have sufficient author rights to the multimedia materials that you submit for publication. The proceedings media will NOT contain readers or players, so be sure to use widely accepted file formats, such as MPEG, Windows WAVE PCM (.wav) or Windows Media Video (.wmv) using standard codecs.

Your multimedia files must be submitted in a single ZIP file for each separate paper. Within the ZIP file you can use folders and filenames to help organize the multimedia files. In the ZIP file you should include a TEXT or HTML index file which describes the purpose and significance of each multimedia file. From within the manuscript, refer to a multimedia illustration by its filename. Use short file names without blanks for clarity.

The ZIP file you submit will be included as-is in the proceedings media and will be linked to your paper in the navigation interface of the proceedings. Causal Productions (the publisher) and the conference committee will not check or change the contents of your ZIP file.

Users of the proceedings who wish to access your multimedia files will click the link to the ZIP file which will then be opened by the operating system of their computer. Access to the contents of the ZIP file will be governed entirely by the operating system of the user's computer.

\subsection{Page numbering}

Final page numbers will be added later to the document electronically. \emph{Do not make any footers or headers!}


\subsection{References}

The reference format is the standard IEEE one. References should be numbered in order of appearance, for example \cite{Davis80-COP}, \cite{Rabiner89-ATO}, \cite[pp.\ 417--422]{Hastie09-TEO}, and \cite{YourName21-XXX}.

\subsection{Abstract}

The total length of the abstract is limited to 200 words. The abstract included in your paper and the one you enter during web-based submission must be identical. Avoid non-ASCII characters or symbols as they may not display correctly in the abstract book.

\subsection{Author affiliation}

Please list country names as part of the affiliation for each country.

\subsection{Number of authors in the author list}

The maximum number of authors in the author list is twenty. If the number of contributing authors is more than twenty, they should be listed in a footnote or in acknowledgement section, as appropriate.

\subsection{Submitted files}

Authors are requested to submit PDF files of their manuscripts. You can use commercially available tools or for instance http://www.pdfforge.org/products/pdfcreator. The PDF file should comply with the following requirements: (a) there must be no PASSWORD protection on the PDF file at all; (b) all fonts must be embedded; and (c) the file must be text searchable (do CTRL-F and try to find a common word such as ``the''). The proceedings editors (Causal Productions) will contact authors of non-complying files to obtain a replacement. In order not to endanger the preparation of the proceedings, papers for which a replacement is not provided in a timely manner will be withdrawn.

\section{Discussion}

This is the discussion. This is the discussion. This is the discussion. Is there any discussion?

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras consequat mollis odio, nec venenatis enim auctor sed. Integer tincidunt fringilla lectus eget condimentum. In eget sapien id eros dapibus interdum vel ac quam. Aenean vitae rutrum erat. Aenean et risus pharetra, lacinia augue ut, fermentum ante. Integer dui arcu, interdum at ornare a, faucibus quis est. Mauris quis quam felis. Etiam pulvinar massa et turpis lacinia, eu posuere mi iaculis. Fusce at velit quis leo dignissim porttitor.

Fusce ut nunc eu sapien venenatis finibus a vel ligula. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut quam eros, volutpat at gravida consectetur, rutrum ut leo. Aenean cursus euismod feugiat. Cras hendrerit, ligula eu feugiat malesuada, neque turpis auctor lacus, sit amet accumsan neque orci a quam. Mauris suscipit ultrices mattis. Nulla at interdum metus, id pharetra diam. Curabitur at vestibulum sem, sed elementum massa. Donec iaculis et arcu ut rutrum. Fusce gravida, mauris porta volutpat eleifend, enim mauris eleifend orci, eu ultrices leo purus vitae metus. In pretium dolor ut magna dictum, at imperdiet lectus porta.

Quisque mollis lectus id risus pretium mattis. Morbi scelerisque posuere est, id efficitur urna luctus non. Praesent quam lacus, facilisis id ante eu, vehicula maximus ex. Nullam mollis in arcu vitae efficitur. Aliquam molestie eleifend ante, in pretium velit ultrices ac. Etiam laoreet nec sem non pulvinar. Integer ligula felis, interdum non lacus id, malesuada imperdiet turpis.

Aenean sit amet volutpat nisi. Aliquam eu erat quis tortor ultrices laoreet. Vivamus fermentum semper metus, non faucibus libero euismod vitae. Sed efficitur porta congue. Aenean in faucibus nisi. Donec suscipit augue vitae orci consequat, sit amet aliquet felis varius. Duis efficitur lacinia dolor sit amet lobortis. Curabitur erat sapien, molestie nec nisi eu, dignissim accumsan ipsum. Fusce id nibh nec risus dictum posuere in ac magna. Donec malesuada massa sed erat lacinia cursus. Suspendisse ornare augue nec volutpat consequat.

Vestibulum et vulputate nisi, a malesuada mi. Nam pellentesque arcu sapien, at placerat odio imperdiet ut. Curabitur nec venenatis tellus, vel aliquet nisi. Curabitur vel ligula sit amet metus auctor pretium. Nullam nulla mi, blandit a mattis id, vulputate sit amet enim. Proin mollis fringilla dictum. Proin lacinia orci purus.

Curabitur porttitor bibendum dolor, nec consectetur sapien pulvinar id. Donec eleifend, est vel dignissim pretium, tortor augue euismod nunc, id fermentum erat felis ac neque. Morbi id lectus ultricies, rutrum justo eu, sollicitudin risus. Suspendisse lobortis efficitur nisi sit amet pellentesque. Ut eget augue at mi aliquet mattis. Proin et feugiat erat, sit amet sodales eros. Integer sed elit quis est mattis ullamcorper. Pellentesque lectus nisi, vulputate a imperdiet tincidunt, auctor nec orci. Pellentesque sagittis nisl orci, vitae placerat massa lacinia nec. Sed egestas magna sed augue sollicitudin luctus. Praesent interdum bibendum tortor, eu porta purus. Aliquam convallis velit id mi fermentum, sed ornare eros cursus.

Quisque congue leo a fringilla pharetra. Phasellus sed tempor est, sed auctor purus. Morbi vel lacus ullamcorper, auctor mauris id, pulvinar lorem. Suspendisse potenti. Nam porta, purus non eleifend bibendum, erat metus pellentesque elit, non luctus nibh nunc ornare nisl. Sed rutrum lacinia nisi ac suscipit. Curabitur non blandit augue. Integer viverra, ipsum vel molestie euismod, sem quam tempus massa, eget efficitur ante turpis non metus. Quisque efficitur posuere velit in iaculis. Cras imperdiet varius urna vitae vestibulum. Donec accumsan eget nisi sed pellentesque. Vestibulum id quam ut urna volutpat ullamcorper gravida sit amet libero. Aliquam bibendum, ligula vitae porta malesuada, arcu diam congue erat, a pharetra diam sem vulputate tortor. Etiam luctus iaculis leo cursus tristique.

Mauris mattis sem dolor, sit amet ullamcorper arcu tincidunt ac. Vestibulum at blandit tortor. Quisque bibendum congue leo, vitae eleifend massa. Vestibulum vitae odio elit. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut sagittis quam vel felis ornare, in gravida felis tempor. Donec molestie dui quis leo venenatis blandit. Nunc sit amet finibus metus. Cras ut lectus ex.

Suspendisse commodo libero vel leo tincidunt, a tempus mauris porta. Integer varius eros ac sapien lacinia vehicula. Donec porttitor, lacus faucibus rhoncus venenatis, neque quam imperdiet nunc, id consectetur metus purus quis sapien. Phasellus interdum nulla vel euismod posuere. Vestibulum finibus magna vel finibus mollis. Curabitur mollis turpis tortor, hendrerit vulputate justo egestas quis. Nam dignissim luctus leo non elementum. Phasellus a metus at leo malesuada bibendum. Mauris quis eleifend magna, nec vehicula ex. Donec venenatis urna fermentum commodo vehicula. Ut mattis scelerisque aliquam. Vivamus pulvinar erat metus, id tempus mi vulputate quis.

Fusce lobortis a urna eget blandit. Vivamus in eleifend neque, at sollicitudin lectus. Quisque faucibus egestas lorem, in commodo diam maximus eu. Morbi finibus ante ac felis porttitor euismod. Donec lobortis aliquam ipsum sit amet luctus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Etiam rutrum neque sapien, eget luctus turpis iaculis pulvinar. Duis quis pulvinar nunc, nec bibendum ligula. Phasellus suscipit sagittis lacus molestie laoreet. Pellentesque lacus diam, tincidunt a aliquam vitae, aliquet non justo.

Etiam lectus lacus, commodo eget consectetur eget, auctor vitae leo. Praesent vitae erat in diam blandit semper vitae et eros. Maecenas auctor pharetra nibh eget egestas. Donec accumsan ut risus eget rhoncus. Nam placerat, erat sit amet gravida mollis, purus arcu accumsan diam, tempus pharetra risus mi ac sapien. Ut et tortor porta, pulvinar elit vitae, tempor mi. Nam interdum, nisl non pharetra molestie, turpis neque commodo ligula, sit amet pretium nisl nibh quis ante. Quisque et ex eget velit lobortis suscipit. Integer aliquam finibus molestie. Sed pellentesque neque eu turpis aliquet, mattis ornare enim finibus. In hac habitasse platea dictumst.

Integer congue quis justo a posuere. Quisque porta, ante et dignissim suscipit, arcu mauris ultrices libero, nec sollicitudin purus lacus a enim. Aliquam feugiat eget lacus molestie sodales. Duis blandit placerat nunc, et venenatis turpis dictum vel. Nulla facilisi. Nam ullamcorper, tellus eu posuere mattis, arcu lacus dictum nulla, vel mattis nisi sem posuere tellus. Etiam quis eros condimentum lectus lobortis eleifend. In ex lacus, sodales scelerisque egestas ac, aliquam nec purus. Nunc sit amet magna non libero ullamcorper dictum. Phasellus porta faucibus tempus. Praesent blandit tortor sed tellus ornare consectetur. Sed sed nisi id neque porta varius eu eu velit. Curabitur varius convallis justo id facilisis. Mauris auctor velit nec aliquam cursus.

Integer suscipit scelerisque leo sed faucibus. Ut commodo nulla luctus diam posuere egestas. Integer ut augue ac velit ullamcorper tempus. Pellentesque in mi rhoncus, sodales sem quis, commodo sem. Aenean dapibus euismod diam id rhoncus. Nullam vehicula placerat eros consectetur luctus. Aliquam auctor ipsum vitae egestas imperdiet. Ut nulla lacus, imperdiet quis urna vel, ornare imperdiet tortor. Mauris nec diam ac nunc laoreet volutpat at id turpis. Nulla eu neque a risus feugiat iaculis ac vel risus. Ut tempus elementum lorem eget porta. Nullam et ullamcorper urna. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus eget dui vitae nulla hendrerit ultrices quis rutrum leo.

Proin consectetur lacus sit amet eleifend varius. Etiam eu blandit risus. Curabitur pellentesque urna sed dolor congue mattis. Vestibulum ut velit posuere, feugiat leo a, dignissim massa. Proin eu nulla risus. Fusce luctus bibendum est, sit amet venenatis nisi finibus non. Donec ultricies ornare nunc at lobortis. Nam a auctor metus. Vestibulum pretium condimentum turpis ac mattis. Curabitur semper sagittis rhoncus. Duis molestie facilisis mattis. Sed pharetra lorem id tortor efficitur, sed maximus leo posuere. Quisque suscipit molestie convallis. Duis imperdiet placerat congue. Morbi placerat, velit ut tempor porta, ex nisi imperdiet purus, non feugiat ex velit nec nulla.

Phasellus mattis at erat eget lobortis. Vivamus sodales odio non erat luctus faucibus. Curabitur aliquam luctus nulla quis consectetur. Fusce vulputate finibus vulputate. Cras at condimentum massa. Duis vestibulum ipsum ac tortor lobortis fringilla. Cras non neque at nunc pellentesque mollis. Sed quis erat mauris. Ut dapibus sem lectus, quis imperdiet diam bibendum et. Maecenas quis venenatis ante. Nunc blandit a risus sed scelerisque. Praesent cursus est sit amet nisi tempus, quis placerat libero rutrum. Phasellus lacinia nisi quis consequat mollis. Phasellus sagittis aliquam lacus.

Sed dolor quam, posuere nec nunc eget, feugiat lobortis ligula. Fusce lacinia fermentum dolor, luctus dapibus ex venenatis feugiat. In hac habitasse platea dictumst. Nullam vitae ligula dignissim, interdum turpis quis, tincidunt metus. Nullam in nisl vitae mauris egestas porta. Nam fringilla aliquet sapien, non dapibus nunc sollicitudin id. Nunc hendrerit felis et vehicula consequat. Praesent varius libero id volutpat iaculis. Aliquam vel dui imperdiet, pharetra augue sed, iaculis nulla. Quisque mollis orci nec odio eleifend, eget laoreet nunc feugiat. Cras varius tortor a fringilla gravida.

Sed posuere erat eu dolor consequat euismod. Donec imperdiet, tellus nec convallis commodo, lorem sem lobortis purus, a lacinia massa ipsum vitae nisl. Vivamus auctor tellus in urna iaculis luctus. In dui nibh, posuere a erat a, lobortis finibus nulla. Sed vel suscipit nisi. Nunc eget nibh risus. Sed posuere tempus eleifend. Nullam ac lacinia ligula, ut blandit erat. In at est sed turpis consequat rutrum. Nunc eget lectus venenatis, convallis ligula non, mollis orci. Aliquam sit amet ligula turpis. Sed finibus laoreet elit nec molestie.

Mauris in nisi et neque euismod aliquam ut eget felis. Sed eget dictum tellus, finibus rutrum nibh. Fusce placerat augue a faucibus semper. Nam sed nisl ligula. Vivamus ante augue, faucibus at risus ut, hendrerit viverra risus. Etiam justo eros, dignissim a nunc sed, porta mollis erat. Fusce in sem accumsan, laoreet nibh porta, porttitor orci. Donec venenatis vehicula ante eget dictum.

Aliquam fermentum a metus pellentesque cursus. Vivamus eleifend ultricies tellus, vel scelerisque diam. Sed hendrerit est at elit suscipit placerat. Praesent nec pretium erat. Quisque blandit nunc in felis bibendum consequat. 

\section{Conclusions}

Authors must proofread their PDF file prior to submission to ensure it is correct. Authors should not rely on proofreading the Word file. Please proofread the PDF file before it is submitted.

\section{Acknowledgements}

The ISCA Board would like to thank the organizing committees of the past INTERSPEECH conferences for their help and for kindly providing the template files. \\
Note to authors: Authors should not use logos in the acknowledgement section; rather authors should acknowledge corporations by naming them only.


\bibliographystyle{IEEEtran}



\section{Introduction}

Speech representations learned in an unsupervised way should not be limited to better acoustic features for supervised ASR, but also provide a path towards understanding of utterances at a linguistic level by enabling modeling of audio signals at acoustic, phoneme, or word levels. However, these different levels of speech understanding require reasoning at different sampling rates: acoustic features are computed at equally spaced time intervals (e.g., every 10ms), while language models change state on each phoneme occurrence, or every word occurrence.  To capture this behavior, representation learning methods should provide mechanisms that produce variable-rate data representations, which change synchronously with the contents of the utterance, rather than an external clock.

Supervised ASR models allow variable rate data processing: during recognition an HMM~\cite{Rabiner89-ATO,graves_connectionist_2006,graves_sequence_2012} classifies jointly several acoustic frames into a single phoneme. Similarly the attention mechanism~\cite{bahdanau_neural_2014,chorowski_attentionbased_2015}
allows a phoneme or a character-synchronous decoder network to access acoustic features sampled uniformly in time. However, training these models requires ground-truth transcriptions. 

In this paper we investigate the possibility of extending  Contrastive Predictive Coding (CPC) \cite{oord_representation_2018}, an unsupervised representation learning method, with a rate-adjusting alignment step. In the proposed Aligned  Contrastive Predictive Coding (ACPC, Figure \ref{fig:acpc}) the model predicts only a short sequence of latent codes which are aligned to a longer series of upcoming ones. 

ACPC changes the behavior of the model in two ways. First, it promotes stability and piece-wise smoothness of the produced encodings: neighboring representations that are aligned to the same prediction are implicitly trained to be similar, while neighboring representations aligned to different predictions are trained to be different. Second, the task learned by the prediction network is altered: rather than predicting precisely timed future representations, the prediction network can now focus on \emph{what} symbols will come next, rather than exactly \emph{when} they will come. In consequence, ACPC leads to representations that are more related to phonemes, both on the linear frame-wise phoneme prediction task and on ABX evaluations. As demonstrated in experiments, ACPC also disincentivises the encoder from extracting location-based features, such as periodical patterns produced by strided convolutions~\cite{odena2016deconvolution} which are induced by classical contrastive coding.

Finally, we note that ACPC brings a small performance increase over CPC. Contrastive scoring of predictions is expensive, because it requires comparisons against hundreds of negative samples. ACPC makes fewer predictions, bringing in a noticeable speedup. 

We publicly release the implementation at \url{https://github.com/chorowski-lab/CPC_audio}.







\begin{figure}
    \centering\includegraphics[width=.95\columnwidth]{figs/arch_acpc3.png}
    \caption{ACPC architecture. The encoder maps chunks of input data into a latent space and the autoregressive model predicts $K$ upcoming latent vectors. They are   aligned using DTW to the $M$ upcoming latent vectors. Training relies on a contrastive loss: the match between a prediction and its aligned latent vectors has to be stronger than the match of the predictor to any other latent vector. Using $K\slimeq M$ is equivalent to CPC~\protect\cite{oord_representation_2018}.}
    \label{fig:acpc}
\end{figure}


\section{Background and Motivation}


CPC \cite{oord_representation_2018} extracts latent representations of sequential data by learning to predict future states of the model. First, an encoding feed-forward network extracts latent representations from consecutive and overlapping chunks of the data, producing a sequence of latent codes. A powerful auto-regressive model is then applied to the latent representations. It is trained to predict the extracted latent representations several steps into the future. The model is supervised using Noise Contrastive Estimation: the prediction $p_t$ of the latent code $z_t$ at time $t$ must be closer to $z_t$ than to some other randomly sampled latent codes called the negative samples. 
When applied to speech signals, CPC was shown to learn good acoustic representations of the data which are useful for phoneme prediction and low-resource speech recognition.




ACPC makes $K$ predictions of the upcoming latent vectors, which will be then matched to $M$ future latent vectors. It then force-aligns predictions to the latent codes, and only then uses a contrastive loss to supervise the model. The alignment can be performed efficiently using a variant of Dynamic Time Warping (DTW) \cite{vintsyuk_speech_1968}. This extra search step recovers the timing information, freeing the autoregressive model from having to learn it. When $K\slimeq M$ our system is equivalent to the vanilla CPC variant: there is only one possible alignment.

\section{ACPC Details}

Figure~\ref{fig:acpc}
demonstrates ACPC. The input sequence $x_1,\ldots,x_T$ is mapped using a strided convolutional encoder to a sequence of latent encodings $z_1,\ldots, z_{T'}$. Typically, the encoder uniformly reduces the temporal resolution of the data. Then, the autoregressive model summarizes all encodings up to time $t$, denoted $z_{\leq t}$, into a context vector $c_t = g_{enc}(z_{\leq t})$. Finally, $K$ predictions $p_t^1,\ldots, p_t^K$ are made conditioned on $c_t$. We match these prediction to $M$ upcoming latent vectors $z_{t+1} ,\dots, z_{t+M}$ using contrastive scores as follows. We first sample $N$ negative latent vectors from the encoder outputs $\bar{z}_t^1 ,\dots, \bar{z}_t^N$. We then compute the match between prediction $p_t^k$ and encoding $z_{t+m}$ using the contrastive score:

\begin{equation}\label{eq:scores}
    s_t^{k,m} = \frac{e^{\left<p_t^k, z_{t+m}\right>}}{e^{\left<p_t^k, z_{t+m}\right>} + \sum_{n=1}^N e^{\left<p_t^k, \bar{z}_t^N\right>}},
\end{equation}
where $\left<\cdot,\cdot\right>$ denotes the scalar product of two vectors.

We can organize the scores as a matrix showing the affinity of predictions made at time $t$ to subsequent latent codes. An alignment is a path through this matrix which connects the corner $k\slimeq 1,m\slimeq 1$ with the corner $k\slimeq K,m\slimeq M$. An example is shown in Figure~\ref{fig:acpc}.
We allow a single prediction to be aligned with several consecutive latent encodings, but each encoding may only be aligned with one prediction.

The model is trained to promote the best alignment between the predictions and actual latent encodings. The alignment can be found using DTW. However, since it is being computed several times for each training minibatch, the computation has to be efficient. We have chosen to approximate DTW using an optimized GPU implementation of CTC \cite{graves_connectionist_2006}. CTC assumes that scores are normalized, and computes the expected cost of the best path. Once a model is trained, a single path is typically dominant, and the expected cost over all paths is close to the cost of the best DTW path. Two technical issues remain. First, CTC uses a blank character. We forbid its use by extending the scoring matrix with a row set to a large negative constant value as its log-probability. Second, CTC requires the scores to be normalized. We use the fact, that when one path dominates, we can add or subtract any number to its scores - and we subtract the softmax normalizing score. The details may be found in the source code.








\section{Experiments}


We conduct our experiments on LibriSpeech train-clean-100, following the setup of \cite{riviere_unsupervised_2020} and the small CPC baseline used in ZeroSpeech 2021 \cite{nguyen_zero_2020}. We keep all design choices, and only replace the CPC loss with ACPC. All models read single channel raw waveforms sampled at 16kHz, chunked into sequences of 20480 samples. The encoder applies five 1D convolutions with internal dimension 256 and filter widths $(10, 8, 4, 4, 4)$. All convolutions are followed by channel-wise magnitude normalization and ReLU activations. Convolutions are strided by $(5, 4, 2, 2, 2)$ respectively, which results in a 160-fold rate reduction, yielding a 256-dimensional latent vector extracted every 10ms. The autoregressive context-building model is a two-layer LSTM network~\cite{hochreiter_long_1997} with 256 hidden units. Finally, each prediction head accesses all past contexts through a single Transformer layer~\cite{vaswani_attention_2017} with 8 scaled dot-product attention heads with internal dimension 2048 and dropout~\cite{srivastava_dropout_2014} with $p=0.1$.

Both CPC and ACPC use 128 negatives. We sample them from other utterances in the same minibatch, and we use minibatches containing 64 utterances of a single speaker. However, there is a subtle interplay between the negative selection and the number of GPUs, that affects the results: the negative selection is restricted to utterances processed on a single GPU. The more GPUs are used, the smaller the pool used for negative selection. In order to simulate the setup of~\cite{nguyen_zero_2020}, we group utterances into 8 groups which we use to simulate 8 GPUs and restrict negative selection to utterances in a group.

\subsection{Qualitative Results}
All results in this section are reported for models trained for 200 epochs. The output of the convolutional encoder ($z_t$ in Figure~\ref{fig:acpc}) is used as the \emph{learned representations} over which all the following experiments are conducted, but similar conclusions apply to  autoregressive contexts $c_t$.  For clarity, we only report ACPC results when $K\slimeq 8$ predictions are matched to a window of $M\slimeq 12$ frames. Several other combinations of predictions and window sizes were also tested ($K\slimeq 10$ over $M\slimeq 16$ and $K\slimeq 12$ over $M\slimeq 20$) leading to similar conclusions. An in-depth investigation of the effects on representation of these settings is left as a scope of future work. 

We investigate three aspects of the learned representations on speech signals: the slow varying temporal evolution, how clustered the representation space is, and the agreement with interpretable views of speech such as the phonetic structure.


\begin{figure}[tb!]
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.44\columnwidth]{figs/vanilla_enc_similarity_small.png} &
    \includegraphics[width=.44\columnwidth]{figs/acpc_enc_similarity_small.png}
    \\
    CPC & ACPC (K=8, M=12)
    \end{tabular}%
   
   
   
   
   
   
    \caption{Dot-product similarity of latent representations at the output of the convolutional encoder. The white vertical bars and tick labels indicate ground-truth phoneme segmentation. Moire-like artifacts are visible for CPC, but less pronounced for ACPC (see main text for details). The pronounced blocks on the diagonal align with human-annotated phonemes. Visually, ACPC yields a representation in which latent vectors that correspond to a single phone are more self-similar.}
    \label{fig:enc_simi}
\end{figure}


\begin{figure}[tb]
    \includegraphics[width=\linewidth]{figs/dist_conv_repr_cosine.pdf}
    \caption{Distribution of cosine similarities between representations of random pairs of frames (top) and consecutive frames (bottom). The histograms are computed on both the representations from the baseline CPC and those of the proposed ACPC.}
    \label{fig:hist_dist}
\end{figure}

\vspace{1ex}\noindent\textbf{Temporal Smoothness } We first compare the self-similarity of representations produced by the convolutional encoder (Figure~\ref{fig:enc_simi}). CPC produces a distinct, moire-like pattern, and we hypothesize that the network learns to abuse strided convolutions to encode the location of the latent vector modulo 4, similarly to the checkerboard pattern visible in deconvolutional image generators~\cite{odena2016deconvolution}. This artifact appears during CPC training because it helps the predictor to guess the upcoming exactly timed features. ACPC does not require exact timing of predictions, and produces only a faint pattern. It enhances the similarity of encodings, as the boundaries of phonemes roughly align with the sharp blocks on the diagonal. We observe this difference between CPC and ACPC repeatedly on other samples.

\vspace{1ex}\noindent\textbf{Quantifying Pairwise Similarities }
We compare pairwise similarities of latent representations between CPC and ACPC with histograms (Figure~\ref{fig:hist_dist}). The distribution between similarities for both models (top) has a similar overall shape. For ACPC there is a second, distinct mode, which is expected for a sharp similarity matrix: a ground truth matrix would be block-diagonal with $O(n)$ ones on the diagonal and $O(n^2)$ zeros elsewhere, motivating two modes unequal in size. We take this idea further and compare self-similarities of only consecutive latent representations (placed directly below/above the diagonal in the distance matrix). There is a significant distribution shift in similarities of consecutive frames between CPC and ACPC.
The alignment of fewer predictions to the sequence of upcoming representations forces the ACPC model to share similar representations for consecutive frames, and therefore creates more slow-varying representation.

\vspace{1ex}\noindent\textbf{Clustering Quality }
While slow-varying features are often expected in speech signals, another aspect which is regularly sought is their readiness to be interpreted categorically. Speech signals are most often segmented into a series of discrete symbols for further processing. Many Automatic Speech Recognition (ASR) perform an alignment of the acoustic stream with regions associated to phone or sub-phone units through classification. We here test how the representations produced by ACPC are clustered in comparison to those of the baseline CPC. We run k-means on the set of latent codes. The clusters are evaluated in terms of concentration by plotting in Figure~\ref{fig:clust} (solid lines and left axis) the average squared distance from the samples to their assigned centroids. We note that the embeddings of both systems have similar norm due to Channel Normalization layers used in the encoder. We observe that ACPC produces consistently a lower mean distance, thus a more clustered representations for every tested number of components.

\begin{figure}[tb]
    \centering
    \includegraphics[width=\linewidth]{figs/clust_conv_repr.pdf}
    \caption{Clustering quality (solid lines, left axis) and agreement with the phone alignments (dashed lines, right axis) of the CPC and ACPC representations for different numbers of centroids. The quality is measured by the average squared distance to the assigned centroid, the agreement  computed by Normalized Mutual Information (NMI) between the cluster assignments and ground truth phones, forced-aligned with ASR. }
    \label{fig:clust}
\end{figure}

Furthermore, the clusters created by k-means on the ACPC latents are more in agreement with supervised ASR generated phonetic alignments than those computed using the baseline CPC. We compute and plot
the Normalized Mutual Information 
between the set of cluster assignments resulting from the k-means and the phones assigned to each frame with forced alignment
(Figure~\ref{fig:clust}, dashed lines and right axis).
The results demonstrate that the modes in the latent space of the ACPC map better to phonetic content than those of the baseline CPC.

\subsection{Quantitative Results: ABX and Phone Classification}

We evaluate the model on two tasks that highlight the usefulness of CPC-derived representations: minimal-pair ABX task from ZeroSpeech 2021~\cite{nguyen_zero_2020}, and supervised linear frame-wise phoneme prediction.
For ABX (Table~\ref{tab:abx}), ACPC model is trained with the same settings as the CPC baseline of ZeroSpeech 2021. We extract the features from the last (second) hidden layer of the autoregressive model. We report results for $M\slimeq 12$, $K\slimeq 8$, but similar results are obtained for $K\slimeq 6$. ACPC improves upon the CPC baseline by relative $8-15\%$, depending on the task. It also takes significantly less wall-clock time to train with a similar implementation and hardware.
\begin{table}[tb]
\centering
\caption{ABX error rates on ZeroSpeech 2021 dev set: the provided baseline CPC checkpoint, our CPC rerun and ACPC with M=12 and K=8.}
\label{tab:abx}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{@{\extracolsep{4pt}}lllllll@{}}
  \toprule
 
  & \multicolumn{2}{l}{CPC (ZeroSpeech \cite{nguyen_zero_2020})} 
  & \multicolumn{2}{l}{CPC (our baseline)}
  & \multicolumn{2}{l}{ACPC M=12 K=8}\\
  \cmidrule{2-3}\cmidrule{4-5}\cmidrule{6-7}
 
 
  Dev   & Within & Across & Within & Across & Within & Across \\
  \midrule
 
  clean & 6.18\% & 8.02\% & 6.68\% & 8.39\% & {\bf 5.37\%} & {\bf 7.09\%} \\
 
  other & 8.46\% & 13.59\% & 9.03\% & 13.87\% & {\bf 7.46\%} & {\bf 12.60\%} \\
  \bottomrule
\end{tabular}%
}
\end{table}

Next, we provide an ablation on the number of predictions $M$ and predicted frames $K$, with phone classification accuracy task.
We train the models for 50 unsupervised epochs, extract the latent vectors after the convolutional encoder ($z_t$ in Figure~\ref{fig:acpc}) and on the hidden states of the autoregressive model ($c_t$), and then train a supervised linear phone classifier with those representations.
Table~\ref{tab:linphone} lists the results. It consistently indicates that optimal parameters for ACPC in this setup are predicting the next $M\slimeq 12$ frames with $K\slimeq 6$ or $K\slimeq 8$ predictions. We note that the baseline CPC model is set up with $M\slimeq K\slimeq12$.
All tested variants of ACPC improve upon CPC, indicating that $M$ and $K$ are easy to adjust.

We measure training step times, collected with the underlying CPC implementation provided for ZeroSpeech 2021. We can see that the training time depends on the amount of predictions, because scoring them is costly - they have to be matched to 128 negative encodings. Thanks to reduced number of predictions, the fastest ACPC setting is 1.7x faster than CPC, and still more accurate on phone classification. Moreover,
ACPC representations are easier to classify from the early training iterations
(Figure~\ref{fig:convergence}).

\begin{table}[tb]
    \centering
    \caption{Frame-wise linear phoneme recognition accuracy after 50 epochs of unsupervised training. We report accuracy obtained on latent representations: outputs $z_t$ of the convolutional encoder, and autoregressive contexts $c_t$ (see Figure~\ref{fig:acpc}). Best results are achieved for M=12 with K=6 or K=8. We report wall-clock training step time.}
    \label{tab:linphone}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{@{\extracolsep{4pt}}lcllll@{}}
    \toprule
                   & \multirow{2}{0.72cm}[-0.17cm]{Step time}  & \multicolumn{2}{c}{Encoded $z_t$}  & \multicolumn{2}{c}{Contextual $c_t$}  \\
                   \cmidrule{3-4} \cmidrule{5-6}
    Model          &       & Train   & Val     & Train   & Val  \\
    \midrule
    CPC            & 2.6ms       & 51.5\%       & 51.2\%        & 67.8\%      & 67.5\% \\
    ACPC M=12 K=10 & 2.4ms       & 51.8\%       & 51.3\%        & 68.8\%      & 68.4\% \\
    ACPC M=12 K=8  & 2.1ms       & 52.0\%       & 51.9\%        & {\bf 69.7\%}& 68.6\% \\
    ACPC M=12 K=6  & 1.8ms       & {\bf 52.2\%} & {\bf 52.1\%}  & 69.2\%      & {\bf 68.8\%} \\
    ACPC M=12 K=4  & {\bf 1.5ms} & 52.1\%       & 51.9\%        & 69.0\%      & 68.7\% \\
    ACPC M=16 K=10 & 2.4ms       & 52.1\%       & 51.9\%        & 69.1\%      & {\bf 68.8\%} \\
    ACPC M=20 K=12 & 2.6ms       & 52.1\%       & 51.9\%        & 68.0\%      & 67.8\% \\
    \bottomrule
    \end{tabular}%
    }
\end{table}

\begin{figure}[tb!]
    \centering
    \includegraphics[width=\linewidth]{figs/convergence.png}
    \caption{Frame-wise linear phoneme prediction accuracy during training. ACPC steadily improves upon CPC, providing representations
             easier to classify.}
    \label{fig:convergence}
\end{figure}

We conclude that ACPC is able to train more accurate models. Qualitatively, the learned representations are smoother and more self-similar with frames belonging to each phone. These characteristics translate to improved linear classification accuracy and lower ABX error rates. 

\section{Related Work}

Slow-feature analysis \cite{wiskott_slow_2002} introduced the general idea that interesting features of a time sequence should be slowly varying and proposed to train models with a penalty on the rate of representation change applied to all subsequent pairs of representations.
We are inspired by this observation, but attempt instead to learn a representation which is piece-wise constant: ideally, the representation would change abruptly at phoneme boundaries, and be fairly stable within a phoneme. In ACPC this is achieved by aligning the representation to a smaller sequence of predictions. Forcibly, some neighboring encodings will be aligned to the same prediction, and thus trained to be more similar, while other neighboring encodings will be aligned to different predictions, and hence trained to be different.

Several solutions were recently proposed to impose slow changes of the latent representations learned in an unsupervised way. So far, most efforts were aimed at auto-encoding models, especially using discrete encodings such as VQ-VAE~\cite{vandenoord_neural_2017}. In \cite{chorowski_unsupervised_2019} a time-jitter regularization was proposed to smooth the latent layer of VQ-VAE, which resulted in improved performance on ZeroSpeech data. \cite{chorowski_unsupervised_2019a} applied a penalty to VQ-VAE latent codes to enforce a piecewise-constant representation with a bounded number of code changes. To this end an optimal cluster assignment problem was solved for each training sample. This approach was refined in \cite{kamper_unsupervised_2020} for phoneme segmentation in a VQ-VAE and VQ-CPC \cite{niekerk_vectorquantized_2020}. Finally, \cite{dieleman_variablerate_2021} have proposed to use slowness penalty and run-length encoding of the latent representation of a VQ-VAE. ACPC is inspired by these solutions, but differs in a fundamental way. All above approaches essentially rely on temporal differences between subsequent latent encodings to enforce a segmentation. This can be seen as a purely bottom-up approach: the segmentation is fully determined given the states of the encoder. ACPC instead uses a top-down signal coming from the predictive network: it aligns the latent encodings to predictions. This makes ACPC somewhat similar to an online variant of pseudo-labeling \cite{likhomanenko_slimipl_2020}: the prediction network produces pseudolabels which after a force-alignment serve as targets for the encoder. \cite{khurana_convolutional_2020} can be seen as another top-down approach which models Markov-based latent transitions and emissions with neural networks, in contrast ACPC does not model explicitly the representation dynamics.

Matching two unaligned sequences of latent states using time warping was explored in the context of ASR by the neural transducer \cite{graves_sequence_2012}. ACPC employs a similar approach for each prediction window.




\section{Conclusion}
We have proposed a modification to the CPC loss which aligns a small sequence of predictions of the upcoming latent vectors to a longer sequence of encoder outputs, then reinforces this alignment using a contrastive training criterion. ACPC changes the training signal to the encoder: it enforces similarity between latent vectors aligned to the same prediction and the task solved by the prediction network: it avoids requiring exact timings of predictions. These changes lead to more stable latent encodings which yield better phoneme prediction accuracies and lower ABX error rates. We treat ACPC as a step towards enforcing meaningful segmentation, such as detecting phoneme boundaries, in a contrastive coding regime. We hope it will be useful in building systems that detect without supervision entities such as phonemes, which are important for a given problem domain. 

\section{Acknowledgments}
The authors thank Polish National Science Center for funding
under the OPUS-18 2019/35/B/ST6/04379 grant and the PlGrid consortium for computational resources.


\bibliographystyle{IEEEtran}
