\section{INTRODUCTION}
\label{sec:intro}

Audio effects are tools employed in audio and video content creation to facilitate the manipulation of sound attributes such as loudness, timbre, spatialization, and dynamics~\cite{wilmering2020history}. 
While audio effects are powerful tools in the hands of trained audio engineers, their complexity poses a significant challenge for less experienced users, often requiring a time-consuming adjustment process even for professionals. 
While expert-designed presets provide users with a starting point in configuring an audio effect, they reproduce only static configurations, without the ability to adapt to variations across input signals.
This limits their potential in aiding users, since further fine-tuning and adjustment are often still required.
Automatic audio production systems aim to overcome this limitation with analysis and subsequent adaptation based on the input signal, which facilities intelligent control~\cite{IMPbook19}.
These systems may provide automatic control of a single effect parameter or over multiple effects and many channels of audio in a cross-adaptive manner.
They may provide a range of interfaces, from fully automatic systems, to systems that only provide suggestions, leaving the final decision to the user.
As a result, successful automatic audio production systems have the ability not only to simplify the process for amateurs, but also expedite the workflow of professionals~\cite{moffat2019approaches}.

Automatic audio production systems often fall into one of two categories: rule-based systems built on audio engineering best-practices, or data-driven machine learning systems~\cite{moffat2019approaches}.
While rule-based systems have seen success~\cite{deman2013b}, they are limited by the inability to formalize rules that cover the diversity that arises in real-word audio engineering tasks~\cite{deman2017towards}.
Classical machine learning approaches on the other hand provide greater flexibility, but have thus far been limited by the ability to collect sufficient parametric data that describes audio production in a standardized way~\cite{scott2011analysis}.
Recently, deep learning approaches have shown promise in overcoming these challenges, fueling an increasing interest in learning audio production techniques directly from large unstructured datasets.

While end-to-end deep learning approaches that utilize neural networks to directly process audio have shown promise in adjacent tasks such as speech enhancement~\cite{pascual2017segan}, source separation~\cite{stoter2019open},
and audio effect modeling~\cite{hawley2019signaltrain, martinez2020deep,wright2020real,martinez2021deep},
the application of these methods in automatic audio production has been limited by a number of challenges.
These include a lack of interpretability and user control,
tendency to produce artifacts, as well as computational challenges when operating at high sample rates.
However, using deep learning to instead directly control existing audio effects by predicting their control parameters provides a potential approach to overcome these limitations.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/style-transfer-headline-alt.drawio.pdf}
    \vspace{-0.5cm}
    \caption{Style transfer of audio effects for the automatic control of audio effects based on a reference recording. Shared-weight encoders are used to extract information from the input signal and the reference signal, which is fed to the controller to estimate effect parameters to match the production style.}
    \vspace{-0.3cm}
    \label{fig:style-transfer-headline}
\end{figure}

Automatic differentiation, a central component of deep learning, facilitates the computation of derivatives of arbitrary functions within a graph of operations~\cite{baydin2018automatic}. 
Differentiable digital signal processing (DDSP), also known as differentiable signal processing, aims to take advantage of domain knowledge by incorporating signal processing operations directly within an automatic differentiation framework, enabling gradient-based optimization~\cite{engel2020ddsp}.
This provides the ability to leverage domain knowledge from signal processing in the construction of models with an inductive bias, often facilitating interpretability and efficiency.  
This approach is not unique to signal processing, and is part of a growing trend of incorporating domain knowledge within gradient-based optimization, with specialized end-to-end differentiability approaches in computer vision~\cite{riba2020kornia,kato2020differentiable}, physics~\cite{de2018end,degrave2019differentiable}, as well as  biology~\cite{alquraishi2019end,alquraishi2021differentiable}. 

Recent works in DDSP have demonstrated the potential of this approach in the construction of both audio synthesizers and audio effects.
Applications in audio effects include IIR filter design~\cite{nercessian2020neural,colonel2021iirnet}, artificial reverberation~\cite{lee2021differentiable}, audio effect modeling~\cite{kuznetsov2020differentiable, nercessian2021lightweight}, and hearing aid tuning~\cite{tu2021dhasp}.
Approaches have also built on these works and demonstrated the optimization of multiple audio effects jointly~\cite{chen2021automatic, colonel2021reverse}.
Commonalities among the signal processing building blocks in audio effects and synthesizers has resulted in similar approaches in the domain of audio synthesis. 
This has facilitated the construction of a range of different synthesizers, all in a differentiable formulation, including additive~\cite{engel2020ddsp}, subtractive~\cite{masuda2021synthesizer}, waveshaping~\cite{hayes2021neural}, and wavetable~\cite{shan2021differentiable} synthesizers.

These DDSP approaches all achieve end-to-end differentiability through explicit definition of signal processing operations, which we refer to as explicit DDSP or EDDSP.
This approach requires manual re-implementation of low-level signal processing operations directly within an automatic differentiation framework, requiring both signal processing expertise and differentiability of all operations.
Even operations that are differentiable may pose challenges during optimization, for example if they are recursive, requiring backpropagation through time (BPTT)~\cite{hochreiter1997long}. 

Recently proposed DDSP approaches avoid explicit DDSP by instead approximating the gradient of signal processing operations, an approach we refer to as approximate DDSP or ADDSP. 
These approaches potentially relax the constraint for explicit differentiability, instead interrupting the computation graph, providing an approximation of the gradient during optimization.
For example, neural proxies achieve differentiability of arbitrary signal processors by first pre-training neural networks to emulate audio effects, which are then used as differentiable replacements of the real effects during optimization~\cite{steinmetz2021automatic}. 
Another ADDP approach involves numerical estimation of gradients for signal processing operations, treating them as black-boxes~\cite{ramirez2021differentiable}.
Both of these approach have shown promise, but remain relatively underinvestigated.
As a result, it is unclear how they perform in comparison to the more common explicit DDSP approaches. 

Besides challenges in effectively integrating audio effects to support gradient-based optimization, challenges remain in constructing an effective training framework due to the subjective and context-dependent nature of the audio production process~\cite{lefford2021context}.
While evidence suggests the existence of ``best-practices'' employed by audio engineers~\cite{pestana2014intelligent}, the multimodal nature of the production process presents difficulty in the supervised learning paradigm.
While a generative modeling approach may alleviate this,
we argue that a style transfer formulation may help to address this challenge by removing the subjective stage of the production process, instead making it explicit by conditioning on a style reference.
While audio production style transfer has been previously investigated~\cite{mimilakis2020one}, this approach did not utilize DDSP techniques, and instead relied on a loss computed in the parameter space.
This approach greatly limits performance and generalization to other classes of audio effects. 

To address these challenges we combine advancements in \emph{differentiable signal processing} for audio effects with a specialized \emph{self-supervised style transfer} training framework.
We demonstrate the first system to provide dynamic control of a chain of audio effects to match the audio production style of user-supplied reference recordings using differentiable audio effects, as illustrated in Figure~\ref{fig:style-transfer-headline}. 
Our system produces human readable control parameters enabling interpretability and user control, along with the ability to operate at variable sample rates, including those not seen during training, with low computational cost.

We perform experiments on both speech and music signals, demonstrating the ability to effectively control audio effects in emulating audio production style, all of without the need for any labelled data from audio engineers. 
In the process, we perform an extensive evaluation of both existing and novel gradient methods for integrating audio effects within a deep learning framework.
While our results indicate that explicit DDSP effects appear to achieve superior performance in the style transfer task, we show that approximate DDSP methods are competitive and are promising direction for future work.
In addition, we demonstrate the ability of our learned representation to encode relevant aspects of audio production, with potential applications in downstream tasks.

\begin{figure*}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/training.drawio.pdf}
    \vspace{-0.6cm}
    \caption{The paired data generation process for the style transfer task begins with an input recording $x$ sampled at random from a large dataset of audio recordings $\mathcal{D}$ and then a set of random augmentations are applied (pitch shifting and time stretching). Two versions of the signal are generated with the application of randomly configured chains of DSP effects, the input signal $x_i$ and the style reference $x_r$. In order to encourage the model to ignore content and instead focus on audio production style, we split the input and reference recordings, using $x_{i,a}$ as the input and $x_{r,b}$ as the reference. Embeddings $e$ from the shared weight encoders }
    \label{fig:data-generation}
    \vspace{-0.2cm}
\end{figure*}

\section{PROPOSED METHOD}
\label{sec:method}

We begin by decomposing the process normally carried out by the audio engineer into into three distinct sub-processes: (1) acoustic analysis, (2) style determination, and (3) parameter adjustment.
In the first step, the audio engineer listens to the input signal in order to establish an understanding of sonic characteristics they may wish to adjust.
Afterward, the audio engineer must then decide how they want the output to sound. 
This is a complex process based on the input signal as well as the context, which incorporates goals from all individuals involved in the process, including the audio engineer, producer, and listener~\cite{lefford2021context}.
After determining the sonic goal for the production, the final step involves configuring a set of audio effects to achieve this goal.
To perform this task, the audio engineer must utilize their expertise and understanding of audio effects in order to adjust the relevant parameters.

We argue this style determination or sonic goal setting task is central to what makes constructing a data-driven automatic audio production system difficult.
Modeling this step of the process requires learning a distribution over potential output styles. 
However, the task of acoustic analysis and audio effect control parameter estimation are objective processes. 
This realization motivates our design of our style transfer training framework, which enables us to construct a model for learning automatic control without a generative formulation by making the style determination step explicit. 
In this formulation, the goal of the model is control the effects so that the style of the reference recording is produced in the output, removing the need for the model to determine the style. 
Since our definition of audio effect style exists within the space of all possible configurations of a set of effects, we can generate effectively limitless training data by producing style references using random configurations of the effects. 

While the style transformation process may appear to place a larger burden on the user compared to a fully automatic system, it actually provides multiple mechanisms for user-interaction.
Complete style transfer, where the user supplies their own reference recording to the guide the process, provides the user with the most agency.
However, less experienced users may not have access to reference recordings.
A simpler paradigm may involve \emph{adaptive presets} where a set of curated reference recordings are provided, each with semantic descriptions (e.g. warm, bright, broadcast). 
Unlike traditional presents, the style transfer formulation allows the system to adapt parameter estimates based on the input.
In cases where a ``one-button'' solution is desired, a single predefined style reference that represents the designers desired output production style may be used.
In addition, beyond the automatic production task, our training framework also supports the possibility for audio effect removal~\cite{steinmetz2021dl4am}. 

Our proposed method is composed of both a data generation framework along with a model architecture that facilitates learning to control audio effects for the style transfer task using differentiable signal processing. 


\subsection{Self-supervised training}

Core to our approach is the design of a domain inspired self-supervised training framework that enables us to generate paired training examples for the style transfer task using nearly any audio dataset. 
Using the style transfer formulation we generate large datasets that enable us to learn both representations that encode information about audio production style as well as how to control audio effects, all without the need for any annotations or labels produced by audio engineers. 
This enables us to overcome one of the fundamental challenges in building audio production models, which is the lack of large scale audio production datasets. 

As shown in Figure~\ref{fig:data-generation}, the data generation process begins by selecting a random recording $x$ from a large dataset of recordings $\mathcal{D}$. 
To further increase the diversity of the source material, we create an augmented version of this recording by applying pitch shifting and time stretching with random parameters.
Following this step, we create two versions of the augmented recording.
First, we create what will become the input recording $x_i$ by applying a set of DSP audio effects that are randomly configured.
In our formulation, this includes a parametric EQ followed by a dynamic range expander, which serve to both perturb the frequency content and dynamic range of the signal.
Then, we create the target or style reference $x_r$ by applying a different random parametric EQ followed by a random compressor, which aims to simulate a potential target style recording. 
The model will then be tasked with configuring a set of audio effects such that after processing the input signal it matches the style reference.

However, in deployment, the style reference will likely contain differing source material. 
This requires that our model learn representations that are largely agnostic to the content, instead focusing only on the production style. 
To encourage this, we split both the input and style reference two recordings in time.
This allows us to generate an $a$ and $b$ section for each, which gives $x_{i,a}$, $x_{i,b}$, $x_{r,a}$, and $x_{r,b}$.
During training, we randomly select either the $a$ or $b$ section to be passed as input, while the other section is used as the style reference.
While the sources present in the $a$ and $b$ sections are likely consistent, we employ an assumption that the content (e.g. phonemes, instrumentation, etc.) present within these two sections differ. 
During computation of the loss, we use the style reference corresponding to the same section as the input in order to have a paired input and groundtruth recording with the same content. 

\subsection{System architecture}




As illustrated in the high-level depiction in Figure~\ref{fig:style-transfer-headline}, the system is composed of three submodules: an encoder $f_\theta$, a controller $g_\phi$, and a set of differentiable audio effects (DDSP). 
The role of the encoder is to extract relevant information from both the input signal and the reference signal relating to the audio production style. 
We concatenate these representations encoder representations ($e_i$ and $e_r$), which are then passed to the controller.
The controller aggregates the representations in order to estimate a total of $P$ control parameters $p \in \mathbb{R}^{P}$ for the set of differentiable audio effects. 
The use of differentiable audio effects then enables the ability to compute gradients with respect to the audio effect parameters, facilitating training of the weights $\theta$ and $\phi$ for the encoder and controller networks, respectively. 

\linesubsec{Encoder} Unlike many common audio representations that aim to encode information about the content within an audio signal, such as the speaker identity~\cite{nagrani2017voxceleb}, musical pitch~\cite{kim2018crepe}, sound events~\cite{salamon2017deep}, or musical tags~\cite{pons2017end}, we desire our encoder capture information solely about the audio production style.
However, we follow the design of common approaches and use a 2D convolutional neural network (CNN) operating on magnitude spectrograms.
We use the magnitude spectrogram instead of the more commonly used melspectrogram since we wish to attend to energy across the frequency range with greater granularity. 

The time domain audio input is first converted to a spectrogram $X$ using the STFT with window size $N$ and hop size $H$. 
The magnitude of the spectrogram is computed and exponential compression is applied following $|X|^{0.3}$. 
The spectrogram is then fed to the CNN, which produces a series of representations as a function of the model receptive field and number of input spectrogram frames. 
We apply temporal average pooling to aggregate information across frames, producing a single embedding of dimension $D$ for each input. 
During training, we apply the encoder both to the input and style reference recordings to generate two representations $e_i$ and $e_r$, which are concatenated and passed to the controller.

\linesubsec{Controller} The role of the controller is to further aggregate information from the encoder representations to decide how the audio effects should be configured in order apply the style from the reference recording to the input recording. 
In our formulation, the controller is a simple three layer multi-layer perceptron (MLP) featuring a single hidden layer, along with PReLU~\cite{he2015delving} activation functions. 
At the output layer, the hidden representation is projected into the audio effect control parameter space $\mathbb{R}^P$, and a sigmoid activation is applied. 
These effect parameters are then passed to the DDSP modules. 
Where needed, the normalized parameter values are rescaled to match the ranges of the respective audio effects. 

\section{GRADIENT METHODS}
\label{sec:gradient}


\begin{figure}[t!]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/basic-system.pdf}
    \caption{A neural network $f$ estimates parameters $\hat{p}$ for an audio effect $g$ as a function of the input $x$. In general, $f$ is trained to estimate parameters that configure $g$ so that loss $\mathcal{L}$ is minimized between the output $\hat{y}$ and the target $y$, where $y$ often is a version of the input $x$ that has been processed by an audio engineer. Training $f$ with gradient-based methods requires gradients be computed for operations in $f$ \textbf{and} $g$.}
    \vspace{-0.0cm}
    \label{fig:basic-system}
\end{figure}

Central to our approach is placing audio effects directly within the computation graph of the neural network to faciliate the use of a gradient-based learning framework. 
Previous approaches in learning to control effects have avoided the need for backpropagation through effects by computing a loss in the effect parameter space~\cite{ramo2019neural,sheng2019feature, mimilakis2020one}.
While this approach simplifies the process, it requires knowledge of the target effect parameters, which are often difficult to obtain and prohibit the use of more commonly available datasets comprised of pairs of unprocessed and processed audio.

Perhaps more importantly, the use of parameter space losses is likely to hinder training since many different configurations of the effect parameters may result in equivalent output signals.
As a result, the parameter space loss will penalize parameter predictions that produce audio close to the target but use different parameters. 
These factors clearly motivate methods that directly incorporate audio effects within the computation graph of a neural network, enabling the ability to instead compute the loss directly in the audio domain.
This enables the us to forgo knowledge of the original parameters and no longer penalizes predictions with differing parameters, instead relying only on similarly in the audio output.

We can formalize this approach as shown in Figure~\ref{fig:basic-system}, where an audio effect is generalized as a function $y = g(x, p)$ that takes as input a signal $x \in \mathbb{R}^T$ with $T$ samples and a set of $P$ parameters $p \in \mathbb{R}^{P}$ to produce a transformed output $y \in \mathbb{R}^T$. 
Learning to control audio effects involves constructing a neural network $\hat{p} = f_{\theta}(x)$ with weights $\theta$, which analyzes the input signal $x$ and produces a set of control parameters $\hat{p}$.

In general, the goal of the network is to produce control parameters that configure the effect so that the output $y$ achieves a specific goal, such as correcting issues with a recording, or applying a certain production style.
To update the weights $\theta$ of the control function $f$, gradient-based optimization approaches are generally employed, which provide a method for updating $\theta$ according to a predefined objective function $\mathcal{L}$, such as the distance between the output and the target are minimized.
To do so, gradients $\nabla$ must be computed for each operation within both the network $f$ and the audio effect $g$.

As mentioned in Section~\ref{sec:intro}, DDSP approaches facilitate integration of audio effects within the computation graph of neural networks for this purpose. 
However, there has yet to be a formal categorization or comparison of DDSP approaches. 
We present a categorization of existing approaches which include: 

\begin{enumerate}
    \item Explicit DDSP~\cite{engel2020ddsp,kuznetsov2020differentiable, nercessian2020neural, colonel2021reverse}, where signal processing operations are re-implemented directly within a deep learning automatic differentiation framework~\cite{abadi2016tensorflow, paszke2019pytorch}.
    \item Approximate DDSP, where the gradient of signal processing operations are instead approximated by different means, avoiding the need for automatic differentiation and differentiability. Existing approaches include:
    \begin{enumerate}
        \item Neural proxies~\cite{tseng2019hyperparameter, steinmetz2021automatic}, where a neural network is first trained to emulate an signal processing operations, including parametric control, and is then connected with another network trained to use this proxy network.
        \item Gradient approximation~\cite{ramirez2021differentiable}, where effects are treated as a black-box and their gradients are estimated with numerical gradient approximation, such as finite differences (FD) or simultaneous perturbation stochastic approximation (SPSA)~\cite{spall1998overview}.
    \end{enumerate}
   
\end{enumerate}



\subsection{Explicit DDSP}
\label{sec:autodiff}



\subsection{Approximate DDSP}

ADDSP approaches provide a number of benefits as compared to their explicit counterparts.
Unlike EDDSP approaches, these approaches do not require the low-level signal processing operations to be re-implemented. 
Instead, they enable the ability to use existing audio effects while approximating the gradient.
This means that we relax constraints present in EDDSP approaches, including both differentiability of the signal processing building blocks, as well as potential computational inefficiencies arising due to backpropagation. 

\subsubsection{Neural proxy}
\label{sec:proxy}

In the neural proxy approach a neural network is first trained to emulate the behavior of the signal processing operations of interest. 
This is generally achieved by producing a dataset containing many input and output measurements from the system. 
In cases where the system features control parameters, these parameters can be randomly configured during the data generation processor. 
After pre-training, the proxy network is then used as a replacement for the original signal processing device. 
Since the neural proxy is composed of traditional neural network building blocks, it is fully differentiable, and provides the ability to compute gradients with respect to the control parameters using automatic differentiation, all without the need for any knowledge of the underlying signal processing operations.
Assuming the neural proxy behaves similarity to the original effect, this gradient can be treated as a approximate gradient for the real effect control parameters. 

This approach was first applied in computer vision, where neural proxies of complex image signal processors (ISP) were used to facilitate gradient-based optimization~\cite{tseng2019hyperparameter}. 
While their approach was successful, their use case did not involve using these neural proxies in conjunction with a neural network that provides dynamic control over the ISP. 
Instead, they focused on performing gradient-descent in order to optimize the control parameters of ISPs to match certain reference images. 

In the context of audio, the neural proxy approach is largely inspired by the recent success in using neural networks to emulate analog audio effects~\cite{hawley2019signaltrain,  martinez2020deep,wright2020real,martinez2021deep, damskagg2019deep, martinez2019modeling, wright2019real, steinmetz2021efficient}. 
While previous approaches have focused on emulating analog hardware, where datasets must be manually constructed through a laborious process, the neural proxy approach instead relies on constructing emulations of existing audio effects. 
Through programmatic control of these effects, large training datasets can be generated for the emulation task by simply applying the effects with random parameter configurations to existing datasets of audio. 
Training a sufficiently expressive network and sampling a sufficient number of configurations of the audio effect allows for the generation of a neural network emulation that acts similarly to the original effect.
This approach has been used in order to construct a channel of a mixing console featuring a parametric EQ, compressor, and artificial reverberation with a neural network that learned dynamic control of these effects to create complete multitrack mixes~\cite{steinmetz2021automatic}. 


\linesubsec{Neural proxy hybrids}
\label{sec:proxy_hybrids}
As an extension of the neural proxy method, we propose two new variations of this approach.
We note that given the neural proxy behaves sufficiently similar to original effect, it may be possible to discard the proxy after training and use the original audio effects at inference, since gradients are no longer required.
We name this approach the \emph{neural proxy hybrid approach}, which has yet to be investigated in the literature.
In our first variant of the hybrid approach (Proxy-H 1), we use a neural network proxy $\hat{g}$ that is trained identically as the original neural proxy method, but during inference, the pre-trained proxy network is replaced with the original digital signal processing (DSP) implementation of the cloned audio effects. 
This approach enables our method to be both more computationally efficient, and approximately sampling rate independent as long as the the control network $f_\theta(x)$ is trained with a reasonable high sampling rate and the original DSP audio effect implementation is sampling-rate independent.

In our second proxy hybrid approach (Proxy-H 2), a more involved integration of the neural proxy and DSP implementation is performed. 
More specifically, we integrate the swapping of the neural proxy and DSP implementation following the Proxy-H 1 method, but here we do this swapping both for inference and the forward pass during backpropagation during training $f(x)$. 
The idea of this approach is to minimize the chance of approximation error between the neural proxy and DSP implementation while training.

\subsubsection{Numerical gradient approximation}
\label{sec:grad}
While the neural proxy approach facilities gradient approximation through the use of pre-trained neural network, numerical approximation of the gradients provides another route to enable gradient-based optimization of signal processing operations.
For example, finite differences (FD)~\cite{milne2000calculus} provides a method to estimate the partial derivative of the system $g$ with respect to the parameter $p$ given by 
\begin{equation}
\frac{\partial g'(x, p_i)}{\partial p_i} \approx \frac{g(x, p + \epsilon) - g(x, p - \epsilon)}{2\epsilon} 
\end{equation}
(\emph{need to unify the notation here with the figures.)}
where $\epsilon$ is some small, non-zero value.
This provides a method to estimate the gradient with respect to the parameters of the signal processing system, which can be passed onward during backpropagation so that automatic differentiation can be used to update the parameters of the neural network $f()$. 
However, in practice, FD poses challenges in scaling as the number of parameters $P$ increases, as $2P+1$ evaluations of $f()$ are required, which can be prohibitively expensive. 
Simultaneous perturbation stochastic approximation (SPSA)~\cite{spall1998overview} is one attempt to address this scaling issue, which instead perturbs all parameters concurrently during the estimation process, requiring only two function evaluations.
When this process is repeated over a number of iterations the resulting gradient estimates become closer to the true gradient of the underlying system.

In the context of audio, this approach has was used to train a neural network controller for a number of automatic audio production tasks~\cite{ramirez2021differentiable}. 

This approach is unique in that it does not require a pre-training stage, and additionally does not require any knowledge of the underlying signal processing operations in the original audio effect.
However, in practice, optimization may be more challenging due to inaccuracies in the gradient approximations, potentially slower due to a larger number of iterations through the audio effect, which are often implemented on CPU and cannot take advantage of parallelization via GPU resources. 

While our implementation in based on this previous work, we make a few key changes to adapt the approach to our training framework.
In particular, we change the length of audio used when computing the loss $\mathcal{L}$ to the encoder $f(x)$ . 
In past work, the loss was instead computed on very short audio windows ($46$\,ms), which required ~\cite{ramirez2021differentiable}. 
In our work, we simplify the architecture and input a three second signal to both the encoder $f(x)$ and loss $\mathcal{L}$, hence making a single parameter prediction for the entirety of this window, with the result that the gradient is averaged across this time window.
While it was shown that using small windows and adjusting the parameter , this provides a less human interpretable result


\section{Differentiable audio effects}
\label{sec:audio-effects}

In order to perform a fair evaluation between gradient methods, we base all methods on the same set of audio effect implementations. 
We utilize a 6-band parametric EQ and single-band feedforward dynamic range compressor connected in series as shown in Figure~\ref{fig:signal-chain}. 
The parametric EQ features a single low-shelf filter, four peaking filters, and single high-shelf filter. 
We implement the EQ with a cascade of biquad filters where the coefficients of each biquad are computed as a function of the 16 control parameters (2 for shelves and 3 for peaking) and the sample rate, following the Audio EQ Cookbook~\cite{bristow2016cookbook}.
Recall that frequency response of a biquad filter is given by 
\begin{equation}\label{eq:biquad-response}
    H(z) = \frac{}{}
\end{equation}.
Thus the approximate response of the system is given by the product of the response of each of the K biquads.
\begin{equation}\label{eq:system-response}
    H_{sys}(z) = \prod_{k=1}^{K} H_k(z).
\end{equation}

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/signal-chain.pdf}
    \vspace{-0.5cm}
    \caption{The reference audio effect chain featuring a 6-band parametric EQ and a single-band dynamic range compressor. The EQ includes low-shelf, high-shelf and four peaking bands, each with their own center frequency, Q-factor, and gain control. The compressor follows a digital feedforward design. In total there are 22 control parameters between the two effects.}
    \label{fig:signal-chain}
    \vspace{-0.2cm}
\end{figure}

With these coefficients, the output is then computed in the time domain applying each biquad in series following the difference equation
\begin{equation}
    \begin{aligned}
    y[n] = b_0 x[n] &+ b_1 x[n-1] + b_2 x[n-2] \\ 
     & - a_1 y[n-1] - a_2 y[n-2],
    \end{aligned} \label{eq:diff-eq}
\end{equation}
where $y[n]$ is the output of the filter at sample $n$.
We constrain the ranges of the parameters, limiting the overall gain values, and limiting overlap in the center frequency between the bands, shown in Table~\ref{tab:parameter-ranges} in the Appendix.

The dynamic range compressor follows the design of a digital feedforward compressor using a second order gain computer peak level detector operating in the log-domain~\cite{giannoulis2012digital}. 
The target output gain $y_G[n]$ is given by,
\begin{equation}
    \resizebox{\linewidth}{!}{
    $y_G[n] =
    \begin{cases}
        x_{\textrm{dB}}[n] & 2 (x_{\textrm{dB}}[n] - T) \leq -W \\
        x_{\textrm{dB}}[n] + \frac{ \big( \frac{1}{R}-1 \big) \big(x_{\textrm{dB}}[n] -T + \frac{W}{2} \big)^2 }{2W} & 2 | x_{\textrm{dB}}[n] - T| \leq W \\ 
        T + \frac{x_{\textrm{dB}}[n] - T}{R} & 2 (x_{\textrm{dB}}[n] - T) > W
    \end{cases}$},
\end{equation}
where $T$ is the threshold and $W$ is the knee width, both in dB, and $R$ is the compression ratio.
We then take the difference between the gain computer estimate and the input gain $x_L[n] = x_{\textrm{dB}}[n] - y_G[n]$, which is then smoothed by the 
These filters are implemented as 1-pole lowpass filters with time constants $\alpha_A$ and $\alpha_R$ as a function of the attack and release times. 

\begin{equation} \label{eq:smoothing}
    \resizebox{.9\hsize}{!}{
   $y_L[n] =
    \begin{cases}
        \alpha_A y_L[n-1] + (1-\alpha_A) x_L[n] & x_L[n] > y_L[n-1] \\
        \alpha_R y_L[n-1] + (1-\alpha_R) x_L[n] & x_L[n] \leq y_L[n-1]
    \end{cases}$       
    }
\end{equation}

The final gain reduction is computed as the negative of the output and converted to the linear domain to be applied as a multiplication.
\begin{equation}
    c_{\textrm{in}}[n] = 10^{-y_L[n]/20}
\end{equation}

\emph{(Equations here still need work. The main rationale to include them is that we want to demonstrate that these ops are in fact differentiable.)}

\subsection{Explicit DDSP audio effects}

\linesubsec{Parametric equalizer}\label{sec:eq} To design our differentiable parametric EQ, we follow the reference design presented in Section~\ref{sec:audio-effects}, featuring low-shelf, peaking, and high-shelf filters. 
Both the coefficient computation and filter application with the difference equation are composed of differentiable operations~\cite{nercessian2020neural}. 
As a result, it is possible to implement this effect following traditional designs within an automatic differentiation framework.
However, in practice, doing so will result in issues arising during backpropagation.

Due to the recurrence in the difference equation shown in (\ref{eq:diff-eq}), backpropagation through time (BPTT) will be required, tracking the state of gradients through the recurrence. 
While this may pose challenges in optimization due to vanishing gradients~\cite{hochreiter1997long}, we found the bigger limitation was in the computational bottleneck imposed by this recurrence, since each gradient must be computed in series, and there is significant computational overhead for each computation.
To address this, instead of implementing IIR filter in the time domain with the recurrence given by the difference equation (\ref{eq:diff-eq}), we instead apply the filter in the frequency domain using an FIR approximation, an approach used in previous work on differentiable biquads~\cite{nercessian2020neural, nercessian2021lightweight, colonel2021iirnet}.

We approximate the complex frequency response $H(e^{j \omega})$ over a linearly spaced set of digital frequencies $\omega$, which is analogous to designing an FIR filter with the frequency sampling method. 
This corresponds to performing the discrete Fourier Transform (DFT) on both the numerator and denominator coefficients as if they were signals and dividing them. 
We perform filtering in the frequency domain by multiplying the complex response of the filter with the complex response of the input signal. 
To achieve this, we zero-pad both the input signal and the filter coefficients to the nearest power of 2 based on the length of the input signal.
After taking the DFT, we can then compute the approximate output of the filter in the time domain by perform the inverse DFT. 
\begin{equation}
    y = \mathcal{F}^{-1} \Bigg\{ \mathcal{F}\{x \} \cdot \frac{ \mathcal{F}\{ \mathbf{b} \} }{ \mathcal{F}\{ \mathbf{a} \} } \Bigg\}
\end{equation}
In practice, the use of fast Fourier transform (FFT) to compute the $\mathcal{F}\{x \}$ and $ \mathcal{F}\{ \mathbf{b} \} / \mathcal{F}\{ \mathbf{a} \} $ enables this operation to be heavily parallelized and run efficiently on GPU.

\linesubsec{Dynamic range compressor}
While there has been work in differentiable signal processing to address effects such as equalizers, distortion, and reverberation, to our knowledge there has been no work on the design of an explicit DDSP dynamic range compressor.
Inspection of the design of the feedforward dynamic range compressor~\cite{giannoulis2012digital} reveals it is in fact composed of differentiable operations.
However, similar to the parametric EQ, we found direct implementation of existing compressor designs problematic due to the presence of recursive structures, in this case due to the recursive smoothing filters used to model the ballistics.

Unfortunately, since this operation involves a set of branching recursive filters, where the activity of each filter is itself based on a recursion, as shown in (\ref{eq:smoothing}), approximating the behavior of this branching level detector is nontrivial. 
To address this, we instead replace this branching filter with a single smoothing filter that results in a combined attack and release coefficients
\begin{equation}
   y_L[n] = \alpha_{AR} y_L[n-1] + (1-\alpha_{AR}) x_L[n].
\end{equation}
As in the parametric EQ, this 1st order recursive filter can be approximated using a feedforward FIR filter following the approach outlined in the previous section.
While this modification does restrict the capabilities of the compressor, enforcing the same time constants for both the attack and release of the device, we found this formulation sufficient.

\subsection{Audio effect neural proxy pre-training}

We first pre-trained the neural proxies to emulate both a the reference parametric EQ and compressor. 
To construct the neural proxies, we utilized the temporal convolutional network (TCN)~\cite{bai2018empirical}, also known as the feedforward WaveNet~\cite{rethage2018wavenet}, which has seen success in modeling distortion~\cite{wright2019real} and compressors~\cite{steinmetz2021automatic}.
We base the architecture on the causal TCN~\cite{steinmetz2021automatic}, which uses feature-wise linear modulation (FiLM)~\cite{perez2018film} to adapt to the audio effect control parameters, along with rapidly growing dilation factors to obtain a large receptive field using a fewer number of blocks.
This TCN configuration has $4$ blocks, kernel size of $13$, channel width of $64$ and a dilation factor that grows as a power of $8$, where the dilation factor at block $b$ is $d_b = [ 1, 8, 64, 512 ]$.
This model feature a total of parameters with a receptive field of $\approx 300$\,ms at a sample rate of 24\,kHz.

We trained a total of 4 proxy models, one for each of the effects, with one model trained on speech and one trained on music. 
Audio examples were generated by processing audio using the reference audio effect implementations, sampling the random parameter values from a uniform distribution across their ranges. 
These models were trained for 400 epochs with an initial learning rate of $3\cdot 10^{-4}$ and a batch size of 16, using input segments $\approx 3$\,s ($65536$ samples) in length. 
One epoch was defined as 20,000 random segments and the learning rate was lowered by a factor of 10 after the validation loss had not improved for 10 epochs. 
 
\subsection{Gradient approximation for audio effects}

Here we will provide some details on using the SPSA method. 
Probably a good place to talk about the $\epsilon$ hyperparameter and also the need for implementing parallel processing for training.













\setlength{\tabcolsep}{3.0pt}
\setlength{\abovecaptionskip}{15pt plus 3pt minus 2pt}
\begin{table*}[t!]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
        \multicolumn{21}{c}{\normalsize{\textbf{Speech}}} \\ \midrule
        & \multicolumn{6}{c}{\textbf{LibriTTS}} & & \multicolumn{6}{c}{\textbf{DAPS}} & & \multicolumn{6}{c}{\textbf{VCTK}} \\ \cmidrule(lr){2-7} \cmidrule(lr){8-14} \cmidrule(lr){15-21}
        \textbf{Method}  &  \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}}\\ \midrule
        {Corrupt}   
        & 3.765           & 1.187           & 2.180           & 687.5           & 6.983           & 2.426           & 
        & 3.684           & 1.179           & 2.151           & 641.7            & 6.900          & 2.314          & 
        & 3.672           & 1.254           &          2.008  & 815.4           & 7.783           & 2.532\\
        {RB-DSP}  
        & 3.856           & 0.943           & 1.955           & 410.3           & 4.204           & 1.674           & 
        & 3.787           & 0.917           & 1.882           & 399.7            & 3.705           & 1.481          & 
        & 3.709           & 1.101           &          1.911  & 657.6           & 5.039           & 2.018\\ \midrule
        \textbf{E2E} \\ \midrule
        {cTCN 1}    
        & 4.258           & 0.405           & 0.887           & 128.4           & 2.237           & 1.066           & 
        & 4.185           & 0.419           & 0.884           & 124.6            & 2.098           & 1.006          & 
        & 4.181           & 0.467           &          0.891  & 173.8           & 2.651           & 1.165\\
        {cTCN 2}    
        & 4.281           & 0.372           & \textbf{0.833}  & 117.5           & 1.927           & 0.925           & 
        & \textbf{4.224}  & \textbf{0.391}  & \textbf{0.841}  & 113.9            & 1.886           & 0.913          & 
        & 4.201           & \textbf{0.441}  & \textbf{0.856}  & 163.8           & 2.431           & 1.086\\ \midrule
        \textbf{ADDSP} \\ \midrule
        {Proxy 0}   
        & 3.643           & 0.676           & 1.405           & 265.0           & 2.812           & 1.340           & 
        & 3.605           & 0.685           & 1.362           & 249.2            & 2.732           & 1.350          & 
        & 3.651           & 0.737           &          1.300  & 321.7           & 3.166           & 1.453\\
        {Proxy-H 1}   
        & 3.999           & 1.038           & 2.179           & 440.2           & 5.472           & 2.679           & 
        & 3.903           & 1.022           & 2.113           & 451.9            & 5.104           & 2.535          & 
        & 3.951           & 1.044           &          1.930  & 591.5           & 5.194           & 2.651\\
        {Proxy-H 2}   
        & 3.945           & 1.058           & 2.088           & 404.9           & 6.820           & 3.197           & 
        & 3.891           & 1.037           & 2.045           & 395.4            & 6.754           & 3.117          & 
        & 3.894           & 1.087           &          1.934  & 514.0           & 7.065           & 3.363\\
       
        {SPSA}      
        & 4.180           & 0.635           & 1.406           & 219.5           & 3.263           & 1.600           & 
        & 4.099           & 0.645           & 1.379           & 213.6            & 2.989           & 1.511          & 
        & 4.023           & 0.730           &          1.359  & 301.6           & 3.535           & 1.737\\ \midrule
        \textbf{EDDSP} \\ \midrule
        {Autodiff}  
        & \textbf{4.310}  & \textbf{0.388}  & 0.882           & \textbf{111.5}  & \textbf{1.828}  & \textbf{0.823}  & 
        & 4.222           & 0.416           & 0.895           & \textbf{109.0}   & \textbf{1.758}  & \textbf{0.799} & 
        & \textbf{4.218}  & 0.481           &          0.924  & \textbf{152.7}  & \textbf{2.317}  & \textbf{1.006} \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
    \caption{Objective metrics on the synthetic style transfer task using models trained with differing gradient methods using the LibriTTS dataset. Held-out speakers and utterances from the LibriTTS dataset are used for this evaluation, while utterances from DAPS and VCTK come from datasets never seen during training.}
    \label{tab:style-transfer-synthetic-speech}
\end{table*}


\section{EXPERIMENTAL DESIGN}

Using the self-supervised training framework described in Section~\ref{sec:method}, we trained a range of models on the style-transfer task using the gradient methods for differentiable audio effects presented in Section~\ref{sec:gradient}. 

\subsection{Datasets}
\label{sec:datasets}

As described in Section~\ref{sec:method}, we employ a specialized data generation pipeline that produces paired training examples for the style transfer task.
Unlike traditional speech enhancement pipelines that require ``clean'' recordings, which will then be corrupted, our approach can utilize any dataset, preferably one that covers a wide range of quality levels and recording conditions. 
We train two sets of models, first using speech recordings sourced from LibriTTS~\cite{zen2019libritts}, which provides recordings from audiobooks of varying quality.
We use recordings only from the \texttt{train-clean-360} subset, which includes 360 hours of audio.
Then we train the same set of model using music recordings from the MTG-Jamendo dataset~\cite{bogdanov2019mtg}, which provides over 55k songs, which we downmix from stereo to mono. 

For training, all audio is resampled to 24\,kHz. 
The data generation process uses random sections of audio $\approx 10$ seconds in length ($262144$ samples). 
To avoid training with examples that contain a significant portion of silence, we measure the energy. 
If the randomly selected segment is below the energy threshold, we continue sampling random patches from the selected recording until a non-silent segment is found.
To ensure there is sufficient headroom for processing within the audio effects, we peak normalize all inputs and target recordings to -12\,dBFS.
These examples are then split in half to generate the input and reference recordings.
For both datasets we generate a 90/5/5 split of the recordings for training, validation, and testing.
In the case of LibriTTS, we ensure that there is no overlap between the speakers in the different subsets.

 
\subsection{Baselines}
\label{sec:baselines}


\linesubsec{Rule-based DSP}
\label{sec:dsp}
This baseline consists of a two stage process that aims to mimic the signal chain of the parametric equalizer and compressor that can be configured in an automatic way based on low-level signal analysis. 
First, an inverse FIR filtering stage is employed, which aims to transform the spectral content of the input speech to match the spectrum of the style reference. 
We compute the average spectrum of the input and style reference by first computing the STFT using a large window size of $N=65536$ and a hop size of $H=16384$, averaging across the frames, yielding $X_i$ and $X_r$.
Each averaged spectrum is then smoothed using a Savitzky-Golay filter~\cite{savitzky1964smoothing}.
This allows us to design an inverse FIR filter with $63$ taps with the window method, with a magnitude response given by $X = X_{i,sm} / X_{r,sm}$, where $X_{i,sm}$ and $X_{r,sm}$ are the smoothed average spectra of the source recording and the target.

The filtered signal is sent to a dynamic range compressor where all parameters except for the threshold are fixed to a set of general values. 
In order to adjust the amount of compression based on the input signal and the reference we perform a heuristic to find an optimal value for the threshold.
This process starts by lowering the threshold in increments of 0.5 dB. 
At each iteration, the loudness in dB LUFS~\cite{steinmetz2021pyloudnorm} of the output is measured and then compared to the loudness of the style reference.
If the difference in the loudness measurements are below a given threshold (e.g 0.5 dB LUFS), then the process halts. 
Otherwise, the process continues until the loudness measurements are within the specific tolerance, or the threshold reaches -80 dB, at which point the process also stops.
While this approach does not enable the configuration of the compressor parameters, we found in practice it was successful in matching the overall amount of compression applied. 

\linesubsec{End-to-end neural network}
\label{sec:e2e}
To create a baseline on the opposite end of the spectrum, we also evaluate an end-to-end (E2E) neural network approach to replace the DDSP audio effects. 
This network utilizes the same architecture as the neural proxy approach, but instead does not use pre-trained weights. 
Instead, the weights of this model are optimized during the training of the encoder and controller models. 
Even though the controller still predicts parameters for this network as in the case of the neural proxy, since the weights of the network are not frozen during training, the model will learn its own set of control parameters. 
While this method provides the most expressive processing out of the methods we evaluate, it imparts a high compute cost at both training and inference, in addition to potential for artifacts, and uninterpretable control parameters.


\setlength{\tabcolsep}{2.5pt}
\begin{table*}[ht]
    \centering
    \resizebox{\linewidth{}}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
    \multicolumn{21}{c}{\normalsize{\textbf{Music}}} \\ \midrule
        &  
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (24 kHz)}}   & & 
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (44.1 kHz)}} & & 
        \multicolumn{6}{c}{\textbf{MUSDB18 (44.1 kHz)}} \\ 
        \cmidrule(lr){2-7} \cmidrule(lr){9-14} \cmidrule(lr){15-21}
        \textbf{Method}   & 
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        Corrupt      
        & 2.849  & 1.175  & 6.186  & 669.298    & 6.667  & 2.510 & 
        & 2.874  & 1.109  & 4.454  & 767.664  & 6.793  & inf   & 
        & 2.900  & 1.252  & 4.342  & 1088.327    & 5.940  & 2.312    \\ 
        RB-DSP     
        & 2.839  & 0.878  & 4.285  & 254.154 & 3.750  & 1.531 & 
        & 2.931  & 0.887  & 3.355  & 341.957   & 4.749  & 1.882 & 
        & 2.994  & 0.821  & 3.052  & 379.400   & 4.078 & 1.665  \\ \midrule
        \textbf{E2E} \\ \midrule
        cTCN 1       & 3.351  & 0.547  & 2.480  & 166.480    & 3.495  & 1.609 &
        & 3.168  & 0.876  & 2.921  & 494.368    & 4.372  & 2.070  & 
        & 3.121  & 0.896  & 2.956  & 730.446    & 4.548  & 2.231  \\ 
        cTCN 2       & 3.320  & 0.559  & 2.485  & 164.577    & 3.136  & 1.501 & 
        &  3.123  & 0.903  & 2.973  & 517.350   & 4.084  & 1.899  &
        &  3.107  & 0.917  & 2.986  & 749.454   & 4.208  & 2.061 \\  \midrule
        \textbf{ADDSP} \\ \midrule
        Proxy 0 S  
        & 2.941  & 0.773  & 2.965  & 221.462 & 3.118  & 1.482 & 
        & 2.764  & 1.033  & 2.869  & 488.273  & 3.710  & 1.824 & 
        & 2.804  & 0.950  & 2.778  & 742.068   & 3.835  & 1.921  \\
        Proxy 0 M  & 2.941  & 0.773  & 2.965  & 221.462    & 3.118  & 1.482 & 
        & 2.699  & 1.042  & 2.928  & 497.022    & 3.942  & 1.961   & 
        & 2.791  & 0.946  & 2.800  & 757.082   & 4.209  & 2.127    \\
        Proxy-H 1 S  & 2.828  & 1.074  & 7.014  & 423.571    & 6.143  & 2.961 & 
        & 2.865  & 1.101  & 5.194  & 689.232    & 6.792  & 3.365 & 
        & 2.853  & 1.165  & 4.852  & 1005.729   & 6.451  & 3.269 \\
        Proxy-H 1 M  & 2.828  & 1.074  & 7.014  & 423.571    & 6.143  & 2.961 & 
        & 2.512  & 1.148  & 5.385  & 854.932   & 5.940  & 2.740 & 
        & 2.493  & 1.198  & 5.090  & 1021.863    & 5.585  & 2.731 \\
        Proxy-H 2 S  & 2.793  & 1.002  & 5.180  & 291.825    & 3.480  & 1.702 & 
        & 2.888  & 0.977  & 3.883  & 429.862    & 3.480  & 1.709   & 
        & 2.857  & 1.045  & 3.809  & 617.259   & 3.932  & 1.971   \\
        Proxy-H 2 M  & 2.793  & 1.002  & 5.180  & 291.825    & 3.480  & 1.702 & 
        & 2.625  & 1.133  & 5.512  & 844.595   & 6.417  & 2.876  & 
        & 2.575  & 1.225  & 5.450  & 1172.493   & 5.973  & 2.913   \\
        SPSA         & 3.123  & 0.712  & 3.203  & 220.400    & 3.003  & 1.396 & 
        & 3.172  & 0.759  & 2.458  & 386.229   & 2.839  & 1.311 & 
        &  3.126  & 0.789  & 2.321  & 574.392   & 2.925  & 1.394    \\ \midrule
        \textbf{EDDSP} \\ \midrule
        Autodiff     & 3.348  & 0.500  & 2.426  & 154.307    & 2.451  & 1.098 & 
        &  3.400  & 0.585  & 1.824  & 304.393    & 2.425  & 1.106 & 
        &  3.396  & 0.608  & 1.695  & 456.131    & 2.559  & 1.197  \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Objective metrics on the synthetic style transfer task using models trained with differing gradient methods using the MTG-Jamendo dataset. Proxy models with (S) were pre-trained using speech from LibriTTS while Proxy models with (M) were trained using music from the MTG-Jamendo dataset.}
    \label{tab:style-transfer-synthetic-music}
\end{table*}

\setlength{\tabcolsep}{3.0pt}
\begin{table}[ht]
    \centering
    \resizebox{.8\hsize}{!}{%
    \begin{tabular}{l c c c c} \toprule
        & \multicolumn{1}{c}{} & \multicolumn{2}{c}{Inference RTF}  \\ \cmidrule(lr){3-4}
        \textbf{Method}  &  \textbf{Min./epoch}   &  \textbf{CPU} &  \textbf{GPU} & \textbf{Parameters} \\ \midrule
        RB-DSP    & - & & - \\ \midrule
        Proxy 0     &  & & &   \\
        Proxy-H 1     &  & & &   \\
        Proxy-H 2     &  & & &   \\
        cTCN 1      &  & &  \\
        cTCN 2      &  & & &  \\
        SPSA        &  & & -   \\
        Autodiff    &  & & \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Runtime comparison across gradient methods.}
    \label{tab:runtime}
\end{table}



\setlength{\tabcolsep}{3.0pt}
\begin{table*}[ht]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c} \toprule
        \multirow{2}{*}{\textbf{Style}} &   \multicolumn{6}{c}{\textbf{Parametric Equalizer}}                                & \multicolumn{4}{c}{\textbf{Compressor}}  \\ \cmidrule(lr){2-7} \cmidrule(lr){8-11}
                       & LS Gain & LS Freq. & Peak Gain & Peak Freq. & HS Gain & HS Freq. & Threshold & Ratio & Attack (s) & Release (s) \\  \midrule
        Telephone & -26,-20 & 200,400 & +12,+16 & 1000,2000 & -26,-20 & 4000,6000 & -30,-10 & 1.5,3.0 & 0.001,0.006 & 0.010,0.020 \\
        Warm & +20,+26 & 200,400 & 0 & 1000 & -26,-20 & 8000,10000 & 0 & 1.0 & 0.050 & 0.100  \\ 
        Bright & -26,-20 & 200,400 & 0 & 1000 & +20,+26 & 8000,10000 & 0 & 1.0 & 0.050 & 0.100 \\ 
        Neutral & +1,+3 & 80,200 & 0 & 1000 & +1,+3 & 6000,8000 & -30,-20 & 1.5,2.0 & 0.005,0.025 & 0.020,0.0250 \\ 
        Broadcast & +2,+6 & 80,200 & 0 & 1000 & +2,+6 & 6000,8000 & -50,-40 & 4.0,6.0 & 0.001,0.005 & 0.020,0.0250 \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
    \caption{Style generation parameter ranges. Style examples are generated by applying the parametric equalizer and compressor using parameter uniformly sampled in the given ranges. Gain is denoted in dB and frequency in Hz.}
    \label{tab:style-ranges}
\end{table*}

\setlength{\tabcolsep}{3.0pt}
\begin{table}[ht]
    \centering
    \resizebox{.8\hsize}{!}{%
    \begin{tabular}{l c c c c } \toprule
        \multicolumn{5}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Method} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Corrupt}       & 10.398        & 1041.654        & 10.414          & 3.649    \\ 
        {RB-DSP}      & 8.892         & 517.179         & 5.842           & 2.423    \\  \midrule
        {L Proxy-H 1 S}     & 9.770         & 636.605         & 12.922          & 5.878            \\
        {L SPSA}          & 7.972         & 360.0733         & 5.121           & 2.511           \\
        {L Autdiff}       & \textbf{7.756}& \textbf{278.112}& \textbf{5.239}  & \textbf{2.368}            \\ \midrule
        \multicolumn{5}{c}{\textbf{MUSDB18}}  \\ \midrule
        \textbf{Method} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Corrupt}       & 8.420         & 2607.409      & 9.356        & 3.795    \\ 
        {RB-DSP}      & 6.534         & \textbf{915.390}       & 8.647         & 3.662    \\  \midrule
        {J Proxy-H 1 S}     & 6.964         & 1512.650      & 10.019        & 4.472            \\
        {J SPSA}          & 5.535         & 1297.033      & 4.578         & 2.0731           \\
        {J Autdiff}       & \textbf{4.820}& 947.609       & \textbf{3.776}& \textbf{1.732}            \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Average objective metrics across all style transfer configurations from the 5 styles in the realistic style transfer task using input utterances from DAPS and music recordings from MUSDB18 with style references containing differing content. L denotes models trained on LibriTTS while J denotes models trained on the MTG-Jamendo dataset.}
    \label{tab:daps-and-podcast}
\end{table}

\subsection{Losses}


The training loss was computed directly on the audio, using a weighted sum of the error in the time domain and the frequency domain.
The loss in the time domain $\mathcal{L}_{\text{time}}$ is given by the $L_1$ distance or mean absolute error (MAE) between the groudtruth $y[n]$ and estimated waveform $\hat{y}[n]$, which both have $N$ samples
\begin{equation}
      \mathcal{L}_{\text{time}}({\hat{y}},y) = \frac{1}{N} \sum_{n=1}^{N} |\hat{y}[n] - y[n] |  \label{eq:time}.
\end{equation}

The loss in the frequency domain $\mathcal{L}_{\text{freq}}$ is given by the multi-resolution short-time Fourier transform loss (MR-STFT)~\cite{wang2019neural}, using the implementation in \texttt{auraloss}~\cite{steinmetz2020auraloss}.
This loss is the sum of the $L_1$ distance between the STFT of the groundtruth and estimated waveforms measured both in the $\log$ and linear domains, at $R$ different STFT resolutions, with different window and hop sizes, where $F$ is the number of output frames.
In our implementation $R=6$ with window sizes  $W \in [32, 128, 512, 2048, 8192, 32768]$ and hop sizes $H = W/2$.
\begin{align} 
    \mathcal{L}_{\text{lin}}({\hat{y}},y) &= \frac{1}{F} \left\| ~|\text{STFT}(y)|~ -  ~|\text{STFT}(\hat{y})|~  \right\|_1 \label{eq:lin-magnitude} \\ 
    \mathcal{L}_{\log}({\hat{y}},y) &= \frac{1}{F} \left\| \log \left( ~|\text{STFT}(y)|~ \right) - \log ( ~|\text{STFT}(\hat{y})|~ ) \right\|_1 \label{eq:log-magnitude} \\
    \mathcal{L}_{\text{freq}}({\hat{y}},y) &= \sum_{r=1}^{R} \bigg( \mathcal{L}_{\text{lin}_{r}}({\hat{y}},y) + \mathcal{L}_{\text{log}_{r}}({\hat{y}},y) \bigg) \label{eq:stft-loss} 
\end{align}
The overall loss function is simply the sum of these terms, with a weighting factor $\alpha = 100$ to normalize the scale of the losses
\begin{equation}
     \mathcal{L}_{\text{overall}}({\hat{y}},y) = \mathcal{L}_{\text{freq}}({\hat{y}},y) + \alpha \cdot \mathcal{L}_{\text{time}}({\hat{y}},y) \label{eq:overall-loss}.
\end{equation}


\subsection{Style transfer training}

After training the neural proxy models, we proceeded to train all models on the style transfer task. 
In our formulation, we used an STFT with a hann window of size $N=4096$ and a hop size of $H=2048$.
The EfficientNet~\cite{tan2019efficientnet} architecture with a random initialization was used to construct the encoder, which has its final output projected to $D=1024$ followed by mean pooling across the temporal dimension.
Finally this representation is then divided by the $L_2$ norm.

All model configurations were trained for a total of 400 epochs with a batch size of 6, where a single epoch is defined as 20,000 audio files randomly sampled from the training set. 
All models are trained with a learning rate of $10^{-4}$ with the exception of the SPSA models, which we found required a lower learning rate of $10^{-5}$ to ensure stability. 
The learning rate was scheduled such that the learning rate was lowered by a factor of 10 at both $80\%$ (epoch 320) and $95\%$ (epoch 380) through training.
We applied gradient clipping with a value of $4.0$
We used a fixed value of $\epsilon = 0.0005$ for the SPSA hyperparameter, since larger values lead to instability later on in training.




\section{RESULTS}
\label{sec:results}

To evaluate the gradient methods and the performance in the style transfer task we performed a multipart evaluation. 
In the first part we investigated the performance of the gradient methods with a synthetic style transfer task that matched the setup used for data generating during training. 
Then we performed a real-world style transfer evaluation, which more closely simulates inference-time performance by conducting style transfer with reference recordings that come from a completely different source then the input. 
Finally, we evaluated the representations learned by the encoder and demonstrate the ability of the encoder to capture relevant information about audio production style by training linear probes for a style classification task.
Code and listening examples are made available online\footnote{\url{https://csteinmetz1.github.io/deepafx2}}

\subsection{Synthetic style transfer}

Since there are no established metrics for evaluating audio production style, we consider a range of metrics that aim to capture different aspects of the system operation. 
For a general notion of perceptual similarity we consider PESQ~\cite{rix2001perceptual}, along with the Multi-resolution STFT~\cite{wang2019neural}, which is utilized as the training objective. 
To evaluate the ability of the system to address capturing EQ-related style components, we report the mel-spectral distance (MSD) and spectral centroid error (SCE). 
To capture behavior related to the dynamics of the signal, we consider the RMS energy error (RMS) and the perceptual loudness error (LUFS). 

\subsubsection{Speech}

We first evaluate models trained on LibriTTS on the synthetic style transfer task using held-out source material with The results shown in Table~\ref{tab:style-transfer-synthetic-speech}.
RB-DSP corresponds to the rule-based DSP baseline while cTCN corresponds to the end-to-end neural network baselines, featuring either 1 TCN network or 2 TCN networks connected in series.
Proxy 0 corresponds to the original neural proxy approach where the proxy network is used during inference, while Proxy-H 1 corresponds to the method using the DSP during inference, and Proxy-H 2 corresponds to the configuration where the DSP is used both during training in the forward pass, and during inference. 

We note the DSP baseline approach improves on the corrupted input across all metrics, but not by a significant margin.
Overall, we find strong performance from the Autodiff method, which performs best on all metrics except MSD, which is only slightly outperformed by cTCN2.
Both cTCN approaches follow closely behind the Autodiff method, demonstrating the strength of end-to-end neural network approaches, however these models do not feature interpretable control parameters, making them less desirable candidates. 
The SPSA model follows with strong performance, but somewhat lower than both Autodiff and cTCN methods.
Surprisingly, the Autodiff approach achieves the best overall performance even though it features no learnable parameters and utilizes a simplification of the effects used in the data generation process. 

In addition to evaluating on held-out speech from LibriTTS, we also evaluate generalization using utterances from both DAPS~\cite{mysore2014can} and VCTK~\cite{yamagishi2019vctk}.
We observe a very similar trend in performance across all methods when evaluating on utterances from DAPS and VCTK.
This indicates that training on the large scale LibriTTS dataset with out data generation pipeline was sufficient for produces models that generalize to speech originating from different datasets. 

\subsubsection{Music}

Following our speech experiments, we performed the same set of experiments again except using the MTG-Jamendo dataset. 
The results of are shown Table~\ref{tab:style-transfer-synthetic-music} and largely follow a similar trend to the speech experiments. 
Importantly, these results indicate that the proposed training framework is not restricted to only speech and produces, however it appears that the music task is somewhat more difficult as all models perform somewhat worse across all metrics as compared to the speech task.

We also investigated the ability of the methods to operate at a sample rate different than training. 
In this case the input signal to the encoder is resampled to match that during training (24\,kHz), however the audio effects instead process audio at 44.1\,kHz.
First we evaluated on the MTG-Jamendo test set at 44.1
This resulted in a significant drop in performance for the neural network based approaches, which include the cTCNs and Proxy 0, since these models operate at a fixed sample. 
Interestingly, these metrics do not indicate the models completely fail at the higher sample rate, but performance does decrease. 
However, the DSP based approaches performed significantly better, including both the Autodiff and SPSA approaches, which performed only slightly worse than at the 

To test out-of-distribution generalization we performed the same experiment except sourced music recordings from the MUSDB18 dataset, which features 100 songs at sample rate of 44.1 kHz~\cite{musdb18}.
While performance does decrease slightly across all models, the objective metrics appear to generalize well both to audio at the higher sample rate and from a different data distribution, further strengthening the generalization capability enabled by our self-supervised training framework when combined with differentiable audio effects. 

\begin{figure*}[!t]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/style_transfer_p314.pdf}
    \vspace{-0.5cm}
    \caption{Visualization of estimated parameters with the ADDSP SPSA model when processing a single utterance from VCTK using varying style references from the five predefined styles.}
    \label{fig:vctk-variable-style}
\end{figure*}


\begin{figure*}[!t]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/style_transfer_p314_modify.pdf}
    \vspace{-0.5cm}
    \caption{Estimated parameters with the ADDSP SPSA model using a single broadcast style reference while the input VCTK utterance is progressively modified to increase the amount of high frequency energy and compression.}
    \label{fig:vctk-variable-input}
\end{figure*}

\subsection{Real-world style transfer}

To simulate a more realistic style transfer scenario where the style reference contains not only different content but also differing sources (e.g. speaker identity or instrumentation), we construct five different audio production ``styles'', which are defined by hand-crafted parameter ranges.
These five different styles along with their respective parameter ranges are shown in Table~\ref{tab:style-ranges}.
The ranges are selected such that each style is audibly unique.
In order to generate examples in each of these styles we use clean utterances from the DAPS dataset, which are all of similar quality and have minimal post-processing. 

However, we do not have paired groundtruth as in the synthetic style transfer case since the style references originate from a recording with different content from the input. 
To evaluate these examples we instead use only non-intrusive metrics that do not require a matching reference. 
These metrics measure the error between high level audio features, like the spectral centroid, RMS energy, and the perceptual loudness and are shown for 

One limitation of our method is that it is not possible to convert to certain styles if the input recording has been heavily processed, for example, it has a significant amount of compression already applied. 
Future work could investigate methods for handling this, for example adding a dynamic range expander to the signal chain,
However, our current formulation makes an assumption that most input recordings will not have already undergone any significant processing.

To visualize the parameter estimates from our model, we plot both the overall frequency response of parametric EQ as well as the compressor configuration when using an utterance from VCTK as input and applying style transfer using one utterance from each of the five different styles as shown in Figure~\ref{fig:vctk-variable-style}.
Since the utterances from VCTK are relatively clean and contain little post-processing we can observe the estimated parameters and determine if they align with our understanding of the predefined styles. 
For example, we find that the telephone style is achieved by reducing the high and low frequencies while boosting frequencies around 1\,kHz.
In terms of compression, we see a clear trend with the broadcast style using the most aggressive compression. 
Interestingly, in this case, the model applies compression for the warm style even though this style features minimal compression.
While the task of estimating these simplistic style parameters may seem trivial recall that the system was never trained on these utterances and has no knowledge of these pre-defined styles. 

As a demonstration of the ability of the model to adapt to the input signal, we then used a fixed style recording from our generated ``broadcast'' style, and instead varied characteristics of the input. 
In this case, the input signal was modified by progressively increasing the gain of a high-shelf filter from 0\,dB to +24\,dB, while also increasing the amount 
compression with a higher ratio and lower threshold.
As shown in Figure~\ref{fig:vctk-variable-input}, the estimated parameters adapt to these changes in an sensible manner. 
This results in a progressively lower boost in the high-shelf filter that was added originally to achieve the presence of the ``broadcast'' style, while also progressively increasing the threshold and decreasing the ratio to reduce the amount of compression applied, as the input becomes more compressed. 


\subsection{Encoder representations}

In addition to the style transfer evaluation, we also investigated the potential utility of the representations produced by the encoder. 
To do so, we created a downstream task of a audio production style classification where the goal is to classify the style of a recording among the 5 style we created for the real-world style evaluation. 
We trained linear logistic regression classifiers, also known as linear probes, on top of a set of different audio representations, a common approach in evaluating the quality of self-supervised representations~\cite{zhang2017split, asano2019critical}.
These linear probes are trained on a small dataset with only 60 examples from each of the 5 styles and class-wise F1 scores on the test set are reported in Table~\ref{tab:style-probes}.

As baseline representations we considered a random projection of mel features, along with large pre-trained encoder representations including OpenL3~\cite{cramer2019look}, which is focused on capturing audio content and CDPAM~\cite{manocha2021cdpam}, which is focused on capturing audio quality and is therefore the most similar to our approach.
In addition we trained linear probes on representations from the Proxy, SPSA, and Autodiff gradient methods on both speech style audio generated from DAPS recordings and music style audio generated from MUSDB18.

The class-wise scores indicated that while most representations were able to capture the ``bright'' style, differentiating among ``broadcast'' and ``neutral'' was most difficult due to the main different between these styles being the presence of more aggressive dynamic range compression. 
However, we found that our encoder representations did not struggle to capture this difference. 

Overall, is is clear that the representations produced by the our self-supervised training framework capture not only style elements related to spectral content, but also signal dynamics, which are not captured well by the other approaches. 
We found similar results between the speech and music style classification, with the appearance that music was slightly more difficult. 
We hypothesize is likely due to the starting diversity of style within MUSDB18 as compared to DAPS, which makes the style classification task more challenging. 



\begin{table}[ht]
    \centering
    \resizebox{0.95\hsize}{!}{%
    \begin{tabular}{l c c c c c c} \toprule
        \multicolumn{7}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Features} & Tele. &  Bright & Warm & Broadcast & Neutral & \textbf{Avg}\\  \midrule
        Random Mel          & 0.87 & 0.78 & 0.73 & 0.39 & 0.00 & 0.55\\
        OpenL3              & 0.19 & 0.61 & 0.08 & 0.10 & 0.18 & 0.23\\ 
        CDPAM               & 1.00 & 1.00 & 0.79 & 0.25 & 0.63 & 0.73\\\midrule
        L Proxy-H 1 S           & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00\\
        L SPSA                & 0.95 & 0.98 & 1.00 & 0.89 & 0.95 & 0.96\\
        L Autodiff            & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00\\ \midrule
        \multicolumn{7}{c}{\textbf{MUSDB18}} \\ \midrule
        \textbf{Features}   & Tele.&  Bright & Warm & Broadcast & Neutral & \textbf{Avg} \\  \midrule
        Random Mel          & 0.80 & 0.98 & 0.62 & 0.17 & 0.00 & 0.51 \\
        OpenL3              & 0.32 & 0.66 & 0.20 & 0.17 & 0.30 & 0.33 \\ 
        CDPAM               & 0.89 & 0.95 & 0.66 & 0.00 & 0.06 & 0.51 \\\midrule
        J Proxy-H 1 S           & 0.98 & 1.00 & 0.92 & 0.59 & 0.60 & 0.82 \\
        J SPSA                & 0.98 & 1.00 & 0.90 & 0.26 & 0.00 & 0.63 \\
        J Autodiff            & 0.98 & 1.00 & 0.95 & 0.54 & 0.50 & 0.79 \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Class-wise F1 scores for 5-class style prediction across classifiers trained on top of different audio representations for speech and music using a single linear layer. L denotes models trained on LibriTTS while J denotes models trained on the MTG-Jamendo dataset}
    \label{tab:style-probes}\    
    \vspace{-0.2cm}
\end{table}



\section{DISCUSSION}

Future work may consider combining different gradient methods for different effects simultaneously. 
For example, our results indicated that while neural proxies are proficient in modeling the dynamic range compressor, they struggle to model EQ effects. 
An approach that utilizes neural proxies to model the compressor along with autodiff parametric EQ may present one method to further improve performance. 

While our results indicate that neural proxy methods are able to learn some level of control, they do not perform as well as other approaches. 
Through further evaluation this appears to be due to a lack in accuracy between the real effects and the proxy networks, specially for the parametric EQ. 
We hypothesize that differences between the neural proxy and the real effect are the major reason for the lower performance we observed. 
However, this result is not necessarily an inherent limitation of neural proxy approach. 
Instead, it is likely a limitation of the architecture use to construct the neural proxies. 
While this family of architectures has been successful for modeling distortion, compression, and reverb effects, modeling the parametric EQ has proven more difficult, perhaps due to the range of processing that may not easily be parameterized by this architecture. 
Future work should consider using other, potentially more expressive network architecture for creating neural proxies. 

\subsection{Gradient methods}

\emph{still need to organize thoughts here on comparison of gradient methods.}

* strict neural network-based methods (TCN) might eventually be the most powerful if the architecture is large enough and there is enough training data, but lack of interpretable control means that these approaches cannot be easily adjusted by users. 
* gradient approximation-based methods are appealing from a product deployment perspective and don't require know the internal implementation of the effect itself, but limitations of the gradient estimation schemes can cause lack of performance or convergence issue for certain applications without more manual hyperparameter tuning effort
* Proxy-based methods that clone effects are reasonable, but have inference-time complexity issues that limit deployment. This approach also requires a multi-stage training process.
* Proxy hybrid-based methods appear to overcome the inference time limitations of proxy methods, but results are currently still lacking in terms of performance due to an apparent mismatch in training vs. testing setup. This motivates proxy hybrid method II, but we have tried this method yet.
* Autodiff-based methods require the most manual and demanding method to implement technically, but show promise in results, particularly when training data is low in resource and the benefit of expert knowledge can be exploited the most.


\section{CONCLUSIONS}
We demonstrated the application of differentiable audio effects in the construction of a automatic audio production system for style transfer.

\section{ACKNOWLEDGMENT}
This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1).


\section{INTRODUCTION}
\label{sec:intro}
Audio effects are commonly used in audio and video content creation to manipulate sound attributes such as loudness, timbre, spatialization, and dynamics~\cite{wilmering2020history}. 
While audio effects can be powerful tools in the hands of trained audio engineers, their complexity poses a significant challenge for less experienced users, often requiring a time-consuming adjustment process even for professionals. 
To simplify their usage, expert-designed presets can be used to configure an audio effect, but this only results in static content-independent configurations of effects.
Therefore, they commonly require fine-tuning and adjustment, without the ability to adapt to variations across input signals.

Automatic audio production methods aim to overcome this limitation with analysis and subsequent adaptation of effects based on an input signal~\cite{IMPbook19}. These methods can provide intelligent, automatic control of a single effect parameter, multiple effects, and/or many channels of audio that are dependent on each other.
Such methods may also provide a range of interfaces, from fully automatic systems to systems that only provide suggestions.
As a result, successful automatic audio production systems have the ability not only to simplify the process for amateurs, but also expedite the workflow of professionals~\cite{moffat2019approaches}.

Automatic audio production systems are commonly categorized into rule-based systems, built on audio engineering best-practices, and data-driven, machine learning systems~\cite{moffat2019approaches}.
While rule-based systems have seen success~\cite{deman2013b}, they are limited by the inability to formalize rules that cover the diversity arising in real-word audio engineering tasks~\cite{deman2017towards}.
Classical machine learning approaches on the other hand provide greater flexibility, but have thus far been limited by an inability to collect sufficient parametric data that describes audio production in a standardized way~\cite{scott2011analysis}. Recently, deep learning approaches have shown promise in overcoming these challenges, fueling an increasing interest in data-driven audio production techniques.

Deep learning approaches for automatic audio production can be organized into direct transformation methods, parameter prediction methods, and differentiable digital signal processing (DDSP) methods. 
Direct transformation methods map untreated audio to a desired target and have shown promise in tasks such as speech enhancement~\cite{pascual2017segan}, source separation~\cite{stoter2019open}, and audio effect modeling~\cite{hawley2019signaltrain, martinez2020deep,wright2020real,martinez2021deep}, but have had limited use for automatic production due to a lack of interpretable control, artifacts, and/or computational complexity~\cite{martinez2021deep}. 
Parameter prediction methods predict the settings of audio effects, but require expensive ground truth parameter data~\cite{hawley2019signaltrain} or optimize over an undesirable loss in the parameter domain~\cite{sheng2019feature, mimilakis2020one}. 
Lastly, DDSP methods~\cite{engel2020ddsp} use neural networks to control digital signal processors (DSP) that are implemented as a differentiable operators to enable training via backpropagation. 
This approach is desirable, but requires manual implementation of audio effects, expert knowledge, and high engineering effort, potentially limiting its use

Recent work on differentiable signal processing, however, has been significant and rapid.
Most notably for our work, two alternative approaches to implementing differentiable processors were proposed that remove the need to explicitly implement differentiable effects.  
In particular, neural proxies (NP) of audio effects~\cite{steinmetz2021automatic} and numerical gradient approximation schemes~\cite{ramirez2021differentiable} were proposed. 
Given that these approaches were proposed for different tasks and in different contexts, it is difficult to fully understand the benefits of each method.



Beyond integrating audio effects as differentiable operators, a central limitation of deep learning systems for automatic audio production lies in the difficulty of sourcing sufficient data for supervised training, requiring both the unprocessed and final produced recordings and/or parameter data. 
In addition, the subjective and context-dependent nature of the audio production process further complicates the task~\cite{lefford2021context}. 
While evidence suggests the existence of ``best-practices''~\cite{pestana2014intelligent}, learning these techniques in a supervised paradigm is challenging.
In particular, these systems must not only learn how to control effects to achieve a desired result, they must also implicitly uncover the distribution of possible styles present within the training data across different genres and contexts.


In this work, we present \emph{DeepAFx-ST} -- a style transfer method that can impose the audio effects and production style from one recording to another by controlling real or approximated audio effects to perform style manipulation as shown in \fref{fig:style-transfer-headline}. 
Using our approach, only a short example style recording is needed at inference for effects style transfer and no labeled training data or retraining is needed to adapt to new styles. 
To do so, we propose 
\begin{itemize}
    \item The first audio effects style transfer architecture to fully integrate audio effects as differentiable operators, optimized end-to-end with an audio-domain loss, 
    \item A simplified self-supervised training procedure that does not required label or paired training data,
    \item A categorization of existing differentiable signal processing approaches and the development of two novel neural proxy hybrid methods,
    \item An overview of a differentiable parametric equalizer and the  novel development of a differentiable dynamic range compressor and,
    \item An evaluation of our style transfer method and in-depth discussion of differentiable signal processing methods. 
\end{itemize}
We find our approach can perform transfer of arbitrary effect styles, enables intervention of control parameters, can operate at unseen sampling rates, learns production style. representations, and uses less compute resources compared to alternatives. 
A demonstration video and listening examples are  online\footnote{\url{https://csteinmetz1.github.io/deepafx-st}}.



\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/generic-style-transfer-headline.drawio.pdf}
    \vspace{-0.0cm}
    \caption{Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another by example. We use a shared-weight encoder to analyze the input and a style reference signal, then compare each with a controller that outputs the parameters of effects that themselves perform style manipulation.}
    \label{fig:style-transfer-headline}
\end{figure}

\begin{figure*}[t!]
    \centering
    \vspace{-0.2cm}
    \includegraphics[width=0.95\linewidth]{figures/training-lines.drawio.pdf}
    \vspace{-0.0cm}
    \caption{The self-supervised training process for our \emph{DeepAFx-ST} approach.
   
    We sample $x$ from a large dataset of recordings $\mathcal{D}$ and apply random augmentations.
    Recordings with two different styles are produced with randomly configured DSP, which become the input $x_i$ and the style reference $x_r$.
    We split these recordings in half, and pass the opposite sections ($x_{i,a}$ and $x_{r,b}$) to shared-weight encoders $f_\theta$. 
    Using the encoder representations ($e_i$ and $e_r$), the controller $g_\phi$ estimates parameters $p$ to apply the reference style to the input recording.}
    \label{fig:data-generation}
    \vspace{-0.2cm}
\end{figure*}

\section{PROPOSED METHOD}
\label{sec:method}
We seek to emulate the creative process of an audio engineer as they produce audio content. We decompose the creative process into three distinct sub-processes: acoustic analysis, style determination, and then parameter adjustment and application.

In the first step, an audio engineer listens to an input signal in order to establish an understanding of sonic characteristics they may wish to adjust.
Second, the engineer must then decide how they want to stylize the output sound -- a complex process based on the input signal and external context such as personal preference, and feedback from musicians, producers, expert listeners, etc.
Third, the engineer leverages their expertise and configures a set of audio effects to achieve their creative goal and style.  
Out of these steps, we argue that style determination is of critical importance and makes constructing data-driven automatic audio production systems difficult. It therefore motivates our proposed style transfer approach. 


\subsection{Architecture}



Our model architecture is designed to mimic the audio engineering process described above. It is shown on the right of \fref{fig:data-generation} and is composed of three submodules.



\linesubsec{Encoder}
Our encoder $f_\theta$ extracts audio production style information from both the input signal and the style reference signal. To do so, it inputs a time domain audio signal and converts it to a 2D time-frequency domain representation or spectrogram $X$ using a short-time Fourier transform (STFT) with window size $N$ and hop size $H$. 
The spectrogram is fed then to a standard convolutional neural network (e.g. ResNet, MobileNet, EfficientNet, etc), which outputs a time series of representations as a function of the model receptive field and number of input spectrogram frames. 
Temporal average pooling is applied to aggregate information across time frames, producing a single embedding of dimension $D$ for each input recording. 
We use a shared-weight encoder to process both the input recording and style reference recording to generate two representations or embeddings $e_i$ and $e_r$, which are concatenated and passed to the controller.


\linesubsec{Controller} Our controller $g_\phi$ takes the concatenated input and style embeddings ($e_i$ and $e_r$) and outputs audio effect parameters $p$.
To operate correctly, our controller must learn to compare the extracted audio production embeddings, and learn how to map the input style to the desired target style by adjusting the control parameters, similar to what an audio engineer must learn.
In our formulation, the controller is a simple three layer multi-layer perceptron (MLP) featuring a single hidden layer, along with PReLU~\cite{he2015delving} activation functions. 
At the output, the hidden representation is projected into the audio effect control parameter space $\mathbb{R}^P$, and a sigmoid activation is applied. 

\linesubsec{Differentiable audio effects}
For the final step, we use differentiable audio effects $h(x, p)$ to process the input audio signal, given the predicted parameters from the controller. 
Using real audio effects to perform style manipulation provides us with a strong inductive bias to reduce processing artifacts, and can significantly reduce computational complexity and/or implementation engineering requirements. 

In contrast to past work~\cite{mimilakis2020one, sheng2019feature} that avoids backpropagating through effects by using a parameter domain loss, we fully integrate audio effects as differentiable operators or layers with standard automatic differentiation tools and enable backpropagation through effects during training as shown with dashed lines in Figure~\ref{fig:data-generation}. 
We do this to facilitate end-to-end training using a loss in the audio domain and directly avoid difficulties in past works such as undesirable parameter quantization schemes, multi-stage optimization, and/or low correlation between parameter loss and audio manipulation. 
The task of computing gradients for audio effects, however, is nontrivial and discussed at length below.


\subsection{Self-supervised training}

Our self-supervised training strategy enables learning control of audio effects without the need for any (expensive) labeled or paired training data.

As shown on the left of Figure~\ref{fig:data-generation}, our data generation process begins by selecting a random audio recording $x$ from a dataset $\mathcal{D}$. 
We create an augmented version $x_{aug}$ by applying pitch shifting and time stretching, and
then produce two unique versions of this recording by applying a set of randomly configured DSP.
This yields the input recording $x_i$ in the top branch, and the style reference $x_r$ in the bottom branch, each with a different production style. 
Our model is then tasked with estimating parameters $p$ to configure the effects $h(x,p)$ so that the processed input signal matches the style reference.

To train our model to vary only with audio production style, 
we split the recordings $x_i$ and $x_r$ in half, generating an $a$ and $b$ section for each to produce $x_{i,a}$, $x_{i,b}$, $x_{r,a}$, and $x_{r,b}$.
During training, we randomly select either the $a$ or $b$ section to be passed as input, while the other section is used as the style reference.
Siince the sources present in the $a$ and $b$ sections are likely constant, we employ an assumption that the content (e.g. phonemes, instrumentation, etc.) within these sections differ. 
Previous works have employed a similar training strategy~\cite{mimilakis2020one, sheng2019feature}, however our approach is unique in that we consider a chain of multiple effects, avoid parameter quantization schemes, and use an audio-domain loss as discussed above.





\section{GRADIENT METHODS}
\label{sec:gradient}






The topic of DDSP is centered on the idea of combining the domain knowledge of signal processing with the gradient-based optimization framework of deep learning or backpropagation. In order to use backpropagation to optimize signal processing operators, however, all such operators on the automatic differentiation graph must have means to compute a gradient. 
We categorize existing approaches for computing gradients:
\begin{enumerate}
    \item\textbf{Automatic differentiation (exact or approx.)}~\cite{engel2020ddsp,kuznetsov2020differentiable, nercessian2020neural, colonel2021reverse}, where signal processing operations are exactly or approximately implemented within automatic differentiation frameworks (e.g. discrete Fourier transform, melspectrogram, finite impulse response filters, etc.)~\cite{abadi2016tensorflow, paszke2019pytorch}.
    \item \textbf{Approximate, alternative differentiation}, where the gradient of signal processing operations are not computed via automatic differentiation and are approximated by different means. Existing approaches include:
    \begin{enumerate}
        \item \textbf{Neural proxies}~\cite{steinmetz2021automatic}, where a neural network is trained to emulate signal processing operations, including parametric control, and is then connected with a network trained with the differentiable proxy.
        \item \textbf{Neural proxy hybrids}~\cite{tseng2019hyperparameter}, where a neural network is trained to emulate signal processing operations and then somehow combined combined together with the original implementation when used for training and/or inference.
        \item \textbf{Gradient approximation}~\cite{ramirez2021differentiable}, where effects are treated as a black-box and their gradients are estimated with numerical gradient approximation, such as finite differences (FD) or simultaneous perturbation stochastic approximation (SPSA)~\cite{spall1998overview}.
    \end{enumerate}
   
\end{enumerate}


\subsection{Automatic Differentiation}
\label{sec:AD}
The majority of work in differentiable signal processing has thus far focused on what we call exact or approximate automatic, DDSP. 
In this approach, the original signal processing building blocks are re-implemented either exactly or approximately using existing frameworks that support automatic differentiation, such as TensorFlow~\cite{abadi2016tensorflow} or PyTorch~\cite{paszke2019pytorch}.
Assuming the operations in question are differentiable, this provides a straightforward way to integrate signal processing operations within the automatic differentiation computation graph of neural networks, supporting end-to-end gradient-based optimization.

This approach, however, often requires expert knowledge and special design choices. In addition, the implementation of fully differentiable signal processors can be difficult, impossible, and/or computationally infeasible, motivating approximate versions or completely new signal processing algorithms. Such works include infinite impulse response (IIR) filters~\cite{kuznetsov2020differentiable,nercessian2020neural,colonel2021iirnet}, hearing aids~\cite{tu2021dhasp}, optimization of echo cancellation~\cite{casebeer2021auto}, automatic DJ transitions~\cite{chen2021automatic}, and reverse engineering of effects~\cite{kuznetsov2020differentiable, nercessian2021lightweight,colonel2021reverse}. 
Similarly, recent work constructing audio synthesis models with DDSP components include additive~\cite{engel2020ddsp}, subtractive~\cite{masuda2021synthesizer}, waveshaping~\cite{hayes2021neural}, and wavetable~\cite{shan2021differentiable} synthesizers. 

\subsection{Approximate, Alternative Differentiation}


\subsubsection{Neural proxies}
\label{sec:proxy}

In the neural proxy (NP) approach, a neural network is trained to emulate the behavior of the signal processing device of interest and then use a differentiable proxy for optimization~\cite{grathwohl2017backpropagation, jacovi2019neural}. 
This is generally achieved by producing a dataset containing measurements of the output of the system given a diverse range of inputs. 
In cases where the system features control parameters, these parameters can be randomly configured during the data generation process. 
Since the neural proxy is composed of neural network building blocks, it is fully differentiable and does not require any knowledge or re-implementation of the underlying signal processing operations.

In the context of audio, the neural proxy approach is largely inspired by the recent success in using neural networks to emulate analog audio effects~\cite{hawley2019signaltrain,  martinez2020deep,wright2020real,martinez2021deep, damskagg2019deep, martinez2019modeling, wright2019real, steinmetz2021efficient}. 
While these approaches have focused on emulating analog hardware, where datasets must be manually constructed through a laborious process, the neural proxy approach instead relies on constructing emulations of digital audio effects, which provide programmatic control to facilitate generating large training datasets.
This approach has been used to construct a channel of a mixing console featuring a parametric equalizer, compressor, and artificial reverberation with a neural network that learned control of these effects to create multitrack mixtures~\cite{steinmetz2021automatic}. 


\subsubsection{Neural proxy hybrids}
\label{sec:proxy_hybrids}
Assuming the neural proxy behaves similarity to the original effect, the gradients computed through this network may be treated as an approximation of the gradient for the real effect. 
Therefore, it may be possible to discard the neural proxy after training and use the original audio effects at inference, since gradients are no longer required.
This approach was first applied in computer vision, where neural proxies of complex image signal processors (ISP) were used to facilitate gradient-based optimization~\cite{tseng2019hyperparameter}. 
While their approach was successful, their use case did not involve using these neural proxies in conjunction with a neural network that provides dynamic control over the ISP. 
We name this approach the \emph{Neural Proxy Half Hybrid} (NP-HH) approach.
We use a neural network proxy that is trained identically as the original neural proxy method, but during inference, the pre-trained proxy network is replaced with the original digital signal processing (DSP) implementation of the cloned audio effects. 

We also propose a second variation called the \emph{Neural Proxy Full Hybrid} (NP-FH) approach, which features a more involved integration of the neural proxy and DSP implementation. 
In this case, we use the DSP implementation both during the forward pass and after training, using the neural proxy only during the backward pass for gradient computation. 
The idea of this approach is to minimize the chance of approximation error between the neural proxy and DSP implementation during training, while facilitating backpropagation through the use of the neural proxy only for gradient computation. 

\subsubsection{Numerical gradient approximation}
\label{sec:grad}
Numerical approximation of gradients using black-box optimization methods provides another route to enable gradient-based learning for our task.
For example, finite differences (FD)~\cite{milne2000calculus} provides an estimate of the gradient by perturbation of the function inputs.
In the formulation of learning to control a signal processing device $h(x, p)$ with input audio $x$ and parameters $p \in \mathbb{R}^{P}$, we only need to estimate the partial derivatives for each parameter $\frac{\partial}{\partial p_i} h(x,p)$ to update the weights of the controller network $g_\phi$. While FD requires $2P$ function evaluations, a similar approach called simultaneous permutation stochastic approximation (SPSA)~\cite{spall1998overview} can be used that only requires two measurements per gradient. 
The estimate of the partial with respect to $p_i$ from the SPSA gradient estimator $\hat{h}$ is given by 
\begin{equation}
\frac{\hat{h}(x, p_i)}{p_i} = \frac{h(x, p + \epsilon \Delta^P) - h(x, p - \epsilon  \Delta^P)}{2 \epsilon \Delta^{P}_{i}}
\end{equation}
where $\epsilon$ is some small, non-zero value and $\Delta^P \in \mathbb{R}^P$ is a random perturbation vector sampled from a symmetric Bernoulli distribution ($\Delta^P_i = \pm 1$)~\cite{spall1992multivariate}. 

In the context of audio, SPSA approximated gradients were used within an automatic differentiation framework and used to train a neural network controller for a number of audio production tasks, including audio effect modeling, removal of breaths, and mastering~\cite{ramirez2021differentiable}. 
This approach does not require a pre-training stage, and additionally does not require any knowledge of the underlying signal processing operations.
However, in practice, optimization may be more challenging due to inaccuracies in the gradient approximations, and potentially slower due to the use of signal processing devices that are often implemented on CPU. 











\section{AN ILLUSTRATION OF AUTOMATIC DDSP}

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth, trim={0.0cm 0.18cm 0.0cm 0.1cm},clip ]{figures/signal-chain.drawio.pdf}
    \caption{Audio effect signal chain consisting of a 6-band parametric equalizer connected in series with a dynamic range compressor. The effect parameters for each device are shown below.}
    \label{fig:signal-chain}
    \vspace{-0.4cm}
\end{figure}

To illustrate the complexity of implementing automatic DDSP audio effects and benchmark how alternative gradient methods compare, we developed fully differentiable versions of two of the most common audio effects used in audio engineering -- a parametric equalizer and dynamic range compressor as shown in \fref{fig:signal-chain}. 









\subsection{Parametric equalizer}\label{sec:eq} 
The parametric equalizer is generally constructed with a cascade of second order IIR filters, also know as biquads~\cite{valimaki2016all}. 
For our design, we consider a six-band equalizer, which features a low-shelf, high-shelf, and four fully parametric peaking bands. 
It has been demonstrated that both the coefficient computation and filter application are composed of differentiable operations~\cite{nercessian2020neural}, enabling implementation of this effect following traditional designs with an exact, automatic DDSP gradient method.

However, this implementation will result in issues arising during backpropagation due to the recursive nature of IIR filters, requiring backpropagation through time (BPTT) across many time steps. 
While this may pose challenges in optimization due to vanishing gradients~\cite{hochreiter1997long}, the more significant limitation manifests as a computational bottleneck due to the overhead in computing these gradients in series over long time steps.
To address this, we instead apply the filter in the frequency domain using a feedforward approximation (FIR filter), an approach used in previous work~\cite{nercessian2020neural, nercessian2021lightweight, colonel2021iirnet} and an approximate, automatic DDSP gradient method.

We can express the complex frequency response of the $k$th biquad $H_k(e^{j \omega})$ in the parametric equalizer as a ratio of the DFT of the numerator and denominator coefficients, $\mathbf{b}_k = [b_{0,k}, b_{1,k}, b_{2,k}]$ and $\mathbf{a}_k = [a_{0,k}, a_{1,k}, a_{2,k}]$,
\begin{equation}
    H_k(e^{j \omega}) = \frac{ \textrm{DFT}(\mathbf{b}_k) }{ \textrm{DFT}(\mathbf{a}_k) } = \frac{\sum_{m=0}^{2} \ b_{m,k} \ e^{-j\omega m}}{\sum_{n=0}^{2} \ b_{n,k} \ e^{-j \omega n}}.
\end{equation}
Recall that the overall response of a cascade of $K$ biquad filters is given by the product of their responses $H_{sys} (e^{j \omega}) = \prod_{k=1}^{K} H_k(e^{j \omega})$.
We can then approximate the equalizer response using an FIR filter by evaluating $H_{sys}(e^{j \omega})$ over a linearly spaced set of digital frequencies $\omega$. 
We can apply this filter to an input $x[n]$ with $N$ samples in the frequency domain by multiplying the response of the system with the response of the input $ X(e^{j \omega})$, and then perform the inverse DFT to get the output $y[n]$.
\begin{equation}\label{eq:approx}
    y[n] = \textrm{IDFT}^{-1} \Big( X(e^{j \omega}) \cdot H_{sys}(e^{j \omega}) \Big)
\end{equation}
In practice, we evaluate over $F$ frequencies by zero-padding the input $x[n]$ and the filter coefficients to the nearest power of 2 given by $F = 2^{\lceil \log_2 (2N - 1) \rceil}$.
The use of fast Fourier transform (FFT) enables this operation to be heavily parallelized and run efficiently using GPU resources.

\subsection{Dynamic range compressor}
To our knowledge there has been no work on the design of an exact DDSP dynamic range compressor.
Inspection of the design of the traditional feedforward dynamic range compressor reveals it is in fact composed of differentiable operations~\cite{giannoulis2012digital} .
However, similar to the parametric equalizer, we found direct implementation of existing compressor designs problematic due to the presence of recursive structures, in this case due to the recursive smoothing filters for the ballistics,
\begin{equation} \label{eq:smoothing}
    \resizebox{.9\hsize}{!}{
   $y_L[n] =
    \begin{cases}
        \alpha_A y_L[n-1] + (1-\alpha_A) x_L[n] & x_L[n] > y_L[n-1] \\
        \alpha_R y_L[n-1] + (1-\alpha_R) x_L[n] & x_L[n] \leq y_L[n-1]
    \end{cases}$,       
    }
\end{equation}
where $y_L[n]$ is the smoothed gain reduction, $x_L[n] = x_G[n] - y_G[n]$ is the result of subtracting the output of the gain computer from the signal envelope, and $\alpha_A$ and $\alpha_R$ are the attack and release time constants. 

Since this operation involves a set of recursive filters where the activity of each filter is itself based on the recursion, as shown in (\ref{eq:smoothing}), this behavior cannot be easily approximated with a FIR filter. 
However, we found using a single attack and release time constant $\alpha_{AR}$ to create a single-pole IIR filter given by,
\begin{equation}
   y_L[n] = \alpha_{AR} y_L[n-1] + (1-\alpha_{AR}) x_L[n],
\end{equation}
produced a compressor-like effect that could be approximated with an FIR filter using the same approach as in (\ref{eq:approx}).
While this modification does restrict the capabilities of the compressor, enforcing the same time constants for both the attack and release of the device, we found this formulation produced something close enough for our purposes to a practical compressor effect.


\section{EXPERIMENTAL DESIGN}

\subsection{Datasets}
\label{sec:datasets}

We employ a self-supervised data generation pipeline that produces paired training examples for the style transfer task, as described in \sref{sec:method}.
Unlike speech enhancement that generally requires ``clean'' recordings, which will then be corrupted, our approach can utilize any audio dataset, preferably one that covers a wide range of quality levels and recording conditions. 
We train two sets of models, using a number of gradient methods, performing the style transfer task for either speech or music. 
Speech recordings are sourced from LibriTTS~\cite{zen2019libritts}, and we use recordings from the \texttt{train-clean-360} subset, which includes 360 hours of audio at $f_s = 24$\,kHz.
We use music recordings from the MTG-Jamendo dataset~\cite{bogdanov2019mtg}, which provides over 55\,k songs, which we downmix to mono and resample to $f_s = 24$\,kHz. 
For both datasets, we generate a $90$/$5$/$5$ split of the recordings for training, validation, and testing.

During the sampling of random segments, if the selected segment $x[n]$ has a mean energy below the threshold, $\frac{1}{N} \sum_{n=1}^{N} | x[n] |^2 \leq 0.001$), we continue sampling random patches from the selected recording until a non-silent segment is found.
To ensure that there is sufficient headroom for processing within the audio effect chain, we peak normalize all inputs and style recordings to $-12$\,dBFS.
These examples are then split in half to generate the $a$ and $b$ sections for the input and reference recordings.

\subsection{Baselines}
\label{sec:baselines}


\linesubsec{Rule-based DSP}
\label{sec:dsp}
The rule-based approach consists of an automatic, two stage process that aims to mimic the signal chain of the parametric equalizer and compressor. 
First, an FIR filtering stage is employed, which aims to transform the spectral content of the input speech to match the spectrum of the style reference. 
We compute the average magnitude spectrum of the input and style reference by computing the STFT using a large window of $N=65536$ and a hop size of $H=16384$, averaging across the frames, yielding $X_i$ and $X_r$.
$X_i$ and $X_r$ are then smoothed using a Savitzky-Golay filter~\cite{savitzky1964smoothing}.
We design an inverse FIR filter with $63$ taps using the window method, with a desired magnitude response given by $X = X_{i,sm} / X_{r,sm}$, where $X_{i,sm}$ and $X_{r,sm}$ are the smoothed average spectra of the input and the style reference.

The filtered signal is sent to a dynamic range compressor where all parameters except for the threshold are fixed. 
In order to adjust the amount of compression based on the input signal and the style reference, the threshold starts at 0 dB and is lowered in increments of 0.5 dB.
At each iteration the loudness in dB LUFS~\cite{steinmetz2021pyloudnorm} of the output is measured and compared to the loudness of the style reference.
If the difference in the loudness measurements are below a given threshold (e.g 0.5 dB LUFS), then the process halts. 
Otherwise, the process continues until the loudness measurements are within the specific tolerance, or the threshold reaches -80 dB.
While this approach only adjusts the threshold, we found in practice it was successful in matching the overall amount of compression. 

\linesubsec{End-to-end neural network}
\label{sec:e2e}
To create a baseline on the opposite end of the spectrum, we also evaluate an end-to-end (E2E) neural network approach to replace the DDSP audio effects. 
This network utilizes the same architecture as the neural proxy approach, but instead does not use pre-trained weights. 
Instead, the weights of this model are optimized during the training of the encoder and controller models. 
Even though the controller still predicts parameters for this network, as in the case of the neural proxy, since the weights of the network are not frozen during training, the model will learn its own set of control parameters. 
We evaluate two variants of this approach, first using a single TCN network (cTCN 1) and a second using two TCNs connected in series (cTCN 2), which mimics the setup in the neural proxy approach.
While this method provides the most expressive processing out of the methods we evaluate, it imparts a high compute cost at both training and inference, in addition to potential for artifacts, and uninterpretable controls.






\subsection{Losses}

In training both the proxy and style transfer models, the loss was computed directly on the audio output, using a weighted sum of the error in time and frequency domains.
The loss in the time domain $\mathcal{L}_{\text{time}}$ is given by the mean absolute error (MAE) and the
the loss in the frequency domain $\mathcal{L}_{\text{freq}}$ is given by the multi-resolution short-time Fourier transform loss (MR-STFT)~\cite{wang2019neural, steinmetz2020auraloss}.
This loss is the sum of the $L_1$ distance between the STFT of the groundtruth and estimated waveforms measured both in the $\log$ and linear domains, at multiple ressolutions, in this case with window sizes $W \in [32, 128, 512, 2048, 8192, 32768]$ and hop sizes $H = W/2$.
The overall loss function is simply the sum of these terms, with a weighting $\alpha = 100$
\begin{equation}
     \mathcal{L}_{\text{overall}}({\hat{y}},y) = \mathcal{L}_{\text{freq}}({\hat{y}},y) + \alpha \cdot \mathcal{L}_{\text{time}}({\hat{y}},y) \label{eq:overall-loss}.
\end{equation}

\setlength{\tabcolsep}{3.0pt}
\setlength{\abovecaptionskip}{15pt plus 3pt minus 2pt}
\begin{table*}[t!]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
        \multicolumn{21}{c}{\normalsize{\textbf{Speech}}} \\ \midrule
        & \multicolumn{6}{c}{\textbf{LibriTTS}} & & \multicolumn{6}{c}{\textbf{DAPS}} & & \multicolumn{6}{c}{\textbf{VCTK}} \\ \cmidrule(lr){2-7} \cmidrule(lr){8-14} \cmidrule(lr){15-21}
        \textbf{Method}  &  \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}}\\ \midrule
        {Input}   
        & 3.765           & 1.187           & 2.180           & 687.5           & 6.983           & 2.426           & 
        & 3.684           & 1.179           & 2.151           & 641.7            & 6.900          & 2.314          & 
        & 3.672           & 1.254           &          2.008  & 815.4           & 7.783           & 2.532\\
        {RB-DSP}  
        & 3.856           & 0.943           & 1.955           & 410.3           & 4.204           & 1.674           & 
        & 3.787           & 0.917           & 1.882           & 399.7            & 3.705           & 1.481          & 
        & 3.709           & 1.101           &          1.911  & 657.6           & 5.039           & 2.018\\ \midrule
        \textbf{End-to-end} \\ \midrule
        {cTCN 1}    
        & 4.258           & 0.405           & 0.887           & 128.4           & 2.237           & 1.066           & 
        & 4.185           & 0.419           & 0.884           & 124.6            & 2.098           & 1.006          & 
        & 4.181           & 0.467           &          0.891  & 173.8           & 2.651           & 1.165\\
        {cTCN 2}    
        & 4.281           & 0.372           & \textbf{0.833}  & 117.5           & 1.927           & 0.925           & 
        & \textbf{4.224}  & \textbf{0.391}  & \textbf{0.841}  & 113.9            & 1.886           & 0.913          & 
        & 4.201           & \textbf{0.441}  & \textbf{0.856}  & 163.8           & 2.431           & 1.086\\ \midrule
        \textbf{Approx. DDSP} \\ \midrule
        {NP}   
        & 3.643           & 0.676           & 1.405           & 265.0           & 2.812           & 1.340           & 
        & 3.605           & 0.685           & 1.362           & 249.2            & 2.732           & 1.350          & 
        & 3.651           & 0.737           &          1.300  & 321.7           & 3.166           & 1.453\\
        {NP-HH}   
        & 3.999           & 1.038           & 2.179           & 440.2           & 5.472           & 2.679           & 
        & 3.903           & 1.022           & 2.113           & 451.9            & 5.104           & 2.535          & 
        & 3.951           & 1.044           &          1.930  & 591.5           & 5.194           & 2.651\\
        {NP-FH}   
        & 3.945           & 1.058           & 2.088           & 404.9           & 6.820           & 3.197           & 
        & 3.891           & 1.037           & 2.045           & 395.4            & 6.754           & 3.117          & 
        & 3.894           & 1.087           &          1.934  & 514.0           & 7.065           & 3.363\\
       
        {SPSA}      
        & 4.180           & 0.635           & 1.406           & 219.5           & 3.263           & 1.600           & 
        & 4.099           & 0.645           & 1.379           & 213.6            & 2.989           & 1.511          & 
        & 4.023           & 0.730           &          1.359  & 301.6           & 3.535           & 1.737\\ \midrule
        \textbf{Exact DDSP} \\ \midrule
        {AD}  
        & \textbf{4.310}  & \textbf{0.388}  & 0.882           & \textbf{111.5}  & \textbf{1.828}  & \textbf{0.823}  & 
        & 4.222           & 0.416           & 0.895           & \textbf{109.0}   & \textbf{1.758}  & \textbf{0.799} & 
        & \textbf{4.218}  & 0.481           &          0.924  & \textbf{152.7}  & \textbf{2.317}  & \textbf{1.006} \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
\caption{Synthetic style transfer with models trained using LibriTTS. Held-out speakers from the LibriTTS dataset are used, while utterances from DAPS and VCTK come from datasets never seen during training. Lower is better for all metrics except PESQ.}
    \vspace{-0.1cm}
    \label{tab:style-transfer-synthetic-speech}
\end{table*}

\subsection{Neural proxy pre-training}

To construct the neural proxies, we utilized the temporal convolutional network (TCN)~\cite{bai2018empirical}, also known as the feedforward WaveNet~\cite{rethage2018wavenet}, which has seen success in modeling distortion~\cite{wright2019real} and compressors~\cite{steinmetz2021automatic}.
We based the architecture on the causal TCN~\cite{steinmetz2021automatic}, which uses feature-wise linear modulation (FiLM)~\cite{perez2018film} to adapt to the effect control parameters.
This TCN configuration has $4$ blocks, kernel size of $13$, channel width of $64$ and a dilation factor that grows as a power of $8$, where the dilation factor at block $b$ is $d_b = [ 1, 8, 64, 512 ]$.
We trained a total of 4 proxy models, one for each of the two effects, with one model for both speech and music. 
Training examples were generated by processing audio using audio effects, sampling the random parameter values from a uniform distribution across their range. 
These models were trained for 400 epochs with an initial learning rate of $3\cdot 10^{-4}$ and a batch size of 16, using inputs $\approx 3$\,s ($65536$ samples) in length. 
One epoch was defined as 20,000 random segments and the learning rate was lowered by a factor of 10 after the validation loss had not improved for 10 epochs. 

\subsection{Style transfer training}

After training the neural proxy models, we proceeded to train all models on the style transfer task. 
In our formulation, we used an input representation obtained with an STFT with a hann window of size $N=4096$ and a hop size of $H=2048$.
To construct the encoder, we use a randomly initialized EfficientNet~\cite{tan2019efficientnet} with a single input channel. 
The final activations of the network are projections to $D=1024$ with a linear layer, followed by mean pooling across the temporal dimension to produce a single representation, which is normalized by the $L_2$ norm.

All model configurations were trained for a total of 400 epochs with a batch size of 6, where a single epoch is defined as 20,000 segments of audio $\approx 10$ seconds in length ($262144$ samples) randomly sampled from the training set. 
All models are trained with a learning rate of $10^{-4}$ with the exception of the SPSA models, which we found required a lower learning rate of $10^{-5}$ to ensure stability. 
The learning rate was scheduled such that the learning rate was lowered by a factor of 10 at both $80\%$ and $95\%$ through training.
We applied gradient clipping with a value of $4.0$
We used a fixed value of $\epsilon = 0.0005$ for the SPSA hyperparameter, since larger values lead to instability later on in training.

\subsection{Metrics} 
Since there are no established metrics for evaluating audio production style, we considered a range of metrics that aim to capture different aspects. 
For a general notion of perceptual similarity we used PESQ~\cite{rix2001perceptual}, along with the Multi-resolution STFT~\cite{wang2019neural} error, which is a component of the training objective. 
To evaluate the ability of the system to address equalizer-related style components, we reported the mel-spectral distance (MSD), which consists of the error between melspectrograms using a large window size of $W=65536$, along with the spectral centroid error (SCE). 
To evaluate aspects related to the dynamic range, we considered the RMS energy error (RMS) and the perceptual loudness error (LUFS)~\cite{steinmetz2021pyloudnorm}. 



\setlength{\tabcolsep}{2.5pt}
\begin{table*}[ht]
    \centering
    \resizebox{\linewidth{}}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
    \multicolumn{21}{c}{\normalsize{\textbf{Music}}} \\ \midrule
        &  
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (24 kHz)}}   & & 
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (44.1 kHz)}} & & 
        \multicolumn{6}{c}{\textbf{MUSDB18 (44.1 kHz)}} \\ 
        \cmidrule(lr){2-7} \cmidrule(lr){9-14} \cmidrule(lr){15-21}
        \textbf{Method}   & 
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        Input      
        & 2.927  & 1.198  & 6.088  & 646.5  & 6.695  & 2.518 & 
        & 2.874  & 1.109  & 4.454  & 767.7  & 6.793  & inf   & 
        & 2.900  & 1.252  & 4.342  & 1088.3    & 5.940  & 2.312    \\ 
        RB-DSP     
        & 2.849  & 0.925  & 4.422  & 263.2  & 4.254  & 1.706  & 
        & 2.931  & 0.887  & 3.355  & 342.0   & 4.749  & 1.882 & 
        & 2.994  & 0.821  & 3.052  & 379.4   & 4.078 & 1.665  \\ \midrule
        \textbf{End-to-end} \\ \midrule
        cTCN 1       
        & 3.402  & 0.547  & 2.294  & 160.9  & 3.261  & 1.483 &
        & 3.168  & 0.876  & 2.921  & 494.4    & 4.372  & 2.070  & 
        & 3.121  & 0.896  & 2.956  & 730.4    & 4.548  & 2.231  \\ 
        cTCN 2       
        & 3.390  & 0.548  & 2.278  & 152.3  & 2.951  & 1.397 & 
        &  3.123  & 0.903  & 2.973  & 517.4   & 4.084  & 1.899  &
        &  3.107  & 0.917  & 2.986  & 749.5   & 4.208  & 2.061 \\  \midrule
        \textbf{Approx. DDSP} \\ \midrule
        NP S  
        & 2.926  & 0.787  & 2.838  & 221.3  & 2.785  & 1.390 &  
        & 2.764  & 1.033  & 2.869  & 488.3  & 3.710  & 1.824 & 
        & 2.804  & 0.950  & 2.778  & 742.1   & 3.835  & 1.921  \\
        NP M  
        & 2.765  & 0.845  & 3.211  & 255.2  & 3.227  & 1.608 & 
        & 2.699  & 1.042  & 2.928  & 497.0    & 3.942  & 1.961   & 
        & 2.791  & 0.946  & 2.800  & 757.1   & 4.209  & 2.127    \\
        NP-HH (S)  
        & 2.819  & 1.092  & 6.791  & 395.3  & 6.276  & 3.032 & 
        & 2.865  & 1.101  & 5.194  & 689.2    & 6.792  & 3.365 & 
        & 2.853  & 1.165  & 4.852  & 1005.7   & 6.451  & 3.269 \\
        NP-HH (M)  
        & 2.532  & 1.166  & 7.070  & 591.9  & 5.660  & 2.593 & 
        & 2.512  & 1.148  & 5.385  & 854.9   & 5.940  & 2.740 & 
        & 2.493  & 1.198  & 5.090  & 1021.9    & 5.585  & 2.731 \\
        NP-FH (S)  
        & 2.833  & 1.016  & 5.005  & 280.8  & 3.377  & 1.634 & 
        & 2.888  & 0.977  & 3.883  & 429.9    & 3.480  & 1.709   & 
        & 2.857  & 1.045  & 3.809  & 617.3   & 3.932  & 1.971   \\
        NP-FH (M)  
        & 2.648  & 1.137  & 6.368  & 605.6  & 5.903  & 2.587 & 
        & 2.625  & 1.133  & 5.512  & 844.6   & 6.417  & 2.876  & 
        & 2.575  & 1.225  & 5.450  & 1172.5   & 5.973  & 2.913   \\
        SPSA         
        & 3.173  & 0.716  & 3.024  & 210.1  & 2.809  & 1.344 & 
        & 3.172  & 0.759  & 2.458  & 386.2   & 2.839  & 1.311 & 
        &  3.126  & 0.789  & 2.321  & 574.4   & 2.925  & 1.394    \\ \midrule
        \textbf{Exact DDSP} \\ \midrule
        AD     
        & 3.355  & 0.488  & 2.149  & 144.7  & 2.167  & 1.005   & 
        &  3.400  & 0.585  & 1.824  & 304.4    & 2.425  & 1.106 & 
        &  3.396  & 0.608  & 1.695  & 456.1    & 2.559  & 1.197  \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Synthetic style transfer with models trained using MTG-Jamendo dataset. Proxy models with (S) were pre-trained using speech from LibriTTS those with (M) were trained using music from the MTG-Jamendo dataset. Lower is better for all metrics except PESQ.}
    \label{tab:style-transfer-synthetic-music}
    \vspace{-0.0cm}
\end{table*}



\section{RESULTS}
\label{sec:results}



\subsection{Synthetic style transfer}

We first evaluate models trained with speech from LibriTTS on the synthetic style transfer task, which mirrors the training setup. 
This facilitates the use of full-reference metrics, since we synthetically generate style references and as a result have inputs and groundtruth with matching content.
Results of the evaluation using held-out speakers from LibriTTS are shown in \tref{tab:style-transfer-synthetic-speech}.

We note the rule-based DSP baseline approach improves on the input across all metrics, though not by a significant margin, indicating it has limited success in the style transfer task.
Overall, we find strong performance from the AD method, which performs best on all metrics except MSD, which is only slightly outperformed by cTCN2.
Both cTCN approaches follow closely behind the AD method, demonstrating the strength of end-to-end neural network approaches. However, these models do not feature interpretable control parameters. 
The SPSA model follows closely behind these approaches with somewhat degraded PESQ and STFT error, yet it outperforms all NP methods. 
While the NP methods are not as competitive, we observe that both hybrid methods, NP-HH and NP-FH, outperform the original NP in terms of PESQ, likely a result of the use of the DSP implementation, which imparts less artifacts than the neural proxy.
However, the NP approach achieves better performance across all other metrics, indicating it likely better matches the style reference. 
Surprisingly, the AD approach achieves the best overall performance even though it features no learnable parameters and utilizes a simplified dynamic range compressor design. 

In addition to evaluating on held-out speech from LibriTTS, we also evaluate generalization using utterances from speech datasets not seen during training including DAPS~\cite{mysore2014can} and VCTK~\cite{yamagishi2019vctk}, which were resampled to 24\,kHz.
We observe a similar trend in performance across all methods, yet the cTCN models slightly outperform AD as compared to on LibriTTS.
Overall, this indicates that training on the large scale LibriTTS dataset with our self-supervised data generation pipeline was sufficient for producing models that generalize to speech originating from different sources with different acoustic conditions. 

We also trained the same set of models using music from the MTG-Jamendo dataset. 
The result. shown in \tref{tab:style-transfer-synthetic-music}, largely follows a similar trend to the speech experiments with regards to the gradient methods, demonstrating the general nature of our approach. 
However, it appears that the music task is somewhat more difficult.
We also compared neural proxies trained on speech (S) or music (M), and found those trained on music performed slightly worse.

We then investigated the ability of the methods to operate at a sample rate different than training. 
Signals to the encoder were first resampled to match training (24\,kHz), however the audio effects operate at 44.1\,kHz.
First, we evaluated on the MTG-Jamendo test set, and we noticed a significant drop in performance for the neural network based approaches, which include the cTCNs and NP, since these models operate at a fixed sample rate. 
However, the DSP-based approaches performed significantly better, including both the AD and SPSA approaches, which performed only slightly worse than at the training sample rate.
To test out-of-distribution generalization we used music from MUSDB18~\cite{musdb18}.
While performance does decrease slightly across most metrics and for all models, they do not indicate that the models fails to perform the style transfer task, with the SPSA and AD approaches far exceeding the baseline. 


\subsection{Realistic style transfer}
\setlength{\tabcolsep}{3.0pt}
\begin{table}[t!]
    \centering
    \resizebox{.75\hsize}{!}{%
    \begin{tabular}{l l c c c c } \toprule
        \multicolumn{6}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Method} & $\mathcal{D}$ & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Input} &  -    & 10.398        & 1041.654        & 10.414          & 3.649    \\ 
        {RB-DSP}  &  -  & 8.892         & 517.179         & 5.842           & 2.423    \\  \midrule
        {NP-HH S} & L    & 9.770         & 636.605         & 12.922          & 5.878            \\
        {SPSA}  & L        & 7.972         & 360.073         & \textbf{5.121}           & 2.511           \\
        {AD}   & L    & \textbf{7.756}& \textbf{278.112}& 5.239  & \textbf{2.368}            \\ \midrule
        \multicolumn{6}{c}{\textbf{MUSDB18}}  \\ \midrule
        \textbf{Method} & $\mathcal{D}$ & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Input} &   -    & 8.420         & 2607.409      & 9.356        & 3.795    \\ 
        {RB-DSP}  &  -  & 6.534         & \textbf{915.390}       & 8.647         & 3.662    \\  \midrule
        {NP-HH S} & J    & 6.964         & 1512.650      & 10.019        & 4.472            \\
        {SPSA}   & J      & 5.535         & 1297.033      & 4.578         & 2.073          \\
        {AD}  & J     & \textbf{4.820}& 947.609       & \textbf{3.776}& \textbf{1.732}            \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.3cm}
    \caption{Average performance across all pairwise style transfer configurations using input utterances from DAPS and music recordings from MUSDB18. L denotes models trained on LibriTTS while J denotes music from the MTG-Jamendo dataset.}
    \vspace{-0.0cm}
    \label{tab:style-transfer-real}
\end{table}

To simulate a more realistic style transfer scenario where the style reference contains not only different content but also differing sources (e.g. speaker identity or instrumentation), we construct a realistic style transfer evaluation task.
We designed five different audio production ``styles'', which are defined by hand-crafted parameter ranges.
We generate audio from five different styles (Telephone, Warm, Bright, Neutral, and Broadcast) by randomly sampling parametric equalizer and compressor parameters from a set of predefined ranges such that each style is audibly unique.
Sampling parameter values randomly provides more variation within audio examples of the same style, which is more complex than applying a fixed preset. 

To generate examples in each of these styles, we use clean utterances from the DAPS dataset to generate speech examples, and we use the same style definitions to generate music examples using songs from MUSDB18.
However, since we do not have paired groundtruth as in the synthetic style transfer case, we performed our evaluation of this task using only non-intrusive metrics that do not require a matching reference. 
These metrics measure the error between high level audio features, like the spectral centroid, RMS energy, and the perceptual loudness.

We performed all pairwise style transfers resulting in 25 configurations, and we performed each configuration ten times using different input and output recordings from each style. 
Here we considered only the best performing methods that produce interpretable control parameters (NP-HH, SPSA, and AD).
To summarize the results of this evaluation, we aggregated the metrics across all 25 configurations reporting the mean for each to provide a general notion of the style transfer performance, as shown in \tref{tab:style-transfer-real}. 

On examples from DAPS, all methods, including the baseline, improve upon the input, but SPSA and AD clearly outperform the other approaches across all metrics, with AD preforming the best. 
In the case of MUSDB18, the AD approach further outperforms the others, with the exception of SCE, where the RB-DSP baseline performs best.



\begin{figure}[!t]
    \centering
    \includegraphics[width=0.9\linewidth, trim={0.4cm 0.0cm 0.15cm 0},clip ]{figures/style_transfer_p314.pdf}
    \vspace{-0.3cm}
    \caption{Estimated parametric equalizer (top) and compressor (bottom) parameters SPSA model with an utterance from VCTK using varying style references from the five predefined styles.}
    \label{fig:vctk-variable-style}
        \vspace{-0.1cm}
\end{figure}

\subsection{Parameter visualization}

To demonstrate the ability or our approach to provided precise control over the effect parameters, we plot both the overall frequency response of the parametric equalizer as well as the compressor configuration using an utterance from VCTK as input and applying style transfer using one utterance from each of the five different styles as shown in Figure~\ref{fig:vctk-variable-style}.
Since the utterances from VCTK are relatively clean and contain little post-processing we observed the estimated parameters aligned with our expectation for the predefined styles. 
For example, we find that the telephone style is achieved by reducing the high and low frequencies while boosting frequencies around 1\,kHz.
In terms of compression, we see a clear trend with the broadcast style using the most aggressive compression, as expected. 
While the task of estimating these style parameters may seem trivial, recall that the system was never trained on these utterances and has no knowledge of our pre-defined styles. 

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.9\linewidth, trim={0.4cm 0.2cm 0.15cm 0},clip ]{figures/style_transfer_p314_modify.pdf}
    \vspace{-0.3cm}
    \caption{Estimated parametric equalizer (top) and compressor (bottom) parameters with the SPSA model using a fixed Broadcast style reference. The input VCTK utterance was progressively modified to increase the high frequency energy and compression.}
        \vspace{-0.0cm}
    \label{fig:vctk-variable-input}
\end{figure}

As a demonstration of the ability of the model to adapt to the input signal like an \emph{adaptive preset}, we used a single style recording from our generated Broadcast style, and varied characteristics of the input. 
In this case, the input signal was modified by progressively increasing the gain of a high-shelf filter from 0\,dB to +24\,dB, while also increasing the amount of compression with a higher ratio (1.0 to 4.0) and lower threshold (0\,dB to -62\,dB).
As shown in Figure~\ref{fig:vctk-variable-input}, the estimated parameters adapt to these changes in an sensible manner. 
This results in a progressively lower boost in the high-shelf filter that was added originally to achieve the presence of the Broadcast style, while also progressively increasing the threshold and decreasing the ratio to reduce the amount of compression applied, as the input becomes more compressed. 

\subsection{Encoder representations}

\setlength{\tabcolsep}{3.0pt}
\begin{table}[t!]
    \centering
    \resizebox{0.95\hsize}{!}{%
    \begin{tabular}{l l c c c c c c} \toprule
        \multicolumn{8}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Features} & $\mathcal{D}$ & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg}\\  \midrule
        Random Mel      & - & 0.87 & 0.78 & 0.73 & 0.39 & 0.00 & 0.55\\
        OpenL3          & -  & 0.19 & 0.61 & 0.08 & 0.10 & 0.18 & 0.23\\ 
        CDPAM           & -  & \textbf{1.00} & \textbf{1.00} & 0.79 & 0.25 & 0.63 & 0.73\\\midrule
        NP-HH S         & L  & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\
        SPSA            & L & 0.95 & 0.98 & \textbf{1.00} & 0.89 & 0.95 & 0.96\\
        AD              & L & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\ \midrule
        \multicolumn{8}{c}{\textbf{MUSDB18}} \\ \midrule
        \textbf{Features} & $\mathcal{D}$  & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg} \\  \midrule
        Random Mel        & - & 0.80 & 0.98 & 0.62 & 0.17 & 0.00 & 0.51 \\
        OpenL3            & - & 0.32 & 0.66 & 0.20 & 0.17 & 0.30 & 0.33 \\ 
        CDPAM             & - & 0.89 & 0.95 & 0.66 & 0.00 & 0.06 & 0.51 \\\midrule
        NP-HH S         & J  & \textbf{0.98} & \textbf{1.00} & 0.92 & \textbf{0.59} & \textbf{0.60} & \textbf{ 0.82} \\
        SPSA            & J  & \textbf{0.98} & \textbf{1.00} & 0.90 & 0.26 & 0.00 & 0.63 \\
        AD              & J  & \textbf{0.98} & \textbf{1.00} & \textbf{0.95} & 0.54 & 0.50 & 0.79 \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Class-wise F1 scores for five-class style prediction with linear classifiers trained on top of audio representations for speech and music using a single linear layer.}
   
    \label{tab:style-probes}\    
    \vspace{-0.8cm}
\end{table}

In addition to the style transfer evaluation, we also investigated the potential utility of the representations produced by the encoder. 
To do so, we created a downstream task of audio production style classification where the goal is to classify the style of a recording among the five styles we created for the realistic style evaluation. 
We trained linear logistic regression classifiers, also known as linear probes, on top of a set of different audio representations, a common approach in evaluating the quality of self-supervised representations~\cite{zhang2017split, asano2019critical}.
These linear probes are trained for a total of 400 epochs on a small dataset with only 60 examples from each of the five styles.
Class-wise and overall F1 scores on the test set are reported in Table~\ref{tab:style-probes}.

As baseline representations we considered a random projection of melspectrogam features, along with large pre-trained representations including OpenL3~\cite{cramer2019look}, which is focused on capturing audio content and CDPAM~\cite{manocha2021cdpam}, which is focused on capturing audio quality and is therefore the most similar to our approach.
In addition, we trained linear probes on representations from the NP-HH, SPSA, and AD gradient methods on both speech styles from DAPS and music styles from MUSDB18.

In the case of DAPS, the class-wise scores indicated that CDPAM and our proposed representations were able to capture the Bright and Telephone styles. 
Differentiating between Broadcast and Neutral was difficult due to the main different between these styles being the presence of more aggressive dynamic range compression. 
However, we found that our encoder representations did not struggle to capture this difference. 
Overall, is is clear that the representations produced by the our self-supervised training framework capture not only style elements related to spectral content, but also signal dynamics, which are not captured well by the other approaches. 
We found similar results between the speech and music style classification, with the appearance that music was slightly more difficult, likely due to the starting diversity of style within MUSDB18.




\section{DISCUSSION}

While the results of our evaluation indicated that the AD exact DDSP approach performed best across most metrics, there are a number of important considerations for each of the the gradient methods that are not reflected in these metrics. 
For example, while the TCN-based models performed well in comparison, and are ultimately more flexible since they make no assumptions about the underlying audio effects, these system do not provide interpretable controls, and also require significant computation, making them difficult to integrate into audio production environments. 

On the other hand, the approximate DDSP approaches provide a unique benefit in that both SPSA and NP variants do not require knowledge of the underlying system or any re-implementation, nor do they explicitly require differentiability.
Since they enable the use of the original DSP implementation at inference, they also require less computation and produce interpretable control parameters. 
However, we observed that both the NP and SPSA methods suffered from other drawbacks. 

The SPSA approach proved more susceptible to training instability, thus requiring careful tuning of the $\epsilon$ hyperparameter and learning rate. 
And, while the neural proxy variants were promising, they were unable to achieve performance on par with the other approaches.
While the NP approaches do not require re-implementation or knowledge of the audio effects, they do require a laborious pre-training process. 
However, we hypothesize that their worse performance likely stemmed from the inaccuracy of the proxy networks, especially for the parametric equalizer. 
Therefore, future directions may involve the design of neural proxy architectures better suited to a wide range of audio effects. 

The exact DDSP AD approach was the most manual and demanding method to implement technically, but showed promise in results. This is especially the case when expert knowledge could be exploited to introduce manual modifications that simplify the system and support efficient operation during training. 








\section{CONCLUSIONS}
Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another, by example.
We demonstrated the applicability of this approach in both speech and music tasks and performed an extensive quantitative and qualitative comparison for a diverse range of differentiable signal processing methods. 
We furthermore proposed a categorization of existing approaches for differentiable signal processing, including the introduction of a differentiable dynamic range compressor, and neural proxy hybrid approaches

Our proposed system demonstrated convincing performance on both the synthetic and realistic style transfer tasks. It also estimates interpretable audio effect control parameters, can operate at variable sample rates, even those not seen during training, and learns representations that reflect  production style..

\section{ACKNOWLEDGMENT}
This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1).



\section{INTRODUCTION}
\label{sec:intro}
Audio effects are commonly used in audio and video content creation to manipulate sound attributes such as loudness, timbre, spatialization, and dynamics~\cite{wilmering2020history}. 
While audio effects can be powerful tools in the hands of trained audio engineers, their complexity poses a significant challenge for less experienced users, often requiring a time-consuming adjustment process even for professionals. 
To simplify their usage, expert-designed presets can be used to configure an audio effect, but this only results in static content-independent configurations of effects.
Therefore, they commonly require fine-tuning and adjustment, without the ability to adapt to variations across input signals.

Automatic audio production methods aim to overcome this limitation with analysis and subsequent adaptation of effects based on an input signal~\cite{IMPbook19}. These methods can provide intelligent, automatic control of a single effect parameter, multiple effects, and/or many channels of audio in a cross-adaptive manner.
Such methods may also provide a range of interfaces, from fully automatic systems to systems that only provide suggestions.
As a result, successful automatic audio production systems have the ability not only to simplify the process for amateurs, but also expedite the workflow of professionals~\cite{moffat2019approaches}.

Automatic audio production systems are commonly categorized into rule-based systems, built on audio engineering best-practices, and data-driven, machine learning systems~\cite{moffat2019approaches}.
While rule-based systems have seen success~\cite{deman2013b}, they are limited by the inability to formalize rules that cover the diversity arising in real-word audio engineering tasks~\cite{deman2017towards}.
Classical machine learning approaches on the other hand provide greater flexibility, but have thus far been limited by an inability to collect sufficient parametric data that describes audio production in a standardized way~\cite{scott2011analysis}. Recently, deep learning approaches have shown promise in overcoming these challenges, fueling an increasing interest in data-driven audio production techniques.

Deep learning approaches for automatic audio production can be organized into direct transformation methods, parameter prediction methods, and differentiable digital signal processing (DDSP) methods. 
Direct transformation methods map untreated audio to a desired target and have shown promise in tasks such as speech enhancement~\cite{pascual2017segan}, source separation~\cite{stoter2019open}, and audio effect modeling~\cite{hawley2019signaltrain, martinez2020deep,wright2020real,martinez2021deep}, but have had limited use for automatic production due to a lack of interpretable control, artifacts, and/or computational complexity~\cite{martinez2021deep}. 
Parameter prediction methods predict the settings of audio effects, but require expensive ground truth parameter data~\cite{hawley2019signaltrain} or optimize over an undesirable loss in the parameter domain~\cite{sheng2019feature, mimilakis2020one}. 
Lastly, DDSP methods~\cite{engel2020ddsp} use neural networks to control digital signal processors (DSP) implemented as differentiable operators enabling training via backpropagation. 
This approach is desirable, but requires manual implementation of audio effects, expert knowledge, and high engineering effort, potentially limiting its use. To work around this, two alternatives methods have been proposed including neural proxies (NP) of audio effects~\cite{steinmetz2021automatic} and numerical gradient approximation schemes~\cite{ramirez2021differentiable}, but given that these approaches were proposed and evaluated in different tasks, it is difficult to fully understand the benefits of each method.




Beyond integrating audio effects as differentiable operators, a central limitation of deep learning systems for automatic audio production lies in the difficulty of sourcing sufficient data for supervised training, requiring both the unprocessed and final produced recordings and/or parameter data. 
In addition, the subjective and context-dependent nature of the audio production process further complicates the task~\cite{lefford2021context}. 
While evidence suggests the existence of ``best-practices''~\cite{pestana2014intelligent}, learning these techniques in a supervised paradigm is challenging.
In particular, these systems must not only learn how to control effects to achieve a desired result, they must also implicitly uncover the distribution of possible styles present within the training data across different genres and contexts.


In this work, we present \emph{DeepAFx-ST} -- a style transfer method that can impose the audio effects and production style from one recording to another by controlling real or approximated audio effects to perform style manipulation as shown in \fref{fig:style-transfer-headline}. 
Using our approach, only a short example style recording is needed at inference for effects style transfer and no labeled training data or retraining is needed to adapt to new styles. 
To do so, we propose 
\begin{itemize}
    \item The first audio effects style transfer method to integrate audio effects as differentiable operators, optimized end-to-end with an audio-domain loss, 
    \item A simplified self-supervised training procedure that does not required label or paired training data,
    \item An overview of differentiation strategies for implementing  differentiable audio effects, including the development of two novel neural proxy hybrid methods, and a differentiable dynamic range compressor,
    \item An evaluation of our style transfer method and in-depth discussion of differentiable signal processing methods. 
\end{itemize}
We find our approach offers several benefits including the ability to perform audio effect style transfer for speech and music signals, yields interpretable control parameters enabling user interaction, and can operate at sampling rates different than those seen during training.

The remainder of our paper is organized as follows. We outline background material in~\sref{sec:background}, discuss our proposed method on~\sref{sec:method}, review our experimental design in~\sref{sec:experimental_design}, and conclude with our results and discussion in~\sref{sec:results} and~\sref{sec:discussion}. A demonstration video and listening examples are available online\footnote{\url{https://csteinmetz1.github.io/deepafx-st}}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/generic-style-transfer-headline.drawio.pdf}
    \vspace{-0.3cm}
    \caption{Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another by example. We use a shared-weight encoder to analyze the input and a style reference signal, then compare each with a controller that outputs the parameters of effects that themselves perform style manipulation.}
    \label{fig:style-transfer-headline}
\end{figure}


\section{BACKGROUND}
\label{sec:background}


\subsection{Audio Effects and Production Style Transfer}
Relevant past work on audio effects and production style transfer include initial work on controlling a dynamic range compressor~\cite{sheng2019feature} using a multi-stage training procedure combining neural networks and random forest as well as another method that uses a fully deep neural network approach to control a parametric frequency equalizer~\cite{mimilakis2020one}. 
Both works, however, only consider controlling one audio effect (compressor or parametric equalization) and use a (quantized) parameter domain loss to avoid backpropagating through differentiable audio effects -- a non-trivial task. 
This approach has disadvantages including undesirable quantization schemes or loss tuning, low correlation between parameters and audio, and the management of multiple systems (i.e. neural networks and DSP). 
However, the use of differentiable signal processing audio effects for the style transfer task have yet to be investigated. 

\subsection{Differentiable Audio Effects}\label{sec:diff_fx}
Typically, differentiable digital signal processors are manually implemented within automatic differentiation frameworks~\cite{abadi2016tensorflow, paszke2019pytorch}.
Assuming the mathematical operations in question are differentiable, this provides a straightforward way to integrate DSP operations into neural networks. 
This approach, however, requires expert knowledge, design trade-offs, and/or can be difficult or impossible to implement exactly. Examples of such audio effects include infinite impulse response (IIR) filters~\cite{kuznetsov2020differentiable, nercessian2020neural, colonel2021iirnet}, hearing aids~\cite{tu2021dhasp}, echo cancellers~\cite{casebeer2021auto}, DJ transitions~\cite{chen2021automatic}, and reverse engineered effects~\cite{kuznetsov2020differentiable, nercessian2021lightweight,colonel2021reverse}. 

Differentiable audio effects of interest to our work include the parametric frequency equalizer (PEQ) and dynamic range compressor (DRC) -- two common audio production effects.
To our knowledge, there is no known past work on differentiable compressors, but differentiable PEQs do exist~\cite{nercessian2020neural}. 
PEQs are typically constructed with a cascade of multiple second-order IIR filters, also know as biquads~\cite{valimaki2016all}. 
While it is possible to implement differentiable IIR filters in the time-domain, the recursive filter structure can cause issues as a result of vanishing/exploding gradients and computational bottlenecks during backpropagation through time (BPTT)~\cite{hochreiter1997long}. This motivates frequency-domain finite impulse response (FIR) approximations~\cite{nercessian2020neural, nercessian2021lightweight, colonel2021iirnet}. In this case, the complex frequency response of the $k^{\textrm{th}}$ biquad $H_k(e^{j \omega})$ is expressed as a ratio of the DFT of the numerator and denominator coefficients, $\mathbf{b}_k = [b_{0,k}, b_{1,k}, b_{2,k}]$ and $\mathbf{a}_k = [a_{0,k}, a_{1,k}, a_{2,k}]$,
\begin{equation}
    H_k(e^{j \omega}) = \frac{ \textrm{DFT}(\mathbf{b}_k) }{ \textrm{DFT}(\mathbf{a}_k) } = \frac{\sum_{m=0}^{2} \ b_{m,k} \ e^{-j\omega m}}{\sum_{n=0}^{2} \ b_{n,k} \ e^{-j \omega n}}.
\end{equation}
The overall response of a cascade of $K$ biquad filters is then given by the product of their responses $H_{sys} (e^{j \omega}) = \prod_{k=1}^{K} H_k(e^{j \omega})$.
We can approximate the response using an FIR filter by evaluating $H_{sys}(e^{j \omega})$ over a linearly spaced, zero-padded frequencies of length $2^{\lceil \log_2 (2N - 1) \rceil}$, applying the filter(s) to an input $x[n]$ with $N$ samples in the frequency domain by multiplying with  the input $ X(e^{j \omega})$, 
and then perform the inverse DFT to get the output $y[n]$.


\subsection{Alternative Differentiation Methods}

In contrast to manually implementing DDSP algorithms, three  alternative approaches are of note. In neural proxy (NP) approaches, a neural network is trained to emulate the behavior of a signal processor~\cite{grathwohl2017backpropagation, jacovi2019neural} including control parameter behavior using a dataset of input/output measurements. 
The proxy model can be used directly~\cite{hawley2019signaltrain, damskagg2019deep, martinez2019modeling, wright2019real, martinez2020deep, wright2020real, martinez2021deep, steinmetz2021efficient} or together with as part of a larger neural network architecture~\cite{steinmetz2021automatic}.
Since neural proxy models are composed of neural network building blocks, they are differentiable by default, but require accurate cloning to work effectively and can be computationally complex.  

In an effort to reduce inference time complexity and reduce approximation error, neural proxy models can be used together with DSP to create what we call neural proxy hybrid models~\cite{tseng2019hyperparameter}. In this case, neural proxies are only used for gradient approximation to reduce approximation error caused by the neural proxy. We denote full hybrid approaches as methods that employ the original DSP during training (forward pass of backpropagation) as well as inference and half hybrid models as methods that employ the original DSP only during inference (easier to implement). Past neural proxy hybrid approaches, however, have only been used for static hyper-parameter optimization for fixed datasets, and not in an adaptive manner.

As a third alternative, non-differentiable DSP implementations can be used directly with a numerical gradient approximations methods~\cite{ramirez2021differentiable}. In context of controlling audio effects $h(x, p)$ with input audio $x$ and parameters $p \in \mathbb{R}^{P}$, we only need to estimate the partial derivatives for each parameter $\frac{\partial}{\partial p_i} h(x,p)$. For this, finite differences (FD)~\cite{milne2000calculus} can be used by perturbing each input parameter forward and backward and evaluating the operator $2P$ times for $P$ input parameters. Alternatively, simultaneous permutation stochastic approximation (SPSA)~\cite{spall1998overview} gradient estimation can be used that only requires $2P$ function evaluations per gradient estimate. In this case, the approximate partial derivative with respect to $p_i$ is given by 
\begin{equation}
\frac{\hat{h}(x, p_i)}{p_i} = \frac{h(x, p + \epsilon \Delta^P) - h(x, p - \epsilon  \Delta^P)}{2 \epsilon \Delta^{P}_{i}},
\end{equation}
where $\epsilon$ is a small, non-zero value and $\Delta^P \in \mathbb{R}^P$ is a random vector sampled from a symmetric Bernoulli distribution ($\Delta^P_i = \pm 1$)~\cite{spall1992multivariate}.  
This approach does not require pre-training or knowledge of signal processing operations, but can be challenging to train due to inaccuracies in the gradient approximations and slow CPU implementations of DSP.
This method has been used for audio effect modeling, removal of breaths, and music mastering~\cite{ramirez2021differentiable}. 

\begin{figure*}[t!]
    \centering
    \vspace{-0.2cm}
    \includegraphics[width=0.95\linewidth]{figures/training-lines.drawio.pdf}
    \vspace{-0.0cm}
    \caption{The self-supervised training process for our \emph{DeepAFx-ST} approach.
   
    We sample $x$ from a large dataset of recordings $\mathcal{D}$ and apply random augmentations.
    Recordings with two different styles are produced with randomly configured DSP, which become the input $x_i$ and the style reference $x_r$.
    We split these recordings in half, and pass the opposite sections ($x_{i,a}$ and $x_{r,b}$) to shared-weight encoders $f_\theta$. 
    Using the encoder representations ($e_i$ and $e_r$), the controller $g_\phi$ estimates parameters $p$ to apply the reference style to the input recording.}
    \label{fig:data-generation}
    \vspace{-0.0cm}
\end{figure*}

\section{PROPOSED METHOD}
\label{sec:method}
Our audio effects style transfer method architecture is described in~\sref{sec:arch} and training procedure in~\sref{sec:ssl}.



\subsection{Model Architecture}
\label{sec:arch}
Our model architecture is designed to emulate the creative process of an audio engineer and consists of three sub-processes including acoustic analysis, style determination, and parameter adjustment as shown in~\fref{fig:data-generation} (Right).

\linesubsec{Encoder} 
Our encoder $f_\theta$ extracts audio production style information from both the input and style reference signal. To do so, it inputs a time domain audio signal and converts it to a 2D representation or spectrogram $X$ using a short-time Fourier transform (STFT) with window size $N$ and hop size $H$. 
Each spectrogram is fed to a shared-weight convolutional neural network, which outputs a time series of representations or embedding sequence per signal. 
Temporal average pooling is applied to aggregate across time, producing a single embedding of dimension $D$ for each input recording. 
The input and style embeddings, $e_i$ and $e_r$, are then concatenated and passed to the controller.

\linesubsec{Controller} Our controller $g_\phi$ takes the concatenated input and style embeddings ($e_i$ and $e_r$) and outputs audio effect parameters $p$. 
To operate correctly, our controller must learn to compare the extracted encoder embeddings, and map the input signal to the desired target style by adjusting the control parameters.
In our formulation, the controller is a simple three layer multi-layer perceptron (MLP) featuring a single hidden layer, along with PReLU~\cite{he2015delving} activation functions. 
At the output, the hidden representation is projected into the audio effect control parameter space $\mathbb{R}^P$, and a sigmoid activation is applied. 


\linesubsec{Differentiable audio effects} 
Finally, we use differentiable audio effects $h(x, p)$ to process the input audio signal, given the predicted parameters from the controller. 
In contrast to past work~\cite{mimilakis2020one, sheng2019feature}, we fully integrate audio effects as differentiable operators or layers with standard automatic differentiation tools and enable backpropagation through effects during training as shown with dashed lines in Figure~\ref{fig:data-generation}. 
We do this to enable end-to-end training using an audio-domain loss and directly avoid difficulties in past works as discussed in~\sref{sec:background}.
We further develop our approach to work with a variety of differentiation strategies including manual automatic differentiation implementations (AD), neural proxies (NP), full neural proxy hybrids (NP-FH), half neural proxy hybrids (NP-HH), and numerical gradient approximations (SPSA) for an in-depth comparison between approaches.

\subsection{Self-supervised Training}
\label{sec:ssl}

Our self-supervised training strategy enables learning control of audio effects without the need for any labeled or paired training data. 
Our data generation process is shown on the left of Figure~\ref{fig:data-generation} and begins by selecting a random audio recording $x$ from a dataset $\mathcal{D}$. 
We then create an augmented version $x_{aug}$ by applying pitch shifting and time stretching, and
then produce two unique versions of this signal by applying a set of randomly configured audio effects.
This yields the input recording $x_i$ in the top branch, and the style reference $x_r$ in the bottom branch, each with a different production style. Our model is then tasked with estimating parameters $p$ to configure the effects $h(x,p)$ so that the processed input signal matches the style reference.

To train our model to only focus on audio production style and operate even when the content differs between the input and style reference, we split the recordings $x_i$ and $x_r$ in half, generating an $a$ and $b$ section for each or $x_{i,a}$, $x_{i,b}$, $x_{r,a}$, and $x_{r,b}$.
During training, we randomly select either the $a$ or $b$ section to be passed as input and use the other section as the style reference.
To compute an audio-domain loss, we use the style reference corresponding to the same section as the input in order to have a generate a paired input and groundtruth recording with the same content.  
Unlike typical audio training, our approach automatically generates paired data and can utilize any audio dataset, preferably with a wide range of quality and recording conditions. 

\section{EXPERIMENTAL DESIGN}
\label{sec:experimental_design}

\subsection{Problem Setup}
\label{sec:setup}

To demonstrate our approach, we focus on the task of imposing the equalization and loudness dynamics from one recording to another using a parametric equalizer and dynamic range compressor as shown in~\fref{fig:signal-chain}. 
This forms the basic signal chain employed by audio engineers with applications in dialogue processing or in music mastering.
We manually implement automatically differentiable PEQ and DRC and compare style transfer performance using alternative differentiation strategies as noted in~\sref{sec:arch}. 
By doing so, we seek to both understand the performance of our style transfer approach as well as the advantageous and disadvantageous of differentiation strategies.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth, trim={0.0cm 0.18cm 0.0cm 0.1cm},clip ]{figures/signal-chain.drawio.pdf}
    \caption{Our audio effect signal chain consisting of a 6-band parametric equalizer and dynamic range compressor.}
    \label{fig:signal-chain}
    \vspace{-0.0cm}
\end{figure}


\subsection{Differentiable Equalizer and Compressor}
\label{sec:eq_drc}
We implement a differentiable PEQ following past work~\cite{nercessian2020neural}, but develop own novel differentiable dynamic range compressor. 
For our differentiable compressor, we use a traditional feedforward dynamic range compressor, which is in fact composed of differentiable operations~\cite{giannoulis2012digital}.
Similar to PEQs, however, we found direct implementation of existing compressor designs problematic due to the presence of recursive filters, in this case due to the smoothing filters for the ballistics,
\begin{equation} \label{eq:smoothing}
    \resizebox{.9\hsize}{!}{
   $y_L[n] =
    \begin{cases}
        \alpha_A y_L[n-1] + (1-\alpha_A) x_L[n] & x_L[n] > y_L[n-1] \\
        \alpha_R y_L[n-1] + (1-\alpha_R) x_L[n] & x_L[n] \leq y_L[n-1]
    \end{cases}$,       
    }
\end{equation}
where $y_L[n]$ is the smoothed gain reduction, $x_L[n] = x_G[n] - y_G[n]$ is the result of subtracting the output of the gain computer from the signal envelope, and $\alpha_A$ and $\alpha_R$ are the attack and release time constants. 
This operation is not easily approximated with a FIR filter, thus we simplify our design using a single attack and release time constant $\alpha$ to create a single-pole IIR ballistics filter,
\begin{equation}
   y_L[n] = \alpha y_L[n-1] + (1-\alpha) x_L[n].
\end{equation}
This filter can can be approximated using an FIR filter using the same approach as differentiable PEQs.
While this modification restricts the capabilities of our compressor, forcing the effective attack and release times to be shared, we seek only a basic compressor to show the complexity in implementing differentiable versions of common audio effects, which we compare against competing alternative differentiation strategies. 

\subsection{Datasets}
\label{sec:datasets}

We employ our data generation pipeline as described in~\sref{sec:method} using both speech and music.
To test our approach on speech, we use \texttt{train-clean-360} subset of LibriTTS~\cite{zen2019libritts}, which totals  360 hours of audio at $f_s = 24$\,kHz.
To test our approach on music, we use the MTG-Jamendo dataset~\cite{bogdanov2019mtg}, which provides over 55\,k songs, which we downmix to mono and resample to $f_s = 24$\,kHz. 
For both datasets, we generate a $90$/$5$/$5$ split of the recordings for training, validation, and testing. When randomly sampling patches of audio from a recording, if the selected segment $x[n]$ has a mean energy below the threshold, $\frac{1}{N} \sum_{n=1}^{N} | x[n] |^2 \leq 0.001$), we continue sampling until a non-silent segment is found.
To ensure that there is sufficient headroom for processing within the audio effects, we peak normalize all inputs and style recordings to $-12$\,dBFS.

\subsection{Baselines}
\label{sec:baselines}

\linesubsec{Rule-based DSP}
\label{sec:dsp}
Our rule-based baseline consists of an automatic, two stage process that aims to mimic the signal chain of the PEQ and DRC. 
First, an FIR filtering stage is employed, which aims to transform the spectral content of the input speech to match the spectrum of the style reference. 
We compute the average magnitude spectrum of the input and style reference by computing the STFT using a large window of $N=65536$ and a hop size of $H=16384$, averaging across the frames, yielding $X_i$ and $X_r$, which
are smoothed using a Savitzky-Golay filter~\cite{savitzky1964smoothing}.
We design an inverse FIR filter with $63$ taps using the window method, with a desired magnitude response given by $X = X_{i,sm} / X_{r,sm}$, where $X_{i,sm}$ and $X_{r,sm}$ are the smoothed average spectra of the input and the style reference.

The filtered signal is sent to a dynamic range compressor where all parameters except for the threshold are fixed. 
In order to adjust the amount of compression based on the input signal and the style reference, the threshold starts at 0 dB and is lowered in increments of 0.5 dB.
At each iteration the loudness in dB LUFS~\cite{steinmetz2021pyloudnorm} of the output is measured and compared to the loudness of the style reference.
If the difference in the loudness measurements are below a given threshold (e.g 0.5 dB LUFS), the process halts. 
Otherwise, the process continues until the loudness is within a specific tolerance, or the threshold reaches -80 dB.

\linesubsec{End-to-end neural network}
\label{sec:e2e}
As a baline on the opposite end of the spectrum, we evaluate a direct, end-to-end neural network approach to replace the DDSP audio effects. 
This network utilizes the same architecture our neural proxy approaches, but instead does not use pre-trained weights. 
Instead, the weights of this model are optimized during the training of the encoder and controller. 
Even though the controller still predicts parameters for this network, as in the case of a neural proxy, since the weights of the network are not frozen during training, the model will learn its own set of control parameters. 
We evaluate two variants of this approach, first using a single TCN network (cTCN 1) and a second using two TCNs connected in series (cTCN 2), which mimics the setup in the neural proxy approach.



\subsection{Losses}
We train both our neural proxy models and style transfer models using an audio-domain loss.
In particular, we compute our loss using the model output and groundtruth audio from our data generation process with a weighted sum of the error in time and frequency domains.
The time domain loss $\mathcal{L}_{\text{time}}$ is the mean absolute error (MAE) and the
frequency domain loss $\mathcal{L}_{\text{freq}}$ is the multi-resolution short-time Fourier transform loss (MR-STFT)~\cite{wang2019neural, steinmetz2020auraloss}.
This loss is the sum of the $L_1$ distance between the STFT of the groundtruth and estimated waveforms measured both in the $\log$ and linear domains, at multiple resolutions, in this case with window sizes $W \in [32, 128, 512, 2048, 8192, 32768]$ and hop sizes $H = W/2$.
The overall loss is  $\mathcal{L}_{\text{overall}}({\hat{y}},y) = \mathcal{L}_{\text{freq}}({\hat{y}},y) + \alpha \cdot \mathcal{L}_{\text{time}}({\hat{y}},y) \label{eq:overall-loss}$, where $\alpha = 100$.
 

\setlength{\tabcolsep}{3.0pt}
\setlength{\abovecaptionskip}{15pt plus 3pt minus 2pt}
\begin{table*}[t!]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
        \multicolumn{21}{c}{\normalsize{\textbf{Speech}}} \\ \midrule
        & \multicolumn{6}{c}{\textbf{LibriTTS}} & & \multicolumn{6}{c}{\textbf{DAPS}} & & \multicolumn{6}{c}{\textbf{VCTK}} \\ \cmidrule(lr){2-7} \cmidrule(lr){8-14} \cmidrule(lr){15-21}
        \textbf{Method}  &  \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}}\\ \midrule
        {Input}   
        & 3.765           & 1.187           & 2.180           & 687.5           & 6.983           & 2.426           & 
        & 3.684           & 1.179           & 2.151           & 641.7            & 6.900          & 2.314          & 
        & 3.672           & 1.254           &          2.008  & 815.4           & 7.783           & 2.532\\
        {RB-DSP}  
        & 3.856           & 0.943           & 1.955           & 410.3           & 4.204           & 1.674           & 
        & 3.787           & 0.917           & 1.882           & 399.7            & 3.705           & 1.481          & 
        & 3.709           & 1.101           &          1.911  & 657.6           & 5.039           & 2.018\\ \midrule
       
        {cTCN 1}    
        & 4.258           & 0.405           & 0.887           & 128.4           & 2.237           & 1.066           & 
        & 4.185           & 0.419           & 0.884           & 124.6            & 2.098           & 1.006          & 
        & 4.181           & 0.467           &          0.891  & 173.8           & 2.651           & 1.165\\
        {cTCN 2}    
        & 4.281           & 0.372           & \textbf{0.833}  & 117.5           & 1.927           & 0.925           & 
        & \textbf{4.224}  & \textbf{0.391}  & \textbf{0.841}  & 113.9            & 1.886           & 0.913          & 
        & 4.201           & \textbf{0.441}  & \textbf{0.856}  & 163.8           & 2.431           & 1.086\\ \midrule
       
        {NP}   
        & 3.643           & 0.676           & 1.405           & 265.0           & 2.812           & 1.340           & 
        & 3.605           & 0.685           & 1.362           & 249.2            & 2.732           & 1.350          & 
        & 3.651           & 0.737           &          1.300  & 321.7           & 3.166           & 1.453\\
        {NP-HH}   
        & 3.999           & 1.038           & 2.179           & 440.2           & 5.472           & 2.679           & 
        & 3.903           & 1.022           & 2.113           & 451.9            & 5.104           & 2.535          & 
        & 3.951           & 1.044           &          1.930  & 591.5           & 5.194           & 2.651\\
        {NP-FH}   
        & 3.945           & 1.058           & 2.088           & 404.9           & 6.820           & 3.197           & 
        & 3.891           & 1.037           & 2.045           & 395.4            & 6.754           & 3.117          & 
        & 3.894           & 1.087           &          1.934  & 514.0           & 7.065           & 3.363\\
       
        {SPSA}      
        & 4.180           & 0.635           & 1.406           & 219.5           & 3.263           & 1.600           & 
        & 4.099           & 0.645           & 1.379           & 213.6            & 2.989           & 1.511          & 
        & 4.023           & 0.730           &          1.359  & 301.6           & 3.535           & 1.737\\
       
        \midrule
        {AD}  
        & \textbf{4.310}  & \textbf{0.388}  & 0.882           & \textbf{111.5}  & \textbf{1.828}  & \textbf{0.823}  & 
        & 4.222           & 0.416           & 0.895           & \textbf{109.0}   & \textbf{1.758}  & \textbf{0.799} & 
        & \textbf{4.218}  & 0.481           &          0.924  & \textbf{152.7}  & \textbf{2.317}  & \textbf{1.006} \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
\caption{Synthetic style transfer with models trained using LibriTTS. Held-out speakers from the LibriTTS dataset are used, while utterances from DAPS and VCTK come from datasets never seen during training. Lower is better for all metrics except PESQ.}
    \vspace{-0.1cm}
    \label{tab:style-transfer-synthetic-speech}
\end{table*}

\subsection{Neural proxy pre-training}
To construct our neural proxy-based models, we utilized a temporal convolutional network (TCN)~\cite{bai2018empirical}, also known as the feedforward WaveNet~\cite{rethage2018wavenet}. This model has seen success in modeling distortion~\cite{wright2019real} and compressors~\cite{steinmetz2021automatic}.
In particular, we use a causal TCN~\cite{steinmetz2021automatic} with feature-wise linear modulation (FiLM)~\cite{perez2018film} to adapt to the effect control parameters.
Our TCN configuration has $4$ blocks, kernel size of $13$, channel width of $64$ and a dilation factor that grows as a power of $8$, where the dilation factor at block $b$ is $d_b = [ 1, 8, 64, 512 ]$.
We trained a total of 4 proxy models, one for each of the two effects, with one model for both speech and music. 
Training examples were generated by using DSP effects with uniformly random parameters. 
We trained each model for 400 epochs, where one epoch is defined was 20,000 samples and used the checkpoint with the lowest validation loss.
We used an initial learning rate of $3\cdot 10^{-4}$, a batch size of 16, and inputs $\approx 3$\,s ($65536$ samples) in length. 
The learning rate was lowered by a factor of 10 after the validation loss had not improved for 10 epochs. 

\subsection{Style transfer training}
After training the neural proxy models, we trained our style transfer model using five different audio effects differentiation methods -- AD, NP, NP-HH, NP-FH, SPSA. 
For our encoder, we used an STFT with a Hann window of size $N=4096$ and a hop size of $H=2048$ as input into a randomly initialized EfficientNet B2 CNN~\cite{tan2019efficientnet} with a single input channel. 
The final activations of the network are projections to $D=1024$ with a linear layer, followed by mean pooling across the temporal dimension to produce a single representation, which is normalized by the $L_2$ norm.
We trained all models for a total of 400 epochs with a batch size of 6, where a single epoch is defined as 20,000 segments of audio $\approx 10$ seconds in length ($262144$ samples) randomly sampled from the training set. 
We used a learning rate of $10^{-4}$ for all models, with the exception of the SPSA models, which required a lower learning rate of $10^{-5}$ for stability. 
In both cases, we scheduled the learning rate such that it is lowered by a factor of 10 at both $80\%$ and $95\%$ through training.
We applied gradient clipping with a value of $4.0$ and used an SPSA $\epsilon$ of $0.0005$.




\setlength{\tabcolsep}{2.5pt}
\begin{table*}[ht]
    \centering
    \resizebox{\linewidth{}}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
    \multicolumn{21}{c}{\normalsize{\textbf{Music}}} \\ \midrule
        &  
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (24 kHz)}}   & & 
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (44.1 kHz)}} & & 
        \multicolumn{6}{c}{\textbf{MUSDB18 (44.1 kHz)}} \\ 
        \cmidrule(lr){2-7} \cmidrule(lr){9-14} \cmidrule(lr){15-21}
        \textbf{Method}   & 
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
       
        Input      
        & 2.927  & 1.198  & 6.088  & 646.5  & 6.695  & 2.518 & 
        & 2.874  & 1.109  & 4.454  & 767.7  & 6.793  & inf   & 
        & 2.900  & 1.252  & 4.342  & 1088.3    & 5.940  & 2.312    \\ 
        RB-DSP     
        & 2.849  & 0.925  & 4.422  & 263.2  & 4.254  & 1.706  & 
        & 2.931  & 0.887  & 3.355  & 342.0   & 4.749  & 1.882 & 
        & 2.994  & 0.821  & 3.052  & 379.4   & 4.078 & 1.665  \\
       
        \midrule
        cTCN 1       
        & 3.402  & 0.547  & 2.294  & 160.9  & 3.261  & 1.483 &
        & 3.168  & 0.876  & 2.921  & 494.4    & 4.372  & 2.070  & 
        & 3.121  & 0.896  & 2.956  & 730.4    & 4.548  & 2.231  \\ 
        cTCN 2       
        & 3.390  & 0.548  & 2.278  & 152.3  & 2.951  & 1.397 & 
        &  3.123  & 0.903  & 2.973  & 517.4   & 4.084  & 1.899  &
        &  3.107  & 0.917  & 2.986  & 749.5   & 4.208  & 2.061 \\  \midrule
       
        NP S  
        & 2.926  & 0.787  & 2.838  & 221.3  & 2.785  & 1.390 &  
        & 2.764  & 1.033  & 2.869  & 488.3  & 3.710  & 1.824 & 
        & 2.804  & 0.950  & 2.778  & 742.1   & 3.835  & 1.921  \\
        NP M  
        & 2.765  & 0.845  & 3.211  & 255.2  & 3.227  & 1.608 & 
        & 2.699  & 1.042  & 2.928  & 497.0    & 3.942  & 1.961   & 
        & 2.791  & 0.946  & 2.800  & 757.1   & 4.209  & 2.127    \\
        NP-HH (S)  
        & 2.819  & 1.092  & 6.791  & 395.3  & 6.276  & 3.032 & 
        & 2.865  & 1.101  & 5.194  & 689.2    & 6.792  & 3.365 & 
        & 2.853  & 1.165  & 4.852  & 1005.7   & 6.451  & 3.269 \\
        NP-HH (M)  
        & 2.532  & 1.166  & 7.070  & 591.9  & 5.660  & 2.593 & 
        & 2.512  & 1.148  & 5.385  & 854.9   & 5.940  & 2.740 & 
        & 2.493  & 1.198  & 5.090  & 1021.9    & 5.585  & 2.731 \\
        NP-FH (S)  
        & 2.833  & 1.016  & 5.005  & 280.8  & 3.377  & 1.634 & 
        & 2.888  & 0.977  & 3.883  & 429.9    & 3.480  & 1.709   & 
        & 2.857  & 1.045  & 3.809  & 617.3   & 3.932  & 1.971   \\
        NP-FH (M)  
        & 2.648  & 1.137  & 6.368  & 605.6  & 5.903  & 2.587 & 
        & 2.625  & 1.133  & 5.512  & 844.6   & 6.417  & 2.876  & 
        & 2.575  & 1.225  & 5.450  & 1172.5   & 5.973  & 2.913   \\
        SPSA         
        & 3.173  & 0.716  & 3.024  & 210.1  & 2.809  & 1.344 & 
        & 3.172  & 0.759  & 2.458  & 386.2   & 2.839  & 1.311 & 
        &  3.126  & 0.789  & 2.321  & 574.4   & 2.925  & 1.394    \\
       
        \midrule
        AD     
        & 3.355  & 0.488  & 2.149  & 144.7  & 2.167  & 1.005   & 
        &  3.400  & 0.585  & 1.824  & 304.4    & 2.425  & 1.106 & 
        &  3.396  & 0.608  & 1.695  & 456.1    & 2.559  & 1.197  \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Synthetic style transfer with models trained using MTG-Jamendo dataset. Proxy models with (S) were pre-trained using speech from LibriTTS those with (M) were trained using music from the MTG-Jamendo dataset. Lower is better for all metrics except PESQ.}
    \label{tab:style-transfer-synthetic-music}
    \vspace{-0.2cm}
\end{table*}

\section{RESULTS}
\label{sec:results}



\subsection{Synthetic style transfer}
We first evaluate on a synthetic style transfer task, which mirrors our training setup. 
This facilitates the use of full-reference metrics, since we synthetically generate style references and as a result have inputs and groundtruth with matching content.
Results of the evaluation using held-out speakers from LibriTTS are shown in \tref{tab:style-transfer-synthetic-speech}.

As there are no established metrics for evaluating audio production style, we propose a range of metrics. 
For a general notion of perceptual similarity we use PESQ~\cite{rix2001perceptual}, along with the Multi-resolution STFT~\cite{wang2019neural} error, which is a component of the training objective. 
To evaluate the ability of the system to address equalizer-related style components, we report the mel-spectral distance (MSD), which consists of the error between melspectrograms using a large window size of $W=65536$, along with the spectral centroid error (SCE). 
To evaluate aspects related to the dynamic range, we consider the RMS energy error (RMS) and the perceptual loudness error (LUFS)~\cite{steinmetz2021pyloudnorm}. 

We note our rule-based DSP baseline approach improves on the input across all metrics, though not by a significant margin, indicating it has limited success in the style transfer task.
Overall, we find strong performance from the AD method, which performs best on all metrics except MSD, which is only slightly outperformed by cTCN2.
Both cTCN approaches follow closely behind the AD method, demonstrating the strength of end-to-end neural network approaches. 
The SPSA model follows closely behind these approaches with somewhat degraded PESQ and STFT error, yet it outperforms all NP methods. 
While the NP methods are not as competitive, we observe that both hybrid methods, NP-HH and NP-FH, outperform the original NP in terms of PESQ, likely a result of the use of the DSP implementation, which imparts less artifacts than the neural proxy.
However, the NP approach achieves better performance across all other metrics, indicating it likely better matches the style reference. 

In addition to evaluating on held-out speech from LibriTTS, we also evaluate generalization using utterances from speech datasets not seen during training including DAPS~\cite{mysore2014can} and VCTK~\cite{yamagishi2019vctk}, which were resampled to 24\,kHz.
We observe a similar trend in performance across all methods, yet the cTCN models slightly outperform AD as compared to on LibriTTS.
Overall, this indicates that training on the large scale LibriTTS dataset with our self-supervised data generation pipeline was sufficient for producing models that generalize to speech originating from different sources with different acoustic conditions. 

We also trained the same set of models using music from the MTG-Jamendo dataset. 
The result, shown in \tref{tab:style-transfer-synthetic-music}, largely follows a similar trend to the speech experiments with regards to the gradient methods, demonstrating the general nature of our approach. 
However, it appears that the music task is somewhat more difficult.
We also compared neural proxies trained on speech (S) or music (M), and found those trained on music performed slightly worse.

We then investigated the ability of our methods to operate at a sample rate different than training. 
Signals to the encoder were first resampled to match training (24\,kHz), however the audio effects operate at 44.1\,kHz.
First, we evaluated on the MTG-Jamendo test set, and we noticed a significant drop in performance for the neural network based approaches, which include the cTCNs and NP, since these models operate at a fixed sample rate. 
However, the DSP-based approaches performed significantly better, including both the AD and SPSA approaches, which performed only slightly worse than at the training sample rate.
To test out-of-distribution generalization we used music from MUSDB18~\cite{musdb18}.
While performance does decrease across most metrics and for all models, they do not indicate that the models fail to perform the style transfer task, with the SPSA and AD approaches exceeding the baseline. 

\subsection{Realistic style transfer}
\setlength{\tabcolsep}{3.0pt}

\begin{table}[t!]
    \centering
    \resizebox{1.0\hsize}{!}{%
    \begin{tabular}{l c c c c c c c c c } \toprule
        & \multicolumn{4}{c}{\textbf{DAPS}} & & \multicolumn{4}{c}{\textbf{MUSDB18}} \\  \cmidrule(lr){2-5} \cmidrule(lr){6-10}
        \textbf{Method} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Input}   & 10.4        & 1041.7        & 10.4          & 3.6   & & 8.4         & 2607.4      & 9.4        & 3.8   \\ 
        {RB-DSP}  & 8.9         & 517.2         & 5.8           & 2.4   & & 6.5         & \textbf{915.4}       & 8.6         & 3.7   \\  \midrule
        {NP-HH S} & 9.8         & 636.6         & 12.9          & 5.9   & & 7.0         & 1512.7      & 10.0        & 4.5          \\
        {SPSA}    & 8.0         & 360.1         & \textbf{5.1} & 2.5    & & 5.5         & 1297.0      & 4.6         & 2.1         \\
        {AD}      & \textbf{7.8}& \textbf{278.1}& 5.2  & \textbf{2.4}   & & \textbf{4.8} & 947.6       & \textbf{3.8} & \textbf{1.7}             \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Average performance across all pairwise style transfer configurations using input utterances from DAPS (trained on LibriTSS) and music from MUSDB18 (trained on MTG-Jamendo). }
    \vspace{-0.6cm}
    \label{tab:style-transfer-real}
\end{table}

To simulate a more realistic style transfer scenario where the style reference contains not only different content but also differing sources (e.g. speaker identity or instrumentation), we construct a realistic style transfer evaluation task.
We designed five different audio production ``styles'', which are defined by hand-crafted parameter ranges.
We generate audio from five different styles (Telephone, Warm, Bright, Neutral, and Broadcast) by randomly sampling parametric equalizer and compressor parameters from a set of predefined ranges such that each style is audibly unique.
Sampling parameter values randomly provides more variation within audio examples of the same style, which is more complex than applying a fixed preset. 

To generate examples in each of these styles, we use clean utterances from DAPS, and we use the same style definitions to generate music examples using songs from MUSDB18.
However, since we do not have paired groundtruth as in the synthetic style transfer case, we performed our evaluation using only non-intrusive metrics that do not require a matching reference. 
These metrics measure the error between high-level audio features, like the spectral centroid, RMS energy, and perceptual loudness.

We performed all pairwise style transfers resulting in 25 configurations, and we performed each configuration ten times using different input and output recordings from each style. 
Here we considered only the best performing methods that produce interpretable control parameters (NP-HH, SPSA, and AD).
To summarize the results of this evaluation, we aggregated the metrics across all 25 configurations reporting the mean for each to provide a general notion of the style transfer performance, as shown in \tref{tab:style-transfer-real}. 

On examples from DAPS, all methods, including the baseline, improve upon the input, but SPSA and AD clearly outperform the other approaches across all metrics, with AD preforming the best. 
In the case of MUSDB18, the AD approach further outperforms the others, with the exception of SCE, where the RB-DSP baseline performs best.



\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (top) and compressor (bottom) parameters SPSA model with an utterance from VCTK using varying style references from the five predefined styles.}
    \label{fig:vctk-variable-style}
\end{figure}

\subsection{Parameter visualization}
To demonstrate the ability or our approach to provide precise control over the effect parameters, we plot both the overall frequency response of the parametric equalizer as well as the compressor configuration using an utterance from VCTK as input and applying style transfer using one utterance from each of the five different styles as shown in Figure~\ref{fig:vctk-variable-style}.
Since the utterances from VCTK are relatively clean and contain little post-processing we observed the estimated parameters aligned with our expectation for the predefined styles. 
For example, we find that the telephone style is achieved by reducing the high and low frequencies while boosting frequencies around 1\,kHz.
In terms of compression, we see a clear trend with the broadcast style using the most aggressive compression, as expected. 
While the task of estimating these style parameters may seem trivial, recall that the system was never trained on these utterances and has no knowledge of our pre-defined styles. 

\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314_modify.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (top) and compressor (bottom) parameters with the SPSA model using a fixed Broadcast style reference. The input VCTK utterance was progressively modified to increase the high frequency energy and compression.}
    \label{fig:vctk-variable-input}
    \vspace{-0.4cm}
\end{figure}

As a demonstration of the ability of the model to adapt to the input signal like an \emph{adaptive preset}, we used a single style recording from our generated Broadcast style, and varied characteristics of the input. 
In this case, the input signal was modified by progressively increasing the gain of a high-shelf filter from 0\,dB to +24\,dB, while also increasing the amount of compression with a higher ratio (1.0 to 4.0) and lower threshold (0\,dB to -62\,dB).
As shown in \fref{fig:vctk-variable-input}, the estimated parameters adapt to these changes in an sensible manner. 
This results in a progressively lower boost in the high-shelf filter that was added originally to achieve the presence of the Broadcast style, while also progressively increasing the threshold and decreasing the ratio to reduce the amount of compression applied, as the input becomes more compressed. 

\subsection{Encoder representations}
\setlength{\tabcolsep}{3.0pt}

In addition to the style transfer evaluation, we also investigated the potential utility of the representations produced by the encoder. 
To do so, we created a downstream task of audio production style classification where the goal is to classify the style of a recording among the five styles we created for the realistic style evaluation. 
We trained linear logistic regression classifiers, also known as linear probes, on top of a set of different audio representations, a common approach in evaluating the quality of self-supervised representations~\cite{zhang2017split, asano2019critical}.
These linear probes are trained for a total of 400 epochs on a small dataset with only 60 examples from each of the five styles.
Class-wise and overall F1 scores on the test set are reported in Table~\ref{tab:style-probes}.

As baseline representations we considered a random projection of melspectrogam features, along with large pre-trained representations including OpenL3~\cite{cramer2019look}, which is focused on capturing audio content and CDPAM~\cite{manocha2021cdpam}, which is focused on capturing audio quality and is therefore the most similar to our approach.
In addition, we trained linear probes on representations from the NP-HH, SPSA, and AD gradient methods on both speech styles from DAPS and music styles from MUSDB18.

In the case of DAPS, the class-wise scores indicated that CDPAM and our proposed representations were able to capture the Bright and Telephone styles. 
Differentiating between Broadcast and Neutral was difficult due to the main different between these styles being the presence of more aggressive dynamic range compression. 
However, we found that our encoder representations did not struggle to capture this difference. 
Overall, is is clear that the representations produced by the our self-supervised training framework capture not only style elements related to spectral content, but also signal dynamics, which are not captured well by the other approaches. 
We found similar results between the speech and music style classification, with the appearance that music was slightly more difficult, likely due to the starting diversity of style within MUSDB18.

\begin{table}[t!]
    \centering
    \resizebox{0.95\hsize}{!}{%
    \begin{tabular}{l c c c c c c} \toprule
        \multicolumn{7}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg}\\  \midrule
        Random Mel      & 0.87 & 0.78 & 0.73 & 0.39 & 0.00 & 0.55\\
        OpenL3          & 0.19 & 0.61 & 0.08 & 0.10 & 0.18 & 0.23\\ 
        CDPAM           & \textbf{1.00} & \textbf{1.00} & 0.79 & 0.25 & 0.63 & 0.73\\\midrule
        NP-HH S         & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\
        SPSA            & 0.95 & 0.98 & \textbf{1.00} & 0.89 & 0.95 & 0.96\\
        AD              & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\ \midrule
        \multicolumn{7}{c}{\textbf{MUSDB18}} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg} \\  \midrule
        Random Mel        & 0.80 & 0.98 & 0.62 & 0.17 & 0.00 & 0.51 \\
        OpenL3            & 0.32 & 0.66 & 0.20 & 0.17 & 0.30 & 0.33 \\ 
        CDPAM             & 0.89 & 0.95 & 0.66 & 0.00 & 0.06 & 0.51 \\\midrule
        NP-HH S         & \textbf{0.98} & \textbf{1.00} & 0.92 & \textbf{0.59} & \textbf{0.60} & \textbf{ 0.82} \\
        SPSA            & \textbf{0.98} & \textbf{1.00} & 0.90 & 0.26 & 0.00 & 0.63 \\
        AD              & \textbf{0.98} & \textbf{1.00} & \textbf{0.95} & 0.54 & 0.50 & 0.79 \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Class-wise F1 scores for five-class style prediction with linear classifiers trained on top of audio representations for speech and music using a single linear layer.}
   
    \label{tab:style-probes}\    
    \vspace{-0.0cm}
\end{table}


\setlength{\tabcolsep}{3.0pt}
\begin{table}[t]
    \centering
    \resizebox{.8\hsize}{!}{%
    \begin{tabular}{l c c c c} \toprule
       
        \textbf{Method}  &  \textbf{Train step (s)}   &  \textbf{CPU} &  \textbf{GPU} & \textbf{Parameters} \\ \midrule
        RB-DSP      & -     & 0.004 & -     & 0 \\ \midrule
        NP          & 0.434 & 0.277 & 0.005 & 336\,k  \\
        NP-HH       & 0.434 & 0.003 & -     & 0 \\
        NP-FH       & 0.434 & 0.003 & -     & 0  \\
        cTCN 1      & 0.438 & 0.132 & 0.002 & 174\,k \\
        cTCN 2      & 0.642 & 0.268 & 0.005 & 336\,k  \\
        SPSA        & 0.413 & 0.003 & -     & 0   \\
        AD          & 0.301 & 0.006 & 0.001 & 0\\
                 \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Runtime comparison across gradient methods including real-time factor for inference on CPU (Intel Xeon CPU E5-2623 v3 @ 3.00GHz) and GPU (GeForce GTX 1080 Ti). 
}
    \label{tab:runtime}
\end{table}


\section{DISCUSSION}
\label{sec:discussion}



While the results of our evaluation indicated that the AD exact DDSP approach performed best across most metrics, there are a number of important considerations for each of the the gradient methods that are not reflected in these metrics. 
For example, while the TCN-based models performed well in comparison, and are ultimately more flexible since they make no assumptions about the underlying audio effects, these system do not provide interpretable controls, and also require significant computation, making them difficult to integrate into audio production environments. 

On the other hand, the approximate DDSP approaches provide a unique benefit in that both SPSA and NP variants do not require knowledge of the underlying system or any re-implementation, nor do they explicitly require differentiability.
Since they enable the use of the original DSP implementation at inference, they also require less computation and produce interpretable control parameters. 
However, we observed that both the NP and SPSA methods suffered from other drawbacks. 

The SPSA approach proved more susceptible to training instability, thus requiring careful tuning of the $\epsilon$ hyperparameter and learning rate. 
And, while the neural proxy variants were promising, they were unable to achieve performance on par with the other approaches.
While the NP approaches do not require re-implementation or knowledge of the audio effects, they do require a laborious pre-training process. 
However, we hypothesize that their worse performance likely stemmed from the inaccuracy of the proxy networks, especially for the parametric equalizer. 
Therefore, future directions may involve the design of neural proxy architectures better suited to a wide range of audio effects. 
The AD approach was the most manual and demanding method to implement technically, but showed promise in results. This is especially the case when expert knowledge can be exploited to introduce manual modifications that simplify the system and support efficient operation during training. 








\section{CONCLUSIONS}
Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another, by example.
We demonstrated the applicability of this approach in both speech and music tasks and performed an extensive quantitative and qualitative comparison for a diverse range of differentiable signal processing methods. 
We furthermore proposed a categorization of existing approaches for differentiable signal processing, including the introduction of a differentiable dynamic range compressor, and neural proxy hybrid approaches
Our proposed system demonstrated convincing performance on both the synthetic and realistic style transfer tasks. It also estimates interpretable audio effect control parameters, can operate at variable sample rates, even those not seen during training, and learns representations that reflect  production style..

\section{ACKNOWLEDGMENT}
This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1).




\section{INTRODUCTION}
\label{sec:intro}
Audio effects are commonly used in audio and video content creation to manipulate sound attributes such as loudness, timbre, spatialization, and dynamics~\cite{wilmering2020history}. 
While audio effects can be powerful tools in the hands of trained audio engineers, their complexity poses a significant challenge for less experienced users, often requiring a time-consuming adjustment process even for professionals. 
To simplify their usage, expert-designed presets can be used to configure an audio effect, but this only results in static content-independent configurations of effects.
Therefore, presets commonly require fine-tuning and adjustment, without the ability to adapt to variations across inputs.

Automatic audio production methods aim to overcome this limitation with analysis and subsequent adaptation of effects based on an input signal~\cite{IMPbook19}. These methods can provide intelligent, automatic control of a single effect parameter, multiple effects, and/or many channels of audio in a cross-adaptive manner.
Such methods may also provide a range of interfaces, from fully automatic systems to systems that only provide suggestions.
As a result, successful automatic audio production systems have the ability not only to simplify the process for amateurs, but also expedite the workflow of professionals~\cite{moffat2019approaches}.

Automatic audio production systems are commonly categorized into rule-based systems, built on audio engineering best-practices, or data-driven, machine learning systems~\cite{moffat2019approaches}.
While rule-based systems have seen success~\cite{deman2013b}, they are limited by the inability to formalize rules that cover the diversity arising in real-world audio engineering tasks~\cite{deman2017towards}.
Classical machine learning approaches on the other hand provide greater flexibility, but have thus far been limited by an inability to collect sufficient parametric data describing audio production in a standardized way~\cite{scott2011analysis}. Recently, deep learning approaches have shown promise in overcoming these challenges, fueling an increasing interest in data-driven audio production techniques.

Deep learning approaches for automatic audio production can be organized into direct transformation, parameter prediction, and differentiable digital signal processing (DDSP) methods. 
Direct transformation methods map untreated audio to a desired target and have shown promise in tasks such as speech enhancement~\cite{pascual2017segan}, source separation~\cite{stoter2019open}, and audio effect modeling~\cite{hawley2019signaltrain, martinez2020deep,wright2020real,martinez2021deep}, but have had limited use for automatic production due to a lack of interpretable control, artifacts, and/or computational complexity~\cite{martinez2021deep}. 
Parameter prediction methods predict the settings of audio effects, but require expensive ground truth parameter data~\cite{hawley2019signaltrain} or optimize over an undesirable loss in the parameter domain~\cite{sheng2019feature, mimilakis2020one}. 
Lastly, DDSP methods~\cite{engel2020ddsp} use neural networks to control digital signal processors (DSP) implemented as differentiable operators enabling training via backpropagation. 
This approach is desirable, but requires manual implementation of audio effects, expert knowledge, and high engineering effort, potentially limiting its use. 
To work around this, two alternative methods have been proposed including neural proxies (NP) of audio effects~\cite{steinmetz2021automatic} and numerical gradient approximation schemes~\cite{ramirez2021differentiable}.
However, given that these approaches were proposed and evaluated in different tasks, it is difficult to fully understand the benefits of each method.



Beyond integrating audio effects as differentiable operators, a central limitation of deep learning systems for automatic audio production lies in the difficulty of sourcing sufficient data for supervised training, requiring both the unprocessed and final produced recordings and/or parameter data. 
In addition, the subjective and context-dependent nature of the audio production process further complicates the task~\cite{lefford2021context}. 
While evidence suggests the existence of ``best-practices''~\cite{pestana2014intelligent}, learning these techniques in a supervised paradigm is challenging.
In particular, these systems must not only learn how to control effects to achieve a desired result, they must also implicitly uncover the distribution of possible styles present within the training data across different genres and contexts.

In this work, we present \emph{DeepAFx-ST} -- a style transfer approach that can impose the audio effects and production style from one recording to another by controlling audio effects as shown in \fref{fig:style-transfer-headline}. 
Only a short example style recording is needed at inference for effects style transfer and no labeled training data or retraining is needed to adapt to new styles. 
To do so, we propose 
\begin{itemize}
    \setlength\itemsep{0.7em}
    \item The first audio effects style transfer method to integrate audio effects as differentiable operators, optimized end-to-end with an audio-domain loss, 
    \item A simplified self-supervised training procedure that enables automatic audio production without the use of label or paired training data,
    \item A benchmark of five differentiation strategies for audio effects, including compute cost, engineering difficulty, and performance. 
    \item The development of novel neural proxy hybrid methods, and a differentiable dynamic range compressor.
   
\end{itemize}
We find our approach offers several benefits including the ability to perform audio effect style transfer for speech and music signals, yield interpretable audio effects control parameters, and operate at sampling rates different than those seen during training.

The remainder of our paper is organized as follows. We outline background material in~\sref{sec:background}, discuss our proposed method in~\sref{sec:method}, review our experimental design in~\sref{sec:experimental_design}, and conclude with our results and discussion in~\sref{sec:results} and~\sref{sec:discussion}. A demonstration video and listening examples are available online\footnote{\url{https://csteinmetz1.github.io/deepafx-st}}.

\begin{figure}[t!]
    \centering
    \vspace{-0.3cm}
    \includegraphics[width=0.95\linewidth]{figures/generic-style-transfer-headline.drawio.pdf}
    \vspace{-0.3cm}
    \caption{Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another by example. We use a shared-weight encoder to analyze the input and a style reference signal, then compare each with a controller that outputs the parameters of effects that themselves perform style manipulation.}
    \label{fig:style-transfer-headline}
\end{figure}


\section{BACKGROUND}
\label{sec:background}


\subsection{Audio Effects and Production Style Transfer}
Relevant works on audio production style transfer include initial work on controlling a dynamic range compressor~\cite{sheng2019feature} with a multi-stage training procedure using neural networks and random forest, as well as a deep neural network approach to control a parametric frequency equalizer~\cite{mimilakis2020one}. 
Both works, however, only consider controlling one audio effect in isolation (compressor or parametric equalization) and use a (quantized) parameter domain loss to avoid backpropagating through differentiable audio effects -- a non-trivial task. 
This approach has notable disadvantages including undesirable quantization schemes or loss tuning and low correlation between parameters and audio, both of which limit performance and generalization across effects. 
However, differentiable signal processing provides the potential to overcome these challenges and has yet to be investigated for the style transfer task. 

\subsection{Differentiable Audio Effects}\label{sec:diff_fx}
Typically, differentiable digital signal processors are manually implemented within automatic differentiation frameworks~\cite{abadi2016tensorflow, paszke2019pytorch}.
Assuming the mathematical operations in question are differentiable, this provides a straightforward way to integrate DSP operations into neural networks. 
This approach, however, requires expert knowledge, design trade-offs, and/or can be difficult or impossible to implement exactly. Examples of such audio effects include infinite impulse response (IIR) filters~\cite{kuznetsov2020differentiable, nercessian2020neural, colonel2021iirnet}, hearing aids~\cite{tu2021dhasp}, echo cancellers~\cite{casebeer2021auto}, DJ transitions~\cite{chen2021automatic}, and reverse engineered effects~\cite{kuznetsov2020differentiable, nercessian2021lightweight,colonel2021reverse}. 
Similarly, recent work constructing audio synthesis models with DDSP components include additive~\cite{engel2020ddsp}, subtractive~\cite{masuda2021synthesizer}, waveshaping~\cite{hayes2021neural}, and wavetable~\cite{shan2021differentiable} synthesizers. 

Differentiable audio effects of interest to our work include the parametric frequency equalizer (PEQ) and dynamic range compressor (DRC) -- two common audio production effects.
To our knowledge, there is no known past work on differentiable compressors, but differentiable PEQs do exist~\cite{nercessian2020neural}. 
PEQs are typically constructed with a cascade of multiple second-order IIR filters, also know as biquads~\cite{valimaki2016all}. 
While it is possible to implement differentiable IIR filters in the time-domain, the recursive filter structure can cause issues as a result of vanishing/exploding gradients and computational bottlenecks during backpropagation through time (BPTT)~\cite{hochreiter1997long}. This motivates frequency-domain finite impulse response (FIR) approximations~\cite{nercessian2020neural, nercessian2021lightweight, colonel2021iirnet}. In this case, the complex frequency response of the $k^{\textrm{th}}$ biquad $H_k(e^{j \omega})$ is expressed as a ratio of the DFT of the numerator and denominator coefficients, $\mathbf{b}_k = [b_{0,k}, b_{1,k}, b_{2,k}]$ and $\mathbf{a}_k = [a_{0,k}, a_{1,k}, a_{2,k}]$,
\begin{equation}
    H_k(e^{j \omega}) = \frac{ \textrm{DFT}(\mathbf{b}_k) }{ \textrm{DFT}(\mathbf{a}_k) } = \frac{\sum_{m=0}^{2} \ b_{m,k} \ e^{-j\omega m}}{\sum_{n=0}^{2} \ b_{n,k} \ e^{-j \omega n}}.
\end{equation}
The overall response of a cascade of $K$ biquad filters is then given by the product of their responses $H_{sys} (e^{j \omega}) = \prod_{k=1}^{K} H_k(e^{j \omega})$.
We can approximate the response using an FIR filter by evaluating $H_{sys}(e^{j \omega})$ over a linearly spaced, zero-padded frequencies of length $2^{\lceil \log_2 (2N - 1) \rceil}$, applying the filter(s) to an input $x[n]$ with $N$ samples in the frequency domain by multiplying with  the input $ X(e^{j \omega})$, 
and then performing the inverse DFT to get the output $y[n]$.


\subsection{Alternative Differentiation Methods}

In contrast to manually implementing DDSP algorithms, three alternative approaches are of note. In neural proxy (NP) approaches, a neural network is trained to emulate the behavior of a signal processor~\cite{grathwohl2017backpropagation, jacovi2019neural} including control parameter behavior using a dataset of input/output measurements. 
While most work in this area has focused on creating emulations of analog audio effects~\cite{hawley2019signaltrain, damskagg2019deep, martinez2019modeling, wright2019real, martinez2020deep, wright2020real, martinez2021deep, steinmetz2021efficient}, this approach can be used together with larger neural network for automatic control of effects, as was demonstrated with applications in automatic mixing~\cite{steinmetz2021automatic}.
Since neural proxy models are composed of neural network building blocks, they are differentiable by default, but require accurate emulation to work effectively and can be computationally complex.  

In an effort to reduce inference time complexity and reduce approximation error, neural proxy models can be used together with DSP to create what we call neural proxy hybrid models~\cite{tseng2019hyperparameter}. In this case, neural proxies are only used for gradient approximation to reduce approximation error caused by the neural proxy. We denote full hybrid approaches as methods that employ the original DSP during training (forward pass of backpropagation) as well as inference and half hybrid models as methods that employ the original DSP only during inference, which are easier to implement. Past neural proxy hybrid approaches, however, have only been used outside audio in hyper-parameter optimization for fixed datasets, and not in an adaptive manner.

As a third alternative, non-differentiable DSP implementations can be used directly with numerical gradient approximation methods, which has been demonstrated in audio effect modeling, removal of breaths, and music mastering~\cite{ramirez2021differentiable}. 
In context of controlling audio effects $h(x, p)$ with input audio $x$ and parameters $p \in \mathbb{R}^{P}$, we only need to estimate the partial derivatives for each parameter $\frac{\partial}{\partial p_i} h(x,p)$. 
For this, finite differences (FD)~\cite{milne2000calculus} can be used by perturbing each input parameter forward and backward and evaluating the operator $2P$ times for $P$ control parameters. 
Alternatively, simultaneous permutation stochastic approximation (SPSA)~\cite{spall1998overview} gradient estimation only requires two function evaluations per gradient vector estimate. 
In this case, the approximate partial derivative with respect to $p_i$ is given by 
\begin{equation}
\frac{\hat{h}(x, p_i)}{p_i} = \frac{h(x, p + \epsilon \Delta^P) - h(x, p - \epsilon  \Delta^P)}{2 \epsilon \Delta^{P}_{i}},
\end{equation}
where $\epsilon$ is a small, non-zero value and $\Delta^P \in \mathbb{R}^P$ is a random vector sampled from a symmetric Bernoulli distribution ($\Delta^P_i = \pm 1$)~\cite{spall1992multivariate}.  
This approach does not require pre-training or knowledge of the DSP, but can be challenging to train due to inaccuracies in gradient approximation and overhead from implementations available only on CPU.

\begin{figure*}[t!]
    \centering
    \vspace{-0.2cm}
    \includegraphics[width=0.95\linewidth]{figures/training-lines.drawio.pdf}
    \vspace{-0.0cm}
    \caption{The self-supervised training process for our \emph{DeepAFx-ST} approach.
   
    We sample $x$ from a large dataset of recordings $\mathcal{D}$ and apply random augmentations.
    Recordings with two different styles are produced with randomly configured DSP, which become the input $x_i$ and the style reference $x_r$.
    We split these recordings in half, and pass the opposite sections ($x_{i,a}$ and $x_{r,b}$) to shared-weight encoders $f_\theta$. 
    Using the encoder representations ($e_i$ and $e_r$), the controller $g_\phi$ estimates parameters $p$ to apply the reference style to the input recording.}
    \label{fig:data-generation}
    \vspace{-0.0cm}
\end{figure*}

\section{PROPOSED METHOD}
\label{sec:method}

\subsection{Model Architecture}
\label{sec:arch}
Our model architecture is designed to emulate the creative process of an audio engineer and consists of three sub-processes including acoustic analysis, style determination, and parameter adjustment as shown in~\fref{fig:data-generation} (Right).

\linesubsec{Encoder} 
Our encoder $f_\theta$ extracts audio production style information from both the input and style reference signal. To do so, it inputs a time domain audio signal and converts it to a 2D representation or spectrogram $X$ using a short-time Fourier transform (STFT) with window size $N$ and hop size $H$. 
Each spectrogram is fed to a shared-weight convolutional neural network, which outputs a time series of representations or embedding sequence per signal. 
Temporal average pooling is applied to aggregate across time, producing a single embedding of dimension $D$ for each input recording. 
The input and style embeddings, $e_i$ and $e_r$, are then concatenated and passed to the controller.

\linesubsec{Controller} Our controller $g_\phi$ takes the concatenated input and style embeddings ($e_i$ and $e_r$) and outputs audio effect parameters $p$. 
To operate correctly, our controller must learn to compare the extracted encoder embeddings, and map the input signal to the desired target style by adjusting the control parameters.
In our formulation, the controller is a simple three layer multi-layer perceptron (MLP) featuring a single hidden layer, along with PReLU~\cite{he2015delving} activation functions. 
At the output, the hidden representation is projected into the audio effect control parameter space $\mathbb{R}^P$, and a sigmoid activation is applied. 

\linesubsec{Differentiable audio effects} 
Finally, we use differentiable audio effects $h(x, p)$ to process the input audio signal, given the predicted parameters from the controller. 
In contrast to past work~\cite{mimilakis2020one, sheng2019feature}, we fully integrate audio effects as differentiable operators or layers with standard automatic differentiation tools and enable backpropagation through effects during training as shown with dashed lines in Figure~\ref{fig:data-generation}. 
We do this to enable end-to-end training using an audio-domain loss and directly avoid difficulties in past works as discussed in~\sref{sec:background}.

\subsection{Differentiation Methods}
\label{sec:diff_methods}
Core to our approach is the integration of audio effects directly within the computation graph of a neural network. 
To enable our style transfer approach to work with audio effects with known implementations (white-box), unknown implementations (black-box), neural implementations, and/or DSP implementations (e.g. plugins), we propose backpropagating through audio effects using five unique differentiation strategies.
These include manually implemented automatic differentiation effects (AD), neural proxy effects (NP), full neural proxy hybrids (NP-FH), half neural proxy hybrids (NP-HH), and numerical gradient approximations (SPSA).
While AD, NP, and SPSA methods have previously been employed in automatic audio production tasks, NP-FH has only been used in static image processing hyperparameter optimization~\cite{tseng2019hyperparameter}, and NP-HH has not been proposed before. 
Moreover, all approaches have never been compared in a unifying manner, leaving uncertainty around their relative efficacy.

\subsection{Self-supervised Training}
\label{sec:ssl}

Our self-supervised training strategy enables learning control of audio effects without the need for any labeled or paired training data. 
Our data generation process is shown on the left of~\fref{fig:data-generation} and begins by selecting a random audio recording $x$ from a dataset $\mathcal{D}$. 
We then create an augmented version $x_{aug}$ by applying pitch shifting and time stretching, and
then produce two unique versions of this signal by applying a set of randomly configured audio effects.
This yields the input recording $x_i$ in the top branch, and the style reference $x_r$ in the bottom branch, each with a different production style. Our model is then tasked with estimating parameters $p$ to configure the effects $h(x,p)$ so that the processed input signal matches the style reference.

To train our model to only focus on audio production style and operate even when the content differs between the input and style reference, we split the recordings $x_i$ and $x_r$ in half, generating an $a$ and $b$ section for each or $x_{i,a}$, $x_{i,b}$, $x_{r,a}$, and $x_{r,b}$.
During training, we randomly select either the $a$ or $b$ section to be passed as input and use the other section as the style reference.
To compute an audio-domain loss, we use the style reference corresponding to the same section as the input in order to have a paired input and ground truth recording with the same content.  
Unlike typical audio training, our approach automatically generates paired data and can utilize any audio dataset, preferably with a wide range of quality and recording conditions. 

\section{EXPERIMENTAL DESIGN}
\label{sec:experimental_design}


\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth, trim={0.0cm 0.18cm 0.0cm 0.1cm},clip ]{figures/signal-chain.drawio.pdf}
    \caption{Our audio effect signal chain consisting of a 6-band parametric equalizer and dynamic range compressor.}
    \label{fig:signal-chain}
    \vspace{-0.0cm}
\end{figure}

\subsection{Problem Setup}
\label{sec:setup}

To demonstrate our approach, we focus on the task of imposing the equalization and loudness dynamics from one recording to another using a parametric equalizer and dynamic range compressor as shown in~\fref{fig:signal-chain}. 
This forms the basic signal chain employed by audio engineers with applications in dialogue processing or in music mastering.
We manually implement automatically differentiable PEQ and DRC and compare style transfer performance using alternative differentiation strategies as noted in~\sref{sec:arch}. 
By doing so, we seek to both understand the performance of our style transfer approach as well as the advantages and disadvantages of differentiation strategies.


\subsection{Differentiable Equalizer and Compressor}
\label{sec:eq_drc}
We implement a differentiable PEQ following past work~\cite{nercessian2020neural}, but develop own novel differentiable dynamic range compressor. 
For our differentiable compressor, we use a traditional feedforward dynamic range compressor, which is in fact composed of differentiable operations~\cite{giannoulis2012digital}.
Similar to PEQs, however, we found direct implementation of existing compressor designs problematic due to the presence of recursive filters, in this case due to the smoothing filters for the ballistics,
\begin{equation} \label{eq:smoothing}
    \resizebox{.9\hsize}{!}{
   $y_L[n] =
    \begin{cases}
        \alpha_A y_L[n-1] + (1-\alpha_A) x_L[n] & x_L[n] > y_L[n-1] \\
        \alpha_R y_L[n-1] + (1-\alpha_R) x_L[n] & x_L[n] \leq y_L[n-1]
    \end{cases}$,       
    }
\end{equation}
where $y_L[n]$ is the smoothed gain reduction, $x_L[n] = x_G[n] - y_G[n]$ is the result of subtracting the output of the gain computer from the signal envelope, and $\alpha_A$ and $\alpha_R$ are the attack and release time constants. 
This operation is not easily approximated with a FIR filter, thus we simplify our design using a single attack and release time constant $\alpha$ to create a single-pole IIR ballistics filter,
\begin{equation}
   y_L[n] = \alpha y_L[n-1] + (1-\alpha) x_L[n].
\end{equation}
This filter can be approximated using an FIR filter by employing the same approach as in differentiable PEQs.
While this modification restricts the capabilities of our compressor, forcing the effective attack and release times to be shared, we seek only a basic compressor to show the complexity in implementing differentiable versions of common audio effects, which we compare against competing alternative differentiation strategies. 

\subsection{Datasets}
\label{sec:datasets}

We employ our data generation pipeline as described in~\sref{sec:method} using both speech and music.
To test our approach on speech, we use \texttt{train-clean-360}, a subset of LibriTTS~\cite{zen2019libritts}, which totals 360 hours of audio at $f_s = 24$\,kHz.
To test our approach on music, we use the MTG-Jamendo dataset~\cite{bogdanov2019mtg}, which provides over 55\,k songs, which we downmix to mono and resample to $f_s = 24$\,kHz. 
For both datasets, we generate a $90$/$5$/$5$ split of the recordings for training, validation, and testing. When randomly sampling patches of audio from a recording, if the selected segment $x[n]$ has a mean energy below the threshold, $\frac{1}{N} \sum_{n=1}^{N} | x[n] |^2 \leq 0.001$), we continue sampling until a non-silent segment is found.
To ensure that there is sufficient headroom for processing within the audio effects, we peak normalize all inputs and style recordings to $-12$\,dBFS.

\subsection{Baselines}
\label{sec:baselines}

\linesubsec{Rule-based DSP}
\label{sec:dsp}
Our rule-based baseline consists of an automatic, two stage process that aims to mimic the signal chain of the PEQ and DRC. 
First, an FIR filtering stage is employed, which aims to transform the spectral content of the input to match the spectrum of the style reference. 
We compute the average magnitude spectrum of the input and style reference by computing the STFT using a large window of $N=65536$ and a hop size of $H=16384$, averaging across the frames, yielding $X_i$ and $X_r$, which
are smoothed using a Savitzky-Golay filter~\cite{savitzky1964smoothing}.
We design an inverse FIR filter with $63$ taps using the window method, with a desired magnitude response given by $X = X_{i,sm} / X_{r,sm}$, where $X_{i,sm}$ and $X_{r,sm}$ are the smoothed average spectra of the input and the style reference.

The filtered signal is sent to a dynamic range compressor where all parameters except for the threshold are fixed. 
In order to adjust the amount of compression based on the input signal and the style reference, the threshold starts at 0 dB and is lowered in increments of 0.5 dB.
At each iteration the loudness in dB LUFS~\cite{steinmetz2021pyloudnorm} of the output is measured and compared to the loudness of the style reference.
If the difference in the loudness measurements are below a given threshold (e.g 0.5 dB LUFS), the process halts. 
Otherwise, the process continues until the loudness is within a specific tolerance, or the threshold reaches -80 dB.

\linesubsec{End-to-end neural network}
\label{sec:e2e}
As a baseline on the opposite end of the spectrum, we evaluate a direct, end-to-end neural network approach to replace the DDSP audio effects. 
This network utilizes the same architecture as our neural proxy approaches, but does not use pre-trained weights. 
Instead, the weights of this model are optimized during the training of the encoder and controller. 
The controller still predicts parameters for this network, as in the case of a neural proxy, since the weights of the network are not frozen during training.
As a result the model will learn its own set of control parameters. 
We evaluate two variants of this approach, first using a single TCN network (cTCN 1) and a second using two TCNs connected in series (cTCN 2), which mimics the setup in the neural proxy approach.



\subsection{Losses}
We train both our neural proxy models and style transfer models using an audio-domain loss.
In particular, we compute our loss using the model output and ground truth audio from our data generation process with a weighted sum of the error in time and frequency domains.
The time domain loss $\mathcal{L}_{\text{time}}$ is the mean absolute error (MAE) and the
frequency domain loss $\mathcal{L}_{\text{freq}}$ is the multi-resolution short-time Fourier transform loss (MR-STFT)~\cite{wang2019neural, steinmetz2020auraloss}.
This loss is the sum of the $L_1$ distance between the STFT of the ground truth and estimated waveforms measured both in the $\log$ and linear domains, at multiple resolutions, in this case with window sizes $W \in [32, 128, 512, 2048, 8192, 32768]$ and hop sizes $H = W/2$.
The overall loss is  $\mathcal{L}_{\text{overall}}({\hat{y}},y) = \mathcal{L}_{\text{freq}}({\hat{y}},y) + \alpha \cdot \mathcal{L}_{\text{time}}({\hat{y}},y) \label{eq:overall-loss}$, where $\alpha = 100$.
 

\setlength{\tabcolsep}{3.0pt}
\setlength{\abovecaptionskip}{15pt plus 3pt minus 2pt}
\begin{table*}[t!]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
        \multicolumn{21}{c}{\normalsize{\textbf{Speech}}} \\ \midrule
        & \multicolumn{6}{c}{\textbf{LibriTTS}} & & \multicolumn{6}{c}{\textbf{DAPS}} & & \multicolumn{6}{c}{\textbf{VCTK}} \\ \cmidrule(lr){2-7} \cmidrule(lr){8-14} \cmidrule(lr){15-21}
        \textbf{Method}  &  \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}}\\ \midrule
        {Input}   
        & 3.765           & 1.187           & 2.180           & 687.5           & 6.983           & 2.426           & 
        & 3.684           & 1.179           & 2.151           & 641.7            & 6.900          & 2.314          & 
        & 3.672           & 1.254           &          2.008  & 815.4           & 7.783           & 2.532\\
        {RB-DSP}  
        & 3.856           & 0.943           & 1.955           & 410.3           & 4.204           & 1.674           & 
        & 3.787           & 0.917           & 1.882           & 399.7            & 3.705           & 1.481          & 
        & 3.709           & 1.101           &          1.911  & 657.6           & 5.039           & 2.018\\ \midrule
       
        {cTCN 1}    
        & 4.258           & 0.405           & 0.887           & 128.4           & 2.237           & 1.066           & 
        & 4.185           & 0.419           & 0.884           & 124.6            & 2.098           & 1.006          & 
        & 4.181           & 0.467           &          0.891  & 173.8           & 2.651           & 1.165\\
        {cTCN 2}    
        & 4.281           & 0.372           & \textbf{0.833}  & 117.5           & 1.927           & 0.925           & 
        & \textbf{4.224}  & \textbf{0.391}  & \textbf{0.841}  & 113.9            & 1.886           & 0.913          & 
        & 4.201           & \textbf{0.441}  & \textbf{0.856}  & 163.8           & 2.431           & 1.086\\ \midrule
       
        {NP}   
        & 3.643           & 0.676           & 1.405           & 265.0           & 2.812           & 1.340           & 
        & 3.605           & 0.685           & 1.362           & 249.2            & 2.732           & 1.350          & 
        & 3.651           & 0.737           &          1.300  & 321.7           & 3.166           & 1.453\\
        {NP-HH}   
        & 3.999           & 1.038           & 2.179           & 440.2           & 5.472           & 2.679           & 
        & 3.903           & 1.022           & 2.113           & 451.9            & 5.104           & 2.535          & 
        & 3.951           & 1.044           &          1.930  & 591.5           & 5.194           & 2.651\\
        {NP-FH}   
        & 3.945           & 1.058           & 2.088           & 404.9           & 6.820           & 3.197           & 
        & 3.891           & 1.037           & 2.045           & 395.4            & 6.754           & 3.117          & 
        & 3.894           & 1.087           &          1.934  & 514.0           & 7.065           & 3.363\\
       
        {SPSA}      
        & 4.180           & 0.635           & 1.406           & 219.5           & 3.263           & 1.600           & 
        & 4.099           & 0.645           & 1.379           & 213.6            & 2.989           & 1.511          & 
        & 4.023           & 0.730           &          1.359  & 301.6           & 3.535           & 1.737\\
       
        \midrule
        {AD}  
        & \textbf{4.310}  & \textbf{0.388}  & 0.882           & \textbf{111.5}  & \textbf{1.828}  & \textbf{0.823}  & 
        & 4.222           & 0.416           & 0.895           & \textbf{109.0}   & \textbf{1.758}  & \textbf{0.799} & 
        & \textbf{4.218}  & 0.481           &          0.924  & \textbf{152.7}  & \textbf{2.317}  & \textbf{1.006} \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
\caption{Synthetic style transfer with models trained using LibriTTS. Held-out speakers from the LibriTTS dataset are used, while utterances from DAPS and VCTK come from datasets never seen during training. Lower is better for all metrics except PESQ.}
    \vspace{-0.1cm}
    \label{tab:style-transfer-synthetic-speech}
\end{table*}

\subsection{Neural Proxy Pre-training}
To construct our neural proxy-based models, we utilized a temporal convolutional network (TCN)~\cite{bai2018empirical}, also known as the feedforward WaveNet~\cite{rethage2018wavenet}. This model has seen success in modeling distortion~\cite{wright2019real} and compressors~\cite{steinmetz2021automatic}.
In particular, we use a causal TCN~\cite{steinmetz2021automatic} with feature-wise linear modulation (FiLM)~\cite{perez2018film} to adapt to the effect control parameters.
Our TCN configuration has $4$ blocks, kernel size of $13$, channel width of $64$ and a dilation factor that grows as a power of $8$, where the dilation factor at block $b$ is $d_b = [ 1, 8, 64, 512 ]$.
We trained a total of 4 proxies, one for each of the two effects, with one for both speech and music. 
Training examples were generated by using DSP effects with uniformly random parameters. 
We trained each model for 400 epochs, where one epoch was defined as 20,000 samples, and used the checkpoint with the lowest validation loss.
We used an initial learning rate of $3\cdot 10^{-4}$, a batch size of 16, and inputs $\approx 3$\,s ($65536$ samples) in length. 
The learning rate was lowered by a factor of 10 after the validation loss had not improved for 10 epochs. 

\subsection{Style Transfer Training}
After training the neural proxy models, we trained our style transfer model using five different audio effects differentiation methods -- AD, NP, NP-HH, NP-FH, SPSA, and end-to-end cTCN1 and cTCN2. 
For our encoder, we used an STFT with a Hann window of size $N=4096$ and a hop size of $H=2048$ as input into a randomly initialized EfficientNet B2 CNN~\cite{tan2019efficientnet} with a single input channel. 
The final activations are projected to $D=1024$ with a linear layer, followed by temporal mean pooling to produce a single representation, which is normalized by the $L_2$ norm.
We trained all models for 400 epochs with a batch size of 6, where a single epoch is defined as 20,000 segments of audio $\approx 10$ seconds in length ($262144$ samples). 
We used a learning rate of $10^{-4}$ for all models, with the exception of the SPSA models, which required a lower learning rate of $10^{-5}$ for stability. 
In both cases, we scheduled the learning rate such that it is lowered by a factor of 10 at both $80\%$ and $95\%$ through training.
We applied gradient clipping with a value of $4.0$ and used an SPSA $\epsilon$ of $5 \cdot 10^{-4}$.




\setlength{\tabcolsep}{2.5pt}
\begin{table*}[ht]
    \centering
    \resizebox{\linewidth{}}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
    \multicolumn{21}{c}{\normalsize{\textbf{Music}}} \\ \midrule
        &  
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (24 kHz)}}   & & 
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (44.1 kHz)}} & & 
        \multicolumn{6}{c}{\textbf{MUSDB18 (44.1 kHz)}} \\ 
        \cmidrule(lr){2-7} \cmidrule(lr){9-14} \cmidrule(lr){15-21}
        \textbf{Method}   & 
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
       
        Input      
        & 2.927  & 1.198  & 6.088  & 646.5  & 6.695  & 2.518 & 
        & 2.874  & 1.109  & 4.454  & 767.7  & 6.793  & inf   & 
        & 2.900  & 1.252  & 4.342  & 1088.3    & 5.940  & 2.312    \\ 
        RB-DSP     
        & 2.849  & 0.925  & 4.422  & 263.2  & 4.254  & 1.706  & 
        & 2.931  & 0.887  & 3.355  & 342.0   & 4.749  & 1.882 & 
        & 2.994  & 0.821  & 3.052  & 379.4   & 4.078 & 1.665  \\
       
        \midrule
        cTCN 1       
        & 3.402  & 0.547  & 2.294  & 160.9  & 3.261  & 1.483 &
        & 3.168  & 0.876  & 2.921  & 494.4    & 4.372  & 2.070  & 
        & 3.121  & 0.896  & 2.956  & 730.4    & 4.548  & 2.231  \\ 
        cTCN 2       
        & 3.390  & 0.548  & 2.278  & 152.3  & 2.951  & 1.397 & 
        &  3.123  & 0.903  & 2.973  & 517.4   & 4.084  & 1.899  &
        &  3.107  & 0.917  & 2.986  & 749.5   & 4.208  & 2.061 \\  \midrule
       
        NP S  
        & 2.926  & 0.787  & 2.838  & 221.3  & 2.785  & 1.390 &  
        & 2.764  & 1.033  & 2.869  & 488.3  & 3.710  & 1.824 & 
        & 2.804  & 0.950  & 2.778  & 742.1   & 3.835  & 1.921  \\
        NP M  
        & 2.765  & 0.845  & 3.211  & 255.2  & 3.227  & 1.608 & 
        & 2.699  & 1.042  & 2.928  & 497.0    & 3.942  & 1.961   & 
        & 2.791  & 0.946  & 2.800  & 757.1   & 4.209  & 2.127    \\
        NP-HH (S)  
        & 2.819  & 1.092  & 6.791  & 395.3  & 6.276  & 3.032 & 
        & 2.865  & 1.101  & 5.194  & 689.2    & 6.792  & 3.365 & 
        & 2.853  & 1.165  & 4.852  & 1005.7   & 6.451  & 3.269 \\
        NP-HH (M)  
        & 2.532  & 1.166  & 7.070  & 591.9  & 5.660  & 2.593 & 
        & 2.512  & 1.148  & 5.385  & 854.9   & 5.940  & 2.740 & 
        & 2.493  & 1.198  & 5.090  & 1021.9    & 5.585  & 2.731 \\
        NP-FH (S)  
        & 2.833  & 1.016  & 5.005  & 280.8  & 3.377  & 1.634 & 
        & 2.888  & 0.977  & 3.883  & 429.9    & 3.480  & 1.709   & 
        & 2.857  & 1.045  & 3.809  & 617.3   & 3.932  & 1.971   \\
        NP-FH (M)  
        & 2.648  & 1.137  & 6.368  & 605.6  & 5.903  & 2.587 & 
        & 2.625  & 1.133  & 5.512  & 844.6   & 6.417  & 2.876  & 
        & 2.575  & 1.225  & 5.450  & 1172.5   & 5.973  & 2.913   \\
        SPSA         
        & 3.173  & 0.716  & 3.024  & 210.1  & 2.809  & 1.344 & 
        & 3.172  & 0.759  & 2.458  & 386.2   & 2.839  & 1.311 & 
        &  3.126  & 0.789  & 2.321  & 574.4   & 2.925  & 1.394    \\
       
        \midrule
        AD     
        & 3.355  & 0.488  & 2.149  & 144.7  & 2.167  & 1.005   & 
        &  3.400  & 0.585  & 1.824  & 304.4    & 2.425  & 1.106 & 
        &  3.396  & 0.608  & 1.695  & 456.1    & 2.559  & 1.197  \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Synthetic style transfer with models trained using MTG-Jamendo dataset. Proxy models with (S) were pre-trained using speech from LibriTTS those with (M) were trained using music from the MTG-Jamendo dataset. Lower is better for all metrics except PESQ.}
    \label{tab:style-transfer-synthetic-music}
    \vspace{-0.2cm}
\end{table*}

\section{RESULTS}
\label{sec:results}



\subsection{Synthetic Style Transfer}
We first evaluate on a synthetic style transfer task, which mirrors our training setup. 
This facilitates the use of full-reference metrics, since we synthetically generate style references and as a result have inputs and ground truth with matching content.
Results of the evaluation using held-out speakers from LibriTTS are shown in \tref{tab:style-transfer-synthetic-speech}.

As there are no established metrics for evaluating audio production style, we propose a range of metrics. 
For a general notion of perceptual similarity we use PESQ~\cite{rix2001perceptual}, along with the Multi-resolution STFT~\cite{wang2019neural} error, which is a component of the training objective. 
To evaluate the ability of the system to address equalizer-related style components, we report the mel-spectral distance (MSD), which consists of the error between melspectrograms using a large window size of $W=65536$, along with the spectral centroid error (SCE). 
To evaluate aspects related to the dynamic range, we consider the RMS energy error (RMS) and the perceptual loudness error (LUFS)~\cite{steinmetz2021pyloudnorm}. 

We note our rule-based DSP baseline approach improves on the input across all metrics, though not by a significant margin, indicating it has limited success.
Overall, we find strong performance from the AD method, which performs best on all metrics except MSD, which is only slightly outperformed by cTCN2.
Both cTCN approaches follow closely behind AD, demonstrating the strength of end-to-end neural network approaches. 
The SPSA model follows closely behind these approaches with somewhat degraded PESQ and STFT error, yet it outperforms all NP methods. 
While the NP methods are not as competitive, both NP-HH and NP-FH outperform NP in terms of PESQ, likely a result of the use of the DSP implementation, which imparts less artifacts than the neural proxy.
However, the NP approach achieves better performance across all other metrics, indicating it likely better matches the style reference. 

In addition to evaluating on held-out speech from LibriTTS, we also evaluate generalization with speech datasets not seen during training including DAPS~\cite{mysore2014can} and VCTK~\cite{yamagishi2019vctk}.
We observe a similar trend in performance, yet the cTCN models slightly outperform AD as compared to on LibriTTS.
This indicates our self-supervised training was sufficient for producing models that generalize to speech from different sources and acoustic conditions. 

We also trained the same set of models using music from the MTG-Jamendo dataset. 
The result, shown in \tref{tab:style-transfer-synthetic-music}, largely follows a similar trend to the speech experiments with regards to the differentiation methods, demonstrating the general nature of our approach. 
However, it appears that the music task is somewhat more difficult.
We also compared neural proxies trained on speech (S) or music (M), and found those trained on music performed slightly worse.

We then investigated the ability of our methods to operate at a sample rate different than training. 
Signals to the encoder were first resampled to match training (24\,kHz), however the audio effects operate at 44.1\,kHz.
First, we evaluated on the MTG-Jamendo test set, and we noticed a significant drop in performance for the neural network based approaches, which include the cTCNs and NP, since these models operate at a fixed sample rate. 
However, the DSP-based approaches performed significantly better, including both the AD and SPSA approaches, which performed only slightly worse than at the training sample rate.
To test out-of-distribution generalization we used music from MUSDB18~\cite{musdb18}.
While performance does decrease across most metrics and for all models, they do not indicate that the models fail to perform the style transfer task, with the SPSA and AD approaches exceeding the baseline. 

\subsection{Realistic Style Transfer}
\setlength{\tabcolsep}{3.0pt}

To simulate a more realistic style transfer scenario where the style reference contains not only different content but also differing sources (e.g. speaker identity or instrumentation), we construct a realistic style transfer evaluation task.
We designed five different audio production ``styles'', which are defined by hand-crafted parameter ranges and inspired by commonly used plugin presets.
We generate audio from five different styles (Telephone, Warm, Bright, Neutral, and Broadcast) by randomly sampling parametric equalizer and compressor parameters from a set of predefined ranges such that each style is audibly unique.
Sampling parameter values randomly provides more variation within audio examples of the same style, which is more complex than applying a fixed preset. 

To generate examples in each of these styles, we use speech from DAPS, and music from MUSDB18.
However, since we do not have paired ground truth as in the synthetic style transfer case, we performed our evaluation using only non-intrusive metrics that do not require a matching reference. 
These metrics measure the error between high-level audio features, like the MSD, spectral centroid, RMS energy, and perceptual loudness.


\begin{table}[t!]
    \centering
    \resizebox{1.0\hsize}{!}{%
    \begin{tabular}{l c c c c c c c c c } \toprule
        & \multicolumn{4}{c}{\textbf{DAPS}} & & \multicolumn{4}{c}{\textbf{MUSDB18}} \\  \cmidrule(lr){2-5} \cmidrule(lr){6-10}
        \textbf{Method} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Input}   & 10.4        & 1041.7        & 10.4          & 3.6   & & 8.4         & 2607.4      & 9.4        & 3.8   \\ 
        {RB-DSP}  & 8.9         & 517.2         & 5.8           & 2.4   & & 6.5         & \textbf{915.4}       & 8.6         & 3.7   \\  \midrule
        {NP-HH S} & 9.8         & 636.6         & 12.9          & 5.9   & & 7.0         & 1512.7      & 10.0        & 4.5          \\
        {SPSA}    & 8.0         & 360.1         & \textbf{5.1} & 2.5    & & 5.5         & 1297.0      & 4.6         & 2.1         \\
        {AD}      & \textbf{7.8}& \textbf{278.1}& 5.2  & \textbf{2.4}   & & \textbf{4.8} & 947.6       & \textbf{3.8} & \textbf{1.7}             \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Average performance across all pairwise style transfer configurations using input utterances from DAPS (trained on LibriTSS) and music from MUSDB18 (trained on MTG-Jamendo). }
    \vspace{-0.6cm}
    \label{tab:style-transfer-real}
\end{table}

We performed all pairwise style transfers resulting in 25 configurations, and we performed each configuration ten times using different input and output recordings from each style. 
Here we considered only the best performing methods that produce interpretable control parameters (NP-HH, SPSA, and AD).
To summarize the results of this evaluation, we aggregated the metrics across all 25 configurations reporting the mean for each to provide a general notion of the style transfer performance, as shown in \tref{tab:style-transfer-real}. 

On examples from DAPS, all methods, including the baseline, improve upon the input, but SPSA and AD clearly outperform the other approaches across all metrics, with AD preforming the best. 
In the case of MUSDB18, the AD approach further outperforms the others, with the exception of SCE, where the RB-DSP baseline performs best.



\subsection{Parameter Visualization}
To demonstrate the ability of our approach to provide precise control over the effect parameters, we plot both the overall frequency response of the parametric equalizer as well as the compressor configuration.
In~\fref{fig:vctk-variable-style}, we show the result of using one utterance from VCTK as the input with five different style references.
Since the utterances from VCTK are relatively clean and contain little post-processing we observed the estimated parameters aligned with our expectation for the predefined styles. 
For example, we find that the telephone style is achieved by reducing the high and low frequencies while boosting frequencies around 1\,kHz.
In terms of compression, we see a clear trend with the broadcast style using the most aggressive compression, as expected. 
While the task of estimating these style parameters may seem trivial, the system was not trained on these recordings and has no knowledge of the styles. 

\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (left) and compressor (right) parameters from our SPSA model with one utterance from VCTK and five unique style references.}
    \label{fig:vctk-variable-style}
\end{figure}

\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314_modify.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (left) and compressor (right) parameters with our SPSA model using a fixed Broadcast style reference. The input VCTK utterance was progressively modified to increase the high frequency energy and compression.}
    \label{fig:vctk-variable-input}
    \vspace{-0.2cm}
\end{figure}

As a demonstration of the ability of the model to adapt to the input signal like an \emph{adaptive preset}, we used a single style recording from our generated Broadcast style, and varied characteristics of the input. 
In this case, the input signal was modified by progressively increasing the gain of a high-shelf filter from 0\,dB to +24\,dB, while also increasing the amount of compression with a higher ratio (1.0 to 4.0) and lower threshold (0\,dB to -62\,dB).
As shown in \fref{fig:vctk-variable-input}, the estimated parameters adapt to these changes in a sensible manner. 
This results in a progressively lower boost in the high-shelf filter that was added originally to achieve the presence of the Broadcast style, while also progressively increasing the threshold and decreasing the ratio to reduce the amount of compression applied, as the input becomes more compressed. 

\subsection{Encoder Representations}
\setlength{\tabcolsep}{3.0pt}

In addition to our style transfer evaluation, we also investigated the potential utility of the representations produced by the encoder. 
To do so, we created a downstream task of audio production style classification where the goal is to classify the style of a recording among the five styles we created for the realistic style evaluation. 
We trained linear logistic regression classifiers, also known as linear probes, on top of a set of different audio representations, a common approach in evaluating the quality of self-supervised representations~\cite{zhang2017split, asano2019critical}.
These linear probes are trained for a total of 400 epochs on a small dataset with only 60 examples from each of the five styles.
Class-wise and overall F1 scores on the test set are reported in Table~\ref{tab:style-probes}.

As baseline representations, we considered a random projection of melspectrogram features, along with large pre-trained representations including OpenL3~\cite{cramer2019look}, which is focused on capturing audio content and CDPAM~\cite{manocha2021cdpam}, which is focused on capturing audio quality, therefore the most similar to our approach.
In addition, we trained linear probes on representations from the NP-HH, SPSA, and AD differentiation methods on both speech styles from DAPS and music styles from MUSDB18.

In the case of DAPS, the class-wise scores indicated that CDPAM and our proposed representations were able to capture the Bright and Telephone styles. 
Differentiating between Broadcast and Neutral was difficult due to the main difference between these styles being the presence of more aggressive dynamic range compression. 
However, we found that our encoder representations did not struggle to capture this difference. 
Overall, is is clear that the representations produced by the our self-supervised training framework capture not only style elements related to spectral content, but also signal dynamics, which are not captured well by the other approaches. 
We found similar results between the speech and music style classification, with the appearance that music was slightly more difficult, likely due to the starting diversity of style within MUSDB18.

\begin{table}[t!]
    \centering
    \resizebox{0.95\hsize}{!}{%
    \begin{tabular}{l c c c c c c} \toprule
        \multicolumn{7}{c}{\textbf{DAPS}} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg}\\  \midrule
        Random Mel      & 0.87 & 0.78 & 0.73 & 0.39 & 0.00 & 0.55\\
        OpenL3          & 0.19 & 0.61 & 0.08 & 0.10 & 0.18 & 0.23\\ 
        CDPAM           & \textbf{1.00} & \textbf{1.00} & 0.79 & 0.25 & 0.63 & 0.73\\\midrule
        NP-HH S         & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\
        SPSA            & 0.95 & 0.98 & \textbf{1.00} & 0.89 & 0.95 & 0.96\\
        AD              & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\ \midrule
        \multicolumn{7}{c}{\textbf{MUSDB18}} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg} \\  \midrule
        Random Mel        & 0.80 & 0.98 & 0.62 & 0.17 & 0.00 & 0.51 \\
        OpenL3            & 0.32 & 0.66 & 0.20 & 0.17 & 0.30 & 0.33 \\ 
        CDPAM             & 0.89 & 0.95 & 0.66 & 0.00 & 0.06 & 0.51 \\\midrule
        NP-HH S         & \textbf{0.98} & \textbf{1.00} & 0.92 & \textbf{0.59} & \textbf{0.60} & \textbf{ 0.82} \\
        SPSA            & \textbf{0.98} & \textbf{1.00} & 0.90 & 0.26 & 0.00 & 0.63 \\
        AD              & \textbf{0.98} & \textbf{1.00} & \textbf{0.95} & 0.54 & 0.50 & 0.79 \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Class-wise F1 scores for five-class style prediction with linear classifiers trained on top of audio representations for speech and music using a single linear layer.}
   
    \label{tab:style-probes}\    
    \vspace{-0.4cm}
\end{table}





\section{DISCUSSION}
\label{sec:discussion}

While our style transfer approach may appear to place a larger burden on the user compared to a fully automatic system, we argue that it provides multiple mechanisms for user-interaction.
Complete style transfer, where the user supplies their own reference, provides the most agency, but requires that users already have a reference recording.
A simpler paradigm may involve \emph{adaptive presets}, where a set of curated references are provided, each with semantic descriptions. 
In cases where a ``one-button'' solution is desired, a single predefined style reference that represents a desired production style may be used.

With regards to the differentiation methods we examined, while the AD approach performed best across most metrics, there are a number of considerations for each of the the differentiation methods that are not reflected in these metrics. 
For example, while the TCN-based models performed well in comparison, and are ultimately more flexible, these system do not provide interpretable controls, and also require significant computation, making them undesireable, and difficult to integrate into audio production environments. 
On the other hand, the approximate DDSP approaches provide a unique benefit in that both SPSA and NP variants do not require knowledge of the underlying system or any re-implementation, nor do they explicitly require differentiability.
Since they enable the use of the original DSP implementation at inference, they also require less computation and produce interpretable control parameters, as shown in~\tref{tab:runtime}. 
However, we observed that both the NP and SPSA methods suffered from other drawbacks. 

\setlength{\tabcolsep}{3.0pt}
\begin{table}[t]
    \centering
    \resizebox{.8\hsize}{!}{%
    \begin{tabular}{l c c c c} \toprule
       
        \textbf{Method}  &  \textbf{Train step (s)}   &  \textbf{CPU} &  \textbf{GPU} & \textbf{Parameters} \\ \midrule
        RB-DSP      & -     & 0.004 & -     & 0 \\ \midrule
        NP          & 0.434 & 0.277 & 0.005 & 336\,k  \\
        NP-HH       & 0.434 & 0.003 & -     & 0 \\
        NP-FH       & 0.434 & 0.003 & -     & 0  \\
        cTCN 1      & 0.438 & 0.132 & 0.002 & 174\,k \\
        cTCN 2      & 0.642 & 0.268 & 0.005 & 336\,k  \\
        SPSA        & 0.413 & 0.003 & -     & 0   \\
        AD          & 0.301 & 0.006 & 0.001 & 0\\
                 \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Runtime comparison across differentiation methods including seconds taken for a single training step, and real-time factor for inference on CPU (Intel Xeon CPU E5-2623 v3 @ 3.00GHz) and GPU (GeForce GTX 1080 Ti). }
    \label{tab:runtime}
     \vspace{-0.2cm}
\end{table}

The SPSA approach proved more susceptible to training instability, thus requiring careful tuning of the $\epsilon$ hyperparameter and learning rate. 
And, while the neural proxy variants were promising, they were unable to achieve performance on par with the other approaches.
While the NP approaches do not require re-implementation or knowledge of the audio effects, they do require a laborious pre-training process. 
However, we hypothesize that their worse performance likely stemmed from the inaccuracy of the proxy networks, especially for the parametric equalizer. 
Future directions may involve the design of neural proxy architectures better suited to a wider range of audio effects. 

The AD approach was the most manual and demanding method to implement technically, but showed promising results. 
While this approach is feasible when expert knowledge can be exploited to introduce manual modifications to support efficient training, generalization of this approach to other effects is a challenging and time consuming process. 







\section{CONCLUSIONS}
Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another enabling automatic audio production and \emph{adaptive presets}.
We demonstrated the applicability of our approach in both speech and music tasks and performed an extensive comparison of a diverse range of differentiable signal processing methods. 
We proposed a categorization of existing approaches for differentiable signal processing, and also introduced differentiable dynamic range compressor and neural proxy hybrid approaches.
Our system demonstrated convincing performance on both the synthetic and realistic style transfer tasks while estimating interpretable audio effect controls, operating at variable sample rates, and producing representations that reflect production style.

\section{ACKNOWLEDGMENT}
This work is supported in part by the EPSRC UKRI CDT in Artificial Intelligence and Music (EP/S022694/1). We would like to thank Oliver Wang, Paris Smaragdis, and Juan-Pablo Caceres for their valuable conversations.


\section{INTRODUCTION}
\label{sec:intro}
Audio effects are commonly used in audio and video content creation to manipulate sound attributes such as loudness, timbre, spatialization, and dynamics~\cite{wilmering2020history}. 
While audio effects can be powerful tools in the hands of trained audio engineers, their complexity poses a significant challenge for less experienced users, often requiring a time-consuming adjustment process even for professionals. 
To simplify their usage, expert-designed presets can be used to configure an audio effect, but this only results in static content-independent configurations of effects.
Therefore, presets commonly require fine-tuning and adjustment, without the ability to adapt to variations across inputs.

Automatic audio production methods aim to overcome this limitation with analysis and subsequent adaptation of effects based on an input signal~\cite{IMPbook19}. These methods can provide intelligent, automatic control of a single effect parameter, multiple effects, and/or many channels of audio in a cross-adaptive manner.
Such methods may also provide a range of interfaces, from fully automatic systems to systems that only provide suggestions.
As a result, successful automatic audio production systems have the ability not only to simplify the process for amateurs, but also expedite the workflow of professionals~\cite{moffat2019approaches}.

Automatic audio production systems are commonly categorized into rule-based systems built on audio engineering best-practices, or data-driven machine learning systems~\cite{moffat2019approaches}.
While rule-based systems have seen success~\cite{deman2013b}, they are limited by the inability to formalize rules that cover the diversity arising in real-world audio engineering tasks~\cite{deman2017towards}.
Classical machine learning approaches on the other hand provide greater flexibility, but have thus far been limited by an inability to collect sufficient parametric data describing audio production in a standardized way~\cite{scott2011analysis}. Recently, deep learning approaches have shown promise in overcoming these challenges, fueling an increasing interest in data-driven audio production techniques.

Deep learning approaches for automatic audio production can be organized into direct transformation, parameter prediction, and differentiable digital signal processing methods. 
Direct transformation methods map untreated audio to a desired target and have shown promise in tasks such as speech enhancement~\cite{pascual2017segan}, source separation~\cite{stoter2019open}, and audio effect modeling~\cite{hawley2019signaltrain, martinez2020deep,wright2020real,martinez2021deep}, but have had limited use for automatic production due to a lack of interpretable control, artifacts, and/or computational complexity~\cite{martinez2021deep}. 
Parameter prediction methods predict the settings of audio effects, but require expensive ground truth parameter data~\cite{hawley2019signaltrain} or optimize over an undesirable loss in the parameter domain~\cite{sheng2019feature, mimilakis2020one}. 

Lastly, differentiable signal processing uses neural networks to control digital signal processors (DSP) implemented as differentiable operators enabling training via backpropagation~\cite{engel2020ddsp}. 
This approach imposes a stronger inductive bias by restricting the flexibility of the model, which may help to reduce the likelihood of processing artifacts, enable user control and refinement of the model predictions, and improve computational efficiency.
However, this approach requires manual implementation and often modification of DSP, imparting high engineering cost, potentially limiting its application. 
To work around this, two alternative methods have been proposed including neural proxies (NP) of audio effects~\cite{steinmetz2021automatic} and numerical gradient approximation schemes~\cite{ramirez2021differentiable}.
However, given that these approaches were proposed and evaluated in different tasks, it is difficult to fully understand their relative performance.


Beyond integrating audio effects as differentiable operators, a central limitation of deep learning systems for automatic audio production lies in the difficulty of sourcing sufficient data for supervised training, requiring both the unprocessed and final produced recordings and/or parameter data. 
In addition, the subjective and context-dependent nature of the audio production process further complicates the task~\cite{lefford2021context}. 
While evidence suggests the existence of ``best-practices''~\cite{pestana2014intelligent}, learning these techniques in a supervised paradigm is challenging.
In particular, these systems must not only learn how to control effects to achieve a desired result, they must also implicitly uncover the distribution of possible styles present within the training data across different genres and contexts.

In this work, we present \emph{DeepAFx-ST} -- an approach that can impose the audio effects and production style from one recording with automatic control of a set of audio effects as shown in \fref{fig:style-transfer-headline}. 
Only a short example style recording is needed at inference and no labeled training data or retraining is needed to adapt to new styles. 
To do so, we propose 
\begin{itemize}
    \setlength\itemsep{0.9em}
    \item The first audio effects style transfer method to integrate audio effects as differentiable operators, optimized end-to-end with an audio-domain loss, 
    \item Self-supervised training that enables automatic audio production without labeled or paired training data,
    \item A benchmark of five differentiation strategies for audio effects, including compute cost, engineering difficulty, and performance,
    \item The development of novel neural proxy hybrid methods, and a differentiable dynamic range compressor.
   
\end{itemize}

We find our approach offers several benefits including the ability to perform audio effect style transfer for both speech and music signals, yield interpretable audio effects control parameters enabling user interaction, and operate at sampling rates different than those seen during training.
Implementation of our code, a demonstration video, and listening examples are made available online\footnote{\url{https://csteinmetz1.github.io/DeepAFx-ST}}.


\begin{figure}[t!]
    \centering
    \vspace{-0.0cm}
    \includegraphics[width=0.95\linewidth]{figures/generic-style-transfer-headline.drawio.pdf}
    \vspace{-0.0cm}
    \caption{Our \emph{DeepAFx-ST} method imposes the audio effects and production style from one recording to another by example. We use a shared-weight encoder to analyze the input and a style reference signal, then compare each with a controller that outputs the parameters of effects that themselves perform style manipulation.}
    \label{fig:style-transfer-headline}
\end{figure}


\section{BACKGROUND}
\label{sec:background}


\subsection{Audio Effects and Production Style Transfer}
Relevant works on audio production style transfer include initial work on controlling a dynamic range compressor with a multi-stage training procedure using neural networks and random forest~\cite{sheng2019feature}, as well as a deep neural network approach to control a parametric frequency equalizer~\cite{mimilakis2020one}. 
Both works, however, only consider controlling one audio effect in isolation and use a (quantized) parameter domain loss to avoid backpropagating through audio effects -- a non-trivial task. 
This approach has notable disadvantages including undesirable quantization schemes or loss tuning and low correlation between parameters and audio, which have been noted in the context of parameter inference for automatic synthesizer control~\cite{yee2018automatic, esling2020flow}. 
Both of these realities limit performance and generalization across classes of effects. 
However, differentiable signal processing provides the potential to overcome these challenges and has yet to be investigated for the style transfer task. 

\subsection{Differentiable Audio Effects}\label{sec:diff_fx}
Typically, differentiable signal processors are manually implemented within automatic differentiation frameworks~\cite{abadi2016tensorflow, paszke2019pytorch}.
Assuming the mathematical operations in question are differentiable, this provides a straightforward way to integrate DSP operations with neural networks. 
This approach, however, requires expert knowledge, design trade-offs, and can be difficult or impossible to implement exactly. 
Existing audio effects implemented in this manner include infinite impulse response (IIR) filters~\cite{nercessian2020neural}, reverberation~\cite{lee2021differentiable}, echo cancellers~\cite{casebeer2021auto}, DJ transitions~\cite{chen2021automatic}, and reverse engineered effects~\cite{colonel2021reverse}. 
These approaches are related to the growing body of work focused on the construction of audio synthesis models with differentiable components, which now include additive~\cite{engel2020ddsp}, subtractive~\cite{masuda2021synthesizer}, waveshaping~\cite{hayes2021neural}, and wavetable~\cite{shan2021differentiable} synthesizers. 


Differentiable audio effects of interest to our work include the parametric frequency equalizer (PEQ) and dynamic range compressor (DRC) -- two common audio production effects.
To our knowledge, there is no known past work on differentiable compressors, but differentiable PEQs do exist~\cite{nercessian2020neural}. 
PEQs are typically constructed with a cascade of multiple second-order IIR filters, also know as biquads~\cite{valimaki2016all}. 
While it is possible to implement differentiable IIR filters in the time-domain~\cite{kuznetsov2020differentiable}, the recursive filter structure can cause issues as a result of vanishing/exploding gradients and computational bottlenecks during backpropagation through time (BPTT)~\cite{hochreiter1997long}. 
This motivates frequency-domain finite impulse response (FIR) approximations~\cite{nercessian2020neural, nercessian2021lightweight, colonel2021iirnet}. In this case, the complex frequency response of the $k^{\textrm{th}}$ biquad $H_k(e^{j \omega})$ is expressed as a ratio of the DFT of the numerator and denominator coefficients, $\mathbf{b}_k = [b_{0,k}, b_{1,k}, b_{2,k}]$ and $\mathbf{a}_k = [a_{0,k}, a_{1,k}, a_{2,k}]$,
\begin{equation}
    H_k(e^{j \omega}) = \frac{ \textrm{DFT}(\mathbf{b}_k) }{ \textrm{DFT}(\mathbf{a}_k) } = \frac{\sum_{m=0}^{2} \ b_{m,k} \ e^{-j\omega m}}{\sum_{n=0}^{2} \ b_{n,k} \ e^{-j \omega n}}.
\end{equation}
The overall response of a cascade of $K$ biquad filters is then given by the product of their responses $H_{sys} (e^{j \omega}) = \prod_{k=1}^{K} H_k(e^{j \omega})$.
We can approximate the response using an FIR filter by evaluating $H_{sys}(e^{j \omega})$ over a linearly spaced, zero-padded frequencies of length $2^{\lceil \log_2 (2N - 1) \rceil}$, applying the filter(s) to an input $x[n]$ with $N$ samples in the frequency domain by multiplying with  the input $ X(e^{j \omega})$, 
and then performing the inverse DFT to get the output $y[n]$.


\subsection{Alternative Differentiation Methods}

In contrast to manually implementing differentiable signal processing operations, three alternative approaches are of note. 
In neural proxy (NP) approaches, a neural network is trained to emulate the behavior of a signal processor~\cite{grathwohl2017backpropagation, jacovi2019neural} including control parameter behavior using a dataset of input/output measurements. 
While most work in this area has focused on creating emulations of analog audio effects~\cite{hawley2019signaltrain, martinez2020deep, wright2020real, martinez2021deep, steinmetz2021efficient, damskagg2019deep, martinez2019modeling, wright2019real}, this approach can be used together with a larger neural network for automatic control of effects, as was demonstrated with applications in automatic mixing~\cite{steinmetz2021automatic}.
Since neural proxy models are composed of neural network building blocks, they are differentiable by default, but require accurate emulation to work effectively and can be computationally complex.  

In an effort to reduce inference time complexity and reduce approximation error, neural proxy models can be used together with the original DSP device to create what we call neural proxy hybrid models~\cite{tseng2019hyperparameter}.
Half hybrid approaches (NP-HH) use the neural proxy during the forward and backward pass of training, but use the DSP device during inference.
Full hybrid approaches (NP-FH) aim to further reduce approximation error caused by the neural proxy by using the DSP device during inference as well as the forward pass during training.
In this case, the neural proxy is used only during the backward pass where gradient computation is required.
Thus far neural proxy hybrid approaches, however, have not been applied in audio, and have only been used in computer vision for hyperparameter optimization, and not in an adaptive manner for automatic control.

As a third alternative, non-differentiable DSP implementations can be used directly with numerical gradient approximation methods, which has been demonstrated in audio effect modeling, removal of breaths, and music mastering~\cite{ramirez2021differentiable}. 
In context of controlling audio effects $h(x, p)$ with input audio $x$ and parameters $p \in \mathbb{R}^{P}$, we only need to estimate the partial derivatives for each parameter $\frac{\partial}{\partial p_i} h(x,p)$. 
For this, finite differences (FD)~\cite{milne2000calculus} perturb each input parameter forward and backward and evaluate the operator $2P$ times for $P$ control parameters. 
Alternatively, simultaneous permutation stochastic approximation (SPSA)~\cite{spall1998overview} gradient estimation only requires two function evaluations per gradient vector estimate. 
In this case, the approximate partial derivative with respect to $p_i$ is given by 
\begin{equation}
\frac{\hat{h}(x, p_i)}{p_i} = \frac{h(x, p + \epsilon \Delta^P) - h(x, p - \epsilon  \Delta^P)}{2 \epsilon \Delta^{P}_{i}},
\end{equation}
where $\epsilon$ is a small, non-zero value and $\Delta^P \in \mathbb{R}^P$ is a random vector sampled from a symmetric Bernoulli distribution ($\Delta^P_i = \pm 1$)~\cite{spall1992multivariate}.  
This approach does not require pre-training or knowledge of the DSP, but can be challenging to train due to inaccuracies in gradient approximation and overhead from implementations available only on CPU.

\begin{figure*}[t!]
    \centering
    \vspace{-0.0cm}
    \includegraphics[width=0.99\linewidth,trim={0cm 0.1cm 0.8cm 0},clip]{figures/training-v2.pdf}
    \vspace{-0.0cm}
    \caption{The self-supervised training process for our \emph{DeepAFx-ST} approach.
   
    We sample $x$ from a large dataset of recordings $\mathcal{D}$ and apply random augmentations.
    Recordings with two different styles are produced with randomly configured DSP, which become the input $x_i$ and the style reference $x_r$.
    We split these recordings in half, and pass the opposite sections ($x_{i,a}$ and $x_{r,b}$) to shared-weight encoders $f_\theta$. 
    Using the concatenated encoder representation $e_{ir}$, the controller $g_\phi$ estimates parameters $p$ to apply the reference style to the input.}
    \label{fig:data-generation}
    \vspace{-0.0cm}
\end{figure*}

\section{PROPOSED METHOD}
\label{sec:method}
Our proposed audio production style transfer method is composed of three components as shown in \fref{fig:data-generation}.
A self-supervised data generation process enables the creation of input and reference recordings, $x_i$ and $x_r$, from a large dataset of audio $\mathcal{D}$.
These input and reference recordings are generated by processing a source recording through two different configurations of a chain of audio effects, resulting in two different production styles.  
Then a shared weight encoder $f_\theta$ is used to extract information about the production style of the input and reference recordings. 
This information is aggregated and then passed to the controller network $g_\phi$ that is tasked with estimating parameters $p$ to configure the differentiable audio effects $h(x,p)$ so that the processed input signal $\hat{x}_r$ matches the style reference $x_r$.

\subsection{Self-Supervised Training}
\label{sec:ssl}

Our self-supervised training strategy enables learning control of audio effects without labeled or paired training data. 
Our data generation process is shown on the left of~\fref{fig:data-generation} and begins by selecting a random audio recording $x$ from a dataset $\mathcal{D}$. 
We then create an augmented version $x_{aug}$ by applying pitch shifting and time stretching to further increase diversity.
Two versions of this augmented recording are created, $x_i$ and $x_r$, each with a different production style. 
The recording in the top branch $x_i$ becomes the input, and the recording in the bottom branch $x_r$ becomes the style reference.
This is achieved by applying a set of audio effects to both recordings, with each set of effects having a different random configuration.
To ensure that sensible outputs are produced, the method for sampling random effect configuration should be carefully tuned, as is common with data augmentation pipelines. 

To train our model to only focus on audio production style and operate when the content differs between the input and style reference, we split the recordings $x_i$ and $x_r$ in half, generating an $a$ and $b$ section, producing four recordings $x_{i,a}$, $x_{i,b}$, $x_{r,a}$, and $x_{r,b}$.
During training, we randomly select either the $a$ or $b$ section to be passed as input and use the other section as the style reference.
To compute an audio-domain loss we use the style reference corresponding to the same section as the input in order to have a paired input and ground truth with the same content. 
Typical approaches to constructing audio datasets often require ground truth paired recordings, or generate them by taking clean recording and degrading them.
However, our approach generates paired data using a dataset of any source material with a range of quality and recording conditions. 

The process of splitting the recordings does not perfectly mimic the case at inference, where the input and reference recordings come from different sources.
However, this splitting operation helps to further increase the difference between the input and reference recordings during training, as opposed to using recordings with the same content.
The process of randomly selecting either the $a$ or $b$ section as input and reference further increases this diversity by ensuring that the model cannot rely on the relative temporal relation of the input and style reference.

\subsection{Model Architecture}
\label{sec:arch}

To carry about the production style transfer task, magnitude spectrograms from the input and style reference recordings are fed to a shared-weight convolutional neural network encoder $f_\theta$, which produces a time series of embeddings for each recording.
Temporal average pooling is used to aggregate these embeddings across time, producing a single embedding of dimension $D$ for both the input $e_i$ and style reference $e_r$. 
These embeddings are then concatenated $e_{ir}$ and passed to the controller network.

The controller network $g_\phi$ is a basic multi-layer perception (MLP) and is tasked with producing control parameters $p$ to configure a set of audio effects $h(x, p)$.
This network should consider the information from the encoder about the production style of the input and the reference.
The estimated control parameters should configure the set of audio effects in such a way that passing the input signal though the effect chain will produce a recording that matches the style reference. 





\subsection{Differentiable Audio Effects}
\label{sec:diff_methods}
Core to our approach is the integration of audio effects directly within the computation graph of a neural network. 
Learning to control audio effects instead of carrying out audio processing with a neural network helps us incorporate signal processing domain-knowledge, impose a strong inductive bias, reduce processing artifacts, and reduce computational complexity~\cite{engel2020ddsp}. 
In contrast to past work~\cite{mimilakis2020one, sheng2019feature}, we fully integrate audio effects as differentiable operators or layers with standard automatic differentiation tools and enable backpropagation through effects during training as shown with dashed lines in Figure~\ref{fig:data-generation}. 

We propose backpropagating through audio effects using five unique differentiation strategies.
These include manually implemented automatic differentiation effects (AD), neural proxy effects (NP), full neural proxy hybrids (NP-FH), half neural proxy hybrids (NP-HH), and numerical gradient approximations (SPSA).
While AD, NP, and SPSA methods have previously been employed in automatic audio production tasks, NP-FH has only been used in static image processing hyperparameter optimization~\cite{tseng2019hyperparameter}, and NP-HH has not been proposed before. 
Moreover, all approaches have never been compared in a unifying manner, leaving uncertainty around their relative efficacy.

\section{EXPERIMENTAL DESIGN}
\label{sec:experimental_design}


To demonstrate our approach, we focus on the task of imposing the equalization and loudness dynamics from one recording to another using a parametric equalizer (PEQ) and dynamic range compressor (DRC) as shown in~\fref{fig:signal-chain}. 
This forms the basic signal chain employed by audio engineers with applications in dialogue processing or in music mastering.
We compare performance of our production style transfer system using each of the differentiation strategies introduced in ~\sref{sec:diff_methods}, keeping the data generation and network architecture fixed.
By doing so, we seek to both understand the performance of our production style transfer approach, as well as the advantages and disadvantages of these differentiation strategies.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth, trim={0.0cm 0.18cm 0.0cm 0.1cm},clip ]{figures/signal-chain.pdf}
    \caption{Our audio effect signal chain consisting of a 6-band parametric equalizer and single-band dynamic range compressor.}
    \label{fig:signal-chain}
    \vspace{-0.0cm}
\end{figure}

\subsection{Differentiable Equalizer and Compressor}
\label{sec:eq_drc}
We implement a differentiable PEQ following past work~\cite{nercessian2020neural}, but develop our own novel differentiable dynamic range compressor. 
For our differentiable compressor, we use a traditional feedforward dynamic range compressor, which is in fact composed of differentiable operations~\cite{giannoulis2012digital}.
Similar to PEQs, however, we found direct implementation of existing compressor designs problematic due to the presence of recursive filters, in this case due to the smoothing filters for the ballistics,
\begin{equation} \label{eq:smoothing}
    \resizebox{.9\hsize}{!}{
   $y_L[n] =
    \begin{cases}
        \alpha_A y_L[n-1] + (1-\alpha_A) x_L[n] & x_L[n] > y_L[n-1] \\
        \alpha_R y_L[n-1] + (1-\alpha_R) x_L[n] & x_L[n] \leq y_L[n-1]
    \end{cases}$,       
    }
\end{equation}
where $y_L[n]$ is the smoothed gain reduction, $x_L[n] = x_G[n] - y_G[n]$ is the result of subtracting the output of the gain computer from the signal envelope, and $\alpha_A$ and $\alpha_R$ are the attack and release time constants. 
This operation is not easily approximated with a FIR filter, thus we simplify our design using a single attack and release time constant $\alpha$ to create a single-pole IIR ballistics filter,
\begin{equation}
   y_L[n] = \alpha y_L[n-1] + (1-\alpha) x_L[n].
\end{equation}
This filter can be approximated using an FIR filter by employing the same approach as in differentiable PEQs.
While this modification restricts the capabilities of our compressor, forcing the effective attack and release times to be shared, we seek only a basic compressor to show the complexity in implementing differentiable versions of common audio effects, which we compare against competing alternative differentiation strategies. 

\subsection{Datasets}
\label{sec:datasets}

We employ our data generation pipeline as described in~\sref{sec:method} using both speech and music.
To test our approach on speech, we use \texttt{train-clean-360}, a subset of LibriTTS~\cite{zen2019libritts}, which totals 360 hours of audio at $f_s = 24$\,kHz.
To test our approach on music, we use the MTG-Jamendo dataset~\cite{bogdanov2019mtg}, which provides over 55\,k songs, which we downmix to mono and resample to $f_s = 24$\,kHz. 
For both datasets, we generate a $90$/$5$/$5$ split of the recordings for training, validation, and testing. When randomly sampling patches of audio from a recording, if the selected segment $x[n]$ has a mean energy below the threshold, $\frac{1}{N} \sum_{n=1}^{N} | x[n] |^2 \leq 0.001$, we continue sampling until a non-silent segment is found.
To ensure that there is sufficient headroom for processing within the audio effects, we peak normalize all inputs and style recordings to $-12$\,dBFS.

\subsection{Baselines}
\label{sec:baselines}

\linesubsec{Rule-based DSP}
\label{sec:dsp}
Our rule-based baseline consists of an automatic, two stage process that aims to mimic the signal chain of the PEQ and DRC. 
First, an FIR filtering stage is employed, which aims to transform the spectral content of the input to match the spectrum of the style reference. 
We compute the average magnitude spectrum of the input and style reference by computing the STFT using a large window of $N=65536$ and a hop size of $H=16384$, averaging across the frames, yielding $X_i$ and $X_r$, which
are smoothed using a Savitzky-Golay filter~\cite{savitzky1964smoothing}.
We design an inverse FIR filter with $63$ taps using the window method, with a desired magnitude response given by $X = X_{i,sm} / X_{r,sm}$, where $X_{i,sm}$ and $X_{r,sm}$ are the smoothed average spectra of the input and the style reference.

The filtered signal is sent to a dynamic range compressor where all parameters except for the threshold are fixed. 
In order to adjust the amount of compression based on the input signal and the style reference, the threshold starts at 0 dB and is lowered in increments of 0.5 dB.
At each iteration the loudness in dB LUFS~\cite{steinmetz2021pyloudnorm} of the output is measured and compared to the loudness of the style reference.
If the difference in the loudness measurements are below a given threshold (e.g. 0.5 dB LUFS), the process halts. 
Otherwise, the process continues until the loudness is within a specific tolerance, or the threshold reaches -80 dB.

\linesubsec{End-to-end neural network}
\label{sec:e2e}
As a baseline on the opposite end of the spectrum, we evaluate a direct, end-to-end neural network approach to replace the differentiable signal processing audio effects. 
This network utilizes the same architecture as our neural proxy approaches, but does not use pre-trained weights. 
Instead, the weights of this model are optimized during the training of the encoder and controller. 
The controller still predicts parameters for this network, as in the case of a neural proxy, since the weights of the network are not frozen during training.
As a result the model will learn its own set of control parameters. 
We evaluate two variants of this approach, first using a single TCN network (cTCN 1) and a second using two TCNs connected in series (cTCN 2), which mimics the setup in the neural proxy approach.



\subsection{Losses}
We train both our neural proxy models and production style transfer models using an audio-domain loss.
In particular, we compute our loss using the model output and ground truth audio from our data generation process with a weighted sum of the error in time and frequency domains.
The time domain loss $\mathcal{L}_{\text{time}}$ is the mean absolute error (MAE) and the
frequency domain loss $\mathcal{L}_{\text{freq}}$ is the multi-resolution short-time Fourier transform loss (MR-STFT)~\cite{wang2019neural, steinmetz2020auraloss}.
This loss is the sum of the $L_1$ distance between the STFT of the ground truth and estimated waveforms measured both in the $\log$ and linear domains, at multiple resolutions, in this case with window sizes $W \in [32, 128, 512, 2048, 8192, 32768]$ and hop sizes $H = W/2$.
The overall loss is  $\mathcal{L}_{\text{overall}}({\hat{y}},y) = \mathcal{L}_{\text{freq}}({\hat{y}},y) + \alpha \cdot \mathcal{L}_{\text{time}}({\hat{y}},y) \label{eq:overall-loss}$, where $\alpha = 100$.
 

\setlength{\tabcolsep}{3.0pt}
\setlength{\abovecaptionskip}{15pt plus 3pt minus 2pt}
\begin{table*}[t!]
    \centering
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
        \multicolumn{21}{c}{\normalsize{\textbf{Speech}}} \\ \midrule
        & \multicolumn{6}{c}{\textbf{LibriTTS}} & & \multicolumn{6}{c}{\textbf{DAPS}} & & \multicolumn{6}{c}{\textbf{VCTK}} \\ \cmidrule(lr){2-7} \cmidrule(lr){8-14} \cmidrule(lr){15-21}
        \textbf{Method}  &  \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}} & & \footnotesize{\textbf{PESQ}} &  \footnotesize{\textbf{STFT}}  & \footnotesize{\textbf{MSD}} & \footnotesize{\textbf{SCE}} & \footnotesize{\textbf{RMS}} & \footnotesize{\textbf{LUFS}}\\ \midrule
        {Input}   
        & 3.765           & 1.187           & 2.180           & 687.5           & 6.983           & 2.426           & 
        & 3.684           & 1.179           & 2.151           & 641.7            & 6.900          & 2.314          & 
        & 3.672           & 1.254           &          2.008  & 815.4           & 7.783           & 2.532\\
        {RB-DSP}  
        & 3.856           & 0.943           & 1.955           & 410.3           & 4.204           & 1.674           & 
        & 3.787           & 0.917           & 1.882           & 399.7            & 3.705           & 1.481          & 
        & 3.709           & 1.101           &          1.911  & 657.6           & 5.039           & 2.018\\ \midrule
       
        {cTCN 1}    
        & 4.258           & 0.405           & 0.887           & 128.4           & 2.237           & 1.066           & 
        & 4.185           & 0.419           & 0.884           & 124.6            & 2.098           & 1.006          & 
        & 4.181           & 0.467           &          0.891  & 173.8           & 2.651           & 1.165\\
        {cTCN 2}    
        & 4.281           & 0.372           & \textbf{0.833}  & 117.5           & 1.927           & 0.925           & 
        & \textbf{4.224}  & \textbf{0.391}  & \textbf{0.841}  & 113.9            & 1.886           & 0.913          & 
        & 4.201           & \textbf{0.441}  & \textbf{0.856}  & 163.8           & 2.431           & 1.086\\ \midrule
       
        {NP}   
        & 3.643           & 0.676           & 1.405           & 265.0           & 2.812           & 1.340           & 
        & 3.605           & 0.685           & 1.362           & 249.2            & 2.732           & 1.350          & 
        & 3.651           & 0.737           &          1.300  & 321.7           & 3.166           & 1.453\\
        {NP-HH}   
        & 3.999           & 1.038           & 2.179           & 440.2           & 5.472           & 2.679           & 
        & 3.903           & 1.022           & 2.113           & 451.9            & 5.104           & 2.535          & 
        & 3.951           & 1.044           &          1.930  & 591.5           & 5.194           & 2.651\\
        {NP-FH}   
        & 3.945           & 1.058           & 2.088           & 404.9           & 6.820           & 3.197           & 
        & 3.891           & 1.037           & 2.045           & 395.4            & 6.754           & 3.117          & 
        & 3.894           & 1.087           &          1.934  & 514.0           & 7.065           & 3.363\\
       
        {SPSA}      
        & 4.180           & 0.635           & 1.406           & 219.5           & 3.263           & 1.600           & 
        & 4.099           & 0.645           & 1.379           & 213.6            & 2.989           & 1.511          & 
        & 4.023           & 0.730           &          1.359  & 301.6           & 3.535           & 1.737\\
       
        \midrule
        {AD}  
        & \textbf{4.310}  & \textbf{0.388}  & 0.882           & \textbf{111.5}  & \textbf{1.828}  & \textbf{0.823}  & 
        & 4.222           & 0.416           & 0.895           & \textbf{109.0}   & \textbf{1.758}  & \textbf{0.799} & 
        & \textbf{4.218}  & 0.481           &          0.924  & \textbf{152.7}  & \textbf{2.317}  & \textbf{1.006} \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.1cm}
\caption{Synthetic production style transfer with models trained using LibriTTS. Held-out speakers from the LibriTTS dataset are used, while utterances from DAPS and VCTK come from datasets never seen during training. Lower is better for all metrics except PESQ.}
    \vspace{-0.1cm}
    \label{tab:style-transfer-synthetic-speech}
\end{table*}

\subsection{Neural Proxy Pre-training}
To construct our neural proxy-based models, we utilized a temporal convolutional network (TCN)~\cite{bai2018empirical}, also known as the feedforward WaveNet~\cite{rethage2018wavenet}. This model has seen success in modeling distortion~\cite{wright2019real} and compressors~\cite{steinmetz2021automatic}.
In particular, we use a causal TCN~\cite{steinmetz2021automatic} with feature-wise linear modulation (FiLM)~\cite{perez2018film} to adapt to the effect control parameters.
Our TCN configuration has $4$ blocks, kernel size of $13$, channel width of $64$ and a dilation factor that grows as a power of $8$, where the dilation factor at block $b$ is $d_b = [ 1, 8, 64, 512 ]$.
We trained a total of 4 proxies, one for each of the two effects, with one for both speech and music. 
Training examples were generated by using DSP effects with uniformly random parameters. 
We trained each model for 400 epochs, where one epoch was defined as 20,000 samples, and used the checkpoint with the lowest validation loss.
We used an initial learning rate of $3\cdot 10^{-4}$, a batch size of 16, and inputs $\approx 3$s ($65536$ samples) in length. 
The learning rate was lowered by a factor of 10 after the validation loss had not improved for 10 epochs. 

\subsection{Production Style Transfer Training}
After training our neural proxy models, we trained our production style transfer model using five different audio effects differentiation methods -- AD, NP, NP-HH, NP-FH, and SPSA as well as end-to-end cTCN1 and cTCN2. 
The encoder operates on spectrograms using a short-time Fourier transform with a Hann window of size $N=4096$ and a hop size of $H=2048$.
The magnitude of the spectrogram was computed and exponential compression was applied following $|X|^{0.3}$. 
The compressed magnitude spectrogram was then passed into a randomly initialized EfficientNet B2 CNN~\cite{tan2019efficientnet} with a single input channel. 
The final activations were projected to $D=1024$ with a linear layer, followed by temporal mean pooling to produce a single representation, which was normalized by the $L_2$ norm.

The controller network was implemented as a simple three layer MLP, along with PReLU~\cite{he2015delving} activation functions. 
At the output of the controller, the hidden representation was projected into the audio effect control parameter space $\mathbb{R}^P$, and a sigmoid activation was applied. 
All parameters were then denormalized following predefined minimum and maximum ranges for parameters before they were passed to the audio effects. 

We trained all models for 400 epochs with a batch size of 6, where a single epoch is defined as 20,000 segments of audio $\approx 10$ seconds in length ($262144$ samples). 
We used a learning rate of $10^{-4}$ for all models, with the exception of the SPSA models, which required a lower learning rate of $10^{-5}$ for stability. 
In both cases, we scheduled the learning rate such that it is lowered by a factor of 10 at both $80\%$ and $95\%$ through training.
We applied gradient clipping with a value of $4.0$ and used an SPSA $\epsilon$ of $5 \cdot 10^{-4}$.




\setlength{\tabcolsep}{2.5pt}
\begin{table*}[ht]
    \vspace{-0.2cm}
    \centering
    \resizebox{\linewidth{}}{!}{%
    \begin{tabular}{l c c c c c c c c c c c c c c c c c c c c} \toprule
    \multicolumn{21}{c}{\normalsize{\textbf{Music}}} \\ \midrule
        &  
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (24 kHz)}}   & & 
        \multicolumn{6}{c}{\textbf{MTG-Jamendo (44.1 kHz)}} & & 
        \multicolumn{6}{c}{\textbf{MUSDB18 (44.1 kHz)}} \\ 
        \cmidrule(lr){2-7} \cmidrule(lr){9-14} \cmidrule(lr){15-21}
        \textbf{Method}   & 
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & &
        \textbf{PESQ} & \textbf{STFT} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
       
        Input      
        & 2.927  & 1.198  & 6.088  & 646.5  & 6.695  & 2.518 & 
        & 2.874  & 1.109  & 4.454  & 767.7  & 6.793  & inf   & 
        & 2.900  & 1.252  & 4.342  & 1088.3    & 5.940  & 2.312    \\ 
        RB-DSP     
        & 2.849  & 0.925  & 4.422  & 263.2  & 4.254  & 1.706  & 
        & 2.931  & 0.887  & 3.355  & 342.0   & 4.749  & 1.882 & 
        & 2.994  & 0.821  & 3.052  & \textbf{379.4}   & 4.078 & 1.665  \\
       
        \midrule
        cTCN 1       
        & \textbf{3.402}  & 0.547  & 2.294  & 160.9  & 3.261  & 1.483 &
        & 3.168  & 0.876  & 2.921  & 494.4    & 4.372  & 2.070  & 
        & 3.121  & 0.896  & 2.956  & 730.4    & 4.548  & 2.231  \\ 
        cTCN 2       
        & 3.390  & 0.548  & 2.278  & 152.3  & 2.951  & 1.397 & 
        &  3.123  & 0.903  & 2.973  & 517.4   & 4.084  & 1.899  &
        &  3.107  & 0.917  & 2.986  & 749.5   & 4.208  & 2.061 \\  \midrule
       
        NP S  
        & 2.926  & 0.787  & 2.838  & 221.3  & 2.785  & 1.390 &  
        & 2.764  & 1.033  & 2.869  & 488.3  & 3.710  & 1.824 & 
        & 2.804  & 0.950  & 2.778  & 742.1   & 3.835  & 1.921  \\
        NP M  
        & 2.765  & 0.845  & 3.211  & 255.2  & 3.227  & 1.608 & 
        & 2.699  & 1.042  & 2.928  & 497.0    & 3.942  & 1.961   & 
        & 2.791  & 0.946  & 2.800  & 757.1   & 4.209  & 2.127    \\
        NP-HH (S)  
        & 2.819  & 1.092  & 6.791  & 395.3  & 6.276  & 3.032 & 
        & 2.865  & 1.101  & 5.194  & 689.2    & 6.792  & 3.365 & 
        & 2.853  & 1.165  & 4.852  & 1005.7   & 6.451  & 3.269 \\
        NP-HH (M)  
        & 2.532  & 1.166  & 7.070  & 591.9  & 5.660  & 2.593 & 
        & 2.512  & 1.148  & 5.385  & 854.9   & 5.940  & 2.740 & 
        & 2.493  & 1.198  & 5.090  & 1021.9    & 5.585  & 2.731 \\
        NP-FH (S)  
        & 2.833  & 1.016  & 5.005  & 280.8  & 3.377  & 1.634 & 
        & 2.888  & 0.977  & 3.883  & 429.9    & 3.480  & 1.709   & 
        & 2.857  & 1.045  & 3.809  & 617.3   & 3.932  & 1.971   \\
        NP-FH (M)  
        & 2.648  & 1.137  & 6.368  & 605.6  & 5.903  & 2.587 & 
        & 2.625  & 1.133  & 5.512  & 844.6   & 6.417  & 2.876  & 
        & 2.575  & 1.225  & 5.450  & 1172.5   & 5.973  & 2.913   \\
        SPSA         
        & 3.173  & 0.716  & 3.024  & 210.1  & 2.809  & 1.344 & 
        & 3.172  & 0.759  & 2.458  & 386.2   & 2.839  & 1.311 & 
        &  3.126  & 0.789  & 2.321  & 574.4   & 2.925  & 1.394    \\
       
        \midrule
        AD     
        & 3.355  & \textbf{0.488}  & \textbf{2.149}  & \textbf{144.7}  & \textbf{2.167}  & \textbf{1.005}   & 
        &  \textbf{3.400}  & \textbf{0.585}  & \textbf{1.824}  & \textbf{304.4}    & \textbf{2.425}  & \textbf{1.106} & 
        &  \textbf{3.396}  & \textbf{0.608}  & \textbf{1.695}  & 456.1    & \textbf{2.559}  & \textbf{1.197}  \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Synthetic production style transfer using the MTG-Jamendo dataset. Proxy models (S) were pre-trained using speech from LibriTTS and (M) were trained using music from the MTG-Jamendo dataset. Lower is better for all metrics except PESQ.}
    \label{tab:style-transfer-synthetic-music}
    \vspace{-0.2cm}
\end{table*}

\section{RESULTS}
\label{sec:results}



\subsection{Synthetic Production Style Transfer}
We first evaluated on a synthetic production style transfer task mirroring our training setup. 
This facilitates the use of full-reference metrics, since we synthetically generate style references and as a result have inputs and ground truth with matching content.
Results of the evaluation using held-out speakers from LibriTTS are shown in \tref{tab:style-transfer-synthetic-speech}.

As there are no established metrics for evaluating audio production style, we propose a range of metrics. 
For a general notion of perceptual similarity we use PESQ~\cite{rix2001perceptual}, along with the Multi-resolution STFT~\cite{wang2019neural} error, which is a component of the training objective. 
To evaluate the ability of the system to address equalizer-related style components, we report the mel-spectral distance (MSD), which consists of the error between melspectrograms using a large window size of $W=65536$, along with the spectral centroid error (SCE). 
To evaluate aspects related to the dynamic range, we consider the RMS energy error (RMS) and the perceptual loudness error (LUFS)~\cite{steinmetz2021pyloudnorm}. 

We note our rule-based DSP baseline improves on the input across all metrics, though not by a significant margin.
Overall, we find strong performance from the AD method, which performs best on all metrics except MSD, which is only slightly outperformed by cTCN2.
Both cTCN approaches follow closely behind AD, demonstrating the strength of end-to-end neural network approaches. 
The SPSA model follows closely behind these approaches with somewhat degraded PESQ and STFT error, yet it outperforms all NP methods. 
While the NP methods are not as competitive, both NP-HH and NP-FH outperform NP in terms of PESQ, likely a result of using a DSP implementation, which imparts less artifacts.
However, the NP approach achieves better performance on all other metrics, indicating it likely better matches the style reference. 

In addition to evaluating on held-out speech from LibriTTS, we also evaluate generalization with speech datasets not seen during training including DAPS~\cite{mysore2014can}, which contains high quality recordings from 20 speakers, and the larger VCTK~\cite{yamagishi2019vctk} dataset, which contains 110 speakers.
We observe a similar trend in performance, yet the cTCN models slightly outperform AD as compared to on LibriTTS.
This indicates our self-supervised training was sufficient for producing models that generalize to speech from different sources and acoustic conditions. 

We also trained the same set of models using music from the MTG-Jamendo dataset. 
The result, shown in \tref{tab:style-transfer-synthetic-music}, largely follows a similar trend to the speech experiments with regards to the differentiation methods, demonstrating the general nature of our approach. 
However, it appears that the music task is somewhat more difficult.
We also compared neural proxies trained on speech (S) or music (M), and found those trained on music performed slightly worse.

We then investigated operation at sample rates different than training. 
To achieve variable sample rate operation we resampled the signals passed to the encoder to match training (24\,kHz), however, the audio effects operate at the sample rate of the input audio (44.1\,kHz).
First, we evaluated on the MTG-Jamendo test set, and we noticed a significant drop in performance for the neural network-based approaches, which include the cTCNs and NP, since these models operate at a fixed sample rate. 
However, the DSP-based approaches performed significantly better, including both the AD and SPSA approaches, which performed only slightly worse than at the training sample rate.

Finally, to test out-of-distribution generalization, we used music from MUSDB18~\cite{musdb18}.
While performance does decrease across most metrics and for all models, the models can still perform the task, with SPSA and AD approaches exceeding the baseline. 

\subsection{Realistic Production Style Transfer}
\setlength{\tabcolsep}{3.0pt}

To simulate a more realistic production style transfer scenario where the style reference contains not only different content but also differing sources (e.g. speaker identity or instrumentation), we construct a realistic production style transfer evaluation task.
We designed five different audio production ``styles'', which are defined by hand-crafted parameter ranges inspired by commonly used plugin presets.
We generate audio from these five different styles (Telephone, Warm, Bright, Neutral, and Broadcast) by randomly sampling parametric equalizer and compressor parameters from these ranges such that each style is audibly unique.
Sampling parameter values randomly provides more variation within audio examples of the same style, which is more complex than applying a fixed preset. 

To generate examples in each of these styles, we use speech from DAPS, and music from MUSDB18.
However, since we do not have paired ground truth as in the synthetic production style transfer case, we performed our evaluation using only non-intrusive metrics that do not require a matching reference. 
These metrics measure the error between high-level audio features, like the MSD, spectral centroid, RMS energy, and perceptual loudness.


\begin{table}[t!]
    \centering
    \resizebox{1.0\hsize}{!}{%
    \begin{tabular}{l c c c c c c c c c } \toprule
        & \multicolumn{4}{c}{\textbf{DAPS}} & & \multicolumn{4}{c}{\textbf{MUSDB18}} \\  \cmidrule(lr){2-5} \cmidrule(lr){6-10}
        \textbf{Method} & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} & & \textbf{MSD} & \textbf{SCE} & \textbf{RMS} & \textbf{LUFS} \\ \midrule
        {Input}   & 10.4        & 1041.7        & 10.4          & 3.6   & & 8.4         & 2607.4      & 9.4        & 3.8   \\ 
        {RB-DSP}  & 8.9         & 517.2         & 5.8           & 2.4   & & 6.5         & \textbf{915.4}       & 8.6         & 3.7   \\  \midrule
        {NP-HH S} & 9.8         & 636.6         & 12.9          & 5.9   & & 7.0         & 1512.7      & 10.0        & 4.5          \\
        {SPSA}    & 8.0         & 360.1         & \textbf{5.1} & 2.5    & & 5.5         & 1297.0      & 4.6         & 2.1         \\
        {AD}      & \textbf{7.8}& \textbf{278.1}& 5.2  & \textbf{2.4}   & & \textbf{4.8} & 947.6       & \textbf{3.8} & \textbf{1.7}             \\ 
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Realistic production style transfer average performance of all pairwise configurations from five predefined styles with speech from DAPS using the model trained on LibriTSS and music from MUSDB18 using the model trained on MTG-Jamendo.}
    \vspace{-0.3cm}
    \label{tab:style-transfer-real}
\end{table}

We performed all pairwise production style transfers resulting in 25 configurations, and we performed each configuration ten times using different input and output recordings from each style. 
Here, we considered only the best performing methods that produce interpretable control parameters (NP-HH, SPSA, and AD).
To summarize the results of this evaluation, we aggregated the metrics across all 25 configurations reporting the mean for each to provide a general notion of the production style transfer performance, as shown in \tref{tab:style-transfer-real}. 

On examples from DAPS, all methods including the baseline improve upon the input, but SPSA and AD clearly outperform the other approaches across all metrics, with AD performing the best. 
In the case of MUSDB18, the AD approach further outperforms the others, with the exception of SCE, where the RB-DSP baseline performs best.



\subsection{Parameter Visualization}
To demonstrate the ability of our approach to provide precise control over the effect parameters, we plot both the overall frequency response of the parametric equalizer as well as the compressor configuration in~\fref{fig:vctk-variable-style}.
We show the result of using one utterance from VCTK as the input with five different style references.
Since the utterances from VCTK are relatively clean and contain little post-processing, we observed the estimated parameters aligned with our expectation for the predefined styles. 
For example, we find that the telephone style is achieved by reducing the high and low frequencies while boosting frequencies around 1\,kHz.
In terms of compression, we see a clear trend with the broadcast style using the most aggressive compression, as expected. 
While estimating these style parameters may seem trivial, our system was not trained on these recordings and has no knowledge of the styles. 

\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (left) and compressor (right) parameters from our SPSA model with one utterance from VCTK and five unique style references.}
    \label{fig:vctk-variable-style}
    \vspace{-0.0cm}
\end{figure}

\begin{figure}[!t]
\vspace{-0.0cm}
    \centering
    \includegraphics[width=\linewidth, trim={0.3cm 0.1cm 0.85cm 0},clip ]{figures/style_transfer_p314_modify.pdf}
    \vspace{-0.8cm}
    \caption{Estimated parametric equalizer (left) and compressor (right) parameters with our SPSA model using a fixed Broadcast style reference. The input VCTK utterance was progressively modified to increase the high frequency energy and compression.}
    \label{fig:vctk-variable-input}
    \vspace{-0.2cm}
\end{figure}

As a demonstration of the ability of the model to adapt to the input signal like an \emph{adaptive preset}, we processed a clean recording using a reference from the Broadcast style while varying characteristics of the input. 
In this case, the input signal was modified by progressively increasing the gain of a high-shelf filter from 0\,dB to +24\,dB, while also increasing the amount of compression with a higher ratio (1.0 to 4.0) and lower threshold (0\,dB to -62\,dB).
As shown in \fref{fig:vctk-variable-input}, the estimated parameters adapt to these changes in a sensible manner. 
This results in a progressively lower boost in the high-shelf filter that was added originally to achieve the presence of the Broadcast style, while also increasing the threshold and decreasing the ratio to reduce the amount of compression, as the input becomes more compressed. 

\subsection{Encoder Representations}
\setlength{\tabcolsep}{3.0pt}

\begin{table}[t!]
    \centering
    \resizebox{1.0\hsize}{!}{%
    \begin{tabular}{l c c c c c c} \toprule
        \multicolumn{7}{c}{\textbf{DAPS} (Speech)} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg}\\  \midrule
        Random Mel      & 0.87 & 0.78 & 0.73 & 0.39 & 0.00 & 0.55\\
        OpenL3          & 0.19 & 0.61 & 0.08 & 0.10 & 0.18 & 0.23\\ 
        CDPAM           & \textbf{1.00} & \textbf{1.00} & 0.79 & 0.25 & 0.63 & 0.73\\\midrule
        NP-HH S         & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\
        SPSA            & 0.95 & 0.98 & \textbf{1.00} & 0.89 & 0.95 & 0.96\\
        AD              & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{1.00}\\ \midrule
        \multicolumn{7}{c}{\textbf{MUSDB18} (Music)} \\ \midrule
        \textbf{Features} & Telephone &  Bright & Warm & Broadcast & Neutral & \textbf{Avg} \\  \midrule
        Random Mel        & 0.80 & 0.98 & 0.62 & 0.17 & 0.00 & 0.51 \\
        OpenL3            & 0.32 & 0.66 & 0.20 & 0.17 & 0.30 & 0.33 \\ 
        CDPAM             & 0.89 & 0.95 & 0.66 & 0.00 & 0.06 & 0.51 \\\midrule
        NP-HH S         & \textbf{0.98} & \textbf{1.00} & 0.92 & \textbf{0.59} & \textbf{0.60} & \textbf{ 0.82} \\
        SPSA            & \textbf{0.98} & \textbf{1.00} & 0.90 & 0.26 & 0.00 & 0.63 \\
        AD              & \textbf{0.98} & \textbf{1.00} & \textbf{0.95} & 0.54 & 0.50 & 0.79 \\
         \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Class-wise F1 scores for five-class style prediction with linear classifiers trained on top of audio representations for speech and music using a single linear layer.}
   
    \label{tab:style-probes}\    
    \vspace{-0.2cm}
\end{table}

In addition to our production style transfer evaluation, we also investigated the potential utility of the representations learned by our encoder. 
To do so, we created a downstream task of audio production style classification, where the goal is to classify the style of a recording among the five styles we created for our realistic style evaluation. 
We trained linear logistic regression classifiers, also known as linear probes, on top of a set of different audio representations, a common approach in evaluating the quality of self-supervised representations~\cite{zhang2017split, asano2019critical}.
These linear probes are trained for a total of 400 epochs on a small dataset with only 60 examples from each of the five styles.
We show class-wise and overall F1 scores on the test set in Table~\ref{tab:style-probes}.

As baseline representations, we considered a random projection of melspectrogram features, along with large pre-trained representations including OpenL3~\cite{cramer2019look}, which is focused on capturing audio content and CDPAM~\cite{manocha2021cdpam}, which is focused on capturing audio quality, therefore the most similar to our approach.
In addition, we trained linear probes on representations from the NP-HH, SPSA, and AD differentiation methods on both speech styles from DAPS and music styles from MUSDB18.

In the case of DAPS, the class-wise scores indicated that CDPAM and our proposed representations were able to capture the Bright and Telephone styles. 
Classifying between Broadcast and Neutral was difficult due to the main difference between these styles being the presence of more aggressive compression. 
However, we found that our encoder representations did not struggle to capture this difference. 
Overall, it is clear that representations from our approach not only capture style elements related to spectral content, but also signal dynamics, which are not captured well by the other approaches. 
We found similar results for both speech and music, where music was slightly more difficult, likely due to MUSDB18 style diversity.


\subsection{Computational Complexity}
Beyond production style transfer performance, we benchmark training time per step, CPU/GPU inference time, and number of parameters for audio effects per method variant in~\fref{tab:runtime}.
To compare the training step we measure the time required for both a forward and backward pass when optimizing the system using the same batch size and input size as training.
We found the neural proxy and TCN approaches impart the highest computational cost since they feature the use of neural networks for audio processing during both inference and training. 
However, the neural proxy hybrids (HP-HH and NP-FH) and SPSA have the lowest computational cost since they all use only the DSP device during inference, which is on par with RB-DSP baseline. 
The AD approach is twice as slow as the DSP-based methods, but is still far more efficient than any of the neural network based effects. 

\setlength{\tabcolsep}{3.0pt}
\begin{table}[t]
    \centering
    \resizebox{1.0\hsize}{!}{%
    \begin{tabular}{l c c c c c} \toprule
       
        \textbf{Method}  &  \textbf{Train step (s)}   &  \textbf{CPU} &  \textbf{GPU} & \textbf{Parameters} & \textbf{Interpretable}\\ \midrule
        RB-DSP      & -     & 0.004 & -     & 0 & $\mathcal{-}$\\ \midrule
        cTCN 1      & 0.438 & 0.132 & 0.002 & 174\,k & - \\
        cTCN 2      & 0.642 & 0.268 & 0.005 & 336\,k & - \\ \midrule
        NP          & 0.434 & 0.277 & 0.005 & 336\,k & \checkmark \\
        NP-HH       & 0.434 & \textbf{0.003} & -     & 0 & \checkmark\\
        NP-FH       & 0.434 & \textbf{0.003} & -     & 0 & \checkmark \\
        SPSA        & 0.413 & \textbf{0.003} & -     & 0 & \checkmark \\
        AD          & \textbf{0.301} & 0.006 & \textbf{0.001} & 0 & \checkmark \\
                 \bottomrule 
    \end{tabular}}
    \vspace{0.2cm}
    \caption{Runtime comparison across differentiation methods including seconds taken for a single training step, and real-time factor for inference on CPU (Intel Xeon CPU E5-2623 v3 @ 3.00GHz) and GPU (GeForce GTX 1080 Ti). }
    \label{tab:runtime}
     \vspace{-0.2cm}
\end{table}


\section{DISCUSSION}
\label{sec:discussion}

Our production style transfer approach provides multiple  user-interaction paradigms. 
At a high-level, a user can supply their own input content and style reference and view the estimated control parameters. 
This affords significant agency, but requires the user have a goal and a suitable reference recording. 
We can simplify this interaction by creating \emph{adaptive presets} that supply a curated, predefined set of style reference recordings, each with semantic descriptions. 
Lastly, we can also empower a fully automatic solution by employing a single, predefined style reference.
 
When we compare differentiation methods, we note the AD approach performed best across most metrics. There are, however, a number of additional considerations. 
In particular, the AD approach is the most manual and demanding method to implement, commonly requires significant expertise and design trade-offs to implement, and can be more difficult to integrate into on-device inference. 
In contrast, generic neural network architectures like our TCN-based baselines performed well in comparison and are ultimately more flexible. 
However, these approaches do not provide interpretable control and require significant computation, making them undesirable and difficult to integrate into audio production environments. 

Lastly, when we compare against the SPSA and NP methods, we note they offer interpretable control (similar to AD), but do not require knowledge or re-implementation of the underlying system (black-box), do not require explicit differentiation, enable the use of the original DSP at inference, and typically require less computation. 
We observed, however, NP and SPSA methods have other drawbacks.
We found SPSA was more susceptible to training instability and required careful tuning of the $\epsilon$ hyperparameter and learning rate. 
And, while the NP variants were promising, we found they were unable to achieve performance on par with the other approaches. We hypothesize this was caused by inaccuracy in the proxy networks, especially for the parametric equalizer. 
Strict NP approaches also require laborious pre-training and design.

We conclude that the preferred differentiation strategy for our task and similar automatic production methods is application dependent.  
Future research directions include the design of better neural proxy methods, a reusable differentiable audio effects library, the ability for dynamic audio signal chain construction, and/or improved numerical gradient approximation methods.






\section{CONCLUSIONS}
Our \emph{DeepAFx-ST} method imposes the production style from one recording to another, enabling automatic audio production and \emph{adaptive presets}.
We demonstrated the applicability of our approach in both speech and music tasks and performed an extensive comparison of a diverse range of differentiable signal processing methods. 
We proposed a categorization of existing approaches for differentiable signal processing, and also introduced differentiable dynamic range compressor and neural proxy hybrid approaches.
Our approach demonstrated convincing performance on both the synthetic and realistic production style transfer tasks, while estimating interpretable audio effect controls, operating at variable sample rates, and producing representations that accurately reflect production style.

\section{ACKNOWLEDGMENT}
This work is supported in part by the EPSRC UKRI CDT in Artificial Intelligence and Music (EP/S022694/1). We would like to thank O. Wang, P. Smaragdis, and J.-P. Caceres for their valuable conversations.

