\section{Research Questions}
\label{sec:research_qs}
We are interested to seek answers for these following research questions:

\begin{itemize}[leftmargin=2.5em]
    \item[\bf RQ1]  When would feature interaction learning methods become more efficient than ReLU-based DNNs? What's the role of each component in the cross network?
    \item[\bf RQ2]  How does the feature-interaction component of each baseline perform without integrating with DNN? 
    \item[\bf RQ3]  How does the proposed mDCN approaches compare to the baselines? Could we achieve healthier trade-off between model accuracy and cost through mDCN and the mixture of low-rank DCN?
    \item[\bf RQ4]  Could cross network replace typical ReLU layers?
    \item[\bf RQ5] How does the settings in mDCN affect model quality?
    \item[\bf RQ6] Is mDCN capturing important feature crosses? Does the model provide good understandability?
\end{itemize}
\section{Introduction}

Learning to rank (LTR) \cite{liu2011learning, cao2007learning} has remained to be one of the most important problems in modern-day machine learning and deep learning. It has a wide range of applications in search, recommendation systems \cite{resnick1997recommender, herlocker2004evaluating, schafer1999recommender}, and computational advertising \cite{broder2008computational, bottou2013counterfactual}. Among the crucial components of LTR models, learning effective feature crosses continues to attract lots of attention from both academia \cite{qu2016product, lian2018xdeepfm, song2019autoint} and industry \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, naumov2019deep}.

Effective feature crosses are crucial to the success of many models. They provide additional interaction information beyond individual features. For example, the combination of ``{\ttfamily{country}}'' and ``{\ttfamily{language}}'' is more informative than either one of them.
In the era of linear models, ML practitioners rely on manually identifying such feature crosses \cite{seide2011feature} to increase model's expressiveness.
Unfortunately, this involves a combinatorial search space, which is large and sparse in web-scale applications where the data is mostly categorical.
Searching in such setting is exhaustive, often requires domain expertise, and makes the model harder to generalize.

Later on, embedding techniques have been widely adopted to project features from high-dimensional sparse vectors to much lower-dimensional dense vectors. Factorization Machines (FMs) \cite{rendle2010factorization, rendle:tist2012} leverage the embedding techniques and construct pairwise feature interactions via the inner-product of two latent vectors. Compared to those traditional feature crosses in linear models, FM brings more generalization capabilities.

In the last decade, with more computing firepower and huge scale of data, LTR models in industry have gradually migrated from linear models and FM-based models to deep neural networks (DNN). This has significantly improved model performance for search and recommendation systems across the board \cite{cheng2016wide, wang2017deep, guo2017deepfm}. People generally consider DNNs as universal function approximators, that could potentially learn all kinds of feature interactions \cite{mhaskar1996neural, valiant2014learning, NIPS2016_6556}. However, recent studies \cite{beutel2018latent, wang2017deep} found that DNNs are inefficient to even approximately model 2nd or 3rd-order feature crosses.

To capture effective feature crosses more accurately, a common remedy is to further increase model capacity through wider or deeper networks. This naturally crafts a double edged sword that we are improving model performance while making models much slower to serve. In many production settings, these models are handling extremely high QPS, thus have very strict latency requirements for real-time inference. Possibly, the serving systems are already pushed to a stretch that cannot afford even larger models. Furthermore, deeper models often introduce trainability issues, making models harder to train.

This has shed light on critical needs to design a model that can efficiently and effectively learn predictive feature interactions, especially in a resource-constraint environment that handles real-time traffic from billions of users. Many recent works \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, qu2016product, lian2018xdeepfm, song2019autoint, naumov2019deep} tried to tackle this challenge. The common theme is to leverage those \emph{implicit} high-order crosses learned from DNNs, with \emph{explicit} and bounded-degree feature crosses which have been found to be effective in linear models. \emph{Implicit} cross means the interaction is learned through an end-to-end function without any explicit formula modeling such cross. \emph{Explicit} cross, on the other hand, is modeled by an explicit formula with controllable interaction order. We defer a detailed discussion of these models in \autoref{sec:related_work}.

Among these, Deep \& Cross Network (DCN) \cite{wang2017deep} is effective and elegant, however, productionizing DCN in large-scale industry systems faces many challenges. The expressiveness of its cross network is limited. The polynomial class reproduced by the cross network is only characterized by $O(\text{input size})$ parameters, largely limiting its flexibility in modeling random cross patterns. 
Moreover, the allocated capacity between the cross network and DNN is unbalanced. This gap significantly increases when applying DCN to large-scale production data. An overwhelming portion of the parameters will be used to learn implicit crosses in the DNN


In this paper, we propose a new model \emph{{DCN-V2}} that improves the original DCN model. We have already successfully deployed {DCN-V2} in quite a few learning to rank systems across Google with significant gains in both offline model accuracy and online business metrics. {DCN-V2} first learns explicit feature interactions of the inputs (typically the embedding layer) through cross layers, and then combines with a deep network to learn complementary implicit interactions. The core of {DCN-V2} is the cross layers, which inherit the simple structure of the cross network from DCN, however significantly more expressive at learning explicit and bounded-degree cross features. The paper studies datasets with clicks as positive labels, however {DCN-V2} is label agnostic and can be applied to any learning to rank systems.
The main contributions of the paper are five-fold:


\begin{itemize}[leftmargin=1em]
    \item We propose a novel model---{DCN-V2}---to learn effective explicit and implicit feature crosses. Compared to existing methods, our model is more expressive yet remains efficient and simple.
    \item Observing the low-rank nature of the learned matrix in {DCN-V2}, we propose to leverage low-rank techniques to approximate feature crosses in a subspace for better performance and latency trade-offs. In addition, we propose a technique based on the Mixture-of-Expert architecture \cite{shazeer2017outrageously, jacobs1991adaptive} to further decompose the matrix into multiple smaller sub-spaces. These sub-spaces are then aggregated through a gating mechanism.
    \item We conduct and provide an extensive study using synthetic datasets, which demonstrates the inefficiency of traditional ReLU-based neural nets to learn high-order feature crosses.
    \item Through comprehensive experimental analysis, we demonstrate that our proposed {DCN-V2} models significantly outperform SOTA algorithms on Criteo and MovieLen-1M benchmark datasets.
   
    \item We provide a case study and share lessons in productionizing {DCN-V2} in a large-scale industrial ranking system, which delivered significant offline and online gains.
\end{itemize}


The paper is organized as follows. \autoref{sec:related_work} summarizes related work. \autoref{sec:dcn-m} describes our proposed model architecture {DCN-V2} along with its memory efficient version. \autoref{sec:dcn_analysis} analyzes {DCN-V2}. \autoref{sec:research_qs} raises a few research questions, which are answered through comprehensive experiments on both synthetic datasets in \autoref{sec:exp_synthetic} and public datasets in \autoref{sec:exp_public}. \autoref{sec:productionization} describes the process of productionizing {DCN-V2} in a web-scale recommender. 

\section{Related Work}
\label{sec:related_work}


The core idea of recent feature interaction learning work is to leverage both explicit and implicit (from DNNs) feature crosses. To model explicit crosses, most recent work introduces multiplicative operations ($x_1 \times x_2$) which is inefficient in DNN, and designs a function $f({\bf x}_1, {\bf x}_2)$ to efficiently and explicitly model the pairwise interactions between features ${\bf x}_1$ and ${\bf x}_2$. We organize the work in terms of how they combine the explicit and implicit components.

{\bf Parallel Structure.} One line of work jointly trains two parallel networks inspired from
the wide and deep model \cite{cheng2016wide}, where the wide component takes inputs as crosses of raw features; and the deep component is a DNN model. However, selecting cross features for the wide component falls back to the feature engineering problem for linear models. Nonetheless, the wide and deep model has inspired many works to adopt this parallel architecture and improve upon the wide component. 

DeepFM \cite{guo2017deepfm} automates the feature interaction learning in the wide component by adopting a FM model. DCN \cite{wang2017deep} introduces a cross network, which learns explicit and bounded-degree feature interactions automatically and efficiently.
xDeepFM \cite{lian2018xdeepfm} increases the expressiveness of DCN by generating multiple feature maps, each encoding all the pairwise interactions between features at current level and the input level. Besides, it also considers each feature embedding ${\bf x}_i$ as a unit instead of each element $x_i$ as a unit. Unfortunately, its computational cost is significantly high (10x of \#params), making it impractical for industrial-scale applications. Moreover, both DeepFM and xDeepFM require all the feature embeddings to be of equal size, yet another limitation when applying to industrial data where the vocab sizes (sizes of categorical features) vary from $O(10)$ to millions. AutoInt \cite{song2019autoint} leverages the multi-head self-attention mechanism with residual connections. InterHAt \cite{li2020interpretable} further employs Hierarchical Attentions.

{\bf Stacked Structure.} Another line of work introduces an interaction layer---which creates explicit feature crosses---in between the embedding layer and a DNN model. This interaction layer captures feature interaction at an early stage, and facilitates the learning of subsequent hidden layers. Product-based neural network (PNN) \cite{qu2016product} introduces inner (IPNN) and outer (OPNN) product layer as the pairwise interaction layers. One downside of OPNN lies in its high computational cost.
Neural FM (NFM) \cite{he2017neural} extends FM by replacing the inner-product with a Hadamard product;
DLRM \cite{naumov2019deep} follows FM to compute the feature crosses through inner products;
These models can only create up to 2nd-order explicit crosses. AFN \cite{cheng2019adaptive} transforms features into a log space and adaptively learns arbitrary-order feature interactions. Similar to DeepFM and xDeepFM, they only accept embeddings of equal sizes.

Despite many advances made, our comprehensive experiments (\autoref{sec:exp_public}) demonstrate that DCN still remains to be a strong baseline. We attribute this to its simple structure that has facilitated the optimization. However, as discussed, its limited expressiveness has prevented it from learning more effective feature crosses in web-scale systems. In the following, we present a new architecture that inherits DCN's simple structure while increasing its expressiveness.




\section{Proposed Architecture: {DCN-V2}}
\label{sec:dcn-m}

This section describes a novel model architecture --- {DCN-V2} --- to learn both explicit and implicit feature interactions. {DCN-V2} starts with an \emph{embedding layer}, followed by a \emph{cross network} containing multiple cross layers that models explicit feature interactions, and then combines with a \emph{deep network} that models implicit feature interactions. The improvements made in {DCN-V2} are {\bf critical for putting {DCN} into practice for highly-optimized production systems}. {DCN-V2} significantly improves the expressiveness of DCN \cite{wang2017deep} in modeling complex explicit cross terms in web-scale production data, while maintaining its elegant formula for easy deployment. The function class modeled by {DCN-V2} is a strict superset of that modeled by DCN. The overall model architecture is depicted in Fig. \ref{fig:dcn-visualization}, with two ways to combine the cross network with the deep network: (1) stacked and (2) parallel.  In addition, observing the low-rank nature of the cross layers, we propose to leverage a mixture of low-rank cross layers to achieve healthier trade-off between model performance and efficiency.

\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.2\textwidth}  
    \includegraphics[width=\textwidth]{dcn-stack.pdf}
    \caption{Stacked}
    \label{fig:dcn-stack}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.27\textwidth}  
    \includegraphics[width=\textwidth]{dcn-parallel.pdf}
    \caption{Parallel}
    \label{fig:dcn-parallel}
    \end{subfigure}
    \caption{Visualization of {DCN-V2}. $\otimes$ represents the cross operation in Eq. \eqref{eq:cross_layer}, \emph{i.e.}, ${\bf x}_{l+1} ={\bf x}_0 \odot (W_l {\bf x}_l +  {\bf b}_l) + {\bf x}_l$.}
    \label{fig:dcn-visualization}
\end{figure}


\subsection{Embedding Layer}

The embedding layer takes input as a combination of categorical (sparse) and dense features, and outputs ${\bf x}_0 \in \mathbb{R}^d$. For the $i$-th categorical feature, we project it from a high-dimensional sparse space to a lower-dimensional dense space via
	${\bf x}_{\text{embed}, i} = W_{\text{embed},i} {\bf e}_i$,
where ${\bf e}_i \in \{0, 1\}^{v_i}$; $W \in \mathbb{R}^{e_i \times v_i}$ is a learned projection matrix; ${\bf x}_{\text{embed}, i} \in \mathbb{R}^{e_i}$ is the dense embedded vector; $v_i$ and $e_i$ represents vocab and embedding sizes respectively. For multivalent features, we use the mean of the embedded vectors as the final vector. 

The output is the concatenation of all the embedded vectors and the normalized dense features:
	${\bf x}_0 = [{\bf x}_{\text{embed}, 1}; \ldots; {\bf x}_{\text{embed}, n}; x_{\text{dense}}]$.

Unlike many related works \cite{song2019autoint, lian2018xdeepfm, qu2016product, guo2017deepfm, naumov2019deep, he2017neural} which requires $e_i=e_j~\forall i, j$, our model accepts arbitrary embedding sizes. This is particularly important for industrial recommenders where the vocab size varies from $O(10)$ to $O(10^5)$. Moreover, our model isn't limited to the above described embedding method; any other embedding techniques such as hashing could be adopted. 

\subsection{Cross Network}
The core of {DCN-V2} lies in the cross layers that create explicit feature crosses. Eq. \eqref{eq:cross_layer} shows the $(l+1)^\text{th}$ cross layer.
\begin{equation}
\label{eq:cross_layer}
	{\bf x}_{l+1} ={\bf x}_0 \odot (W_l {\bf x}_l +  {\bf b}_l) + {\bf x}_l
\end{equation}

where ${\bf x}_0 \in \mathbb{R}^{d}$ is the base layer that contains the original features of order 1, and is normally set as the embedding (input) layer. ${\bf x}_l,{\bf x}_{l+1} \in \mathbb{R}^{d}$, respectively, represents the input and output of the  $(l+1)$-th cross layer. $W_l \in \mathbb{R}^{d \times d}$ and ${\bf b}_l \in \mathbb{R}^{d}$ are the learned weight matrix and bias vector. \autoref{fig:cross_sub_network} shows how an individual cross layer functions.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=3in]{dcn-formula.png}
\vspace{-2ex}
\caption{Visualization of a cross layer.}
  \label{fig:cross_sub_network}
\vspace{-1ex}
\end{figure}

For an $l$-layered cross network, the highest polynomial order is $l+1$ and the network contains all the feature crosses up to the highest order. Please see \autoref{sec:poly_analysis} for a detailed analysis, both from bitwise and feature-wise point of views. When $W = {\bf 1} \times {\bf w}^\top$, where ${\bf 1}$ represents a vector of ones, {DCN-V2} falls back to {DCN}.

The cross layers could only reproduce polynomial function classes of bounded degree; any other complex function space could only be approximated\footnote{Any function with certain smoothness assumptions can be well-approximated by polynomials. In fact, we've observed in our experiments that cross network alone was able to achieve similar performance as traditional deep networks.}. Hence, we introduce a deep network next to complement the modeling of the inherent distribution in the data.


\subsection{Deep Network}
The $l^\text{th}$ deep layer's formula is given by
	${\bf h}_{l+1} = f(W_l {\bf h}_l +  {\bf b}_l)$,
where ${\bf h}_l \in \mathbb{R}^{d_l}, {\bf h}_{l+1} \in \mathbb{R}^{d_{l+1}}$, respectively, are the input and output of the $l$-th deep layer; $W_{l} \in \mathbb{R}^{d_{l} \times d_{l+1}}$ is the weight matrix and ${\bf b}_l \in \mathbb{R}^{d_{l+1}}$ is the bias vector; $f(\cdot)$ is an elementwise activation function and we set it to be ReLU; any other activation functions are also suitable.

\subsection{Deep and Cross Combination}
We seek structures to combine the cross network and deep network. Recent literature adopted two structures: stacked and parallel. In practice, we have found that which architecture works better is data dependent. Hence, we present both structures:

{\bf Stacked Structure (\autoref{fig:dcn-stack}):} The input ${\bf x}_0$ is fed to the cross network followed by the deep network, and the final layer is given by ${\bf x}_{\text{final}} = {\bf h}_{L_d}, ~ {\bf h}_{0} = {\bf x}_{L_c}$, which models the data as $f_{\text{deep}} \circ f_{\text{cross}}$.

{\bf Parallel Structure (\autoref{fig:dcn-parallel}):} The input ${\bf x}_0$ is fed in parallel to both the cross and deep networks; then, the outputs ${\bf x}_{L_c}$ and ${\bf h}_{L_d}$ are concatenated to create the final output layer ${\bf x}_{\text{final}} = [{\bf x}_{L_c}; {\bf h}_{L_d}]$. This structure models the data as $f_{\text{cross}} + f_{\text{deep}}$.
	

In the end, the prediction $\hat y_i$ is computed as:
	$\hat y_i = \sigma({\bf w}_{\text{logit}}^\top{\bf x}_{\text{final} })$,
where ${\bf w}_{\text{logit}}$ is the weight vector for the logit, and $\sigma(x) = 1 / (1 + \exp(-x))$. For the final loss, we use the Log Loss that is commonly used for learning to rank systems especially with a binary label (e.g., click). Note that {DCN-V2} itself is both prediction-task and loss-function agnostic.
\begin{equation*}
	\begin{split}
	\text{loss} = &-\frac{1}{N} \sum_{i = 1}^N y_i \log(\hat y_i) + (1 - y_i) \log(1 - \hat y_i) + 
    \lambda \sum_l \|W_l\|^2_2,
    \end{split}
\end{equation*}
where $\hat y_i$'s are predictions; $y_i$'s are the true labels; $N$ is the total number of inputs; and $\lambda$ is the $L_2$ regularization parameter. 

\subsection{Cost-Effective Mixture of Low-Rank DCN}
In real production models, the model capacity is often constrained by limited serving resources and strict latency requirements. It is often the case that we have to seek methods to reduce cost while maintaining the accuracy. Low-rank techniques \cite{golub1996matrix} are widely used \cite{jaderberg2014speeding, yu2017compressing, chen2018adaptive, wang2019block, halko2011finding, drineas2005nystrom} to reduce the computational cost. It approximates a dense matrix $M \in \mathbb{R}^{d \times d}$ by two tall and skinny matrices $U, V \in \mathbb{R}^{d \times r}$. When $r \le d/2$, the cost will be reduced.
However, they are most effective when the matrix shows a large gap in singular values or a fast spectrum decay. In many settings, we indeed observe that the learned matrix is numerically low-rank in practice.

Fig. \ref{fig:dcn-sval} shows the singular decay pattern of the learned matrix $W$ in {DCN-V2} (see Eq. \eqref{eq:cross_layer}) from a production model. Compared to the initial matrix, the learned matrix shows a much faster spectrum decay pattern. Let's define the numerical rank $R_T$ with tolerance T to be $\text{argmin}_k (\sigma_k < T \cdot \sigma_1)$, where $\sigma_1 \ge \sigma_2 \ge, \ldots, \ge \sigma_n$ are the singular values. Then, $R_T$ means majority of the mass up to tolerance $T$, is preserved in the top $k$ singular values. In the field of machine learning and deep learning, a model could still work surprisingly well with a reasonably high tolerance $T$ \footnote{This is very different from the filed of scientific computing (\emph{e.g.}, solving linear systems), where the approximation accuracy need to be very high. For problems such as CTR prediction, some errors could introduce regularization effect to the model.}.


\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.2\textwidth}  
    \includegraphics[width=\textwidth]{dcn-sval.png}
    \caption{Singular Values}
    \label{fig:dcn-sval}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.23\textwidth}  
    \includegraphics[width=\textwidth]{mixture-dcn.pdf}
    \caption{Mixture of Low-rank Experts}
    \label{fig:mixture-dcn}
    \end{subfigure}
    \caption{Left: Singular value decay of the learned {DCN-V2} weight matrix. The singular values are normalized and $1 = \sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k$. {\color{orange}$+$} represents the randomly initialized truncated normal matrix; {\color{blue}$\times$} represents the final learned matrix. Right: Visualization of mixture of low-rank cross layer.}
\end{figure}


Hence, it is well-motivated to impose a low-rank structure on $W$. Eq \eqref{eq:low_rank_cross_layer} shows the resulting $(l+1)$-th low-rank cross layer
\begin{equation}
\label{eq:low_rank_cross_layer}
	{\bf x}_{l+1} = {\bf x}_0 \odot \Big(U_l \big(V_l^\top {\bf x}_i\big) + {\bf b}_l \Big) + {\bf x}_i
\end{equation}
where $U_l, V_l \in \mathbb{R}^{d \times r}$ and $r \ll d$. Eq \eqref{eq:low_rank_cross_layer} has two \emph{interpretations}: 1) we learn feature crosses in a subspace; 2) we project the input ${\bf x}$ to lower-dimensional $\mathbb{R}^r$, and then project it back to $\mathbb{R}^d$. The two interpretations have inspired the following two model improvements.

Interpretation 1 inspires us to adopt the idea from Mixture-of-Experts (MoE) \cite{shazeer2017outrageously, jacobs1991adaptive, eigen2013learning, ma2018modeling}. MoE-based models consist of two components: experts (typically a small network) and gating (a function of inputs). In our case, instead of relying on one single expert (Eq \eqref{eq:low_rank_cross_layer}) to learn feature crosses, we leverage multiple such experts, each learning feature interactions in a different subspaces, and adaptively combine the learned crosses using a gating mechanism that depends on input ${\bf x}$. The resulting mixture of low-rank cross layer formulation is shown in Eq. \eqref{eq:mixture_low_rank_cross_layer} and depicted in \autoref{fig:mixture-dcn}.
\begin{equation}
\label{eq:mixture_low_rank_cross_layer}
    \begin{split}
	{\bf x}_{l+1} &= \sum\nolimits_{i=1}^K G_i({\bf x}_l) E_i({\bf x}_l) + {\bf x}_l \\
	E_i({\bf x}_l) &= {\bf x}_0 \odot \Big(U_l^i \big(V_l^{i\top}{\bf x}_l \big)  + {\bf b}_l \Big )
	\end{split}
\end{equation}
where $K$ is the number of experts; $G_i(\cdot): \mathbb{R}^d \mapsto \mathbb{R}$ is the gating function, common sigmoid or softmax; $E_i(\cdot): \mathbb{R}^d \mapsto \mathbb{R}^d$ is the $i^\text{th}$ expert in learning feature crosses. $G(\cdot)$ dynamically weights each expert for input ${\bf x}$, and when $G(\cdot) \equiv 1$, Eq \eqref{eq:mixture_low_rank_cross_layer} falls back to Eq \eqref{eq:low_rank_cross_layer}.

Interpretation 2 inspires us to leverage the low-dimensional nature of the projected space. Instead of immediately projecting back from dimension $d'$ to $d$ ($d' \ll d$), we further apply nonlinear transformations in the projected space to refine the representation \cite{fan2019multiscale}
\begin{equation}
\label{eq:mixture_low_rank_with_c_cross_layer}

	E_i({\bf x}_l) = {\bf x}_0 \odot \Big(
	U_l^i \cdot g \big( C_l^i \cdot g\big(V_l^{i\top}{\bf x}_l \big) \big)  + {\bf b}_l 
	\Big )
\end{equation}
where $g(\cdot)$ represents any nonlinear activation function.

{\bf Discussions.} This section aims to make effective use of the fixed memory/time budget to learn meaningful feature crosses. From Eqs \eqref{eq:cross_layer}--\eqref{eq:mixture_low_rank_with_c_cross_layer}, each formula represents a strictly larger function class assuming a fixed \#params.

Different from many model compression techniques where the compression is conducted post-training, our model imposes the structure prior to training and jointly learn the associated parameters with the rest of the parameters. Due to that, the cross layer is an integral part of the nonlinear system $f({\bf x}) = \big(f_k(W_k) \circ \cdots \circ f_1(W_1)\big) ({\bf x})$, where $(f_{i+1} \circ f_{i}) (\cdot) \coloneqq f_{i+1}(f_i(\cdot))$. Hence, the training dynamics of the overall system might be affected, and it would be interesting to see how the global statistics, such as Jacobian and Hession matrices of $f({\bf x})$, are affected. We leave such investigations to future work.





\subsection{Complexity Analysis}
Let $d$ denote the embedding size, $L_c$ denote the number of cross layers, $K$ denote the number of low-rank DCN experts. Further, for simplicity, we assume each expert has the same smaller dimension $r$ (upper bound on the rank). 
The time and space complexity for the cross network is $O(d^2 L_c)$, and for mixture of low-rank DCN ({DCN-Mix}) it's efficient when $rK \ll d$ with $O(2drKL_c)$. 



\section{Model Analysis}
\label{sec:dcn_analysis}
This section analyzes {DCN-V2} from polynomial approximation point of view, and makes connections to related work. We adopt the notations from \cite{wang2017deep}.

{\bf Notations.} Let the embedding vector ${\bf x} = [{\bf x}_1; {\bf x}_2; \ldots; {\bf x}_k] = [x_1, x_2, \ldots, x_d] \in \mathbb{R}^d$ be a column vector, where ${\bf x}_i \in \mathbb{R}^{e_i}$ represents the $i$-th feature embedding, and $x_i$ represents the $i$-th element in ${\bf x}$. Let multi-index ${\bm \alpha} = [\alpha_1, \cdots, \alpha_d] \in \mathbb{N}^d$ and $|{\bm \alpha}| = \sum_{i=1}^d \alpha_i$. $C_a^b \coloneqq \bigl\{{\bf y} \in \{1,\cdots, a\}^b \mathrel{\big|} \forall i < j, y_i > y_j  \bigr\}$. Let $\mathbf{1}$ be a vector of all 1's, and $I$ be an identity matrix. We use capital letters for matrices, bold lower-case letters for vectors, and normal lower-case letters for scalars.

\subsection{Polynomial Approximation}
\label{sec:poly_analysis}
We analyze {DCN-V2} from two perspectives of polynomial approximation ---
1) Considering each element (bit) $x_i$ as a unit, and analyzes interactions among the elements (\autoref{thm:cross_x0_bitwise}); and 2) Considering each feature embedding ${\bf x}_i$ as a unit, and only analyzes the feature-wise interactions (\autoref{thm:cross_x0_featurewise} ) (proofs in Appendix). 

\begin{theorem}[Bitwise]
\label{thm:cross_x0_bitwise}
	Assume the input to an $l$-layer cross network be ${\bf x} \in \mathbb{R}^d$, the output be $f_l({\bf x})={\bf 1}^\top {\bf x}^l$, and the $i^\text{th}$ layer is defined as ${\bf x}^i ={\bf x} \odot W^{(i-1)}{\bf x}^{i-1} +{\bf x}^{i-1}$. Then, the multivariate polynomial $f_l({\bf x})$ reproduces polynomials in the following class:
    $$\biggl\{\sum_{{\bm \alpha}} c_{{\bm \alpha}}\left(W^{(1)}, \ldots, W^{(l)}\right) x_1^{\alpha_1}x_2^{\alpha_2}\ldots x_d^{\alpha_d} \mathrel{\bigg|}  0 \le |{\bm \alpha}| \le l+1, {\bm \alpha} \in \mathbb{N}^d \biggr\},$$
    where $c_{\bm \alpha} = \sum_{{\bf j} \in C_l^{|{\bm \alpha}|-1}}  \sum_{{\bf i} \in P_{\bm \alpha}} \prod_{k=1}^{|{\bm \alpha}|-1} w_{i_k i_{k+1}}^{(j_k)}$, $w_{i j}^{(k)}$ is the $(i,j)^\text{th}$ element of matrix $W^{(k)}$, and $P_{\bm \alpha} = \text{Permutations}~(\cup_i \{\underbrace{i, \ldots, i}_{\alpha_i \text{times}} \mathrel{|} \alpha_i \neq 0\})$.
\end{theorem}


\begin{theorem}[feature-wise]
\label{thm:cross_x0_featurewise}
  With the same setting as in \autoref{thm:cross_x0_bitwise}, we further assume input ${\bf x} = [{\bf x}_1; \ldots; {\bf x}_k]$ contains $k$ feature embeddings and consider each ${\bf x}_i$ as a unit. Then, the output ${\bf x}^l$ of an $l$-layer cross network creates all the feature interactions up to order $l+1$. Specifically, for features with their (repeated) indices in $I$, let $P_I = Permutations (I)$, then their order-$p$ interaction is given by:
\begin{equation*}
\begin{split}
\sum_{{\bf i} \in P_I} \sum_{{\bf j} \in C_p^{p-1}} {\bf x}_{i_1} \odot \left(W_{i_1, i_2}^{(j_1)} {\bf x}_{i_2} \odot \ldots \odot \left(W_{i_k, i_{k+1}}^{(j_{k})} {\bf x}_{i_{l+1}}\right) \right)
\end{split}
\end{equation*}

\end{theorem}


From both bitwise and feature-wise perspectives, the cross network is able to create all the feature interactions up to order $l+1$ for an $l$-layered cross network. Compared to DCN-V, {DCN-V2} characterizes the same polynomial class with more parameters and is more expressive. Moreover, the feature interactions in {DCN-V2} is more expressive and can be viewed both bitwise and feature-wise, whereas in DCN it is only bitwise \cite{wang2017deep, lian2018xdeepfm, song2019autoint}.

\subsection{Connections to Related Work}
We study the connections between {DCN-V2} and other SOTA feature interaction learning methods; we only focus on the feature interaction component of each model and ignore the DNN component. For comparison purposes, we assume the feature embeddings are of equal size $e$.

{\bf DCN.} Our proposed model was largely inspired from DCN \cite{wang2017deep}. Let's take the efficient projection view of DCN \cite{wang2017deep}, \emph{i.e.}, it implicitly generates all the pairwise crosses and then projects it to a lower-dimensional space; {DCN-V2} is similar with a different projection structure.
    \begin{equation*}
    	\begin{split}
    	{\bf x}_{\text{DCN}}^\top = 
        {\bf x}_{\text{pairs}}
        \left[\begin{smallmatrix}
        	{\bf w} & {\bf 0} & \ldots & {\bf 0} \\
            {\bf 0} & {\bf w} & \ldots & {\bf 0} \vspace{-1ex}\\
            \vdots   & \vdots& \ddots & \vdots \\
            {\bf 0} & {\bf 0} & \ldots & {\bf w}
        \end{smallmatrix}\right], 
        {\bf x}_{\text{{DCN-V2}}}^\top = 
        {\bf x}_{\text{pairs}}
         \left[\begin{smallmatrix}
        	{\bf w}_1 & {\bf 0} & \ldots & {\bf 0} \\
            {\bf 0} & {\bf w}_2 & \ldots & {\bf 0} \vspace{-1ex}\\
            \vdots   & \vdots& \ddots & \vdots \\
            {\bf 0} & {\bf 0} & \ldots & {\bf w}_d
        \end{smallmatrix}\right]
        \end{split}
    \end{equation*}
where
${\bf x}_{\text{pairs}} = [x_i \tilde x_j]_{\forall i, j}$
contains all the $d^2$ pairwise interactions between ${\bf x}_0$ and $\tilde {\bf x}$; ${\bf w} \in \mathbb{R}^d$ is the weight vector in DCN-V; ${\bf w}_i \in \mathbb{R}^d$ is the $i^\text{th}$ column of the weight matrix in {DCN-V2} (Eq.\eqref{eq:cross_layer}).

{\bf DLRM and DeepFM.} Both are essentially 2nd-order FM without the DNN component (ignoring small differences). Hence, we simplify our analysis and compare with FM which has formula
	${\bf x}^\top {\bm \beta} + \sum_{i<j} w_{ij} \langle {\bf x}_i, {\bf x}_j \rangle$.
This is equivalent to 1-layer {DCN-V2} (Eq. \eqref{eq:cross_layer} without residual term) with a structured weight matrix.
\begin{equation*}
\label{eq:dcn-fm}
\begin{split}
    \mathbf{1}^\top \left(
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    \odot
    \left(
    \left[\begin{smallmatrix}
        {\bf 0} & w_{12} I & \cdots & w_{1k}I \\
        {\bf 0} & {\bf 0} & \cdots & w_{2k} I \vspace{-1ex}\\
        \vdots  & \vdots  & \ddots & \vdots  \\
        {\bf 0} & {\bf 0} & \cdots & {\bf 0}
    \end{smallmatrix}\right]
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    +
    \left[\begin{smallmatrix}
        {\bm \beta}_1 \\
        {\bm \beta}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bm \beta}_k
    \end{smallmatrix}\right]
    \right)
    \right)
\end{split}
\end{equation*}

{\bf xDeepFM.} The $h$-th feature map at the $k$-th layer is given by:
\begin{equation*}
\label{eq:xdeepfm}
    {\bf x}_{h, *}^k = \sum\nolimits_{i=1}^{k-1} \sum\nolimits_{j=1}^m w_{ij}^{k, h} ({\bf x}_{i, *}^{k-1} \odot {\bf x}_j)
\end{equation*}
The $h$-th feature map at the 1st layer is equivalent to 1-layer {DCN-V2} (Eq. \eqref{eq:cross_layer} without residual term).
\begin{equation*}
\label{eq:dcn-fm2}
  {\bf x}_{h, *}^1 = [I, I, \cdots, I]\left({\bf x} \odot (W{\bf x})\right) = \sum\nolimits_{i=1}^k {\bf x}_i \odot (W_{i,:} {\bf x})
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
\end{equation*}
where the $(i,j)$-th block $W_{i,j} = w_{ij} \cdot I$, and $W_{i,:} \coloneqq [W_{i,1}, \ldots, W_{i,k}]$.

{\bf AutoInt.} The interaction layer of AutoInt adopted the multi-head self-attention mechanism. For simplicity, we assume a single head is used in AutoInt; multi-head case could be compared summarily using concatenated cross layers.  

From a high-level view, the 1st layer of AutoInt outputs $\widetilde {\bf x} = [\widetilde {\bf x}_1; \widetilde {\bf x}_2; \ldots; \widetilde {\bf x}_k]$, where $\widetilde {\bf x}_i$ encodes all the 2nd-order feature interactions with the i-th feature. Then, $\widetilde {\bf x}$ is fed to the 2nd layer to learn higher-order interactions. This is the same as {DCN-V2}. 

From a low-level view (ignoring the residual terms),
\begin{equation*}
\small
    \label{eq:autoint}
    \begin{split}
        \widetilde {\bf x}_i
        &= ReLU\left(\sum\nolimits_{j=1}^k \frac{\exp \left(\langle W_{\text{q}} {\bf x}_i, W_{\text{k}} {\bf x}_j \rangle \right)}{\sum\nolimits_j \exp \left(\langle W_{\text{q}} {\bf x}_i, W_{\text{k}} {\bf x}_j \rangle \right)} (W_{\text{v}} {\bf x}_j)\right) \\
        &= ReLU\big(\sum\nolimits_{j=1}^k \text{softmax}( {\bf x}_i^\top \widetilde W {\bf x}_j)~ W_{\text{v}} {\bf x}_j \big)
\end{split}
\end{equation*}
where $\langle \cdot, \cdot \rangle$ represents inner (dot) product, and $\widetilde W = W_{\text{q}} W_{\text{k}}$.
While in {DCN-V2}, 
\begin{equation}
    \label{eq:autoint_dcn}
    \begin{split}
        \widetilde {\bf x}_i = \sum\nolimits_{j=1}^k {\bf x}_i \odot (W_{i,j} {\bf x}_j) = {\bf x}_i \odot (W_{i,:} {\bf x})
    \end{split}
\end{equation}
where $W_{i,j}$ represents the $(i,j)$-th block of $W$. It is clear that the difference lies in how we model the feature interactions. AutoInt claims the non-linearity was from ReLU($\cdot$); we consider each summation term to also contribute. Differently, {DCN-V2} used ${\bf x}_i \odot W_{i,j} {\bf x}_j$. 

{\bf PNN.} The inner-product version (IPNN) is similar to FM. For the outer-product version (OPNN), it first explicitly creates all the $d^2$ pairwise interactions,
and then projects them to a lower dimensional space $d'$ using a $d'$ by $d^2$ dense matrix. Differently, {DCN-V2} implicitly creates the interactions using a structured matrix.

\section{Research Questions}
\label{sec:research_qs}
We are interested to seek answers for these following research questions:

\begin{itemize}[leftmargin=2.5em]
    \item[\bf RQ1]  When would feature interaction learning methods become more efficient than ReLU-based DNNs?
    \item[\bf RQ2]  How does the feature-interaction component of each baseline perform without integrating with DNN? 
    \item[\bf RQ3]  How does the proposed mDCN approaches compare to the baselines? Could we achieve healthier trade-off between model accuracy and cost through mDCN and the mixture of low-rank DCN?
    \item[\bf RQ4] How does the settings in mDCN affect model quality?
    \item[\bf RQ5] Is mDCN capturing important feature crosses? Does the model provide good understandability?
\end{itemize}

Throughout the paper, ``CrossNet" or ``CN" represents the cross network; suffix ``Mix" denotes the mixture of low-rank version. 

\section{Empirical understanding of feature crossing techniques (RQ1)}
\label{sec:exp_synthetic}
Many recent works \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, qu2016product, lian2018xdeepfm, naumov2019deep} proposed to model explicit feature crosses that couldn't be learned efficiently from traditional neural networks. However, most works only studied public datasets with unknown cross patterns and noisy data; few work has studied in a clean setting with known ground-truth models.
Hence, it's important to understand : 1) in which cases would traditional neural nets become inefficient; 2) the role of each component in the cross network of {DCN-V2}.

We use the cross network in DCN models to represent those feature cross methods and compare with ReLUs, which are commonly used in industrial recommender systems.
To simplify experiments and ease understanding, we assume each feature $x_i$ is of dimension one, and monomial $x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d}$ represents a $|{\bm \alpha}|$-order interaction between features.

{\bf Performance with increasing difficulty.}
Consider only 2nd-order feature crosses and let the ground-truth model be $f({\bf x}) = \sum_{|{\bm \alpha}|=2} w_{{\bm \alpha}} x_1^{\alpha_1}x_2^{\alpha_2}\ldots x_d^{\alpha_d}$.
Then, the difficulty of learning $f({\bf x})$ depends on: 1) sparsity ($w_{{\bm \alpha}}=0$), the number of crosses, and 2) similarity of the cross patterns (characterized by $\mathrm{Var}(w_{{\bm \alpha}})$), meaning a change in one feature would simultaneously affect most feature crosses by similar amount. We create synthetic datasets with increasing difficulty in Eq. \eqref{eq:2nd-order}.
\begin{equation}
    \label{eq:2nd-order}
    \begin{split}
    f_1({\bf x}) &= x_1^2+x_1x_2+x_3x_1+x_4x_1 \\
    f_2({\bf x}) &= x_1^2+0.1x_1x_2+x_2x_3+0.1x_3^2 \\
    f_3({\bf x}) &= \sum\nolimits_{(i,j) \in S} w_{ij} x_i x_j, ~~{\bf x} \in \mathbb{R}^{100}, |S|=100
    \end{split}
\end{equation}
where set $S$ and weights $w_{ij}$ are randomly assigned, and $x_i$'s are uniformly sampled from interval [-1, 1].

\autoref{tab:2nd-order} reports mean RMSE out of 5 runs and the model size.
When the cross patterns are simple ($f_1$), both {DCN-V2} and DCN are efficient. When the patterns become more complicated ($f_3$), {DCN-V2} remains accurate while DCN degrades. DNN's performance remains poor even with a wider and deeper structure (layer sizes [200, 200] for $f_1$ and $f_2$, [1024, 512, 256] for $f_3$). This suggests the inefficiency of DNN in modeling monomial patterns.

\begin{table}[htpb]
\footnotesize
\vspace{-5pt}
\caption{RMSE and Model Size (\# Parameters) for Polynomial Fitting of Increasing Difficulty.}
\vspace{-3.5ex}
\label{tab:2nd-order}
\begin{center}
\begin{tabular}{c|ll|ll|ll|ll}
\toprule
& \multicolumn{2}{c|}{DCN (1Layer)}  & \multicolumn{2}{c|}{DCN-V2 (1Layer)}  & \multicolumn{2}{c|}{DNN (1Layer)} & \multicolumn{2}{c}{DNN (large)}\\
& RMSE & Size & RMSE & Size & RMSE & Size & RMSE & Size \\
\midrule
$f_1$& 8.9E-13 & 12 & {\bf 5.1E-13} & 24 & 2.7E-2 & 24  & 4.7E-3 & 41K 
\\
$f_2$ & 1.0E-01 & 9  & {\bf 4.5E-15} & 15 & 3.0E-2 & 15 & 1.4E-3 & 41K\\
$f_3$ & 2.6E+00 & 300 & {\bf 6.7E-07} & 10K & 2.7E-1 & 10K & 7.8E-2 & 758K\\
\bottomrule
\end{tabular}
\end{center}
\vspace{-5pt}
\end{table}

{\bf Role of each component.}
We also conducted ablation studies on homogeneous polynomials of order 3 and 4, respectively. For each order, we randomly selected 20 cross terms from ${\bf x} \in \mathbb{R}^{50}$.

\autoref{fig:dcn-poly-order} shows the change in mean RMSE with layer depth. Clearly, ${\bf x}_0 \odot (W{\bf x}_i)$ models order-$d$ crosses at layer $d$-1, which is verified by that the best performance for order-3 polynomial is achieved at layer 2 (similar for order-4). At other layers, however, the performance significantly degrades. This is where the bias and residual terms are helpful --- they create and maintain all the crosses up to the highest order. This reduces the performance gap between layers, and stabilizes the model when redundant crosses are introduced. This is particularly important for real-world applications with unknown cross patterns.

Fig. \ref{fig:dcn-poly-order} also reveals the limited expressiveness of {DCN} in modeling complicated cross patterns.


\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth-order3-v1.png}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.208\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth-order4-v1.png}
    \end{subfigure}
    \vspace{-1ex}
    \caption{Homogeneous polynomial fitting of order 3 and 4. $x$-axis represents the number of layers used; $y$-axis represents RMSE (the lower the better). In the legend, the top 3 models are {DCN-V2} with different component(s) included.}
\label{fig:dcn-poly-order}
\end{figure}

{\bf Performance with increasing layer depth.} We now study scenarios closer to real-world settings, where the cross terms are of a combined order.  
\begin{equation*}
\small
    \label{eq:combined-order}
    \begin{split}
    f({\bf x}) = & {\bf x}^\top {\bf w} + 
\sum_{{\bm \alpha} \in S} w_{{\bm \alpha}} x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d}  +
0.1 \sin(2{\bf x}^\top {\bf w}_{s} + 0.1) +
0.01 \epsilon
    \end{split}
\end{equation*}
where the randomly chosen set $S = S_2 \cup S_3 \cup S_4$, $|S_2|=20, |S_3|=10, |S_4|=5$, and $\forall {\bm \alpha} \in S_i, |{\bm \alpha}|=i$; sine introduces perturbations and $\epsilon$ represents Gaussian noises.

\autoref{tab:combined-order} reports the mean RMSE out of 5 runs. With the increase of layer depth, CN-M was able to capture higher-order feature crosses in the data, resulting in improved performance. Thanks to the bias and residual terms, the performance didn't degrade beyond layer 3, where redundant feature interactions were introduced.

\begin{table}[htbp]
\small
\caption{Combined-order (1 - 4) Polynomial Fitting.}
\vspace{-3.ex}
\label{tab:combined-order}
    \begin{subtable}[h]{0.45\textwidth}
        \centering
\begin{tabular}{c|ccccc}
\toprule
{\bf \#Layers} & 1 & 2 & 3 & 4 & 5\\
\midrule
{DCN-V2} & 1.43E-01 & 2.89E-02 & \bf 9.82E-03 & 9.87E-03 & 9.92E-03 \\
DNN& 1.32E-01 & 1.03E-01 & 1.03E-01 &  1.09E-01 & 1.05E-01\\
\bottomrule
\end{tabular}
    \end{subtable}
\end{table}

To summarize, ReLUs are inefficient in capturing explicit feature crosses (multiplicative relations) even with a deeper and larger network. This is well aligned with previous studies \cite{beutel2018latent}.
The accuracy considerably degrades when the cross patterns become more complicated. DCN accurately captures simple cross patterns but fails at more complicated ones. {DCN-V2}, on the other hand, remains accurate and efficient for complicated cross patterns.

\section{Experimental Results (RQ2 - RQ5)}
\label{sec:exp_public}
This section empirically verifies the effectiveness of {DCN-V2} in feature interaction learning across 3 datasets and 2 platforms, compared with SOTA. In light of recent concerns about poor reproducibility of published results \cite{dacrema2019we, musgrave2020metric, rendle2020neural}, we conducted a fair and comprehensive experimental study with extensive hyper-parameter search to properly tune all the baselines and proposed approaches. In addition, for each optimal setup, we train 5 models with different random initialization, and report the mean and standard deviation. 

\autoref{sec:performance_feature_interaction} studies the performance of the feature-cross learning components (\textbf{RQ2}) between baselines \emph{without} integrating with DNN ReLU layers (similar to \cite{lian2018xdeepfm, song2019autoint}); only sparse features are considered for a clean comparison. \autoref{sec:performance_baselines} compares {DCN-V2} with all the baselines comprehensively (\textbf{RQ3}). \autoref{sec:hyper-parameters} evaluates the influence of hyper-parameters on the performance of {DCN-V2} (\textbf{RQ4}). \autoref{sec:model_understanding} focuses on model understanding (\textbf{RQ5}) of whether we are indeed discovering meaningful feature crosses with {DCN-V2}. 


\subsection{Experiment Setup}
\label{sec:experiment_setup}
This section describes the experiment setup, including training datasets, baseline approaches, and details of the hyper-parameter search and training process.
\subsubsection{Datasets}
\autoref{tab:datasets} lists the statistics of each dataset:


\begin{table}[htpb]
\small
\caption{Datasets.}
\vspace{-3.5ex}
\label{tab:datasets}
\begin{center}
\begin{tabular}{c|ccc}
\toprule
{\bf Data} & \# Examples & \# Features & Vocab Size \\
\midrule
Criteo & 45M & 39 & 2.3M\\
MovieLen-1M &  740k & 7 & 3.5k\\
Production & $>$ 100B & NA & NA\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

{\bf Criteo\footnote{http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset}.} The most popular click-through-rate (CTR) prediction benchmark dataset contains user logs over a period of 7 days. We follow \cite{wang2017deep, song2019autoint} and use first 6 days for training, and randomly split the last day's data into validation and test set equally. We log-normalize ($\log(x+4)$ for feature-2 and $\log(x+1)$ for others) the 13 continuous features and embed the remaining 26 categorical features.

{\bf MovieLen-1M\footnote{https://grouplens.org/datasets/movielens}.} The most popular dataset for recommendation systems research. Each training example includes a {\ttfamily{$\langle$\small user-features, \small movie-features, \small rating$\rangle$}} triplet. Similar to AutoInt~\cite{song2019autoint}, we formalize the task as a regression problem. All the ratings for 1s and 2s are normalized to be 0s; 4s and 5s to be 1s; and rating 3s are removed. 6 non-multivalent categorical features are used and embedded. The data is randomly split into 80\% for training, 10\% for validation and 10\% for testing.

\subsubsection{Baselines.}
\label{sec:baselines}
We compare our proposed approaches with 6 SOTA feature interaction learning algorithms. A brief comparison between the approaches is highlighted in \autoref{tab:model_comparisons}.


\begin{table}[htpb]
\small
\caption{High-level comparison between models. Assuming the input ${\bf x}_0 = [{\bf v}_1; \ldots; {\bf v}_k]$ contains $k$ feature embeddings that each represented as ${\bf v}_i$. $\oplus$ denotes concatenation; $\otimes$ denotes outer-product; $\odot$ denotes Hadamard-product. $f_i(\cdot)$ represents implicit feature interactions, \emph{i.e.,} ReLU layers. In the last column, the `+' sign is on the logit level.}
\vspace{-2ex}
\label{tab:model_comparisons}
\begin{center}
\begin{tabular}{l|p{5mm}l|l}
\toprule
\multirow{ 2}{*}{{\bf Model}}  & \multicolumn{2}{c|}{ Explicit Interactions ($f_e$)} &  \multirow{ 2}{*}{\begin{tabular}{@{}l@{}}Final \\ Objective  \end{tabular}}  \\
\cmidrule(){2-3}
& Order & \multicolumn{1}{c|}{(Simplified) Key Formula}  & \\
\midrule
\multirow{ 2}{*}{{PNN \cite{qu2016product}}}  &\multirow{ 2}{*}{2}& ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$ (IPNN) & \multirow{ 2}{*}{$f_i \circ f_e$}   \\
&&${\bf x}_o = [\text{vec}({\bf v}_i \otimes {\bf v}_j) \mathrel{|} \forall i, j]$ (OPNN) & \\
DeepFM \cite{guo2017deepfm} & $2$ & ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$ &  $f_i + f_e$ \\
DLRM \cite{naumov2019deep} & 2 & ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$& $f_i \circ f_e$\\
DCN \cite{wang2017deep} & $\ge 2$ & ${\bf x}_{i+1} = {\bf x}_0 \otimes {\bf x}_i {\bf w}_i$ &$f_i + f_e$ \\
xDeepFM \cite{lian2018xdeepfm} & $\ge 2$ & ${\bf v}_{h}^k = \sum_{i, j} w_{ij}^{kh} ({\bf v}_{i}^{k-1} \odot {\bf v}_j)$  & $f_i + f_e$ \\
AutoInt \cite{song2019autoint} & NA & $ \widetilde {\bf v}_i = g \left(\frac{\sum_{j}\exp (\langle W_q {\bf v}_i, W_k {\bf v}_j \rangle )W_v {\bf v}_j}{\sum_j \exp \left(\langle W_q {\bf v}_i, W_k {\bf v}_j \rangle \right)})\right)$ & $f_i + f_e$\\
\midrule
\multirow{ 2}{*}{{DCN-V2} (ours)} & \multirow{ 2}{*}{ $\ge 2$} & \multirow{ 2}{*}{${\bf x}_i = {\bf x}_0 \odot (W_i {\bf x}_i)$} & $f_i \circ f_e$\\
&&&$f_i + f_e$ \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\subsubsection{Implementation Details.}
\label{sec:implementation_details}
All the baselines and our approaches are implemented in TensorFlow v1. For a fair comparison, all the implementations were identical across all the models except for the feature interaction component \footnote{We adopted implementation from \url{https://github.com/Leavingseason/xDeepFM}, \url{https://github.com/facebookresearch/dlrm} and \url{https://github.com/shenweichen/DeepCTR}}.

{\bf Embeddings.} All the baselines require each feature's embedding size to be the same except for DNN and DCN. Hence, we fixed it to be $ \text{Avg}\big(\sum_{\text{vocab}} 6\cdot(\text{vocab cardinality})^{\frac{1}{4}}\big)$ (39 for Criteo and 30 for Movielen-1M) for all the models\footnote{This formula is a rule-of-thumb number that is widely used \cite{wang2017deep}, also see \url{https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html}}. 

{\bf Optimization.} We used Adam \cite{kingma2014adam} with a batch size of $512$ (128 for MovieLen). The kernels were initialized with He Normal \cite{he2015delving},
and biases to ${\bf 0}$; the gradient clipping norm was 10; an exponential moving average with decay 0.9999 to trained parameters was applied.

{\bf Reproducibility and fair comparisons: hyper-parameters tuning and results reporting.} For all the baselines, we conducted a coarse-level (larger-range) grid search over the hyper-parameters, followed by a finer-level (smaller-range) search. To ensure reproducibility and mitigate model variance, for each approach and dataset, we report the mean and stddev out of 5 independent runs for the best configuration. We describe detailed settings below for Criteo; and follow a similar process for MovieLens with different ranges.

We first describe the hyper-parameters shared across the baselines. The learning rate was tuned from $10^{-4}$ to $10^{-1}$ on a log scale and then narrowed down to $10^{-4}$ to $5 \times 10^{-4}$ on a linear scale. The training steps were searched over \{150k, 160k, 200k, 250k, 300k\}. The number of hidden layers ranged in \{1, 2, 3, 4\} with their layer sizes in \{562, 768, 1024\}. And the regularization parameter $\lambda$ was in \{0, $3 \times 10^{-5}$, $10^{-4}$\}.

We then describe each model's own hyper-parameters, where the search space is designed based on reported setting. For DCN, the number of cross layers ranged from 1 to 4. For AutoInt, the number of attention layers was from 2 to 4; the attention embedding size was in \{20, 32, 40\}; the number of attention head was from 2 to 3; and the residual connection was either on or off. For xDeepFM, the CIN layer size was in \{100, 200\}, depth in \{2, 3, 4\}, activation was identity, computation was either direct or indirect. For DLRM, the bottom MLP layer sizes and numbers was in \{(512,256,64), (256,64)\}. For PNN, we ran for IPNN, OPNN and PNN*, and for the latter two, the kernel type ranged in \{full matrix, vector, number\}.
For all the models, the total number of parameters was capped at $1024^2 \times 5$ to limit the search space and avoid overly expensive computations.

\subsection{Performance of Feature Interaction Component Alone (RQ2)}
\label{sec:performance_feature_interaction}
We consider the feature interaction component alone of each model {\bf without their DNN component}. Moreover, we only consider the categorical features, as the dense features were processed differently among baselines. \autoref{tab:cross-only} shows the results on Criteo dataset. Each baseline was tuned similarly as in \autoref{sec:implementation_details}. There are two major observations. 1). Higher-order methods demonstrate a superior performance over 2nd-order methods. This suggests high-order crosses are meaningful in this dataset. 2). Among the high-order methods, cross network achieved the best performance and was on-par or slightly better compared to DNN.

\begin{table}[htpb]
\small
\caption{LogLoss (test) of feature interaction component of each model (no DNN). Only categorical features were used. In the `Setting' column, $l$ stands for number of layers.}
\vspace{-2ex}
\label{tab:cross-only}
\begin{center}
\begin{tabular}{c|l|cl}
\toprule
& Model & LogLoss & \multicolumn{1}{c}{Best Setting}\\
\midrule
\multirow{ 3}{*}{2nd} 
&PNN \cite{qu2016product} &0.4715 $\pm$ 4.430e-04 & OPNN, kernel=matrix\\
&FM      & 0.4736 $\pm$ 3.04E-04& \multicolumn{1}{c}{--} \\
\midrule
\multirow{ 5}{*}{$>$2} 
&CIN \cite{lian2018xdeepfm}  & 0.4719 $\pm$ 9.41E-04& l=3, cinLayerSize=100\\
&AutoInt \cite{song2019autoint}   & 0.4711 $\pm$ 1.62E-04& l=2, head=3, attEmbed=40\\
&DNN        & 0.4704 $\pm$ 1.57E-04& l=2, size=1024\\
\cline{2-4}
&CrossNet     & 0.4702 $\pm$ 3.80E-04& l=2\\
&CrossNet-Mix & \bf 0.4694 $\pm$ 4.35E-04 & l=5, expert=4, gate=$\frac{1}{1+e^{-x}}$\\

\bottomrule
\end{tabular}
\end{center}
\end{table}

\subsection{Performance of Baselines (RQ3)}
\label{sec:performance_baselines}
This section compares the performance between {DCN-V2} approaches and the baselines in an end-to-end fashion. Note that the best setting reported for each model was searched over a {\bf wide-ranged model capacity and hyper-parameter space} including the baselines. And if two settings performed on-par, we report the {\bf lower-cost} one. \autoref{tab:baseline_comparison} shows the best LogLoss and AUC (Area Under the ROC Curve) on testset for Criteo and MovieLen. For Criteo, a {\bf 0.001-level improvement} is considered significant (see \cite{song2019autoint, wang2017deep, guo2017deepfm}). 
We see that {DCN-V2} consistently outperformed the baselines (including DNN) and achieved a healthy quality/cost trade-off. It's also worth mentioning that the baselines' performances reported in \autoref{tab:baseline_comparison} were improved over the numbers reported by previous papers (see \autoref{tab:metrics_in_papers} in Appendix); however, when integrated with DNN, their performance gaps are closing up (compared to \autoref{tab:cross-only}) with their performances on-par and sometimes worse than the ReLU-based DNN with fine-granular model tuning.

{\bf Best Settings.} The optimal hyper-parameters are in \autoref{tab:baseline_comparison}. For {DCN-V2} models, both the `stacked' and `parallel' structures outperformed all the baselines, while `stacked' worked better on Criteo and `parallel' worked better on Movielen-1M. On Criteo, the setting was gate as constant, hard\_tanh activation for {DCN-Mix}; gate as softmax and identity activation for CrossNet. The best training steps was 150k for all the baselines; learning rate varies for all the models.


{\bf Model Quality --- Comparisons among baselines.} When integrating the feature cross learning component with a DNN, the advantage of higher-order methods is less pronounced, and the performance gap among all the models are closing up on Criteo (compared to \autoref{tab:cross-only}). \textbf{This suggests the importance of implicit feature interactions and the power of a well-tuned DNN model.}

For 2nd-order methods, DLRM performed inferiorly to DeepFM although they are both derived from FM. This might be due to DLRM's omission of the 1st-order sparse features after the dot-product layer. PNN models 2nd-order crosses more expressively and delivered better performance on MovieLen-1M; however on Criteo, its mean LogLoss was driven up by its high standard deviation. For higher-order methods, xDeepFM, AutoInt and DCN behaved similarly on Criteo, while on MovieLens xDeepFm showed a high variance.

{DCN-V2} achieved the best performance (0.001 considered to be significant on Criteo \cite{wang2017deep, lian2018xdeepfm, song2019autoint}) by explicitly modeling up to 3rd-order crosses beyond those implicit ones from DNN. {DCN-Mix}, the mixture of low-rank DCN, efficiently utilized the memory and reduced the cost by 30\% while maintaining the accuracy. Interestingly, CrossNet alone outperformed DNN on both datasets; we defer more discussions to \autoref{sec:crossnet_performance}.


{\bf Model Quality --- Comparisons with DNN.} DNNs are universal approximators and are tough-to-beat baselines when highly-optimized. Hence, we finely tuned DNN along with all the baselines, and used a larger layer size than those used in literature (\emph{e.g.}, 200 - 400 in \cite{lian2018xdeepfm, song2019autoint}). \textbf{To our surprise, DNN performed neck to neck with most baselines and even outperformed certain models.}

Our hypothesis is that those explicit feature crosses from baselines were not modeled in an {\bf expressive} and {\bf easy-to-optimize} manner. The former makes its performanc easy to be matched by a DNN with large capacity. The latter would easily lead to trainability issues, making the model unstable, hard to identify a good local optima or to generalize. Hence, when integrated with DNN, the overall performance is dominated by the DNN component. This becomes especially true with a large-capacity DNN, which could already approximate some simple cross patterns.

In terms of expressiveness, consider the 2nd-order methods. PNN models crosses more expressively than DeepFM and DLRM, which resulted in its superior performance on MovieLen-1M. This also explains the inferior performance of DCN compared to {DCN-V2}.

In terms of trainability, certain models might be inherently more difficult to train and resulted in unsatisfying performance. Consider PNN. On MoiveLen-1M, it outperformed DNN, suggesting the effectiveness of those 2nd-order crosses. On Criteo, however, PNN's advantage has diminished and the averaged performance was on-par with DNN. This was caused by the instability of PNN. Although its best run was better than DNN, its high stddev from multiple trials has driven up the mean loss. xDeepFM also suffers from trainability issue (see its high stddev on MovieLens). In xDeepFM, each feature map encodes all the pair-wise crosses while only relies on a single variable to learn the importance of each cross. In practice, a single variable is difficult to be learned when jointly trained with magnitudes more parameters. Then, an improperly learned variable would lead to noises.


{DCN-V2}, on the other hand, consistently outperforms DNN. It successfully leveraged both the explicit and implicit feature interactions. We attribute this to the balanced number of parameters between the cross network and the deep network ({\bf expressive}), as well as the simple structure of cross net which eased the optimization ({\bf easy-to-optimize}). It's worth noting that the high-level structure of {DCN-V2} shares a similar spirit of the self-attention mechanism adopted in AutoInt, where each feature embedding attends to a weighed combination of other features. The difference is that during the attention, higher-order interactions were modeled explicitly in {DCN-V2} but implicitly in AutoInt.

{\bf Model Efficiency.} \autoref{tab:baseline_comparison} also provides details for model size and FLOPS\footnote{FLOPS is a close estimation of run time, which is subjective to implementation details.}. The reported setting was properly tuned over the hyper-parameters of each model and the DNN component. For most models, the FLOPS is roughly 2x of the \#params; for xDeepFM, however, the FLOPS is one magnitude higher, making it impractical in industrial-scale applications (also observed in \cite{song2019autoint}). Note that for DeepFM and DLRM, we've also searched over larger-capacity models; however, they didn't deliver better quality. Among all the methods, {DCN-V2} delivers the best performance while remaining relatively efficient; {DCN-Mix} further reduced the cost, achieving a better trade-off between model efficiency and quality.

\begin{table*}[htpb]
\small
\caption{LogLoss and AUC (test) on Criteo and Movielen-1M. The metrics were averaged over 5 independent runs with their stddev in the parenthesis. In the `Best Setting' column, the left reports DNN setting and the right reports model-specific setting. $l$ denotes layer depth; $n$ denotes CIN layer size; $h$ and $e$, respectively, denotes \#heads and att-embed-size; $K$ denotes \#experts and $r$ denotes total rank.}
\vspace{-3ex}
\label{tab:baseline_comparison}
\begin{center}
\begin{tabular}{l|ccp{2.5em}p{2.5em}ll|ccp{2.5em}p{2.5em}}
\toprule
\multirow{ 2}{*}{{\bf Baseline}} & \multicolumn{6}{c|}{\bf Criteo} & \multicolumn{4}{c}{\bf MovieLens-1M} \\
& Logloss  & AUC & Params & FLOPS &  \multicolumn{2}{c|}{Best Setting} &Logloss & AUC  & Params & FLOPS  \\
\midrule
PNN     & 0.4421 (5.8E-4) & 0.8099 (6.1E-4) & 3.1M & 6.1M & (3, 1024) & OPNN
         & 0.3182 (1.4E-3) & 0.8955 (3.3E-4) & 54K & 110K\\
DeepFm  & 0.4420 (1.4E-4) & 0.8099 (1.5E-4) & 1.4M & 2.8M & (2, 768) & \multicolumn{1}{c|}{--}
         & 0.3202 (1.0E-3) & 0.8932 (7.7E-4) & 46K & 93K\\
DLRM    & 0.4427 (3.1E-4) & 0.8092 (3.1E-4) & 1.1M & 2.2M & (2, 768) &  [512,256,64]
         & 0.3245 (1.1E-3) & 0.8890 (1.1E-3) & 7.7K & 16K\\
xDeepFm & 0.4421 (1.6E-4) & 0.8099 (1.8E-4) & 3.7M & 32M & (3, 1024) & $l$=2, $n$=100
         & 0.3251 (4.3E-3) & 0.8923 (8.6E-4) & 160K & 990K\\
AutoInt+ & 0.4420 (5.7E-5) & 0.8101 (2.6E-5) & 4.2M & 8.7M & (4, 1024) & $l$=2, $h$=2, $e$=40
         & 0.3204 (4.4E-4) & 0.8928 (3.9E-4) & 260K & 500K\\
DCN     & 0.4420 (1.6E-4) & 0.8099 (1.7E-4) & 2.1M & 4.2M & (2, 1024) & $l$=4
         & 0.3197 (1.9E-4) & 0.8935 (2.1E-4) & 110K & 220K\\
DNN    & 0.4421 (6.5E-5) & 0.8098 (5.9E-5) & 3.2M & 6.3M & (3, 1024) &  \multicolumn{1}{c|}{--}
         & 0.3201 (4.1E-4) & 0.8929 (2.3E-4) & 46K & 92K\\
\midrule
\multicolumn{1}{c}{\bf Ours}& & & & & & \multicolumn{1}{c}{}& & & &\\
{DCN-V2}  & \bf 0.4406 (6.2E-5) & \bf 0.8115 (7.1E-5) & 3.5M & 7.0M & (2, 768) & $l$=2
         & 0.3170 (3.6E-4) & 0.8950 (2.7E-4) & 110K & 220K \\
{DCN-Mix} & 0.4408 (1.0E-4) & 0.8112 (9.8E-5) & 2.4M & 4.8M  & (2, 512) & $l$=3, $K$=4, $r$=258
         & \bf 0.3160 (4.9E-4) & \bf 0.8964 (2.9E-4) & 110K & 210K \\
CrossNet &0.4413 (2.5E-4) & 0.8107 (2.4E-4) & 2.1M & 4.2M &  -- & $l$=4, $K$=4, $r$=258
         & 0.3185 (3.0E-4) & 0.8937 (2.7E-4) & 65K & 130K \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-5pt}
\end{table*}

\subsection{Can Cross Layers Replace ReLU layers?}
\label{sec:crossnet_performance}
The solid performance of {DCN-V2} approaches has inspired us to further study the efficiency of their cross layers (CrossNet) in learning explicit high-order feature crosses.

In a realistic setting with resource constraints, we often have to limit model capacity. Hence, we fixed the model capacity (memory / \# of parameters) at different levels, and compared the performance between a model with only cross layers (Cross Net), and a ReLU based DNN. \autoref{tab:logloss_memory} reports the best test LogLoss for different memory constraints. The memory was controlled by varying the number of cross layers and its rank (\{128, 256\}), the number of hidden layers and their sizes. The best performance was achieved by the cross network (5-layer), suggesting the ground-truth model could be well-approximated by polynomials. Moreover, the best performance per memory limit was also achieved by the cross network, indicating both solid effectiveness and efficiency.

It is well known that ReLU layers are the backbone for various Neural Nets models including DNN, Recurrent Neural Net (RNN) \cite{rumelhart1985learning, hochreiter1997long, mikolov2011extensions} and Convolutional Neural Net (CNN) \cite{lecun1989backpropagation, schmidhuber2015deep, lawrence1997face}. It is quite surprising and encouraging to us that we may potentially replace ReLU layers by Cross Layers entirely for certain applications. Obviously we need significant more analysis and experiments to verify the hypothesis. Nonetheless, this is a very interesting preliminary study and sheds light for our future explorations on cross layers.

\begin{table}[htbp]
\small
\vspace{-4pt}
\caption{Logloss and AUC (test) with a fixed memory budget.}
\vspace{-10pt}
\label{tab:logloss_memory}
\begin{center}
\begin{tabular}{cc|cccc}
\toprule
\multicolumn{2}{c|}{\bf \#Params} & 7.9E+05 & 1.3E+06 & 2.1E+06 & 2.6E+06\\
\midrule
\multirow{ 2}{*}{LogLoss}& CrossNet & \bf 0.4424 & \bf 0.4417 & \bf 0.4416 & \bf 0.4415\\
 &  DNN & 0.4427 & 0.4426 & 0.4423 & 0.4423\\
 \midrule
\multirow{ 2}{*}{AUC} &  CrossNet & \bf 0.8096 & \bf 0.8104 & \bf 0.8105  & \bf 0.8106\\
& DNN & 0.8091 & 0.8094  & 0.8096 & 0.80961 \\

\bottomrule
\end{tabular}
\end{center}
\vspace{-4pt}
\end{table}

\subsection{How the Choice of Hyper-parameters Affect {DCN-V2} Model Performance (RQ4)} 
\label{sec:hyper-parameters}
This section examines the model performance as a function of hyper-parameters that include 1) depth of cross layers; 2) matrix rank of {DCN-Mix}; 3) number of experts in {DCN-Mix}.

{\bf Depth of Cross Layers.}
By design, the highest feature cross order captured by the cross net increases with layer depth. Hence, we constrain ourselves to the full-rank cross layers, and evaluate the performance change with layer depth

\autoref{fig:dcn-layer-depth} shows the test LogLoss and AUC while increasing layer depth on the Criteo dataset. We see a steady quality improvement with a deeper cross network, indicating that it's able to capture more meaningful crosses. The rate of improvement, however, slowed down when more layers were used. This suggests the contribution from that of higher-order crosses is less significant than those from lower-order crosses. We also used a same-sized DNN as a reference. When there were $\le 2$ layers, DNN outperformed the cross network; when more layers became available, the cross network started to close the performance gap and even outperformed DNN. In the small-layer regime, the cross network could only approximate very low-order crosses (\emph{e.g.,} 1 $\sim$ 2); in the large-layer regime, those low-order crosses were characterized with more parameters, and those high-order interactions were started to be captured.

\begin{figure}[htbp]
\small
\centering
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth.png}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-rank-v2.png}
    \end{subfigure}\\[-2ex]
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-auc.png}
    \caption{Layer depth}
    \label{fig:dcn-layer-depth}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-rank-auc.png}
    \caption{Matrix rank}
    \label{fig:dcn-rank}
    \end{subfigure}
    \vspace{-2ex}
    \caption{Logloss and AUC (test) v.s. depth \& matrix rank.}
\vspace{-10pt}
\end{figure}

{\bf Rank of Matrix.}
The rank of the weight matrix controls the number of parameters as well as the portion of low-frequency signals passing through the cross layers. Hence, we study its influence on model quality. The model is based on a well-performed setting with 3 cross layers followed by 3 hidden layers of size 512. We approximate the dense matrix $W$ in each cross layer by $UV^\top$ where $U, V \in \mathbb{R}^{d \times r}$, and we vary $r$. We loosely consider the smaller dimension $r$ to be the rank

\autoref{fig:dcn-rank} shows the test LogLoss and AUC v.s. matrix's rank $r$ on Criteo. When $r$ was as small as 4, the performance was on-par with other baselines. When $r$ was increased from 4 to 64, the LogLoss decreased almost linearly with $r$ (\emph{i.e.}, model's improving). When $r$ was further increased from 64 to full, the improvement on LogLoss slowed down. We refer to 64 as the \emph{threshold rank}. The significant slow down from 64 suggests that the important signals characterizing feature crosses could be captured in the top-64 singular values.

Our hypothesis for the value of this \emph{threshold rank} is $O(k)$ where $k$ represents \# features (39 for Criteo). Consider the $(i,j)$-th block of matrix $W$, we can view $\small W_{i,j} = W_{i,j}^L + W_{i,j}^H$, where $W_{i,j}^L$ stores the dominant signal (low-frequency) and $\small W_{i,j}^H$ stores the rest (high-frequency). In the simplest case where $\small W_{i,j}^L = c_{ij} {\bf 1} {\bf 1}^\top$, the entire matrix $\small W^L$ will be of rank $k$. The effectiveness of this hypothesis remains to be verified across multiple datasets

{\bf Number of Experts.}
We study how the number of low-rank experts affects the quality. We've observed that 1) best-performed setting (\#expert, gate, matrix activation type) was subjective to datasets and model architectures; 2) the best-performed model of each setting yielded similar results. For example, for a 2-layered cross net with total rank 256 on Criteo, the LogLoss for 1, 4, 8, 16, and 32 experts, respectively, was 0.4418, 0.4416, 0.4416, 0.4422, and 0.4420. The fact that more lower-ranked experts wasn't performing better than a single higher-ranked expert might be caused by the na\"ive gating functions and optimizations adopted. We believe more sophisticated gating \cite{jang2016categorical, louizos2017learning, ma2019snr} and optimization techniques (\emph{e.g.}, alternative training, special initialization, temperature adjustment) would leverage more from a mixture of experts. This, however, is beyond the scope of this paper and we leave it to future work.


\subsection{Model Understanding (RQ5)}
\label{sec:model_understanding}
One key research question is whether the proposed approaches are indeed learning meaningful feature crosses. A good understanding about the learned feature crosses helps improve model understandability, and is especially crucial to fields like ML fairness and ML for health. Fortunately, the weight matrix $W$ in {DCN-V2} exactly reveals what feature crosses the model has learned to be important. Specifically, we assume that each input ${\bf x} = [{\bf x}_1; {\bf x}_2; \ldots; {\bf x}_k]$ contains $k$ features with each represented by an embedding ${\bf x}_i$. Then, the block-wise view of the feature crossing component (ignoring the bias) in Eq. \eqref{eq:block-mat} shows that the importance of feature interaction between $i$-th and $j$-th feature is characterized by the $(i,j)$-th block $W_{i,j}$.
\begin{equation} \label{eq:block-mat}
    {\bf x} \odot W {\bf x} = 
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    \odot
    \left[\begin{smallmatrix}
        W_{1,1} & W_{1,2} & \cdots & W_{1,k} \\
        W_{2,1} & W_{2,2} & \cdots & W_{2,k} \vspace{-1ex}\\
        \vdots  & \vdots  & \ddots & \vdots  \\
        W_{k,1} & W_{k,2} & \cdots & W_{k,k}
    \end{smallmatrix}\right]
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
\end{equation}

\autoref{fig:dcn-mat-visualization} shows the learned weight matrix $W$ in the first cross layer. Subplot (a) shows the entire matrix with orange boxes highlighting some notable feature crosses. The off-diagonal block corresponds to crosses that are known to be important, suggesting the effectiveness of {DCN-V2}. The diagonal block represents self-interaction ( $x^2$'s).
Subplot (b) shows each block's Frobenius norm and indicates some strong interactions learned, \emph{e.g.}, {\ttfamily{\small Gender}} $\times$ {\ttfamily{\small UserId}}, {\ttfamily{\small MovieId}} $\times$ {\ttfamily{\small UserId}}.

\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.18\textwidth}  
    \includegraphics[width=\textwidth,valign=t]{dcn-prod-visualization.png}
    \caption{Production data}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth,valign=t]{dcn-movielen-visualization.png}
    \caption{Movielen-1M}
    \end{subfigure}
    \caption{Visualization of learned weight matrix in {DCN-V2}. Rows and columns represents real features. For (a), feature names were not shown for proprietary reasons; darker pixel represents larger weight in its absolute value. For (b), each block represents the Frobenius norm of each matrix block.}
\label{fig:dcn-mat-visualization}
\end{figure}

\section{Productionizing {DCN-V2} at Google}
\label{sec:productionization}
This section provides a case study to share our experience productionizing {DCN-V2} in a large-scale recommender system in Google. We've achieved significant gains through {DCN-V2} in both offline model accuracy, and online key business metrics.

{\bf The Ranking Problem:}
Given a user and a large set of candidates, our problem is to return the top-$k$ items the user is most likely to engage with. Let's denote the training data to be $\{({\bf x}_i, y_i)\}_{i=1}^N$, where ${\bf x}_i$'s represents features of multiple modalities, such as user's interests, an item's metadata and contextual features; $y_i$'s are labels representing a user's action (\emph{e.g.}, a click). The goal is to learn a function $f: \mathbb{R}^d \mapsto \mathbb{R}$ that predicts the probability $P(y | {\bf x})$, the user's action $y$ given features ${\bf x}$.

{\bf Production Data and Model:}
The production data are sampled user logs consisting of hundreds of billions of training examples. The vocabulary sizes of sparse features vary from 2 to millions. The baseline model is a fully-connected multi-layer perceptron (MLP) with ReLU activations.




{\bf Comparisons with Production Models:} When compared with production model, {DCN-V2} yielded 0.6\% AUCLoss (1 - AUC) improvement. For this particular model, a gain of 0.1\% on AUCLoss is considered a significant improvement.  We also observed significant online performance gains on key metrics.
\autoref{tab:dcn-relu-prod} further verifies the amount of gain from {DCN-V2} by replacing cross layers with same-sized ReLU layers.

\begin{table}[htpb]
\small
    \caption{Relative AUCLoss of {DCN-V2} v.s. same-sized ReLUs}
    \label{tab:dcn-relu-prod}
    \vspace{-3ex}
        \centering
        \begin{tabular}{cccc}
        \toprule
        { 1layer ReLU}  & {2layer ReLU} & {1layer {DCN-V2}} & {2layer {DCN-V2}}\\
        \midrule
         0\%   & -0.15\% & -0.19\% &  -0.45\% \\
        \bottomrule
        \end{tabular}
\end{table}

{\bf Practical Learnings.} We share some practical lessons we have learned through productionizing {DCN-V2}. 
\begin{itemize}[leftmargin=1em]
\item It's better to insert the cross layers in between the input and the hidden layers of DNN (also observed in \cite{shan2016deep}). Our hypothesis is that the physical meaning of feature representations and their interactions becomes weaker as it goes farther away from the input layer.
\item We saw consistent accuracy gains by stacking or concatenating 1 - 2 cross layers. Beyond 2 cross layers, the gains start to plateau.
\item We observed that both stacking cross layers and concatenating cross layers work well. Stacking layers learns higher-order feature interactions, while concatenating layers (similar to multi-head mechanism \cite{vaswani2017attention}) captures complimentary interactions.
\item We observed that using low-rank DCN with rank $\text{(input size)}/4$ consistently preserved the accuracy of a full-rank {DCN-V2}.  
\end{itemize}


\section{Conclusions and Future Work}

In this paper, we propose a new model---{DCN-V2}---to model explicit crosses in an expressive yet simple manner. Observing the low-rank nature of the weight matrix in the cross network, we also propose a mixture of low-rank DCN ({DCN-Mix}) to achieve a healthier trade-off between model performance and latency. {DCN-V2} has been successfully deployed in multiple web-scale learning to rank systems with significant offline model accuracy and online business metric gains. Our experimental results also have demonstrated {DCN-V2}'s effectiveness over SOTA methods.


For future work, we are interested in advancing our understanding of 1). the interactions between {DCN-V2} and optimization algorithms such as second-order methods; 2). the relation between embedding, {DCN-V2} and its rank of matrix. Further, we would like to improve the gating mechanism in {DCN-Mix}. Moreover, observing that cross layers in {DCN-V2} may serve as potential alternatives to ReLU layers in DNNs, we are very interested to verify this observation across more complex model architectures (\emph{e.g.}, RNN, CNN).

  
\medskip
{\bf Acknowledgement.} We would like to thank Bin Fu, Gang (Thomas) Fu, and Mingliang Wang for their early contributions of {DCN-V2}; Tianshuo Deng, Wenjing Ma, Yayang Tian, Shuying Zhang, Jie (Jerry) Zhang, Evan Ettinger, Samuel Ieong and many others for their efforts and supports in productionizing {DCN-V2}; Ting Chen for his initial idea of mixture of low-rank; and Jiaxi Tang for his valuable comments.

\bibliographystyle{ACM-Reference-Format}

\section{Introduction}

Learning to rank (LTR) \cite{liu2011learning, cao2007learning} has remained to be one of the most important problems in modern-day machine learning and deep learning. It has a wide range of applications in search, recommendation systems \cite{resnick1997recommender, herlocker2004evaluating, schafer1999recommender}, and computational advertising \cite{broder2008computational, bottou2013counterfactual}. Among the crucial components of LTR models, learning effective feature crosses continues to attract lots of attention from both academia \cite{qu2016product, lian2018xdeepfm, song2019autoint} and industry \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, naumov2019deep}.

Effective feature crosses are crucial to the success of many models. They provide additional interaction information beyond individual features. For example, the combination of ``{\ttfamily{country}}'' and ``{\ttfamily{language}}'' is more informative than either one of them.
In the era of linear models, ML practitioners rely on manually identifying such feature crosses \cite{seide2011feature} to increase model's expressiveness.
Unfortunately, this involves a combinatorial search space, which is large and sparse in web-scale applications where the data is mostly categorical.
Searching in such setting is exhaustive, often requires domain expertise, and makes the model harder to generalize.

Later on, embedding techniques have been widely adopted to project features from high-dimensional sparse vectors to much lower-dimensional dense vectors. Factorization Machines (FMs) \cite{rendle2010factorization, rendle:tist2012} leverage the embedding techniques and construct pairwise feature interactions via the inner-product of two latent vectors. Compared to those traditional feature crosses in linear models, FM brings more generalization capabilities.

In the last decade, with more computing firepower and huge scale of data, LTR models in industry have gradually migrated from linear models and FM-based models to deep neural networks (DNN). This has significantly improved model performance for search and recommendation systems across the board \cite{cheng2016wide, wang2017deep, guo2017deepfm}. People generally consider DNNs as universal function approximators, that could potentially learn all kinds of feature interactions \cite{mhaskar1996neural, valiant2014learning, NIPS2016_6556}. However, recent studies \cite{beutel2018latent, wang2017deep} found that DNNs are inefficient to even approximately model 2nd or 3rd-order feature crosses.

To capture effective feature crosses more accurately, a common remedy is to further increase model capacity through wider or deeper networks. This naturally crafts a double edged sword that we are improving model performance while making models much slower to serve. In many production settings, these models are handling extremely high QPS, thus have very strict latency requirements for real-time inference. Possibly, the serving systems are already pushed to a stretch that cannot afford even larger models. Furthermore, deeper models often introduce trainability issues, making models harder to train.

This has shed light on critical needs to design a model that can efficiently and effectively learn predictive feature interactions, especially in a resource-constraint environment that handles real-time traffic from billions of users. Many recent works \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, qu2016product, lian2018xdeepfm, song2019autoint, naumov2019deep} tried to tackle this challenge. The common theme is to leverage those \emph{implicit} high-order crosses learned from DNNs, with \emph{explicit} and bounded-degree feature crosses which have been found to be effective in linear models. \emph{Implicit} cross means the interaction is learned through an end-to-end function without any explicit formula modeling such cross. \emph{Explicit} cross, on the other hand, is modeled by an explicit formula with controllable interaction order. We defer a detailed discussion of these models in \autoref{sec:related_work}.

Among these, Deep \& Cross Network (DCN) \cite{wang2017deep} is effective and elegant, however, productionizing DCN in large-scale industry systems faces many challenges. The expressiveness of its cross network is limited. The polynomial class reproduced by the cross network is only characterized by $O(\text{input size})$ parameters, largely limiting its flexibility in modeling random cross patterns. 
Moreover, the allocated capacity between the cross network and DNN is unbalanced. This gap significantly increases when applying DCN to large-scale production data. An overwhelming portion of the parameters will be used to learn implicit crosses in the DNN


In this paper, we propose a new model \emph{{DCN-V2}} that improves the original DCN model. We have already successfully deployed {DCN-V2} in quite a few learning to rank systems across Google with significant gains in both offline model accuracy and online business metrics. {DCN-V2} first learns explicit feature interactions of the inputs (typically the embedding layer) through cross layers, and then combines with a deep network to learn complementary implicit interactions. The core of {DCN-V2} is the cross layers, which inherit the simple structure of the cross network from DCN, however significantly more expressive at learning explicit and bounded-degree cross features. The paper studies datasets with clicks as positive labels, however {DCN-V2} is label agnostic and can be applied to any learning to rank systems.
The main contributions of the paper are five-fold:


\begin{itemize}[leftmargin=1em]
    \item We propose a novel model---{DCN-V2}---to learn effective explicit and implicit feature crosses. Compared to existing methods, our model is more expressive yet remains efficient and simple.
    \item Observing the low-rank nature of the learned matrix in {DCN-V2}, we propose to leverage low-rank techniques to approximate feature crosses in a subspace for better performance and latency trade-offs. In addition, we propose a technique based on the Mixture-of-Expert architecture \cite{shazeer2017outrageously, jacobs1991adaptive} to further decompose the matrix into multiple smaller sub-spaces. These sub-spaces are then aggregated through a gating mechanism.
    \item We conduct and provide an extensive study using synthetic datasets, which demonstrates the inefficiency of traditional ReLU-based neural nets to learn high-order feature crosses.
    \item Through comprehensive experimental analysis, we demonstrate that our proposed {DCN-V2} models significantly outperform SOTA algorithms on Criteo and MovieLen-1M benchmark datasets.
   
    \item We provide a case study and share lessons in productionizing {DCN-V2} in a large-scale industrial ranking system, which delivered significant offline and online gains.
\end{itemize}


The paper is organized as follows. \autoref{sec:related_work} summarizes related work. \autoref{sec:dcn-m} describes our proposed model architecture {DCN-V2} along with its memory efficient version. \autoref{sec:dcn_analysis} analyzes {DCN-V2}. \autoref{sec:research_qs} raises a few research questions, which are answered through comprehensive experiments on both synthetic datasets in \autoref{sec:exp_synthetic} and public datasets in \autoref{sec:exp_public}. \autoref{sec:productionization} describes the process of productionizing {DCN-V2} in a web-scale recommender. 

\section{Related Work}
\label{sec:related_work}


The core idea of recent feature interaction learning work is to leverage both explicit and implicit (from DNNs) feature crosses. To model explicit crosses, most recent work introduces multiplicative operations ($x_1 \times x_2$) which is inefficient in DNN, and designs a function $f({\bf x}_1, {\bf x}_2)$ to efficiently and explicitly model the pairwise interactions between features ${\bf x}_1$ and ${\bf x}_2$. We organize the work in terms of how they combine the explicit and implicit components.

{\bf Parallel Structure.} One line of work jointly trains two parallel networks inspired from
the wide and deep model \cite{cheng2016wide}, where the wide component takes inputs as crosses of raw features; and the deep component is a DNN model. However, selecting cross features for the wide component falls back to the feature engineering problem for linear models. Nonetheless, the wide and deep model has inspired many works to adopt this parallel architecture and improve upon the wide component. 

DeepFM \cite{guo2017deepfm} automates the feature interaction learning in the wide component by adopting a FM model. DCN \cite{wang2017deep} introduces a cross network, which learns explicit and bounded-degree feature interactions automatically and efficiently.
xDeepFM \cite{lian2018xdeepfm} increases the expressiveness of DCN by generating multiple feature maps, each encoding all the pairwise interactions between features at current level and the input level. Besides, it also considers each feature embedding ${\bf x}_i$ as a unit instead of each element $x_i$ as a unit. Unfortunately, its computational cost is significantly high (10x of \#params), making it impractical for industrial-scale applications. Moreover, both DeepFM and xDeepFM require all the feature embeddings to be of equal size, yet another limitation when applying to industrial data where the vocab sizes (sizes of categorical features) vary from $O(10)$ to millions. AutoInt \cite{song2019autoint} leverages the multi-head self-attention mechanism with residual connections. InterHAt \cite{li2020interpretable} further employs Hierarchical Attentions.

{\bf Stacked Structure.} Another line of work introduces an interaction layer---which creates explicit feature crosses---in between the embedding layer and a DNN model. This interaction layer captures feature interaction at an early stage, and facilitates the learning of subsequent hidden layers. Product-based neural network (PNN) \cite{qu2016product} introduces inner (IPNN) and outer (OPNN) product layer as the pairwise interaction layers. One downside of OPNN lies in its high computational cost.
Neural FM (NFM) \cite{he2017neural} extends FM by replacing the inner-product with a Hadamard product;
DLRM \cite{naumov2019deep} follows FM to compute the feature crosses through inner products;
These models can only create up to 2nd-order explicit crosses. AFN \cite{cheng2019adaptive} transforms features into a log space and adaptively learns arbitrary-order feature interactions. Similar to DeepFM and xDeepFM, they only accept embeddings of equal sizes.

Despite many advances made, our comprehensive experiments (\autoref{sec:exp_public}) demonstrate that DCN still remains to be a strong baseline. We attribute this to its simple structure that has facilitated the optimization. However, as discussed, its limited expressiveness has prevented it from learning more effective feature crosses in web-scale systems. In the following, we present a new architecture that inherits DCN's simple structure while increasing its expressiveness.




\section{Proposed Architecture: {DCN-V2}}
\label{sec:dcn-m}

This section describes a novel model architecture --- {DCN-V2} --- to learn both explicit and implicit feature interactions. {DCN-V2} starts with an \emph{embedding layer}, followed by a \emph{cross network} containing multiple cross layers that models explicit feature interactions, and then combines with a \emph{deep network} that models implicit feature interactions. The improvements made in {DCN-V2} are {\bf critical for putting {DCN} into practice for highly-optimized production systems}. {DCN-V2} significantly improves the expressiveness of DCN \cite{wang2017deep} in modeling complex explicit cross terms in web-scale production data, while maintaining its elegant formula for easy deployment. The function class modeled by {DCN-V2} is a strict superset of that modeled by DCN. The overall model architecture is depicted in Fig. \ref{fig:dcn-visualization}, with two ways to combine the cross network with the deep network: (1) stacked and (2) parallel.  In addition, observing the low-rank nature of the cross layers, we propose to leverage a mixture of low-rank cross layers to achieve healthier trade-off between model performance and efficiency.

\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.2\textwidth}  
    \includegraphics[width=\textwidth]{dcn-stack.pdf}
    \caption{Stacked}
    \label{fig:dcn-stack}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.27\textwidth}  
    \includegraphics[width=\textwidth]{dcn-parallel.pdf}
    \caption{Parallel}
    \label{fig:dcn-parallel}
    \end{subfigure}
    \caption{Visualization of {DCN-V2}. $\otimes$ represents the cross operation in Eq. \eqref{eq:cross_layer}, \emph{i.e.}, ${\bf x}_{l+1} ={\bf x}_0 \odot (W_l {\bf x}_l +  {\bf b}_l) + {\bf x}_l$.}
    \label{fig:dcn-visualization}
\end{figure}


\subsection{Embedding Layer}

The embedding layer takes input as a combination of categorical (sparse) and dense features, and outputs ${\bf x}_0 \in \mathbb{R}^d$. For the $i$-th categorical feature, we project it from a high-dimensional sparse space to a lower-dimensional dense space via
	${\bf x}_{\text{embed}, i} = W_{\text{embed},i} {\bf e}_i$,
where ${\bf e}_i \in \{0, 1\}^{v_i}$; $W \in \mathbb{R}^{e_i \times v_i}$ is a learned projection matrix; ${\bf x}_{\text{embed}, i} \in \mathbb{R}^{e_i}$ is the dense embedded vector; $v_i$ and $e_i$ represents vocab and embedding sizes respectively. For multivalent features, we use the mean of the embedded vectors as the final vector. 

The output is the concatenation of all the embedded vectors and the normalized dense features:
	${\bf x}_0 = [{\bf x}_{\text{embed}, 1}; \ldots; {\bf x}_{\text{embed}, n}; x_{\text{dense}}]$.

Unlike many related works \cite{song2019autoint, lian2018xdeepfm, qu2016product, guo2017deepfm, naumov2019deep, he2017neural} which requires $e_i=e_j~\forall i, j$, our model accepts arbitrary embedding sizes. This is particularly important for industrial recommenders where the vocab size varies from $O(10)$ to $O(10^5)$. Moreover, our model isn't limited to the above described embedding method; any other embedding techniques such as hashing could be adopted. 

\subsection{Cross Network}
The core of {DCN-V2} lies in the cross layers that create explicit feature crosses. Eq. \eqref{eq:cross_layer} shows the $(l+1)^\text{th}$ cross layer.
\begin{equation}
\label{eq:cross_layer}
	{\bf x}_{l+1} ={\bf x}_0 \odot (W_l {\bf x}_l +  {\bf b}_l) + {\bf x}_l
\end{equation}

where ${\bf x}_0 \in \mathbb{R}^{d}$ is the base layer that contains the original features of order 1, and is normally set as the embedding (input) layer. ${\bf x}_l,{\bf x}_{l+1} \in \mathbb{R}^{d}$, respectively, represents the input and output of the  $(l+1)$-th cross layer. $W_l \in \mathbb{R}^{d \times d}$ and ${\bf b}_l \in \mathbb{R}^{d}$ are the learned weight matrix and bias vector. \autoref{fig:cross_sub_network} shows how an individual cross layer functions.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=3in]{dcn-formula.png}
\vspace{-2ex}
\caption{Visualization of a cross layer.}
  \label{fig:cross_sub_network}
\vspace{-1ex}
\end{figure}

For an $l$-layered cross network, the highest polynomial order is $l+1$ and the network contains all the feature crosses up to the highest order. Please see \autoref{sec:poly_analysis} for a detailed analysis, both from bitwise and feature-wise point of views. When $W = {\bf 1} \times {\bf w}^\top$, where ${\bf 1}$ represents a vector of ones, {DCN-V2} falls back to {DCN}.

The cross layers could only reproduce polynomial function classes of bounded degree; any other complex function space could only be approximated\footnote{Any function with certain smoothness assumptions can be well-approximated by polynomials. In fact, we've observed in our experiments that cross network alone was able to achieve similar performance as traditional deep networks.}. Hence, we introduce a deep network next to complement the modeling of the inherent distribution in the data.


\subsection{Deep Network}
The $l^\text{th}$ deep layer's formula is given by
	${\bf h}_{l+1} = f(W_l {\bf h}_l +  {\bf b}_l)$,
where ${\bf h}_l \in \mathbb{R}^{d_l}, {\bf h}_{l+1} \in \mathbb{R}^{d_{l+1}}$, respectively, are the input and output of the $l$-th deep layer; $W_{l} \in \mathbb{R}^{d_{l} \times d_{l+1}}$ is the weight matrix and ${\bf b}_l \in \mathbb{R}^{d_{l+1}}$ is the bias vector; $f(\cdot)$ is an elementwise activation function and we set it to be ReLU; any other activation functions are also suitable.

\subsection{Deep and Cross Combination}
We seek structures to combine the cross network and deep network. Recent literature adopted two structures: stacked and parallel. In practice, we have found that which architecture works better is data dependent. Hence, we present both structures:

{\bf Stacked Structure (\autoref{fig:dcn-stack}):} The input ${\bf x}_0$ is fed to the cross network followed by the deep network, and the final layer is given by ${\bf x}_{\text{final}} = {\bf h}_{L_d}, ~ {\bf h}_{0} = {\bf x}_{L_c}$, which models the data as $f_{\text{deep}} \circ f_{\text{cross}}$.

{\bf Parallel Structure (\autoref{fig:dcn-parallel}):} The input ${\bf x}_0$ is fed in parallel to both the cross and deep networks; then, the outputs ${\bf x}_{L_c}$ and ${\bf h}_{L_d}$ are concatenated to create the final output layer ${\bf x}_{\text{final}} = [{\bf x}_{L_c}; {\bf h}_{L_d}]$. This structure models the data as $f_{\text{cross}} + f_{\text{deep}}$.
	

In the end, the prediction $\hat y_i$ is computed as:
	$\hat y_i = \sigma({\bf w}_{\text{logit}}^\top{\bf x}_{\text{final} })$,
where ${\bf w}_{\text{logit}}$ is the weight vector for the logit, and $\sigma(x) = 1 / (1 + \exp(-x))$. For the final loss, we use the Log Loss that is commonly used for learning to rank systems especially with a binary label (e.g., click). Note that {DCN-V2} itself is both prediction-task and loss-function agnostic.
\begin{equation*}
	\begin{split}
	\text{loss} = &-\frac{1}{N} \sum_{i = 1}^N y_i \log(\hat y_i) + (1 - y_i) \log(1 - \hat y_i) + 
    \lambda \sum_l \|W_l\|^2_2,
    \end{split}
\end{equation*}
where $\hat y_i$'s are predictions; $y_i$'s are the true labels; $N$ is the total number of inputs; and $\lambda$ is the $L_2$ regularization parameter. 

\subsection{Cost-Effective Mixture of Low-Rank DCN}
In real production models, the model capacity is often constrained by limited serving resources and strict latency requirements. It is often the case that we have to seek methods to reduce cost while maintaining the accuracy. Low-rank techniques \cite{golub1996matrix} are widely used \cite{jaderberg2014speeding, yu2017compressing, chen2018adaptive, wang2019block, halko2011finding, drineas2005nystrom} to reduce the computational cost. It approximates a dense matrix $M \in \mathbb{R}^{d \times d}$ by two tall and skinny matrices $U, V \in \mathbb{R}^{d \times r}$. When $r \le d/2$, the cost will be reduced.
However, they are most effective when the matrix shows a large gap in singular values or a fast spectrum decay. In many settings, we indeed observe that the learned matrix is numerically low-rank in practice.

Fig. \ref{fig:dcn-sval} shows the singular decay pattern of the learned matrix $W$ in {DCN-V2} (see Eq. \eqref{eq:cross_layer}) from a production model. Compared to the initial matrix, the learned matrix shows a much faster spectrum decay pattern. Let's define the numerical rank $R_T$ with tolerance T to be $\text{argmin}_k (\sigma_k < T \cdot \sigma_1)$, where $\sigma_1 \ge \sigma_2 \ge, \ldots, \ge \sigma_n$ are the singular values. Then, $R_T$ means majority of the mass up to tolerance $T$, is preserved in the top $k$ singular values. In the field of machine learning and deep learning, a model could still work surprisingly well with a reasonably high tolerance $T$ \footnote{This is very different from the filed of scientific computing (\emph{e.g.}, solving linear systems), where the approximation accuracy need to be very high. For problems such as CTR prediction, some errors could introduce regularization effect to the model.}.


\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.2\textwidth}  
    \includegraphics[width=\textwidth]{dcn-sval.png}
    \caption{Singular Values}
    \label{fig:dcn-sval}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.23\textwidth}  
    \includegraphics[width=\textwidth]{mixture-dcn.pdf}
    \caption{Mixture of Low-rank Experts}
    \label{fig:mixture-dcn}
    \end{subfigure}
    \caption{Left: Singular value decay of the learned {DCN-V2} weight matrix. The singular values are normalized and $1 = \sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k$. {\color{orange}$+$} represents the randomly initialized truncated normal matrix; {\color{blue}$\times$} represents the final learned matrix. Right: Visualization of mixture of low-rank cross layer.}
\end{figure}


Hence, it is well-motivated to impose a low-rank structure on $W$. Eq \eqref{eq:low_rank_cross_layer} shows the resulting $(l+1)$-th low-rank cross layer
\begin{equation}
\label{eq:low_rank_cross_layer}
	{\bf x}_{l+1} = {\bf x}_0 \odot \Big(U_l \big(V_l^\top {\bf x}_i\big) + {\bf b}_l \Big) + {\bf x}_i
\end{equation}
where $U_l, V_l \in \mathbb{R}^{d \times r}$ and $r \ll d$. Eq \eqref{eq:low_rank_cross_layer} has two \emph{interpretations}: 1) we learn feature crosses in a subspace; 2) we project the input ${\bf x}$ to lower-dimensional $\mathbb{R}^r$, and then project it back to $\mathbb{R}^d$. The two interpretations have inspired the following two model improvements.

Interpretation 1 inspires us to adopt the idea from Mixture-of-Experts (MoE) \cite{shazeer2017outrageously, jacobs1991adaptive, eigen2013learning, ma2018modeling}. MoE-based models consist of two components: experts (typically a small network) and gating (a function of inputs). In our case, instead of relying on one single expert (Eq \eqref{eq:low_rank_cross_layer}) to learn feature crosses, we leverage multiple such experts, each learning feature interactions in a different subspaces, and adaptively combine the learned crosses using a gating mechanism that depends on input ${\bf x}$. The resulting mixture of low-rank cross layer formulation is shown in Eq. \eqref{eq:mixture_low_rank_cross_layer} and depicted in \autoref{fig:mixture-dcn}.
\begin{equation}
\label{eq:mixture_low_rank_cross_layer}
    \begin{split}
	{\bf x}_{l+1} &= \sum\nolimits_{i=1}^K G_i({\bf x}_l) E_i({\bf x}_l) + {\bf x}_l \\
	E_i({\bf x}_l) &= {\bf x}_0 \odot \Big(U_l^i \big(V_l^{i\top}{\bf x}_l \big)  + {\bf b}_l \Big )
	\end{split}
\end{equation}
where $K$ is the number of experts; $G_i(\cdot): \mathbb{R}^d \mapsto \mathbb{R}$ is the gating function, common sigmoid or softmax; $E_i(\cdot): \mathbb{R}^d \mapsto \mathbb{R}^d$ is the $i^\text{th}$ expert in learning feature crosses. $G(\cdot)$ dynamically weights each expert for input ${\bf x}$, and when $G(\cdot) \equiv 1$, Eq \eqref{eq:mixture_low_rank_cross_layer} falls back to Eq \eqref{eq:low_rank_cross_layer}.

Interpretation 2 inspires us to leverage the low-dimensional nature of the projected space. Instead of immediately projecting back from dimension $d'$ to $d$ ($d' \ll d$), we further apply nonlinear transformations in the projected space to refine the representation \cite{fan2019multiscale}
\begin{equation}
\label{eq:mixture_low_rank_with_c_cross_layer}

	E_i({\bf x}_l) = {\bf x}_0 \odot \Big(
	U_l^i \cdot g \big( C_l^i \cdot g\big(V_l^{i\top}{\bf x}_l \big) \big)  + {\bf b}_l 
	\Big )
\end{equation}
where $g(\cdot)$ represents any nonlinear activation function.

{\bf Discussions.} This section aims to make effective use of the fixed memory/time budget to learn meaningful feature crosses. From Eqs \eqref{eq:cross_layer}--\eqref{eq:mixture_low_rank_with_c_cross_layer}, each formula represents a strictly larger function class assuming a fixed \#params.

Different from many model compression techniques where the compression is conducted post-training, our model imposes the structure prior to training and jointly learn the associated parameters with the rest of the parameters. Due to that, the cross layer is an integral part of the nonlinear system $f({\bf x}) = \big(f_k(W_k) \circ \cdots \circ f_1(W_1)\big) ({\bf x})$, where $(f_{i+1} \circ f_{i}) (\cdot) \coloneqq f_{i+1}(f_i(\cdot))$. Hence, the training dynamics of the overall system might be affected, and it would be interesting to see how the global statistics, such as Jacobian and Hession matrices of $f({\bf x})$, are affected. We leave such investigations to future work.





\subsection{Complexity Analysis}
Let $d$ denote the embedding size, $L_c$ denote the number of cross layers, $K$ denote the number of low-rank DCN experts. Further, for simplicity, we assume each expert has the same smaller dimension $r$ (upper bound on the rank). 
The time and space complexity for the cross network is $O(d^2 L_c)$, and for mixture of low-rank DCN ({DCN-Mix}) it's efficient when $rK \ll d$ with $O(2drKL_c)$. 



\section{Model Analysis}
\label{sec:dcn_analysis}
This section analyzes {DCN-V2} from polynomial approximation point of view, and makes connections to related work. We adopt the notations from \cite{wang2017deep}.

{\bf Notations.} Let the embedding vector ${\bf x} = [{\bf x}_1; {\bf x}_2; \ldots; {\bf x}_k] = [x_1, x_2, \ldots, x_d] \in \mathbb{R}^d$ be a column vector, where ${\bf x}_i \in \mathbb{R}^{e_i}$ represents the $i$-th feature embedding, and $x_i$ represents the $i$-th element in ${\bf x}$. Let multi-index ${\bm \alpha} = [\alpha_1, \cdots, \alpha_d] \in \mathbb{N}^d$ and $|{\bm \alpha}| = \sum_{i=1}^d \alpha_i$. $C_a^b \coloneqq \bigl\{{\bf y} \in \{1,\cdots, a\}^b \mathrel{\big|} \forall i < j, y_i > y_j  \bigr\}$. Let $\mathbf{1}$ be a vector of all 1's, and $I$ be an identity matrix. We use capital letters for matrices, bold lower-case letters for vectors, and normal lower-case letters for scalars.

\subsection{Polynomial Approximation}
\label{sec:poly_analysis}
We analyze {DCN-V2} from two perspectives of polynomial approximation ---
1) Considering each element (bit) $x_i$ as a unit, and analyzes interactions among the elements (\autoref{thm:cross_x0_bitwise}); and 2) Considering each feature embedding ${\bf x}_i$ as a unit, and only analyzes the feature-wise interactions (\autoref{thm:cross_x0_featurewise} ) (proofs in Appendix). 

\begin{theorem}[Bitwise]
\label{thm:cross_x0_bitwise}
	Assume the input to an $l$-layer cross network be ${\bf x} \in \mathbb{R}^d$, the output be $f_l({\bf x})={\bf 1}^\top {\bf x}^l$, and the $i^\text{th}$ layer is defined as ${\bf x}^i ={\bf x} \odot W^{(i-1)}{\bf x}^{i-1} +{\bf x}^{i-1}$. Then, the multivariate polynomial $f_l({\bf x})$ reproduces polynomials in the following class:
    $$\biggl\{\sum_{{\bm \alpha}} c_{{\bm \alpha}}\left(W^{(1)}, \ldots, W^{(l)}\right) x_1^{\alpha_1}x_2^{\alpha_2}\ldots x_d^{\alpha_d} \mathrel{\bigg|}  0 \le |{\bm \alpha}| \le l+1, {\bm \alpha} \in \mathbb{N}^d \biggr\},$$
    where $c_{\bm \alpha} = \sum_{{\bf j} \in C_l^{|{\bm \alpha}|-1}}  \sum_{{\bf i} \in P_{\bm \alpha}} \prod_{k=1}^{|{\bm \alpha}|-1} w_{i_k i_{k+1}}^{(j_k)}$, $w_{i j}^{(k)}$ is the $(i,j)^\text{th}$ element of matrix $W^{(k)}$, and $P_{\bm \alpha} = \text{Permutations}~(\cup_i \{\underbrace{i, \ldots, i}_{\alpha_i \text{times}} \mathrel{|} \alpha_i \neq 0\})$.
\end{theorem}


\begin{theorem}[feature-wise]
\label{thm:cross_x0_featurewise}
  With the same setting as in \autoref{thm:cross_x0_bitwise}, we further assume input ${\bf x} = [{\bf x}_1; \ldots; {\bf x}_k]$ contains $k$ feature embeddings and consider each ${\bf x}_i$ as a unit. Then, the output ${\bf x}^l$ of an $l$-layer cross network creates all the feature interactions up to order $l+1$. Specifically, for features with their (repeated) indices in $I$, let $P_I = Permutations (I)$, then their order-$p$ interaction is given by:
\begin{equation*}
\begin{split}
\sum_{{\bf i} \in P_I} \sum_{{\bf j} \in C_p^{p-1}} {\bf x}_{i_1} \odot \left(W_{i_1, i_2}^{(j_1)} {\bf x}_{i_2} \odot \ldots \odot \left(W_{i_k, i_{k+1}}^{(j_{k})} {\bf x}_{i_{l+1}}\right) \right)
\end{split}
\end{equation*}

\end{theorem}


From both bitwise and feature-wise perspectives, the cross network is able to create all the feature interactions up to order $l+1$ for an $l$-layered cross network. Compared to DCN-V, {DCN-V2} characterizes the same polynomial class with more parameters and is more expressive. Moreover, the feature interactions in {DCN-V2} is more expressive and can be viewed both bitwise and feature-wise, whereas in DCN it is only bitwise \cite{wang2017deep, lian2018xdeepfm, song2019autoint}.

\subsection{Connections to Related Work}
We study the connections between {DCN-V2} and other SOTA feature interaction learning methods; we only focus on the feature interaction component of each model and ignore the DNN component. For comparison purposes, we assume the feature embeddings are of equal size $e$.

{\bf DCN.} Our proposed model was largely inspired from DCN \cite{wang2017deep}. Let's take the efficient projection view of DCN \cite{wang2017deep}, \emph{i.e.}, it implicitly generates all the pairwise crosses and then projects it to a lower-dimensional space; {DCN-V2} is similar with a different projection structure.
    \begin{equation*}
    	\begin{split}
    	{\bf x}_{\text{DCN}}^\top = 
        {\bf x}_{\text{pairs}}
        \left[\begin{smallmatrix}
        	{\bf w} & {\bf 0} & \ldots & {\bf 0} \\
            {\bf 0} & {\bf w} & \ldots & {\bf 0} \vspace{-1ex}\\
            \vdots   & \vdots& \ddots & \vdots \\
            {\bf 0} & {\bf 0} & \ldots & {\bf w}
        \end{smallmatrix}\right], 
        {\bf x}_{\text{{DCN-V2}}}^\top = 
        {\bf x}_{\text{pairs}}
         \left[\begin{smallmatrix}
        	{\bf w}_1 & {\bf 0} & \ldots & {\bf 0} \\
            {\bf 0} & {\bf w}_2 & \ldots & {\bf 0} \vspace{-1ex}\\
            \vdots   & \vdots& \ddots & \vdots \\
            {\bf 0} & {\bf 0} & \ldots & {\bf w}_d
        \end{smallmatrix}\right]
        \end{split}
    \end{equation*}
where
${\bf x}_{\text{pairs}} = [x_i \tilde x_j]_{\forall i, j}$
contains all the $d^2$ pairwise interactions between ${\bf x}_0$ and $\tilde {\bf x}$; ${\bf w} \in \mathbb{R}^d$ is the weight vector in DCN-V; ${\bf w}_i \in \mathbb{R}^d$ is the $i^\text{th}$ column of the weight matrix in {DCN-V2} (Eq.\eqref{eq:cross_layer}).

{\bf DLRM and DeepFM.} Both are essentially 2nd-order FM without the DNN component (ignoring small differences). Hence, we simplify our analysis and compare with FM which has formula
	${\bf x}^\top {\bm \beta} + \sum_{i<j} w_{ij} \langle {\bf x}_i, {\bf x}_j \rangle$.
This is equivalent to 1-layer {DCN-V2} (Eq. \eqref{eq:cross_layer} without residual term) with a structured weight matrix.
\begin{equation*}
\label{eq:dcn-fm}
\begin{split}
    \mathbf{1}^\top \left(
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    \odot
    \left(
    \left[\begin{smallmatrix}
        {\bf 0} & w_{12} I & \cdots & w_{1k}I \\
        {\bf 0} & {\bf 0} & \cdots & w_{2k} I \vspace{-1ex}\\
        \vdots  & \vdots  & \ddots & \vdots  \\
        {\bf 0} & {\bf 0} & \cdots & {\bf 0}
    \end{smallmatrix}\right]
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    +
    \left[\begin{smallmatrix}
        {\bm \beta}_1 \\
        {\bm \beta}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bm \beta}_k
    \end{smallmatrix}\right]
    \right)
    \right)
\end{split}
\end{equation*}

{\bf xDeepFM.} The $h$-th feature map at the $k$-th layer is given by:
\begin{equation*}
\label{eq:xdeepfm}
    {\bf x}_{h, *}^k = \sum\nolimits_{i=1}^{k-1} \sum\nolimits_{j=1}^m w_{ij}^{k, h} ({\bf x}_{i, *}^{k-1} \odot {\bf x}_j)
\end{equation*}
The $h$-th feature map at the 1st layer is equivalent to 1-layer {DCN-V2} (Eq. \eqref{eq:cross_layer} without residual term).
\begin{equation*}
\label{eq:dcn-fm2}
  {\bf x}_{h, *}^1 = [I, I, \cdots, I]\left({\bf x} \odot (W{\bf x})\right) = \sum\nolimits_{i=1}^k {\bf x}_i \odot (W_{i,:} {\bf x})
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
\end{equation*}
where the $(i,j)$-th block $W_{i,j} = w_{ij} \cdot I$, and $W_{i,:} \coloneqq [W_{i,1}, \ldots, W_{i,k}]$.

{\bf AutoInt.} The interaction layer of AutoInt adopted the multi-head self-attention mechanism. For simplicity, we assume a single head is used in AutoInt; multi-head case could be compared summarily using concatenated cross layers.  

From a high-level view, the 1st layer of AutoInt outputs $\widetilde {\bf x} = [\widetilde {\bf x}_1; \widetilde {\bf x}_2; \ldots; \widetilde {\bf x}_k]$, where $\widetilde {\bf x}_i$ encodes all the 2nd-order feature interactions with the i-th feature. Then, $\widetilde {\bf x}$ is fed to the 2nd layer to learn higher-order interactions. This is the same as {DCN-V2}. 

From a low-level view (ignoring the residual terms),
\begin{equation*}
\small
    \label{eq:autoint}
    \begin{split}
        \widetilde {\bf x}_i
        &= ReLU\left(\sum\nolimits_{j=1}^k \frac{\exp \left(\langle W_{\text{q}} {\bf x}_i, W_{\text{k}} {\bf x}_j \rangle \right)}{\sum\nolimits_j \exp \left(\langle W_{\text{q}} {\bf x}_i, W_{\text{k}} {\bf x}_j \rangle \right)} (W_{\text{v}} {\bf x}_j)\right) \\
        &= ReLU\big(\sum\nolimits_{j=1}^k \text{softmax}( {\bf x}_i^\top \widetilde W {\bf x}_j)~ W_{\text{v}} {\bf x}_j \big)
\end{split}
\end{equation*}
where $\langle \cdot, \cdot \rangle$ represents inner (dot) product, and $\widetilde W = W_{\text{q}} W_{\text{k}}$.
While in {DCN-V2}, 
\begin{equation}
    \label{eq:autoint_dcn}
    \begin{split}
        \widetilde {\bf x}_i = \sum\nolimits_{j=1}^k {\bf x}_i \odot (W_{i,j} {\bf x}_j) = {\bf x}_i \odot (W_{i,:} {\bf x})
    \end{split}
\end{equation}
where $W_{i,j}$ represents the $(i,j)$-th block of $W$. It is clear that the difference lies in how we model the feature interactions. AutoInt claims the non-linearity was from ReLU($\cdot$); we consider each summation term to also contribute. Differently, {DCN-V2} used ${\bf x}_i \odot W_{i,j} {\bf x}_j$. 

{\bf PNN.} The inner-product version (IPNN) is similar to FM. For the outer-product version (OPNN), it first explicitly creates all the $d^2$ pairwise interactions,
and then projects them to a lower dimensional space $d'$ using a $d'$ by $d^2$ dense matrix. Differently, {DCN-V2} implicitly creates the interactions using a structured matrix.

\section{Research Questions}
\label{sec:research_qs}
We are interested to seek answers for these following research questions:

\begin{itemize}[leftmargin=2.5em]
    \item[\bf RQ1]  When would feature interaction learning methods become more efficient than ReLU-based DNNs?
    \item[\bf RQ2]  How does the feature-interaction component of each baseline perform without integrating with DNN? 
    \item[\bf RQ3]  How does the proposed mDCN approaches compare to the baselines? Could we achieve healthier trade-off between model accuracy and cost through mDCN and the mixture of low-rank DCN?
    \item[\bf RQ4] How does the settings in mDCN affect model quality?
    \item[\bf RQ5] Is mDCN capturing important feature crosses? Does the model provide good understandability?
\end{itemize}

Throughout the paper, ``CrossNet" or ``CN" represents the cross network; suffix ``Mix" denotes the mixture of low-rank version. 

\section{Empirical understanding of feature crossing techniques (RQ1)}
\label{sec:exp_synthetic}
Many recent works \cite{wang2017deep, cheng2016wide, guo2017deepfm, beutel2018latent, qu2016product, lian2018xdeepfm, naumov2019deep} proposed to model explicit feature crosses that couldn't be learned efficiently from traditional neural networks. However, most works only studied public datasets with unknown cross patterns and noisy data; few work has studied in a clean setting with known ground-truth models.
Hence, it's important to understand : 1) in which cases would traditional neural nets become inefficient; 2) the role of each component in the cross network of {DCN-V2}.

We use the cross network in DCN models to represent those feature cross methods and compare with ReLUs, which are commonly used in industrial recommender systems.
To simplify experiments and ease understanding, we assume each feature $x_i$ is of dimension one, and monomial $x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d}$ represents a $|{\bm \alpha}|$-order interaction between features.

{\bf Performance with increasing difficulty.}
Consider only 2nd-order feature crosses and let the ground-truth model be $f({\bf x}) = \sum_{|{\bm \alpha}|=2} w_{{\bm \alpha}} x_1^{\alpha_1}x_2^{\alpha_2}\ldots x_d^{\alpha_d}$.
Then, the difficulty of learning $f({\bf x})$ depends on: 1) sparsity ($w_{{\bm \alpha}}=0$), the number of crosses, and 2) similarity of the cross patterns (characterized by $\mathrm{Var}(w_{{\bm \alpha}})$), meaning a change in one feature would simultaneously affect most feature crosses by similar amount. We create synthetic datasets with increasing difficulty in Eq. \eqref{eq:2nd-order}.
\begin{equation}
    \label{eq:2nd-order}
    \begin{split}
    f_1({\bf x}) &= x_1^2+x_1x_2+x_3x_1+x_4x_1 \\
    f_2({\bf x}) &= x_1^2+0.1x_1x_2+x_2x_3+0.1x_3^2 \\
    f_3({\bf x}) &= \sum\nolimits_{(i,j) \in S} w_{ij} x_i x_j, ~~{\bf x} \in \mathbb{R}^{100}, |S|=100
    \end{split}
\end{equation}
where set $S$ and weights $w_{ij}$ are randomly assigned, and $x_i$'s are uniformly sampled from interval [-1, 1].

\autoref{tab:2nd-order} reports mean RMSE out of 5 runs and the model size.
When the cross patterns are simple ($f_1$), both {DCN-V2} and DCN are efficient. When the patterns become more complicated ($f_3$), {DCN-V2} remains accurate while DCN degrades. DNN's performance remains poor even with a wider and deeper structure (layer sizes [200, 200] for $f_1$ and $f_2$, [1024, 512, 256] for $f_3$). This suggests the inefficiency of DNN in modeling monomial patterns.

\begin{table}[htpb]
\footnotesize
\vspace{-5pt}
\caption{RMSE and Model Size (\# Parameters) for Polynomial Fitting of Increasing Difficulty.}
\vspace{-3.5ex}
\label{tab:2nd-order}
\begin{center}
\begin{tabular}{c|ll|ll|ll|ll}
\toprule
& \multicolumn{2}{c|}{DCN (1Layer)}  & \multicolumn{2}{c|}{DCN-V2 (1Layer)}  & \multicolumn{2}{c|}{DNN (1Layer)} & \multicolumn{2}{c}{DNN (large)}\\
& RMSE & Size & RMSE & Size & RMSE & Size & RMSE & Size \\
\midrule
$f_1$& 8.9E-13 & 12 & {\bf 5.1E-13} & 24 & 2.7E-2 & 24  & 4.7E-3 & 41K 
\\
$f_2$ & 1.0E-01 & 9  & {\bf 4.5E-15} & 15 & 3.0E-2 & 15 & 1.4E-3 & 41K\\
$f_3$ & 2.6E+00 & 300 & {\bf 6.7E-07} & 10K & 2.7E-1 & 10K & 7.8E-2 & 758K\\
\bottomrule
\end{tabular}
\end{center}
\vspace{-5pt}
\end{table}

{\bf Role of each component.}
We also conducted ablation studies on homogeneous polynomials of order 3 and 4, respectively. For each order, we randomly selected 20 cross terms from ${\bf x} \in \mathbb{R}^{50}$.

\autoref{fig:dcn-poly-order} shows the change in mean RMSE with layer depth. Clearly, ${\bf x}_0 \odot (W{\bf x}_i)$ models order-$d$ crosses at layer $d$-1, which is verified by that the best performance for order-3 polynomial is achieved at layer 2 (similar for order-4). At other layers, however, the performance significantly degrades. This is where the bias and residual terms are helpful --- they create and maintain all the crosses up to the highest order. This reduces the performance gap between layers, and stabilizes the model when redundant crosses are introduced. This is particularly important for real-world applications with unknown cross patterns.

Fig. \ref{fig:dcn-poly-order} also reveals the limited expressiveness of {DCN} in modeling complicated cross patterns.


\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth-order3-v1.png}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.208\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth-order4-v1.png}
    \end{subfigure}
    \vspace{-1ex}
    \caption{Homogeneous polynomial fitting of order 3 and 4. $x$-axis represents the number of layers used; $y$-axis represents RMSE (the lower the better). In the legend, the top 3 models are {DCN-V2} with different component(s) included.}
\label{fig:dcn-poly-order}
\end{figure}

{\bf Performance with increasing layer depth.} We now study scenarios closer to real-world settings, where the cross terms are of a combined order.  
\begin{equation*}
\small
    \label{eq:combined-order}
    \begin{split}
    f({\bf x}) = & {\bf x}^\top {\bf w} + 
\sum_{{\bm \alpha} \in S} w_{{\bm \alpha}} x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d}  +
0.1 \sin(2{\bf x}^\top {\bf w}_{s} + 0.1) +
0.01 \epsilon
    \end{split}
\end{equation*}
where the randomly chosen set $S = S_2 \cup S_3 \cup S_4$, $|S_2|=20, |S_3|=10, |S_4|=5$, and $\forall {\bm \alpha} \in S_i, |{\bm \alpha}|=i$; sine introduces perturbations and $\epsilon$ represents Gaussian noises.

\autoref{tab:combined-order} reports the mean RMSE out of 5 runs. With the increase of layer depth, CN-M was able to capture higher-order feature crosses in the data, resulting in improved performance. Thanks to the bias and residual terms, the performance didn't degrade beyond layer 3, where redundant feature interactions were introduced.

\begin{table}[htbp]
\small
\caption{Combined-order (1 - 4) Polynomial Fitting.}
\vspace{-3.ex}
\label{tab:combined-order}
    \begin{subtable}[h]{0.45\textwidth}
        \centering
\begin{tabular}{c|ccccc}
\toprule
{\bf \#Layers} & 1 & 2 & 3 & 4 & 5\\
\midrule
{DCN-V2} & 1.43E-01 & 2.89E-02 & \bf 9.82E-03 & 9.87E-03 & 9.92E-03 \\
DNN& 1.32E-01 & 1.03E-01 & 1.03E-01 &  1.09E-01 & 1.05E-01\\
\bottomrule
\end{tabular}
    \end{subtable}
\end{table}

To summarize, ReLUs are inefficient in capturing explicit feature crosses (multiplicative relations) even with a deeper and larger network. This is well aligned with previous studies \cite{beutel2018latent}.
The accuracy considerably degrades when the cross patterns become more complicated. DCN accurately captures simple cross patterns but fails at more complicated ones. {DCN-V2}, on the other hand, remains accurate and efficient for complicated cross patterns.

\section{Experimental Results (RQ2 - RQ5)}
\label{sec:exp_public}
This section empirically verifies the effectiveness of {DCN-V2} in feature interaction learning across 3 datasets and 2 platforms, compared with SOTA. In light of recent concerns about poor reproducibility of published results \cite{dacrema2019we, musgrave2020metric, rendle2020neural}, we conducted a fair and comprehensive experimental study with extensive hyper-parameter search to properly tune all the baselines and proposed approaches. In addition, for each optimal setup, we train 5 models with different random initialization, and report the mean and standard deviation. 

\autoref{sec:performance_feature_interaction} studies the performance of the feature-cross learning components (\textbf{RQ2}) between baselines \emph{without} integrating with DNN ReLU layers (similar to \cite{lian2018xdeepfm, song2019autoint}); only sparse features are considered for a clean comparison. \autoref{sec:performance_baselines} compares {DCN-V2} with all the baselines comprehensively (\textbf{RQ3}). \autoref{sec:hyper-parameters} evaluates the influence of hyper-parameters on the performance of {DCN-V2} (\textbf{RQ4}). \autoref{sec:model_understanding} focuses on model understanding (\textbf{RQ5}) of whether we are indeed discovering meaningful feature crosses with {DCN-V2}. 


\subsection{Experiment Setup}
\label{sec:experiment_setup}
This section describes the experiment setup, including training datasets, baseline approaches, and details of the hyper-parameter search and training process.
\subsubsection{Datasets}
\autoref{tab:datasets} lists the statistics of each dataset:


\begin{table}[htpb]
\small
\caption{Datasets.}
\vspace{-3.5ex}
\label{tab:datasets}
\begin{center}
\begin{tabular}{c|ccc}
\toprule
{\bf Data} & \# Examples & \# Features & Vocab Size \\
\midrule
Criteo & 45M & 39 & 2.3M\\
MovieLen-1M &  740k & 7 & 3.5k\\
Production & $>$ 100B & NA & NA\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

{\bf Criteo\footnote{http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset}.} The most popular click-through-rate (CTR) prediction benchmark dataset contains user logs over a period of 7 days. We follow \cite{wang2017deep, song2019autoint} and use first 6 days for training, and randomly split the last day's data into validation and test set equally. We log-normalize ($\log(x+4)$ for feature-2 and $\log(x+1)$ for others) the 13 continuous features and embed the remaining 26 categorical features.

{\bf MovieLen-1M\footnote{https://grouplens.org/datasets/movielens}.} The most popular dataset for recommendation systems research. Each training example includes a {\ttfamily{$\langle$\small user-features, \small movie-features, \small rating$\rangle$}} triplet. Similar to AutoInt~\cite{song2019autoint}, we formalize the task as a regression problem. All the ratings for 1s and 2s are normalized to be 0s; 4s and 5s to be 1s; and rating 3s are removed. 6 non-multivalent categorical features are used and embedded. The data is randomly split into 80\% for training, 10\% for validation and 10\% for testing.

\subsubsection{Baselines.}
\label{sec:baselines}
We compare our proposed approaches with 6 SOTA feature interaction learning algorithms. A brief comparison between the approaches is highlighted in \autoref{tab:model_comparisons}.


\begin{table}[htpb]
\small
\caption{High-level comparison between models. Assuming the input ${\bf x}_0 = [{\bf v}_1; \ldots; {\bf v}_k]$ contains $k$ feature embeddings that each represented as ${\bf v}_i$. $\oplus$ denotes concatenation; $\otimes$ denotes outer-product; $\odot$ denotes Hadamard-product. $f_i(\cdot)$ represents implicit feature interactions, \emph{i.e.,} ReLU layers. In the last column, the `+' sign is on the logit level.}
\vspace{-2ex}
\label{tab:model_comparisons}
\begin{center}
\begin{tabular}{l|p{5mm}l|l}
\toprule
\multirow{ 2}{*}{{\bf Model}}  & \multicolumn{2}{c|}{ Explicit Interactions ($f_e$)} &  \multirow{ 2}{*}{\begin{tabular}{@{}l@{}}Final \\ Objective  \end{tabular}}  \\
\cmidrule(){2-3}
& Order & \multicolumn{1}{c|}{(Simplified) Key Formula}  & \\
\midrule
\multirow{ 2}{*}{{PNN \cite{qu2016product}}}  &\multirow{ 2}{*}{2}& ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$ (IPNN) & \multirow{ 2}{*}{$f_i \circ f_e$}   \\
&&${\bf x}_o = [\text{vec}({\bf v}_i \otimes {\bf v}_j) \mathrel{|} \forall i, j]$ (OPNN) & \\
DeepFM \cite{guo2017deepfm} & $2$ & ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$ &  $f_i + f_e$ \\
DLRM \cite{naumov2019deep} & 2 & ${\bf x}_o = [{\bf v}_i^\top {\bf v}_j \mathrel{|} \forall i, j]$& $f_i \circ f_e$\\
DCN \cite{wang2017deep} & $\ge 2$ & ${\bf x}_{i+1} = {\bf x}_0 \otimes {\bf x}_i {\bf w}_i$ &$f_i + f_e$ \\
xDeepFM \cite{lian2018xdeepfm} & $\ge 2$ & ${\bf v}_{h}^k = \sum_{i, j} w_{ij}^{kh} ({\bf v}_{i}^{k-1} \odot {\bf v}_j)$  & $f_i + f_e$ \\
AutoInt \cite{song2019autoint} & NA & $ \widetilde {\bf v}_i = g \left(\frac{\sum_{j}\exp (\langle W_q {\bf v}_i, W_k {\bf v}_j \rangle )W_v {\bf v}_j}{\sum_j \exp \left(\langle W_q {\bf v}_i, W_k {\bf v}_j \rangle \right)})\right)$ & $f_i + f_e$\\
\midrule
\multirow{ 2}{*}{{DCN-V2} (ours)} & \multirow{ 2}{*}{ $\ge 2$} & \multirow{ 2}{*}{${\bf x}_i = {\bf x}_0 \odot (W_i {\bf x}_i)$} & $f_i \circ f_e$\\
&&&$f_i + f_e$ \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\subsubsection{Implementation Details.}
\label{sec:implementation_details}
All the baselines and our approaches are implemented in TensorFlow v1. For a fair comparison, all the implementations were identical across all the models except for the feature interaction component \footnote{We adopted implementation from \url{https://github.com/Leavingseason/xDeepFM}, \url{https://github.com/facebookresearch/dlrm} and \url{https://github.com/shenweichen/DeepCTR}}.

{\bf Embeddings.} All the baselines require each feature's embedding size to be the same except for DNN and DCN. Hence, we fixed it to be $ \text{Avg}\big(\sum_{\text{vocab}} 6\cdot(\text{vocab cardinality})^{\frac{1}{4}}\big)$ (39 for Criteo and 30 for Movielen-1M) for all the models\footnote{This formula is a rule-of-thumb number that is widely used \cite{wang2017deep}, also see \url{https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html}}. 

{\bf Optimization.} We used Adam \cite{kingma2014adam} with a batch size of $512$ (128 for MovieLen). The kernels were initialized with He Normal \cite{he2015delving},
and biases to ${\bf 0}$; the gradient clipping norm was 10; an exponential moving average with decay 0.9999 to trained parameters was applied.

{\bf Reproducibility and fair comparisons: hyper-parameters tuning and results reporting.} For all the baselines, we conducted a coarse-level (larger-range) grid search over the hyper-parameters, followed by a finer-level (smaller-range) search. To ensure reproducibility and mitigate model variance, for each approach and dataset, we report the mean and stddev out of 5 independent runs for the best configuration. We describe detailed settings below for Criteo; and follow a similar process for MovieLens with different ranges.

We first describe the hyper-parameters shared across the baselines. The learning rate was tuned from $10^{-4}$ to $10^{-1}$ on a log scale and then narrowed down to $10^{-4}$ to $5 \times 10^{-4}$ on a linear scale. The training steps were searched over \{150k, 160k, 200k, 250k, 300k\}. The number of hidden layers ranged in \{1, 2, 3, 4\} with their layer sizes in \{562, 768, 1024\}. And the regularization parameter $\lambda$ was in \{0, $3 \times 10^{-5}$, $10^{-4}$\}.

We then describe each model's own hyper-parameters, where the search space is designed based on reported setting. For DCN, the number of cross layers ranged from 1 to 4. For AutoInt, the number of attention layers was from 2 to 4; the attention embedding size was in \{20, 32, 40\}; the number of attention head was from 2 to 3; and the residual connection was either on or off. For xDeepFM, the CIN layer size was in \{100, 200\}, depth in \{2, 3, 4\}, activation was identity, computation was either direct or indirect. For DLRM, the bottom MLP layer sizes and numbers was in \{(512,256,64), (256,64)\}. For PNN, we ran for IPNN, OPNN and PNN*, and for the latter two, the kernel type ranged in \{full matrix, vector, number\}.
For all the models, the total number of parameters was capped at $1024^2 \times 5$ to limit the search space and avoid overly expensive computations.

\subsection{Performance of Feature Interaction Component Alone (RQ2)}
\label{sec:performance_feature_interaction}
We consider the feature interaction component alone of each model {\bf without their DNN component}. Moreover, we only consider the categorical features, as the dense features were processed differently among baselines. \autoref{tab:cross-only} shows the results on Criteo dataset. Each baseline was tuned similarly as in \autoref{sec:implementation_details}. There are two major observations. 1). Higher-order methods demonstrate a superior performance over 2nd-order methods. This suggests high-order crosses are meaningful in this dataset. 2). Among the high-order methods, cross network achieved the best performance and was on-par or slightly better compared to DNN.

\begin{table}[htpb]
\small
\caption{LogLoss (test) of feature interaction component of each model (no DNN). Only categorical features were used. In the `Setting' column, $l$ stands for number of layers.}
\vspace{-2ex}
\label{tab:cross-only}
\begin{center}
\begin{tabular}{c|l|cl}
\toprule
& Model & LogLoss & \multicolumn{1}{c}{Best Setting}\\
\midrule
\multirow{ 3}{*}{2nd} 
&PNN \cite{qu2016product} &0.4715 $\pm$ 4.430e-04 & OPNN, kernel=matrix\\
&FM      & 0.4736 $\pm$ 3.04E-04& \multicolumn{1}{c}{--} \\
\midrule
\multirow{ 5}{*}{$>$2} 
&CIN \cite{lian2018xdeepfm}  & 0.4719 $\pm$ 9.41E-04& l=3, cinLayerSize=100\\
&AutoInt \cite{song2019autoint}   & 0.4711 $\pm$ 1.62E-04& l=2, head=3, attEmbed=40\\
&DNN        & 0.4704 $\pm$ 1.57E-04& l=2, size=1024\\
\cline{2-4}
&CrossNet     & 0.4702 $\pm$ 3.80E-04& l=2\\
&CrossNet-Mix & \bf 0.4694 $\pm$ 4.35E-04 & l=5, expert=4, gate=$\frac{1}{1+e^{-x}}$\\

\bottomrule
\end{tabular}
\end{center}
\end{table}

\subsection{Performance of Baselines (RQ3)}
\label{sec:performance_baselines}
This section compares the performance between {DCN-V2} approaches and the baselines in an end-to-end fashion. Note that the best setting reported for each model was searched over a {\bf wide-ranged model capacity and hyper-parameter space} including the baselines. And if two settings performed on-par, we report the {\bf lower-cost} one. \autoref{tab:baseline_comparison} shows the best LogLoss and AUC (Area Under the ROC Curve) on testset for Criteo and MovieLen. For Criteo, a {\bf 0.001-level improvement} is considered significant (see \cite{song2019autoint, wang2017deep, guo2017deepfm}). 
We see that {DCN-V2} consistently outperformed the baselines (including DNN) and achieved a healthy quality/cost trade-off. It's also worth mentioning that the baselines' performances reported in \autoref{tab:baseline_comparison} were improved over the numbers reported by previous papers (see \autoref{tab:metrics_in_papers} in Appendix); however, when integrated with DNN, their performance gaps are closing up (compared to \autoref{tab:cross-only}) with their performances on-par and sometimes worse than the ReLU-based DNN with fine-granular model tuning.

{\bf Best Settings.} The optimal hyper-parameters are in \autoref{tab:baseline_comparison}. For {DCN-V2} models, both the `stacked' and `parallel' structures outperformed all the baselines, while `stacked' worked better on Criteo and `parallel' worked better on Movielen-1M. On Criteo, the setting was gate as constant, hard\_tanh activation for {DCN-Mix}; gate as softmax and identity activation for CrossNet. The best training steps was 150k for all the baselines; learning rate varies for all the models.


{\bf Model Quality --- Comparisons among baselines.} When integrating the feature cross learning component with a DNN, the advantage of higher-order methods is less pronounced, and the performance gap among all the models are closing up on Criteo (compared to \autoref{tab:cross-only}). \textbf{This suggests the importance of implicit feature interactions and the power of a well-tuned DNN model.}

For 2nd-order methods, DLRM performed inferiorly to DeepFM although they are both derived from FM. This might be due to DLRM's omission of the 1st-order sparse features after the dot-product layer. PNN models 2nd-order crosses more expressively and delivered better performance on MovieLen-1M; however on Criteo, its mean LogLoss was driven up by its high standard deviation. For higher-order methods, xDeepFM, AutoInt and DCN behaved similarly on Criteo, while on MovieLens xDeepFm showed a high variance.

{DCN-V2} achieved the best performance (0.001 considered to be significant on Criteo \cite{wang2017deep, lian2018xdeepfm, song2019autoint}) by explicitly modeling up to 3rd-order crosses beyond those implicit ones from DNN. {DCN-Mix}, the mixture of low-rank DCN, efficiently utilized the memory and reduced the cost by 30\% while maintaining the accuracy. Interestingly, CrossNet alone outperformed DNN on both datasets; we defer more discussions to \autoref{sec:crossnet_performance}.


{\bf Model Quality --- Comparisons with DNN.} DNNs are universal approximators and are tough-to-beat baselines when highly-optimized. Hence, we finely tuned DNN along with all the baselines, and used a larger layer size than those used in literature (\emph{e.g.}, 200 - 400 in \cite{lian2018xdeepfm, song2019autoint}). \textbf{To our surprise, DNN performed neck to neck with most baselines and even outperformed certain models.}

Our hypothesis is that those explicit feature crosses from baselines were not modeled in an {\bf expressive} and {\bf easy-to-optimize} manner. The former makes its performanc easy to be matched by a DNN with large capacity. The latter would easily lead to trainability issues, making the model unstable, hard to identify a good local optima or to generalize. Hence, when integrated with DNN, the overall performance is dominated by the DNN component. This becomes especially true with a large-capacity DNN, which could already approximate some simple cross patterns.

In terms of expressiveness, consider the 2nd-order methods. PNN models crosses more expressively than DeepFM and DLRM, which resulted in its superior performance on MovieLen-1M. This also explains the inferior performance of DCN compared to {DCN-V2}.

In terms of trainability, certain models might be inherently more difficult to train and resulted in unsatisfying performance. Consider PNN. On MoiveLen-1M, it outperformed DNN, suggesting the effectiveness of those 2nd-order crosses. On Criteo, however, PNN's advantage has diminished and the averaged performance was on-par with DNN. This was caused by the instability of PNN. Although its best run was better than DNN, its high stddev from multiple trials has driven up the mean loss. xDeepFM also suffers from trainability issue (see its high stddev on MovieLens). In xDeepFM, each feature map encodes all the pair-wise crosses while only relies on a single variable to learn the importance of each cross. In practice, a single variable is difficult to be learned when jointly trained with magnitudes more parameters. Then, an improperly learned variable would lead to noises.


{DCN-V2}, on the other hand, consistently outperforms DNN. It successfully leveraged both the explicit and implicit feature interactions. We attribute this to the balanced number of parameters between the cross network and the deep network ({\bf expressive}), as well as the simple structure of cross net which eased the optimization ({\bf easy-to-optimize}). It's worth noting that the high-level structure of {DCN-V2} shares a similar spirit of the self-attention mechanism adopted in AutoInt, where each feature embedding attends to a weighed combination of other features. The difference is that during the attention, higher-order interactions were modeled explicitly in {DCN-V2} but implicitly in AutoInt.

{\bf Model Efficiency.} \autoref{tab:baseline_comparison} also provides details for model size and FLOPS\footnote{FLOPS is a close estimation of run time, which is subjective to implementation details.}. The reported setting was properly tuned over the hyper-parameters of each model and the DNN component. For most models, the FLOPS is roughly 2x of the \#params; for xDeepFM, however, the FLOPS is one magnitude higher, making it impractical in industrial-scale applications (also observed in \cite{song2019autoint}). Note that for DeepFM and DLRM, we've also searched over larger-capacity models; however, they didn't deliver better quality. Among all the methods, {DCN-V2} delivers the best performance while remaining relatively efficient; {DCN-Mix} further reduced the cost, achieving a better trade-off between model efficiency and quality.

\begin{table*}[htpb]
\small
\caption{LogLoss and AUC (test) on Criteo and Movielen-1M. The metrics were averaged over 5 independent runs with their stddev in the parenthesis. In the `Best Setting' column, the left reports DNN setting and the right reports model-specific setting. $l$ denotes layer depth; $n$ denotes CIN layer size; $h$ and $e$, respectively, denotes \#heads and att-embed-size; $K$ denotes \#experts and $r$ denotes total rank.}
\vspace{-3ex}
\label{tab:baseline_comparison}
\begin{center}
\begin{tabular}{l|ccp{2.5em}p{2.5em}ll|ccp{2.5em}p{2.5em}}
\toprule
\multirow{ 2}{*}{{\bf Baseline}} & \multicolumn{6}{c|}{\bf Criteo} & \multicolumn{4}{c}{\bf MovieLens-1M} \\
& Logloss  & AUC & Params & FLOPS &  \multicolumn{2}{c|}{Best Setting} &Logloss & AUC  & Params & FLOPS  \\
\midrule
PNN     & 0.4421 (5.8E-4) & 0.8099 (6.1E-4) & 3.1M & 6.1M & (3, 1024) & OPNN
         & 0.3182 (1.4E-3) & 0.8955 (3.3E-4) & 54K & 110K\\
DeepFm  & 0.4420 (1.4E-4) & 0.8099 (1.5E-4) & 1.4M & 2.8M & (2, 768) & \multicolumn{1}{c|}{--}
         & 0.3202 (1.0E-3) & 0.8932 (7.7E-4) & 46K & 93K\\
DLRM    & 0.4427 (3.1E-4) & 0.8092 (3.1E-4) & 1.1M & 2.2M & (2, 768) &  [512,256,64]
         & 0.3245 (1.1E-3) & 0.8890 (1.1E-3) & 7.7K & 16K\\
xDeepFm & 0.4421 (1.6E-4) & 0.8099 (1.8E-4) & 3.7M & 32M & (3, 1024) & $l$=2, $n$=100
         & 0.3251 (4.3E-3) & 0.8923 (8.6E-4) & 160K & 990K\\
AutoInt+ & 0.4420 (5.7E-5) & 0.8101 (2.6E-5) & 4.2M & 8.7M & (4, 1024) & $l$=2, $h$=2, $e$=40
         & 0.3204 (4.4E-4) & 0.8928 (3.9E-4) & 260K & 500K\\
DCN     & 0.4420 (1.6E-4) & 0.8099 (1.7E-4) & 2.1M & 4.2M & (2, 1024) & $l$=4
         & 0.3197 (1.9E-4) & 0.8935 (2.1E-4) & 110K & 220K\\
DNN    & 0.4421 (6.5E-5) & 0.8098 (5.9E-5) & 3.2M & 6.3M & (3, 1024) &  \multicolumn{1}{c|}{--}
         & 0.3201 (4.1E-4) & 0.8929 (2.3E-4) & 46K & 92K\\
\midrule
\multicolumn{1}{c}{\bf Ours}& & & & & & \multicolumn{1}{c}{}& & & &\\
{DCN-V2}  & \bf 0.4406 (6.2E-5) & \bf 0.8115 (7.1E-5) & 3.5M & 7.0M & (2, 768) & $l$=2
         & 0.3170 (3.6E-4) & 0.8950 (2.7E-4) & 110K & 220K \\
{DCN-Mix} & 0.4408 (1.0E-4) & 0.8112 (9.8E-5) & 2.4M & 4.8M  & (2, 512) & $l$=3, $K$=4, $r$=258
         & \bf 0.3160 (4.9E-4) & \bf 0.8964 (2.9E-4) & 110K & 210K \\
CrossNet &0.4413 (2.5E-4) & 0.8107 (2.4E-4) & 2.1M & 4.2M &  -- & $l$=4, $K$=4, $r$=258
         & 0.3185 (3.0E-4) & 0.8937 (2.7E-4) & 65K & 130K \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-5pt}
\end{table*}

\subsection{Can Cross Layers Replace ReLU layers?}
\label{sec:crossnet_performance}
The solid performance of {DCN-V2} approaches has inspired us to further study the efficiency of their cross layers (CrossNet) in learning explicit high-order feature crosses.

In a realistic setting with resource constraints, we often have to limit model capacity. Hence, we fixed the model capacity (memory / \# of parameters) at different levels, and compared the performance between a model with only cross layers (Cross Net), and a ReLU based DNN. \autoref{tab:logloss_memory} reports the best test LogLoss for different memory constraints. The memory was controlled by varying the number of cross layers and its rank (\{128, 256\}), the number of hidden layers and their sizes. The best performance was achieved by the cross network (5-layer), suggesting the ground-truth model could be well-approximated by polynomials. Moreover, the best performance per memory limit was also achieved by the cross network, indicating both solid effectiveness and efficiency.

It is well known that ReLU layers are the backbone for various Neural Nets models including DNN, Recurrent Neural Net (RNN) \cite{rumelhart1985learning, hochreiter1997long, mikolov2011extensions} and Convolutional Neural Net (CNN) \cite{lecun1989backpropagation, schmidhuber2015deep, lawrence1997face}. It is quite surprising and encouraging to us that we may potentially replace ReLU layers by Cross Layers entirely for certain applications. Obviously we need significant more analysis and experiments to verify the hypothesis. Nonetheless, this is a very interesting preliminary study and sheds light for our future explorations on cross layers.

\begin{table}[htbp]
\small
\vspace{-4pt}
\caption{Logloss and AUC (test) with a fixed memory budget.}
\vspace{-10pt}
\label{tab:logloss_memory}
\begin{center}
\begin{tabular}{cc|cccc}
\toprule
\multicolumn{2}{c|}{\bf \#Params} & 7.9E+05 & 1.3E+06 & 2.1E+06 & 2.6E+06\\
\midrule
\multirow{ 2}{*}{LogLoss}& CrossNet & \bf 0.4424 & \bf 0.4417 & \bf 0.4416 & \bf 0.4415\\
 &  DNN & 0.4427 & 0.4426 & 0.4423 & 0.4423\\
 \midrule
\multirow{ 2}{*}{AUC} &  CrossNet & \bf 0.8096 & \bf 0.8104 & \bf 0.8105  & \bf 0.8106\\
& DNN & 0.8091 & 0.8094  & 0.8096 & 0.80961 \\

\bottomrule
\end{tabular}
\end{center}
\vspace{-4pt}
\end{table}

\subsection{How the Choice of Hyper-parameters Affect {DCN-V2} Model Performance (RQ4)} 
\label{sec:hyper-parameters}
This section examines the model performance as a function of hyper-parameters that include 1) depth of cross layers; 2) matrix rank of {DCN-Mix}; 3) number of experts in {DCN-Mix}.

{\bf Depth of Cross Layers.}
By design, the highest feature cross order captured by the cross net increases with layer depth. Hence, we constrain ourselves to the full-rank cross layers, and evaluate the performance change with layer depth

\autoref{fig:dcn-layer-depth} shows the test LogLoss and AUC while increasing layer depth on the Criteo dataset. We see a steady quality improvement with a deeper cross network, indicating that it's able to capture more meaningful crosses. The rate of improvement, however, slowed down when more layers were used. This suggests the contribution from that of higher-order crosses is less significant than those from lower-order crosses. We also used a same-sized DNN as a reference. When there were $\le 2$ layers, DNN outperformed the cross network; when more layers became available, the cross network started to close the performance gap and even outperformed DNN. In the small-layer regime, the cross network could only approximate very low-order crosses (\emph{e.g.,} 1 $\sim$ 2); in the large-layer regime, those low-order crosses were characterized with more parameters, and those high-order interactions were started to be captured.

\begin{figure}[htbp]
\small
\centering
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-depth.png}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-rank-v2.png}
    \end{subfigure}\\[-2ex]
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-layer-auc.png}
    \caption{Layer depth}
    \label{fig:dcn-layer-depth}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth]{dcn-rank-auc.png}
    \caption{Matrix rank}
    \label{fig:dcn-rank}
    \end{subfigure}
    \vspace{-2ex}
    \caption{Logloss and AUC (test) v.s. depth \& matrix rank.}
\vspace{-10pt}
\end{figure}

{\bf Rank of Matrix.}
The rank of the weight matrix controls the number of parameters as well as the portion of low-frequency signals passing through the cross layers. Hence, we study its influence on model quality. The model is based on a well-performed setting with 3 cross layers followed by 3 hidden layers of size 512. We approximate the dense matrix $W$ in each cross layer by $UV^\top$ where $U, V \in \mathbb{R}^{d \times r}$, and we vary $r$. We loosely consider the smaller dimension $r$ to be the rank

\autoref{fig:dcn-rank} shows the test LogLoss and AUC v.s. matrix's rank $r$ on Criteo. When $r$ was as small as 4, the performance was on-par with other baselines. When $r$ was increased from 4 to 64, the LogLoss decreased almost linearly with $r$ (\emph{i.e.}, model's improving). When $r$ was further increased from 64 to full, the improvement on LogLoss slowed down. We refer to 64 as the \emph{threshold rank}. The significant slow down from 64 suggests that the important signals characterizing feature crosses could be captured in the top-64 singular values.

Our hypothesis for the value of this \emph{threshold rank} is $O(k)$ where $k$ represents \# features (39 for Criteo). Consider the $(i,j)$-th block of matrix $W$, we can view $\small W_{i,j} = W_{i,j}^L + W_{i,j}^H$, where $W_{i,j}^L$ stores the dominant signal (low-frequency) and $\small W_{i,j}^H$ stores the rest (high-frequency). In the simplest case where $\small W_{i,j}^L = c_{ij} {\bf 1} {\bf 1}^\top$, the entire matrix $\small W^L$ will be of rank $k$. The effectiveness of this hypothesis remains to be verified across multiple datasets

{\bf Number of Experts.}
We study how the number of low-rank experts affects the quality. We've observed that 1) best-performed setting (\#expert, gate, matrix activation type) was subjective to datasets and model architectures; 2) the best-performed model of each setting yielded similar results. For example, for a 2-layered cross net with total rank 256 on Criteo, the LogLoss for 1, 4, 8, 16, and 32 experts, respectively, was 0.4418, 0.4416, 0.4416, 0.4422, and 0.4420. The fact that more lower-ranked experts wasn't performing better than a single higher-ranked expert might be caused by the na\"ive gating functions and optimizations adopted. We believe more sophisticated gating \cite{jang2016categorical, louizos2017learning, ma2019snr} and optimization techniques (\emph{e.g.}, alternative training, special initialization, temperature adjustment) would leverage more from a mixture of experts. This, however, is beyond the scope of this paper and we leave it to future work.


\subsection{Model Understanding (RQ5)}
\label{sec:model_understanding}
One key research question is whether the proposed approaches are indeed learning meaningful feature crosses. A good understanding about the learned feature crosses helps improve model understandability, and is especially crucial to fields like ML fairness and ML for health. Fortunately, the weight matrix $W$ in {DCN-V2} exactly reveals what feature crosses the model has learned to be important. Specifically, we assume that each input ${\bf x} = [{\bf x}_1; {\bf x}_2; \ldots; {\bf x}_k]$ contains $k$ features with each represented by an embedding ${\bf x}_i$. Then, the block-wise view of the feature crossing component (ignoring the bias) in Eq. \eqref{eq:block-mat} shows that the importance of feature interaction between $i$-th and $j$-th feature is characterized by the $(i,j)$-th block $W_{i,j}$.
\begin{equation} \label{eq:block-mat}
    {\bf x} \odot W {\bf x} = 
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
    \odot
    \left[\begin{smallmatrix}
        W_{1,1} & W_{1,2} & \cdots & W_{1,k} \\
        W_{2,1} & W_{2,2} & \cdots & W_{2,k} \vspace{-1ex}\\
        \vdots  & \vdots  & \ddots & \vdots  \\
        W_{k,1} & W_{k,2} & \cdots & W_{k,k}
    \end{smallmatrix}\right]
    \left[\begin{smallmatrix}
        {\bf x}_1 \\
        {\bf x}_2 \vspace{-1ex}\\
        \vdots  \\
        {\bf x}_k
    \end{smallmatrix}\right]
\end{equation}

\autoref{fig:dcn-mat-visualization} shows the learned weight matrix $W$ in the first cross layer. Subplot (a) shows the entire matrix with orange boxes highlighting some notable feature crosses. The off-diagonal block corresponds to crosses that are known to be important, suggesting the effectiveness of {DCN-V2}. The diagonal block represents self-interaction ( $x^2$'s).
Subplot (b) shows each block's Frobenius norm and indicates some strong interactions learned, \emph{e.g.}, {\ttfamily{\small Gender}} $\times$ {\ttfamily{\small UserId}}, {\ttfamily{\small MovieId}} $\times$ {\ttfamily{\small UserId}}.

\begin{figure}[htbp]
\centering
    \begin{subfigure}[b]{0.18\textwidth}  
    \includegraphics[width=\textwidth,valign=t]{dcn-prod-visualization.png}
    \caption{Production data}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.22\textwidth}  
    \includegraphics[width=\textwidth,valign=t]{dcn-movielen-visualization.png}
    \caption{Movielen-1M}
    \end{subfigure}
    \caption{Visualization of learned weight matrix in {DCN-V2}. Rows and columns represents real features. For (a), feature names were not shown for proprietary reasons; darker pixel represents larger weight in its absolute value. For (b), each block represents the Frobenius norm of each matrix block.}
\label{fig:dcn-mat-visualization}
\end{figure}

\section{Productionizing {DCN-V2} at Google}
\label{sec:productionization}
This section provides a case study to share our experience productionizing {DCN-V2} in a large-scale recommender system in Google. We've achieved significant gains through {DCN-V2} in both offline model accuracy, and online key business metrics.

{\bf The Ranking Problem:}
Given a user and a large set of candidates, our problem is to return the top-$k$ items the user is most likely to engage with. Let's denote the training data to be $\{({\bf x}_i, y_i)\}_{i=1}^N$, where ${\bf x}_i$'s represents features of multiple modalities, such as user's interests, an item's metadata and contextual features; $y_i$'s are labels representing a user's action (\emph{e.g.}, a click). The goal is to learn a function $f: \mathbb{R}^d \mapsto \mathbb{R}$ that predicts the probability $P(y | {\bf x})$, the user's action $y$ given features ${\bf x}$.

{\bf Production Data and Model:}
The production data are sampled user logs consisting of hundreds of billions of training examples. The vocabulary sizes of sparse features vary from 2 to millions. The baseline model is a fully-connected multi-layer perceptron (MLP) with ReLU activations.




{\bf Comparisons with Production Models:} When compared with production model, {DCN-V2} yielded 0.6\% AUCLoss (1 - AUC) improvement. For this particular model, a gain of 0.1\% on AUCLoss is considered a significant improvement.  We also observed significant online performance gains on key metrics.
\autoref{tab:dcn-relu-prod} further verifies the amount of gain from {DCN-V2} by replacing cross layers with same-sized ReLU layers.

\begin{table}[htpb]
\small
    \caption{Relative AUCLoss of {DCN-V2} v.s. same-sized ReLUs}
    \label{tab:dcn-relu-prod}
    \vspace{-3ex}
        \centering
        \begin{tabular}{cccc}
        \toprule
        { 1layer ReLU}  & {2layer ReLU} & {1layer {DCN-V2}} & {2layer {DCN-V2}}\\
        \midrule
         0\%   & -0.15\% & -0.19\% &  -0.45\% \\
        \bottomrule
        \end{tabular}
\end{table}

{\bf Practical Learnings.} We share some practical lessons we have learned through productionizing {DCN-V2}. 
\begin{itemize}[leftmargin=1em]
\item It's better to insert the cross layers in between the input and the hidden layers of DNN (also observed in \cite{shan2016deep}). Our hypothesis is that the physical meaning of feature representations and their interactions becomes weaker as it goes farther away from the input layer.
\item We saw consistent accuracy gains by stacking or concatenating 1 - 2 cross layers. Beyond 2 cross layers, the gains start to plateau.
\item We observed that both stacking cross layers and concatenating cross layers work well. Stacking layers learns higher-order feature interactions, while concatenating layers (similar to multi-head mechanism \cite{vaswani2017attention}) captures complimentary interactions.
\item We observed that using low-rank DCN with rank $\text{(input size)}/4$ consistently preserved the accuracy of a full-rank {DCN-V2}.  
\end{itemize}


\section{Conclusions and Future Work}

In this paper, we propose a new model---{DCN-V2}---to model explicit crosses in an expressive yet simple manner. Observing the low-rank nature of the weight matrix in the cross network, we also propose a mixture of low-rank DCN ({DCN-Mix}) to achieve a healthier trade-off between model performance and latency. {DCN-V2} has been successfully deployed in multiple web-scale learning to rank systems with significant offline model accuracy and online business metric gains. Our experimental results also have demonstrated {DCN-V2}'s effectiveness over SOTA methods.


For future work, we are interested in advancing our understanding of 1). the interactions between {DCN-V2} and optimization algorithms such as second-order methods; 2). the relation between embedding, {DCN-V2} and its rank of matrix. Further, we would like to improve the gating mechanism in {DCN-Mix}. Moreover, observing that cross layers in {DCN-V2} may serve as potential alternatives to ReLU layers in DNNs, we are very interested to verify this observation across more complex model architectures (\emph{e.g.}, RNN, CNN).

  
\medskip
{\bf Acknowledgement.} We would like to thank Bin Fu, Gang (Thomas) Fu, and Mingliang Wang for their early contributions of {DCN-V2}; Tianshuo Deng, Wenjing Ma, Yayang Tian, Shuying Zhang, Jie (Jerry) Zhang, Evan Ettinger, Samuel Ieong and many others for their efforts and supports in productionizing {DCN-V2}; Ting Chen for his initial idea of mixture of low-rank; and Jiaxi Tang for his valuable comments.

\bibliographystyle{ACM-Reference-Format}

\section{Research Questions}
\label{sec:research_qs}
We are interested to seek answers for these following research questions:

\begin{itemize}[leftmargin=2.5em]
    \item[\bf RQ1]  When would feature interaction learning methods become more efficient than ReLU-based DNNs? What's the role of each component in the cross network?
    \item[\bf RQ2]  How does the feature-interaction component of each baseline perform without integrating with DNN? 
    \item[\bf RQ3]  How does the proposed mDCN approaches compare to the baselines? Could we achieve healthier trade-off between model accuracy and cost through mDCN and the mixture of low-rank DCN?
    \item[\bf RQ4]  Could cross network replace typical ReLU layers?
    \item[\bf RQ5] How does the settings in mDCN affect model quality?
    \item[\bf RQ6] Is mDCN capturing important feature crosses? Does the model provide good understandability?
\end{itemize}