\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there have been huge amounts of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM,NewsBert}. Despite the improvement of recommendation quality, the PLMs based news recommenders are severely limited in efficiency. This is because the recommenders need to encode users' historical news browsing behaviors to capture their underlying interests. However, many real-world users are associated with super long histories, which will incur considerable computation costs when PLMs are employed as the encoding backbones. Although accelerations can be made with recent approaches, like distilled lightweight PLMs \cite{sanh2019distilbert} and efficient transformers \cite{tay2020efficient}, the computation cost will still grow dramatically with the input size. As a result, it remains a great challenge to make timely recommendations for active users.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendations is more of capturing a user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user. Therefore, instead of relying on the entire content of the historically browsed news articles, the user's underlying interest is likely to be well represented with a small number of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user-side input can be filtered for making news recommendations. In GateFormer, a user is represented with two steps. Firstly, each input news article is processed by the gating module, which selects the top-$K$ informative keywords from the news text. Secondly, the filtered keywords from the entire user history are concatenated and encoded by transformers, generating the user representation. By setting $K$ to a reasonably small number, the encoding cost will be substantially reduced, as the user-side input becomes significantly smaller than its original size.

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user-side input, the gating module estimates each word's importance w.r.t. user's underlying interest; if one word strongly correlates with the user interest, it will be selected by the gating module with a high priority. In this way, we are able to maximally preserve the keywords that truly matter to news recommendation. 2) The gating module is designed to be \textit{lightweight}. Knowing that the gating module needs to inspect the entire input, we implement it with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable to the computation cost incurred by transformers. 3) Considering that there is no supervision data for input filtering, the gating module is designed to be a differentiable component cascaded with transformer module; thus, it can be \textit{end-to-end learned} to select proper keywords for the optimal recommendation performance. 

Note that the filtered user input may naturally support inverted index (it can be used as an enhanced BM25 system) and provide keyword-based explanations for recommendation results, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work that introduces a gating mechanism to accelerate the recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormers' effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly compressed input.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories \cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in \cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in \cite{wu2019neural}, the hierarchical attention networks (HAN) \cite{yang2016hierarchical} is adapted for the multi-view representation of news articles \cite{wu2019neural}; and in \cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pre-trained language models \cite{wu2021newsbert,xiao2021training}, where PLMs like BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta} are employed as the backbones of news encoder. With the adoption of such highly expressive models, rich-semantic  embeddings can be generated, which substantially benefits the recommendation quality.



Despite the improved quality, the PLMs based news recommenders are limited in multiple perspectives. For example, the recommendation results are lack of explainability \cite{wang2019explainable,liu2020kred}; besides, the deep models are also incompatible with the existing inverted index recall systems \cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is efficiency. While making news recommendations for a user, her entire browsing history needs to be encoded by pre-trained language models. Given the expensive computation costs of PLMs, it will be extremely difficult to make real-time recommendations for those active users who are associated with lots of historical news clicks. In recent years, many approaches are proposed for the acceleration of PLMs. One popular way is to distill large-scale PLMs into lightweight models \cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Besides, efficient transformers are proposed, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer \cite{wang2020linformer} and Performer \cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers \cite{child2019generating} and Big Bird \cite{zaheer2020big} utilize sparse self-attention; Reformer introduces learnable attention patterns, and Synthesizer \cite{tay2021synthesizer} introduces randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not fast enough to handle the challenges in news recommendation. This is because online users are constantly generating large amounts of news clicks. Therefore, even with the proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers, there will still be formidable costs for the recommenders. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup. 

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions \cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_N\}$, the recommender learns to represent the user's underlying interest, and predicts the user's future news clicks based on it. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user-side input can be super large: one user may generate a huge amount of historical news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user-side input with PLMs will be prohibitive. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer2.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user-side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interests. The filtered input (i.e., the selected keywords) is concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of candidate news.}
\vspace{-10pt}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown in Figure \ref{fig:1}. There are two basic components in GateFormer. First, the Gating Module, which processes the entire user-side input and selects the top-$K$ keywords for each historical news article. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Second, the Transformer Module, which encodes the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder-like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword selection based on the user's interest. For efficiency concerns, the user interest is encoded by the following \textbf{lightweight} models.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). Firstly, we leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we further aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, $\mathbf{u}^g$ is the aggregation result used for the representation of user interest (by the gating module); we take the last output hidden-state for $\mathbf{u}^g$. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that may fully represent the user interest. Therefore, we use the correlation with the extracted user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input token (e.g., the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) that produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token and masking the duplicated ones in the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article. 

It should be noted that there is no explicit supervision signal for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with the transformer module in the resulted computation graph; therefore, it can learn to select the tokens that optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation \cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|I_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with more than 10-fold compression of input, which means over $10\times$ lossless acceleration of the news recommender.

\begin{table*}[t]
    \centering
    \small
    \caption{\small Evaluations of accuracy and efficiency on MIND. The highest performances are bolded, the strongest baselines (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \vspace{-5pt}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{4}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule
    \multirow{2}{0.12\linewidth}{Variational Schemes}
    & GFM (Global) & ${70.48}$ & ${35.14}$ & ${38.53}$ & ${44.22}$ & ${0.67}$ & ${3158}$ & ${20.4}$\\
    & GFM (TTW) & ${70.40}$ & ${34.65}$ & ${37.89}$ & ${43.68}$ & ${0.89}$ & ${3202}$ & ${20.0}$\\
    \midrule
    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}   
  \vspace{-10pt}
\end{table*}

\section{Experiments}
Our experiments are mainly dedicated to study GateFormer's impact on news recommendation accuracy and efficiency. Besides, we explore its application to keyword based news retrieval, demonstrate its interpretability, and evaluate its effectiveness on other corpus. 

$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title and abstract of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{\scriptsize  https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in applications beyond news recommendation. This dataset contains academic papers and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. The \textit{reference recommendation} task is evaluated by: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in the reference list.

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard Methods}. 1)
\textbf{NAML} \cite{wu2019neural}, 2) \textbf{LSTUR} \cite{an2019neural},
3) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 4) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 5) \textbf{Big Bird}~\cite{zaheer2020big}, 6) \textbf{LongFormer}~\cite{beltagy2020longformer}, 7) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 8) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 9) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 10) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering input with the following methods. 11) \textbf{GFM (BM25)}, where the user-side input is filtered by BM25 score; 12) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 13) \textbf{GFM (First-$K$)}, where the first $K$ tokens are kept; and 14) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pre-trained model for keyword extraction) is used to generate the filtered input.

Finally, we test different networks for the gating module of GateFormer. 14) The news encoder in gate is switched to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. The news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. Our implementation is based on PyTorch-1.9.1. The experiments are performed on a cluster of 2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. \textit{Supplementary experiments, configurations about training/testing are specified in Appendix. Our code will be open-sourced after the review stage. }

\begin{table*}[t]
  \centering
  \small
  \caption{\small News Recall Evaluations.}
  \vspace{-5pt}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the comparison methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.7$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold in GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully capture users' underlying interests. On the other hand, given that the user-side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result.

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods that may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration, we find the cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for \textit{GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user-side input. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.92\linewidth]{Figures/case.pdf}
    \caption{\small Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \vspace{-10pt}
    \label{fig:case}
    \vspace{-10pt}
\end{figure}

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences in time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, analyze the dense recall scenario, where the candidate news are recalled based on the similarity between the user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we analyze the sparse recall scenario, where the candidate news is recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. We also analyze the effect of sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as a strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{table}[t]
    \centering
    \small
    \caption{\small Evaluation on DBLP.}
    \vspace{-5pt}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{MRR} & \textbf{NDCG}\\
    \midrule
   
   
    GFM (Random) & $40.23$ & $66.82$ & $52.50$ & $62.36$ \\
    GFM (First) & $50.73$ & $75.30$ & $61.21$ & $69.51$\\
    GFM (BM25) & $43.89$ & $69.48$ & $55.70$ & $64.93$\\
    GFM (KeyBERT) & $46.46$ & $72.51$ & $58.38$ & $67.17$ \\
    \midrule
    GateFormer & $\textbf{64.86}$ & $\textbf{89.03}$ & $\textbf{75.41}$ & $\textbf{81.06}$\\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table}

$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words do not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represents the news semantics and indicates the underlying reason why it catches the user's attention. 

    
   
   
   
   
   
   
   
   
   
   

$\bullet$ \textit{Evaluation on DBLP}. We make comparison with other gating methods: \textit{GFM (Random)} (due to the limited entity coverage, the entity based method is switched to random selection), \textit{GFM (First)}, \textit{GFM (BM25)} and \textit{GFM (KeyBERT)}. The experiment results in Table \ref{tab:effectiveness_dblp} are consistent with our previous observations in Table \ref{tab:effectiveness_mind}, and GateFormer's advantages are even highlighted. Such a finding suggests that GateFormer may stand as a generic input filtering method, which facilitates the speedup and keyword selection for text oriented recommendation/retrieval tasks. 

\section{Conclusion}
In this paper, we proposed GateFormer for the speedup of news recommendations. It introduced a gating mechanism, where informative keywords can be filtered from historical news clicks to represent the user's interest. The gating module was designed to be personalized, lightweight, and end-to-end learnable, such that the input filtering can be accurately conducted with little running costs. Our experimental studies verified the effectiveness of GateFormer: it achieved notable improvements in both recommendation accuracy and efficiency against the existing acceleration methods. 

\newpage

\newpage

\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there are a huge amount of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM, NewsBert}. Despite the effectiveness improvement, the PLMs based news recommenders are severely limited in efficiency. Specifically, when PLMs are employed as the encoding backbone, it becomes a heavy burden for the recommender to encode the historical news articles for millions of users to capture their underlying interests. This problem is even worse in face of some active users with super long browsing histories. Although accelerations can be made with recent approaches, like distilled lightweight PLMs~\cite{sanh2019distilbert} and efficient transformers~\cite{tay2020efficient}, the computation cost still grows drastically with the input size. 
In consequence, it remains a great challenge to make recommendation that makes the best of both quality and efficiency.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendations is more of capturing users' underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user.\blue{references?} Therefore, instead of relying on the entire content of the historical news articles, a user's underlying interest is likely to be well represented with a small number of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be quickly selected before running a heavy PLM. In GateFormer, a user is represented with two steps. Firstly, each historical news article is processed by the gating module to select top-$K$ informative keywords. Secondly, these keywords from the entire user history are concatenated and encoded by transformers, generating the final user embedding. By setting $K$ to a reasonably small number, we can limit the user-side input to a significantly smaller size than its original one, thereby substantially reducing the encoding cost.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them set a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former group fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter group requires an extra training step and labels to train the cascaded selector, which consumes too much time and labor given a large-scale dataset. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's interests, and selects the more important ones.
In this way, we are able to largely preserve the keywords which truly matter to represent the user. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable compared with that incurred by transformers. 3) The gating module is designed to be \textit{end-to-end learnable.} Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; during model training, it can be optimized towards better performance in the news recommendation task. 

Note that the filtered user input may naturally support the inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanations for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces a gating mechanism for the acceleration of the recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormer's effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly faster speed.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories~\cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in~\cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in~\cite{wu2019neural}, the hierarchical attention networks (HAN)~\cite{yang2016hierarchical} is adapted for the multi-view representation of news articles~\cite{wu2019neural}; and in~\cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pre-trained language models~\cite{wu2021newsbert,xiao2021training}: PLMs like BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta} are employed as the backbones of news encoding. Such highly expressive models generate rich-semantic embeddings that substantially improve the recommendation quality.



Despite improving quality, the PLMs based news recommenders are limited in multiple perspectives. For example, they lacks explainability for the recommendation results~\cite{wang2019explainable,liu2020kred}; they are also incompatible with the existing inverted index recall systems~\cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is efficiency. While making recommendations for a user, all of her historically browsed news needs to be encoded by pre-trained language models. Given the expensive computation costs of PLMs, it is extremely difficult to make real-time recommendations for those active users who are associated with lots of historical news clicks. In recent years, many approaches are proposed to accelerate PLMs. One popular way is to distill large-scale PLMs into lightweight models~\cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Another line of research focuses on designing efficient transformers, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer~\cite{wang2020linformer} and Performer~\cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers~\cite{child2019generating} and Big Bird~\cite{zaheer2020big} utilize sparse self-attention; Synthesizer~\cite{tay2021synthesizer} introduces learnable and randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because online users are constantly generating large amounts of news clicks. The proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers are still formidable for timely recommendations. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions~\cite{wu2020mind}. Particularly, given a user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which the user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a lot of news clicks, each of which is an article consisting of multiple sentences; therefore, it will take considerable costs to encode the entire user side input

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected from each news article based on user's underlying interests. The filtered input (i.e., the selected keywords) is concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news article. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which further encodes the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder-like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on the user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). We leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, we take the last hidden state for $\mathbf{u}^g$ to represent the user interest, which is used for the next gating module. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that fully represent the user interest. Therefore, we use the correlation with the embedded user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g. the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token, masking the duplicated ones when generating the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article.   

It should be noted that no explicit supervision signal is available for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with the transformer module in the resulted computation graph; therefore, it can learn to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation~\cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|\mathcal{I}_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with a 10-fold compression of input, which means nealy $10\times$ lossless acceleration of the news recommender.  

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
\subsection{Settings}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News\footnote{https://www.msn.com/}. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title, abstract, and body of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in more general scenarios beyond news recommendation. The \textit{reference recommendation} task is evaluated on this dataset: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in reference list. This dataset contains academic literatures and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. 

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. 2) \textbf{FIM}~\cite{wang_FIM}, which achieves on-par performance as NRMS with the joint utilization of dilated convolution and deep attention matching networks. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 3) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering the input with the following methods. 10) \textbf{GFM (BM25)}, where the user side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we make trial of different network architectures for the gating module of GateFormer. 14) The news encoder in gate is switched from CNN to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched from LSTM to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. Besides, the news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. 

Our methods are implemented with \blue{PyTorch-1.9.1}. The experiments are performed on a cluster of \blue{2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. More details about the training and testing are specified in Appendix.} Our code will be open-sourced after the review stage. 

\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the compared methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interests. On the other hand, given that the user side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods that may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration effect, we find that cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined rules for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, 









\newpage

\begin{table}[t]
    \centering
    \small
    \caption{Effectiveness results on DBLP dataset.}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{P@10} & \textbf{MRR} & \textbf{NDCG}\\
    
    \midrule
    GSage & $69.42$ & $90.71$ & $94.90$ & $78.96$ & $83.82$\\
    \midrule
    Random & $40.23$ & $66.82$ & $76.49$ & $52.50$ & $62.36$ \\
    First & $50.73$ & $75.30$ & $83.47$ & $61.21$ & $69.51$\\
    BM25 & $43.89$ & $69.48$ & $78.78$ & $55.70$ & $64.93$\\
    Key & $46.46$ & $72.51$ & $81.06$ & $58.38$ & $67.17$ \\
    \midrule
    GFM-Twotower & $64.86$ & $89.03$ & $93.86$ & $75.41$ & $81.06$\\
    GFM-Onetower & $64.83$ & $88.75$ & $93.70$ & $75.38$ & $81.01$\\
    \bottomrule
  \end{tabular}
\end{table}


\begin{table}[t]
  \centering
  \small
  \caption{Recall results.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.1\linewidth}{D} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GFM & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S}
    & Key & $2.35$ & $10.42$ & $16.35$\\
    & GFM & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S+D}
    & Key & $5.57$ & $14.66$ & $21.81$\\
    & GFM & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table}


\section{Case Study}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/case.pdf}
    \caption{Case study of GateFormer. The original news text is shown in gray box, and the resulting keywords of each gating module are shown below.}
    \label{fig:case}
\end{figure}

First, GateFormers achieve a slightly better effectiveness against the SOTA brute-force method with more than 16x speed up and 2x less gpu consumption. This observation validates: 1) Our gating module can intelligently selects proper terms to represent a news. The over 30x compressed input to BERT boosts the efficiency as expected; 2) The underlying information in the pruned and concatenated user-side input can be fully captured by a 12-layer bert model.

Second, GateFormers consistently outperforms all efficient transformer baselines in terms of recommendation effectiveness, especially those designed for long text with linear complexity. A possible reason is that the input long document is essentially a concatenation of multiple historical news, which contradicts with the pre-training task and may harm the effectiveness of these transformers. Particularly, NewsBERT performs the best in this category because it is fine-tuned on MIND in advance. 

In terms of efficiency, GateFormers peaks among all efficient transformer baselines: compared with the most efficient one, ReFormer, GateFormers consume only half of the time to encode all users while providing a $2.39\%$ better quality; compared with the most effective one, NewsBert, GateFormer achieves a 6x speedup and reduces over half of the parameters while outperforming it by $0.4\%$. Note that our gating implementation is model-agnostic, which can be easily inserted before the efficient transformers to further improve their efficiency.

Third, on both dataset, our proposed end-to-end extraction approach beats all static baselines that identifies key terms by heuristics or pre-trained neural models. This observation validates that learning to extract according to the final ranking loss is a practical and robust strategy with effectiveness guarantee. 

We also notice that statically extracts the first $k$ tokens per news performs best among all baselines, which is another evidence for the feasibility of input simplification. One reason for its strong performance is that news editors tend to put the most important text in the very first of an article. In comparison, BM25 extractor incline to extract terms that merely occur in the corpus, which may lead to its worst performance among all extractors. 

Finally, more sophisticated extraction encoder hardly improves recommendation quality. In particular, a transformer-based extraction encoder TSF-N slightly underperforms simple CNN structure. This observation suggests that aggregating local context of a term is quite enough to guide extraction operation. On the other hand, AGG-U notably underperforms the default GateFormers. This is as expected because a degraded user encoder which applies PLM over each historical news' key terms instead of the concatenated sequence fails to model deep-semantic interactions across historical news pieces.

\subsection{Extraction Analysis}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/topk.pdf}
    \caption{Recommendation effectiveness (y-axes) versus efficiency (x-axes) at different number of $k$. The number next to the marker indicates the number of extracted terms.}
    \label{fig:k}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/kid.pdf}
    \caption{The position of extracted terms.}
    \label{fig:position distribution}
\end{figure}

\label{subsec:k}
The number of key terms extracted from each historical news piece, $k$, is the most important parameter in our approach, which is tightly associated with efficiency. To evaluate its influence, we compare the encoding time of all users in validation set and the corresponding AUC of \textit{end-to-end} extractor and \textit{first} extractor in Figure~\ref{fig:k}. 

Here we find the \textit{end-to-end} extractor consistently outperforms the \textit{first} extractor across the entire span of $k$. This is because the \textit{first} extractor cannot inspect terms beyond the top $k$ position, while the \textit{end-to-end} counterpart is quite able to leverage all of them and extract proper terms. Besides, we notice that extracting 1 or 2 terms per news reduces the effectiveness strongly for both extractors, however we observe different patterns starting from $k=3$. For the \textit{first} extractor, the effectiveness grows but the efficiency drops with the increase of $k$. This is expected since larger $k$ implies providing richer textual features for BERT to learn, while it also means longer sequence that leads to more computation cost. Due to the quadratic complexity of full self-attention layer, the encoding time grows beyond linear w.r.t. $k$, which can be easily verified in the figure. Unlike \textit{first} extractor, the effectiveness of \textit{end-to-end} saturates after $k>=3$, which results in the best overall performance i.e. higher AUC with faster speed at $k=3$. Such a saturation suggests 3 terms are enough to identify what the user's really interested in, and extracting more hardly contributes to profiling the user.

\subsubsection{More Analysis of Gating Module}
After validating the efficiency and effectiveness improvement, a straightforward question rises: \textit{Which terms exactly does the model learn to extract?} To address this research question, we first study the position distribution of the extracted term in the original news text and report the result in Figure~\ref{fig:position distribution}. Generally, the extraction rate is decreasing along the position, which means the model learns to assign a declining weight to the token buried deeper in the text. This finding again accords with the fact that news article displays important information in the front. Apart from that, we observe a strong focus on the first token of the news text. This super high pick rate of the first token is expected since news editors tends to put the most important entity at the beginning of the title. However, tokens situated at other positions are also likely to be picked, which justifies the improvement of \textit{end-to-end} extractor over its \textit{first} variant. 

\subsubsection{More Analysis of Sparse Candidate Recall}
\begin{table}[t]
  \centering
  \small
  \caption{Recall results.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.1\linewidth}{D} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GFM & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S}
    & Key & $2.35$ & $10.42$ & $16.35$\\
    & GFM & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S+D}
    & Key & $5.57$ & $14.66$ & $21.81$\\
    & GFM & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table}


One additional benefit of GateFormers is that the filtered user input may constitutes an enhanced BM25 system that naturally supports inverted index for candidate recall. We investigates its effectiveness against traditional dense recall methods. Specifically, we store the embeddings of every user and news that appears in at least one impression. Then we employ faiss~\cite{faiss} to retrieve $K$ nearest neighbors of a user vector from this news collection (denoted as \textbf{D}). Within the same corpus, we construct a sparse recall system by caching the extracted terms of each user and constructing an inverted-index accordingly. We rank the retrieved articles by BM25 scores (denoted as \textbf{S}) or the matching scores of the news and user embedding (denoted as \textbf{S+D}). Using Recall@K as the metric, we report the results in Table~\ref{table:recall}.

According to the table, all three variants of GateFormers beat the SOTA method in recall effectiveness. In particular, the sparse recall system built upon the extracted terms outperform SOTA by a huge margin, which is even bigger (about 13x better than SOTA) when substiting the BM25 ranking with the neural ranking mechanism. This verifies that the user's interests can be well captured by the representative keywords in her browsing history. 


\section{Case Study}
Figure~\ref{fig:case} illustrates the case study of two sample users.

\section{Conclusion}
Applting BERT as the news encoder yields state-of-the-art effectiveness in a variety of text recommendation task. However, it faces the efficiency bottleneck on account of intensive encoding cost. In this work, we propose an end-to-end scheme, GateFormers, for efficient PLM-based recommendation. It for the first time adopt an input simplification strategy that directly prunes the input to heavy BERT encoders. Combined with a one-tower user encoder, GateFormers improves state-of-the-art effectiveness at 10x faster speed on a large-scale news recommendation dataset. Additional experiments on a paper recommendation dataset also validates the superiority of our approach.



\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there has been a huge amount of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM, NewsBert}. Despite the effectiveness improvement, the PLMs based news recommenders are severely limited in efficiency. Specifically, when PLMs are employed as the encoding backbone, it becomes a heavy burden for the recommender to encode the historical news articles for millions of users to capture their underlying interests. This problem is even worse in face of some active users with super long browsing histories. Although accelerations can be made with recent approaches, like distilled lightweight PLMs~\cite{sanh2019distilbert} and efficient transformers~\cite{tay2020efficient}, the computation cost still grows drastically with the input size. 
In consequence, it remains a great challenge to make recommendation that makes the best of both quality and efficiency.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendation is more of capturing user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user.\blue{references?} Therefore, instead of relying on the entire content of the browsed news articles, user's underlying interest is likely to be well represented with a small amount of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be quickly selected before running a heavy PLM. In GateFormer, a user is represented with two steps. Firstly, each historical news article is processed by the gating module to select top-$K$ informative keywords. Secondly, these keywords from the entire user history are concatenated and encoded by transformers, generating the final user embedding. By setting $K$ to a reasonably small number, we can limit the user-side input to a significantly smaller size than its original one, thereby substantially reducing the encoding cost.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them sets a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former class fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter class requires extra labels to train the cascaded selector. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's interest, and selects the more important ones.
In this way, we are able to largely preserve the keywords which truly matter to represent the user. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable compared with that incurred by transformers. 3) The gating module is designed to be \textit{end-to-end learnable.} Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; during model training, it can be optimized towards better performance in the news recommendation task. 

Note that the filtered user input may naturally support inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanation for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces gating mechanism for the acceleration of recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormer's effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly faster speed.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories~\cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in~\cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in~\cite{wu2019neural}, the hierarchical attention networks (HAN)~\cite{yang2016hierarchical} is adapted for the multi-view representation of news articles~\cite{wu2019neural}; and in~\cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pretrained language models~\cite{wu2021newsbert,xiao2021training}, where PLMs like BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta} are employed as the backbones of news encoding. Such highly expressive models generate rich-semantic  embeddings that substantially improves the recommendation quality.



Despite improving quality, the PLMs based news recommenders are limited in multiple perspectives. For example, they lacks explainability for the recommendation results~\cite{wang2019explainable,liu2020kred}; they are also incompatible with the existing inverted index recall systems~\cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is the efficiency. While making recommendation for a user, all of her historically browsed news needs to be encoded by pretrained language models. Given the expensive computation costs of PLMs, it is extremely difficult to make realtime recommendation for those active users who are associate with lots of historical news clicks. In recent years, many approaches are proposed to acceleration PLMs. One popular way is to distill large-scale PLMs into lightweight models~\cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Another line of research focuses on designing efficient transformers, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer~\cite{wang2020linformer} and Performer~\cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers~\cite{child2019generating} and Big Bird~\cite{zaheer2020big} utilize sparse self-attention; Synthesizer~\cite{tay2021synthesizer} introduces learnable and randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because the online users are constantly generating large amounts of news clicks. The proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers are still formidable for timely recommendation. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions~\cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a lot of news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user side input with PLMs will be considerable. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interest. The filtered input (i.e., the selected keywords) are concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of a candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news articles. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which further encode the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). We leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, we take the last hidden state for $\mathbf{u}^g$ to represent the user interest, which is used for the next gating module. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that fully represent the user interest. Therefore, we use the correlation with the embedded user interest as the indicator as a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g. the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token, masking the duplicated ones when generating the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article.   

It should be noted that no explicit supervision signal is available for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with transformer module in the resulted computation graph; therefore, it can learn to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation~\cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|\mathcal{I}_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with a 10-fold compression of input, which means nealy $10\times$ lossless acceleration of the news recommender.  

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title, abstract, and body of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{\scriptsize  https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in applications beyond news recommendation. This dataset contains academic papers and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. The \textit{reference recommendation} task is evaluated here: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in the reference list.

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. 2) \textbf{FIM}~\cite{wang_FIM}, which achieves on-par performance as NRMS with the joint utilization of dilated convolution and deep attention matching networks. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 3) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering input with the following methods. 10) \textbf{GFM (BM25)}, where the user-side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we test different networks for the gating module of GateFormer. 14) The news encoder in gate is switched to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. The news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. Our implementation is based on PyTorch-1.9.1. The experiments are performed on a cluster of 2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. \textit{Supplementary experiments, configurations about training/testing are specified in Appendix. Our code will be open-sourced after the review stage. }

\begin{table*}[t]
  \centering
  \small
  \caption{\small News Recall Evaluations.}
  \vspace{-5pt}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the comparison methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interests. On the other hand, given that the user-side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result.

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration, we find that cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user-side input. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{Figures/case.pdf}
    \caption{\small Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \vspace{-8pt}
    \label{fig:case}
    \vspace{-10pt}
\end{figure}

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences in time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, analyze the dense recall scenario, where the candidate news are recalled based on the similarity between the user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we analyze the sparse recall scenario, where the candidate news is recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. Finally, we also analyze the effect of jointly making sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as a strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{table}[t]
    \centering
    \small
    \caption{\small Evaluation on DBLP.}
    \vspace{-5pt}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{MRR} & \textbf{NDCG}\\
    \midrule
   
   
    GFM (Random) & $40.23$ & $66.82$ & $52.50$ & $62.36$ \\
    GFM (First) & $50.73$ & $75.30$ & $61.21$ & $69.51$\\
    GFM (BM25) & $43.89$ & $69.48$ & $55.70$ & $64.93$\\
    GFM (KeyBERT) & $46.46$ & $72.51$ & $58.38$ & $67.17$ \\
    \midrule
    GateFormer & $\textbf{64.86}$ & $\textbf{89.03}$ & $\textbf{75.41}$ & $\textbf{81.06}$\\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table}

$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words do not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represents the news semantics and indicates the underlying reason why it catches the user's attention. 


$\bullet$ \textit{Evaluation on DBLP}. We compare GateFormer against other gating methods: \textit{GFM (Random)} (due to the limited entity coverage, the entity based method is switched to random selection), \textit{GFM (First)}, \textit{GFM (BM25)} and \textit{GFM (KeyBERT)}. The experiment results in Table \ref{tab:effectiveness_dblp} are consistent with our previous observations in Table \ref{tab:effectiveness_mind}, and GateFormer's advantages are even expanded. Such a finding suggests that GateFormer may stand as a generic input filtering method, which facilitates the speedup and keyword selection for text oriented recommendation/retrieval tasks. 

\section{Conclusion}
In this paper, we proposed GateFormer for the speedup of news recommendations. It introduced a gating mechanism, where informative keywords can be filtered from historical news clicks to represent user's interests. The gating module was designed to be personalized, lightweight, and end-to-end learnable, such that the input filtering can be accurately conducted with little running costs. Our experimental studies verified the effectiveness of GateFormer, where notable advantages in recommendation accuracy and efficiency were achieved against the existing acceleration methods. 

\newpage

\newpage

\section{Introduction}
Online news platforms have become people's major access to realtime information. Nowadays, there have been huge amounts of daily news articles published by different sources, making it imperative to recommend users' with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pretrained language models (PLMs), e.g., BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta}, are being intensively utilized for high-quality news recommendation (\blue{cite}). Despite the improvement of recommendation quality, the PLMs based news recommenders are severely limited in efficiency. This is because the recommenders need to encode users' historical news browsing behaviors so as to capture their underlying interests. However, many real-world users are associated with super long histories, which will incur considerable computation cost when PLMs are employed as the encoding backbones. Although accelerations can be made with recent approaches, like distilled lightweight PLMs \cite{sanh2019distilbert} and efficient transformers \cite{tay2020efficient}, the computation cost will still grow dramatically with the input size. As a result, it remains a great challenge to make timely recommendation for the active users.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendation is more of capturing user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user. Therefore, instead of relying on the entire content of the historically browsed news articles, user's underlying interest is likely to be well represented with a small amount of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be filtered for making news recommendation. In GateFormer, an user is represented with two steps. Firstly, each input news article is processed by the gating module, from which the top-$K$ keywords are filtered from each news article. Secondly, the filtered keywords from the entire user history are concatenated and encoded by transformers, based on which the user representation is generated. By setting $K$ to a reasonably small number, the encoding cost will be substantially reduced, as the user side input becomes significantly smaller than its original size.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them set a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former group fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter group requires an extra training step and labels to train the cascaded selector, which consumes too much time and labor given a large-scale dataset. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \blue{\textit{personalized}}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's underlying interest; if one word is strongly correlated with the user interest, it will be selected by the gating module with a high priority. In this way, we are able to largely preserve the keywords which truly matter to news recommendation. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable to the computation cost incurred by transformers. 3) Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; thus, it can be \textit{end-to-end learned} for the optimal recommendation performance. 

Note that the filtered user input may naturally support inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanation for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces gating mechanism for the acceleration of recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormers' effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly compressed input.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories \cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in \cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in \cite{wu2019neural}, the hierarchical attention networks (HAN) \cite{yang2016hierarchical} is adapted for the multi-view representation of news articles \cite{wu2019neural}; and in \cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pretrained language models \cite{wu2021newsbert,xiao2021training}, where PLMs like BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta} are employed as the backbones of news representation. With adoption of such highly expressive models, rich-semantic  embeddings can be generated, which substantially benefits the recommendation quality



Despite the improved quality, the PLMs based news recommenders are limited in multiple perspectives. For example, the recommendation results are lack of explainability \cite{wang2019explainable,liu2020kred}; besides, the deep models are also incompatible with the existing inverted index recall systems \cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is the efficiency. While making news recommendation for a user, the entire user history needs to be encoded by the pretrained language models. Given the expensive computation costs of PLMs, it will be extremely difficult to make realtime recommendation for those active users who are associate with lots of historical news clicks. In recent years, many approaches are proposed for the acceleration of PLMs. One popular way is to distill large-scale PLMs into lightweight models \cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Besides, the efficient transformers are proposed, which may reduce the time complexity of self-attention reduced from quadratic to linear (or log-linear). For example, Linformer \cite{wang2020linformer} and Performer \cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers \cite{child2019generating} and Big Bird \cite{zaheer2020big} utilize sparse self-attention; Reformer introduces learnable attention patterns, and Synthesizer \cite{tay2021synthesizer} introduces randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because the online users are constantly generating large amounts of news clicks. Therefore, even with the proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers, there will still be formidable costs for the recommenders. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the  computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions \cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a large amount of historical news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user side input with PLMs will be considerable. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interest. The filtered input (i.e., the selected keywords) are concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of a candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news articles. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which will be further encode the filtered input for the user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations (\blue{experiments to support}).

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). Firstly, we leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \blue{\mathbf{b}}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $i$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\blue{\mathbf{b}}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we further aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, $\mathbf{u}^g$ is the aggregation result used for the representation of user interest (by the gating module); we take the last output hidden-state for $\mathbf{u}^g$. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that may fully represent the user interest. Therefore, we use the correlation with the extracted user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g., the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \bar{\mathbf{h}}_j^i).
\end{equation}
Note that the token's embedding is modified from Eq. \ref{eq:1}: $\bar{\mathbf{h}}_j^i = \mathsf{avg}(\{\mathbf{h}_*^i\}_{N(j)})$, where ``$\mathsf{avg}(\cdot)$'' is the average pooling operator, and $N(j)$ denotes the neighbouring tokens which are within the same word of the $j$-th token (i.e., the importance is calculated on the word rather than the token basis). 

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable. Besides, each duplicated token is only counted for its first appearance in the news articles (other appearances are masked while generating the template matrix $\mathbf{y}^i$); thus, there will be no duplications for the selected tokens within each news article.   

It should be noted that there is no explicit supervision signal for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with transformer module in the resulted computation graph; therefore, it can be learned to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation \cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|I_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+\blue{\lambda_2}{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with more than 10-fold compression of input, which means over $10\times$ lossless acceleration of the news recommender.

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
\subsection{Settings}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News\footnote{https://www.msn.com/}. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The \blue{title, abstract, and body} of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in more general scenarios beyond news recommendation. The \textit{reference recommendation} task is evaluated on this dataset: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in reference list. This dataset contains academic literatures and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. 

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. \blue{2) \textbf{NAML}~\cite{wang_FIM}, which encodes all textual fields of news and attentively aggregates their embeddings. 3) \textbf{LSTUR}~\cite{wu_LSTUR}, which learns a user representation by user-specific LSTM. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders.} we include 4) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering the input with the following methods. 10) \textbf{GFM (BM25)}, where the user side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we make trial of different network architectures for the gating module of GateFormer. 14) The news encoder in gate is switched from CNN to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched from LSTM to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. Besides, the news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. 

Our methods are implemented with \blue{PyTorch-1.9.1}. The experiments are performed on a cluster of \blue{2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. More details about the training and testing are specified in Appendix.} Our code will be open-sourced after the review stage. 

\begin{table*}[t]
  \centering
  \small
  \caption{News Recall Evaluations.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the methods in comparison. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interest. On the other hand, given that the user side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news articles, but also get aware of the information within the entire user history. As a result, we may derive in-depth understanding about user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs layer-wise reduction of the hidden states. Despite the acceleration effect in theory, we find that cost reduction in reality is very limited; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we make analysis for the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user side input. 

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences on time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, we make analysis for the dense recall scenario, where the candidate news are recalled based on the similarity between user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we make analysis for the sparse recall scenario, where the candidate news are recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. Finally, we also analyze the effect of jointly making sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as an strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/case.pdf}
    \caption{Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \label{fig:case}
\end{figure}


$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words does not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to the poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represent the news semantics and indicates the underlying reason of why it catches the user's attention. 

\begin{table}[t]
    \centering
    \small
    \caption{Effectiveness results on DBLP dataset.}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{P@10} & \textbf{MRR} & \textbf{NDCG}\\
    
    \midrule
    GSage & $69.42$ & $90.71$ & $94.90$ & $78.96$ & $83.82$\\
    \midrule
    Random & $40.23$ & $66.82$ & $76.49$ & $52.50$ & $62.36$ \\
    First & $50.73$ & $75.30$ & $83.47$ & $61.21$ & $69.51$\\
    BM25 & $43.89$ & $69.48$ & $78.78$ & $55.70$ & $64.93$\\
    Key & $46.46$ & $72.51$ & $81.06$ & $58.38$ & $67.17$ \\
    \midrule
    GFM-Twotower & $64.86$ & $89.03$ & $93.86$ & $75.41$ & $81.06$\\
    GFM-Onetower & $64.83$ & $88.75$ & $93.70$ & $75.38$ & $81.01$\\
    \bottomrule
  \end{tabular}
\end{table}

$\bullet$ \textit{Evaluation on DBLP}. We 

\newpage

\newpage

\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there are a huge amount of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM, NewsBert}. Despite the effectiveness improvement, the PLMs based news recommenders are severely limited in efficiency. Specifically, when PLMs are employed as the encoding backbone, it becomes a heavy burden for the recommender to encode the historical news articles for millions of users to capture their underlying interests. This problem is even worse in face of some active users with super long browsing histories. Although accelerations can be made with recent approaches, like distilled lightweight PLMs~\cite{sanh2019distilbert} and efficient transformers~\cite{tay2020efficient}, the computation cost still grows drastically with the input size. 
In consequence, it remains a great challenge to make recommendation that makes the best of both quality and efficiency.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendations is more of capturing users' underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user.\blue{references?} Therefore, instead of relying on the entire content of the historical news articles, a user's underlying interest is likely to be well represented with a small number of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be quickly selected before running a heavy PLM. In GateFormer, a user is represented with two steps. Firstly, each historical news article is processed by the gating module to select top-$K$ informative keywords. Secondly, these keywords from the entire user history are concatenated and encoded by transformers, generating the final user embedding. By setting $K$ to a reasonably small number, we can limit the user-side input to a significantly smaller size than its original one, thereby substantially reducing the encoding cost.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them set a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former group fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter group requires an extra training step and labels to train the cascaded selector, which consumes too much time and labor given a large-scale dataset. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's interests, and selects the more important ones.
In this way, we are able to largely preserve the keywords which truly matter to represent the user. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable compared with that incurred by transformers. 3) The gating module is designed to be \textit{end-to-end learnable.} Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; during model training, it can be optimized towards better performance in the news recommendation task. 

Note that the filtered user input may naturally support the inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanations for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces a gating mechanism for the acceleration of the recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormer's effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly faster speed.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories~\cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in~\cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in~\cite{wu2019neural}, the hierarchical attention networks (HAN)~\cite{yang2016hierarchical} is adapted for the multi-view representation of news articles~\cite{wu2019neural}; and in~\cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pre-trained language models~\cite{wu2021newsbert,xiao2021training}: PLMs like BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta} are employed as the backbones of news encoding. Such highly expressive models generate rich-semantic embeddings that substantially improve the recommendation quality.



Despite improving quality, the PLMs based news recommenders are limited in multiple perspectives. For example, they lacks explainability for the recommendation results~\cite{wang2019explainable,liu2020kred}; they are also incompatible with the existing inverted index recall systems~\cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is efficiency. While making recommendations for a user, all of her historically browsed news needs to be encoded by pre-trained language models. Given the expensive computation costs of PLMs, it is extremely difficult to make real-time recommendations for those active users who are associated with lots of historical news clicks. In recent years, many approaches are proposed to accelerate PLMs. One popular way is to distill large-scale PLMs into lightweight models~\cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Another line of research focuses on designing efficient transformers, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer~\cite{wang2020linformer} and Performer~\cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers~\cite{child2019generating} and Big Bird~\cite{zaheer2020big} utilize sparse self-attention; Synthesizer~\cite{tay2021synthesizer} introduces learnable and randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because online users are constantly generating large amounts of news clicks. The proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers are still formidable for timely recommendations. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions~\cite{wu2020mind}. Particularly, given a user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which the user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a lot of news clicks, each of which is an article consisting of multiple sentences; therefore, it will take considerable costs to encode the entire user side input

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected from each news article based on user's underlying interests. The filtered input (i.e., the selected keywords) is concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news article. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which further encodes the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder-like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on the user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). We leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, we take the last hidden state for $\mathbf{u}^g$ to represent the user interest, which is used for the next gating module. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that fully represent the user interest. Therefore, we use the correlation with the embedded user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g. the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token, masking the duplicated ones when generating the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article.   

It should be noted that no explicit supervision signal is available for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with the transformer module in the resulted computation graph; therefore, it can learn to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation~\cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|\mathcal{I}_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with a 10-fold compression of input, which means nealy $10\times$ lossless acceleration of the news recommender.  

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
\subsection{Settings}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News\footnote{https://www.msn.com/}. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title, abstract, and body of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in more general scenarios beyond news recommendation. The \textit{reference recommendation} task is evaluated on this dataset: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in reference list. This dataset contains academic literatures and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. 

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. 2) \textbf{FIM}~\cite{wang_FIM}, which achieves on-par performance as NRMS with the joint utilization of dilated convolution and deep attention matching networks. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 3) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering the input with the following methods. 10) \textbf{GFM (BM25)}, where the user side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we make trial of different network architectures for the gating module of GateFormer. 14) The news encoder in gate is switched from CNN to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched from LSTM to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. Besides, the news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. 

Our methods are implemented with \blue{PyTorch-1.9.1}. The experiments are performed on a cluster of \blue{2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. More details about the training and testing are specified in Appendix.} Our code will be open-sourced after the review stage. 

\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the compared methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interests. On the other hand, given that the user side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods that may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration effect, we find that cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined rules for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, 









\newpage

\begin{table}[t]
    \centering
    \small
    \caption{Effectiveness results on DBLP dataset.}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{P@10} & \textbf{MRR} & \textbf{NDCG}\\
    
    \midrule
    GSage & $69.42$ & $90.71$ & $94.90$ & $78.96$ & $83.82$\\
    \midrule
    Random & $40.23$ & $66.82$ & $76.49$ & $52.50$ & $62.36$ \\
    First & $50.73$ & $75.30$ & $83.47$ & $61.21$ & $69.51$\\
    BM25 & $43.89$ & $69.48$ & $78.78$ & $55.70$ & $64.93$\\
    Key & $46.46$ & $72.51$ & $81.06$ & $58.38$ & $67.17$ \\
    \midrule
    GFM-Twotower & $64.86$ & $89.03$ & $93.86$ & $75.41$ & $81.06$\\
    GFM-Onetower & $64.83$ & $88.75$ & $93.70$ & $75.38$ & $81.01$\\
    \bottomrule
  \end{tabular}
\end{table}


\begin{table}[t]
  \centering
  \small
  \caption{Recall results.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.1\linewidth}{D} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GFM & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S}
    & Key & $2.35$ & $10.42$ & $16.35$\\
    & GFM & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S+D}
    & Key & $5.57$ & $14.66$ & $21.81$\\
    & GFM & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table}


\section{Case Study}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/case.pdf}
    \caption{Case study of GateFormer. The original news text is shown in gray box, and the resulting keywords of each gating module are shown below.}
    \label{fig:case}
\end{figure}

First, GateFormers achieve a slightly better effectiveness against the SOTA brute-force method with more than 16x speed up and 2x less gpu consumption. This observation validates: 1) Our gating module can intelligently selects proper terms to represent a news. The over 30x compressed input to BERT boosts the efficiency as expected; 2) The underlying information in the pruned and concatenated user-side input can be fully captured by a 12-layer bert model.

Second, GateFormers consistently outperforms all efficient transformer baselines in terms of recommendation effectiveness, especially those designed for long text with linear complexity. A possible reason is that the input long document is essentially a concatenation of multiple historical news, which contradicts with the pre-training task and may harm the effectiveness of these transformers. Particularly, NewsBERT performs the best in this category because it is fine-tuned on MIND in advance. 

In terms of efficiency, GateFormers peaks among all efficient transformer baselines: compared with the most efficient one, ReFormer, GateFormers consume only half of the time to encode all users while providing a $2.39\%$ better quality; compared with the most effective one, NewsBert, GateFormer achieves a 6x speedup and reduces over half of the parameters while outperforming it by $0.4\%$. Note that our gating implementation is model-agnostic, which can be easily inserted before the efficient transformers to further improve their efficiency.

Third, on both dataset, our proposed end-to-end extraction approach beats all static baselines that identifies key terms by heuristics or pre-trained neural models. This observation validates that learning to extract according to the final ranking loss is a practical and robust strategy with effectiveness guarantee. 

We also notice that statically extracts the first $k$ tokens per news performs best among all baselines, which is another evidence for the feasibility of input simplification. One reason for its strong performance is that news editors tend to put the most important text in the very first of an article. In comparison, BM25 extractor incline to extract terms that merely occur in the corpus, which may lead to its worst performance among all extractors. 

Finally, more sophisticated extraction encoder hardly improves recommendation quality. In particular, a transformer-based extraction encoder TSF-N slightly underperforms simple CNN structure. This observation suggests that aggregating local context of a term is quite enough to guide extraction operation. On the other hand, AGG-U notably underperforms the default GateFormers. This is as expected because a degraded user encoder which applies PLM over each historical news' key terms instead of the concatenated sequence fails to model deep-semantic interactions across historical news pieces.

\subsection{Extraction Analysis}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/topk.pdf}
    \caption{Recommendation effectiveness (y-axes) versus efficiency (x-axes) at different number of $k$. The number next to the marker indicates the number of extracted terms.}
    \label{fig:k}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/kid.pdf}
    \caption{The position of extracted terms.}
    \label{fig:position distribution}
\end{figure}

\label{subsec:k}
The number of key terms extracted from each historical news piece, $k$, is the most important parameter in our approach, which is tightly associated with efficiency. To evaluate its influence, we compare the encoding time of all users in validation set and the corresponding AUC of \textit{end-to-end} extractor and \textit{first} extractor in Figure~\ref{fig:k}. 

Here we find the \textit{end-to-end} extractor consistently outperforms the \textit{first} extractor across the entire span of $k$. This is because the \textit{first} extractor cannot inspect terms beyond the top $k$ position, while the \textit{end-to-end} counterpart is quite able to leverage all of them and extract proper terms. Besides, we notice that extracting 1 or 2 terms per news reduces the effectiveness strongly for both extractors, however we observe different patterns starting from $k=3$. For the \textit{first} extractor, the effectiveness grows but the efficiency drops with the increase of $k$. This is expected since larger $k$ implies providing richer textual features for BERT to learn, while it also means longer sequence that leads to more computation cost. Due to the quadratic complexity of full self-attention layer, the encoding time grows beyond linear w.r.t. $k$, which can be easily verified in the figure. Unlike \textit{first} extractor, the effectiveness of \textit{end-to-end} saturates after $k>=3$, which results in the best overall performance i.e. higher AUC with faster speed at $k=3$. Such a saturation suggests 3 terms are enough to identify what the user's really interested in, and extracting more hardly contributes to profiling the user.

\subsubsection{More Analysis of Gating Module}
After validating the efficiency and effectiveness improvement, a straightforward question rises: \textit{Which terms exactly does the model learn to extract?} To address this research question, we first study the position distribution of the extracted term in the original news text and report the result in Figure~\ref{fig:position distribution}. Generally, the extraction rate is decreasing along the position, which means the model learns to assign a declining weight to the token buried deeper in the text. This finding again accords with the fact that news article displays important information in the front. Apart from that, we observe a strong focus on the first token of the news text. This super high pick rate of the first token is expected since news editors tends to put the most important entity at the beginning of the title. However, tokens situated at other positions are also likely to be picked, which justifies the improvement of \textit{end-to-end} extractor over its \textit{first} variant. 

\subsubsection{More Analysis of Sparse Candidate Recall}
\begin{table}[t]
  \centering
  \small
  \caption{Recall results.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.1\linewidth}{D} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GFM & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S}
    & Key & $2.35$ & $10.42$ & $16.35$\\
    & GFM & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.1\linewidth}{S+D}
    & Key & $5.57$ & $14.66$ & $21.81$\\
    & GFM & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table}


One additional benefit of GateFormers is that the filtered user input may constitutes an enhanced BM25 system that naturally supports inverted index for candidate recall. We investigates its effectiveness against traditional dense recall methods. Specifically, we store the embeddings of every user and news that appears in at least one impression. Then we employ faiss~\cite{faiss} to retrieve $K$ nearest neighbors of a user vector from this news collection (denoted as \textbf{D}). Within the same corpus, we construct a sparse recall system by caching the extracted terms of each user and constructing an inverted-index accordingly. We rank the retrieved articles by BM25 scores (denoted as \textbf{S}) or the matching scores of the news and user embedding (denoted as \textbf{S+D}). Using Recall@K as the metric, we report the results in Table~\ref{table:recall}.

According to the table, all three variants of GateFormers beat the SOTA method in recall effectiveness. In particular, the sparse recall system built upon the extracted terms outperform SOTA by a huge margin, which is even bigger (about 13x better than SOTA) when substiting the BM25 ranking with the neural ranking mechanism. This verifies that the user's interests can be well captured by the representative keywords in her browsing history. 


\section{Case Study}
Figure~\ref{fig:case} illustrates the case study of two sample users.

\section{Conclusion}
Applting BERT as the news encoder yields state-of-the-art effectiveness in a variety of text recommendation task. However, it faces the efficiency bottleneck on account of intensive encoding cost. In this work, we propose an end-to-end scheme, GateFormers, for efficient PLM-based recommendation. It for the first time adopt an input simplification strategy that directly prunes the input to heavy BERT encoders. Combined with a one-tower user encoder, GateFormers improves state-of-the-art effectiveness at 10x faster speed on a large-scale news recommendation dataset. Additional experiments on a paper recommendation dataset also validates the superiority of our approach.



\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there has been a huge amount of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM, NewsBert}. Despite the effectiveness improvement, the PLMs based news recommenders are severely limited in efficiency. Specifically, when PLMs are employed as the encoding backbone, it becomes a heavy burden for the recommender to encode the historical news articles for millions of users to capture their underlying interests. This problem is even worse in face of some active users with super long browsing histories. Although accelerations can be made with recent approaches, like distilled lightweight PLMs~\cite{sanh2019distilbert} and efficient transformers~\cite{tay2020efficient}, the computation cost still grows drastically with the input size. 
In consequence, it remains a great challenge to make recommendation that makes the best of both quality and efficiency.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendation is more of capturing user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user.\blue{references?} Therefore, instead of relying on the entire content of the browsed news articles, user's underlying interest is likely to be well represented with a small amount of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be quickly selected before running a heavy PLM. In GateFormer, a user is represented with two steps. Firstly, each historical news article is processed by the gating module to select top-$K$ informative keywords. Secondly, these keywords from the entire user history are concatenated and encoded by transformers, generating the final user embedding. By setting $K$ to a reasonably small number, we can limit the user-side input to a significantly smaller size than its original one, thereby substantially reducing the encoding cost.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them sets a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former class fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter class requires extra labels to train the cascaded selector. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's interest, and selects the more important ones.
In this way, we are able to largely preserve the keywords which truly matter to represent the user. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable compared with that incurred by transformers. 3) The gating module is designed to be \textit{end-to-end learnable.} Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; during model training, it can be optimized towards better performance in the news recommendation task. 

Note that the filtered user input may naturally support inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanation for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces gating mechanism for the acceleration of recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormer's effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly faster speed.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories~\cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in~\cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in~\cite{wu2019neural}, the hierarchical attention networks (HAN)~\cite{yang2016hierarchical} is adapted for the multi-view representation of news articles~\cite{wu2019neural}; and in~\cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pretrained language models~\cite{wu2021newsbert,xiao2021training}, where PLMs like BERT~\cite{Devlin2019BERT} and RoBERTa~\cite{Liu2019Roberta} are employed as the backbones of news encoding. Such highly expressive models generate rich-semantic  embeddings that substantially improves the recommendation quality.



Despite improving quality, the PLMs based news recommenders are limited in multiple perspectives. For example, they lacks explainability for the recommendation results~\cite{wang2019explainable,liu2020kred}; they are also incompatible with the existing inverted index recall systems~\cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is the efficiency. While making recommendation for a user, all of her historically browsed news needs to be encoded by pretrained language models. Given the expensive computation costs of PLMs, it is extremely difficult to make realtime recommendation for those active users who are associate with lots of historical news clicks. In recent years, many approaches are proposed to acceleration PLMs. One popular way is to distill large-scale PLMs into lightweight models~\cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Another line of research focuses on designing efficient transformers, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer~\cite{wang2020linformer} and Performer~\cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers~\cite{child2019generating} and Big Bird~\cite{zaheer2020big} utilize sparse self-attention; Synthesizer~\cite{tay2021synthesizer} introduces learnable and randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because the online users are constantly generating large amounts of news clicks. The proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers are still formidable for timely recommendation. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions~\cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a lot of news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user side input with PLMs will be considerable. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interest. The filtered input (i.e., the selected keywords) are concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of a candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news articles. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which further encode the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). We leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, we take the last hidden state for $\mathbf{u}^g$ to represent the user interest, which is used for the next gating module. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that fully represent the user interest. Therefore, we use the correlation with the embedded user interest as the indicator as a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g. the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token, masking the duplicated ones when generating the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article.   

It should be noted that no explicit supervision signal is available for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with transformer module in the resulted computation graph; therefore, it can learn to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation~\cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|\mathcal{I}_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with a 10-fold compression of input, which means nealy $10\times$ lossless acceleration of the news recommender.  

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title, abstract, and body of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{\scriptsize  https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in applications beyond news recommendation. This dataset contains academic papers and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. The \textit{reference recommendation} task is evaluated here: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in the reference list.

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. 2) \textbf{FIM}~\cite{wang_FIM}, which achieves on-par performance as NRMS with the joint utilization of dilated convolution and deep attention matching networks. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 3) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering input with the following methods. 10) \textbf{GFM (BM25)}, where the user-side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we test different networks for the gating module of GateFormer. 14) The news encoder in gate is switched to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. The news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. Our implementation is based on PyTorch-1.9.1. The experiments are performed on a cluster of 2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. \textit{Supplementary experiments, configurations about training/testing are specified in Appendix. Our code will be open-sourced after the review stage. }

\begin{table*}[t]
  \centering
  \small
  \caption{\small News Recall Evaluations.}
  \vspace{-5pt}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the comparison methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interests. On the other hand, given that the user-side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result.

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration, we find that cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user-side input. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{Figures/case.pdf}
    \caption{\small Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \vspace{-8pt}
    \label{fig:case}
    \vspace{-10pt}
\end{figure}

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences in time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, analyze the dense recall scenario, where the candidate news are recalled based on the similarity between the user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we analyze the sparse recall scenario, where the candidate news is recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. Finally, we also analyze the effect of jointly making sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as a strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{table}[t]
    \centering
    \small
    \caption{\small Evaluation on DBLP.}
    \vspace{-5pt}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{MRR} & \textbf{NDCG}\\
    \midrule
   
   
    GFM (Random) & $40.23$ & $66.82$ & $52.50$ & $62.36$ \\
    GFM (First) & $50.73$ & $75.30$ & $61.21$ & $69.51$\\
    GFM (BM25) & $43.89$ & $69.48$ & $55.70$ & $64.93$\\
    GFM (KeyBERT) & $46.46$ & $72.51$ & $58.38$ & $67.17$ \\
    \midrule
    GateFormer & $\textbf{64.86}$ & $\textbf{89.03}$ & $\textbf{75.41}$ & $\textbf{81.06}$\\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table}

$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words do not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represents the news semantics and indicates the underlying reason why it catches the user's attention. 


$\bullet$ \textit{Evaluation on DBLP}. We compare GateFormer against other gating methods: \textit{GFM (Random)} (due to the limited entity coverage, the entity based method is switched to random selection), \textit{GFM (First)}, \textit{GFM (BM25)} and \textit{GFM (KeyBERT)}. The experiment results in Table \ref{tab:effectiveness_dblp} are consistent with our previous observations in Table \ref{tab:effectiveness_mind}, and GateFormer's advantages are even expanded. Such a finding suggests that GateFormer may stand as a generic input filtering method, which facilitates the speedup and keyword selection for text oriented recommendation/retrieval tasks. 

\section{Conclusion}
In this paper, we proposed GateFormer for the speedup of news recommendations. It introduced a gating mechanism, where informative keywords can be filtered from historical news clicks to represent user's interests. The gating module was designed to be personalized, lightweight, and end-to-end learnable, such that the input filtering can be accurately conducted with little running costs. Our experimental studies verified the effectiveness of GateFormer, where notable advantages in recommendation accuracy and efficiency were achieved against the existing acceleration methods. 

\newpage

\newpage

\section{Introduction}
Online news platforms have become people's major access to realtime information. Nowadays, there have been huge amounts of daily news articles published by different sources, making it imperative to recommend users' with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pretrained language models (PLMs), e.g., BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta}, are being intensively utilized for high-quality news recommendation (\blue{cite}). Despite the improvement of recommendation quality, the PLMs based news recommenders are severely limited in efficiency. This is because the recommenders need to encode users' historical news browsing behaviors so as to capture their underlying interests. However, many real-world users are associated with super long histories, which will incur considerable computation cost when PLMs are employed as the encoding backbones. Although accelerations can be made with recent approaches, like distilled lightweight PLMs \cite{sanh2019distilbert} and efficient transformers \cite{tay2020efficient}, the computation cost will still grow dramatically with the input size. As a result, it remains a great challenge to make timely recommendation for the active users.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendation is more of capturing user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user. Therefore, instead of relying on the entire content of the historically browsed news articles, user's underlying interest is likely to be well represented with a small amount of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user side input can be filtered for making news recommendation. In GateFormer, an user is represented with two steps. Firstly, each input news article is processed by the gating module, from which the top-$K$ keywords are filtered from each news article. Secondly, the filtered keywords from the entire user history are concatenated and encoded by transformers, based on which the user representation is generated. By setting $K$ to a reasonably small number, the encoding cost will be substantially reduced, as the user side input becomes significantly smaller than its original size.

\blue{In fact, input gating has been adopted by several previous works, mainly in two different ways. Some of them set a global threshold and mask the input whose importance is below it~\cite{learn_token_prune,zhou_2018_topk_retrieval}. Others cascade an extra selector pre-trained in a teacher-student regime to find out top-$K$ important input~\cite{Intra-Document-Cascading}. However, the former group fails to accelerate the model because all input still involves in the following computation regardless of whether they are masked; the latter group requires an extra training step and labels to train the cascaded selector, which consumes too much time and labor given a large-scale dataset. Unlike both of them, we address the challenge to speed up the model without resorting to extra training stages and labels.}

The gating module is highlighted by the following features. 1) The gating module is designed to be \blue{\textit{personalized}}. While filtering the user side input, the gating module estimates each word's importance w.r.t. user's underlying interest; if one word is strongly correlated with the user interest, it will be selected by the gating module with a high priority. In this way, we are able to largely preserve the keywords which truly matter to news recommendation. 2) The gating module is designed to be \textit{lightweight}. Knowing that the entire input needs to be gated before getting processed by transformers, the gating module is realized with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable to the computation cost incurred by transformers. 3) Considering that there is no supervision data for input filtering, the gating module is made as an differentiable component cascaded with transformer module; thus, it can be \textit{end-to-end learned} for the optimal recommendation performance. 

Note that the filtered user input may naturally support inverted index (it can be regarded as an enhanced BM25 system) and provide keyword-based explanation for the recommendation result, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work which introduces gating mechanism for the acceleration of recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormers' effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly compressed input.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories \cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in \cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in \cite{wu2019neural}, the hierarchical attention networks (HAN) \cite{yang2016hierarchical} is adapted for the multi-view representation of news articles \cite{wu2019neural}; and in \cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pretrained language models \cite{wu2021newsbert,xiao2021training}, where PLMs like BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta} are employed as the backbones of news representation. With adoption of such highly expressive models, rich-semantic  embeddings can be generated, which substantially benefits the recommendation quality



Despite the improved quality, the PLMs based news recommenders are limited in multiple perspectives. For example, the recommendation results are lack of explainability \cite{wang2019explainable,liu2020kred}; besides, the deep models are also incompatible with the existing inverted index recall systems \cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is the efficiency. While making news recommendation for a user, the entire user history needs to be encoded by the pretrained language models. Given the expensive computation costs of PLMs, it will be extremely difficult to make realtime recommendation for those active users who are associate with lots of historical news clicks. In recent years, many approaches are proposed for the acceleration of PLMs. One popular way is to distill large-scale PLMs into lightweight models \cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Besides, the efficient transformers are proposed, which may reduce the time complexity of self-attention reduced from quadratic to linear (or log-linear). For example, Linformer \cite{wang2020linformer} and Performer \cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers \cite{child2019generating} and Big Bird \cite{zaheer2020big} utilize sparse self-attention; Reformer introduces learnable attention patterns, and Synthesizer \cite{tay2021synthesizer} introduces randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not scalable enough to handle the challenges in news recommendation. This is because the online users are constantly generating large amounts of news clicks. Therefore, even with the proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers, there will still be formidable costs for the recommenders. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the  computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup.  

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions \cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_n\}$, the recommender is learned to represent the user's underlying interest, with which user's future news clicks can be predicted. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user side input can be super large: one user may generate a large amount of historical news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user side input with PLMs will be considerable. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interest. The filtered input (i.e., the selected keywords) are concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of a candidate news.}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown as Figure \ref{fig:1}. There are two basic components in GateFormer. Firstly, the Gating Module, which processes the entire user side input and selects the top-$K$ keywords for each historical news articles. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Secondly, the Transformer Module, which will be further encode the filtered input for the user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword filtering based on user's individual interest. For efficiency concerns, the user interest is encoded from the following \textbf{lightweight} computations (\blue{experiments to support}).

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). Firstly, we leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \blue{\mathbf{b}}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $i$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\blue{\mathbf{b}}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we further aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, $\mathbf{u}^g$ is the aggregation result used for the representation of user interest (by the gating module); we take the last output hidden-state for $\mathbf{u}^g$. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that may fully represent the user interest. Therefore, we use the correlation with the extracted user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input tokens (e.g., the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \bar{\mathbf{h}}_j^i).
\end{equation}
Note that the token's embedding is modified from Eq. \ref{eq:1}: $\bar{\mathbf{h}}_j^i = \mathsf{avg}(\{\mathbf{h}_*^i\}_{N(j)})$, where ``$\mathsf{avg}(\cdot)$'' is the average pooling operator, and $N(j)$ denotes the neighbouring tokens which are within the same word of the $j$-th token (i.e., the importance is calculated on the word rather than the token basis). 

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) which produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable. Besides, each duplicated token is only counted for its first appearance in the news articles (other appearances are masked while generating the template matrix $\mathbf{y}^i$); thus, there will be no duplications for the selected tokens within each news article.   

It should be noted that there is no explicit supervision signal for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with transformer module in the resulted computation graph; therefore, it can be learned to select the tokens which optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation \cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|I_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+\blue{\lambda_2}{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with more than 10-fold compression of input, which means over $10\times$ lossless acceleration of the news recommender.

\begin{table*}[t]
    \centering
    \small
    \caption{Evaluations of accuracy and efficiency on MIND dataset. The highest performances are bolded, the strongest baseline performances (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{3}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule

    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}    
\end{table*}

\section{Experiments}
\subsection{Settings}
$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News\footnote{https://www.msn.com/}. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The \blue{title, abstract, and body} of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in more general scenarios beyond news recommendation. The \textit{reference recommendation} task is evaluated on this dataset: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in reference list. This dataset contains academic literatures and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. 

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard  Mathods}. 1) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. \blue{2) \textbf{NAML}~\cite{wang_FIM}, which encodes all textual fields of news and attentively aggregates their embeddings. 3) \textbf{LSTUR}~\cite{wu_LSTUR}, which learns a user representation by user-specific LSTM. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders.} we include 4) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 4) \textbf{Big Bird}~\cite{zaheer2020big}, 5) \textbf{LongFormer}~\cite{beltagy2020longformer}, 6) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 7) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 8) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 9) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering the input with the following methods. 10) \textbf{GFM (BM25)}, where the user side input is filtered by BM25 score; 11) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 12) \textbf{GFM (First-$K$)}, where the first $K$ tokens are truncated; and 13) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pretrained model for keyword extraction) is used to generate the filtered input.

Finally, we make trial of different network architectures for the gating module of GateFormer. 14) The news encoder in gate is switched from CNN to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched from LSTM to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. Besides, the news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. 

Our methods are implemented with \blue{PyTorch-1.9.1}. The experiments are performed on a cluster of \blue{2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. More details about the training and testing are specified in Appendix.} Our code will be open-sourced after the review stage. 

\begin{table*}[t]
  \centering
  \small
  \caption{News Recall Evaluations.}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{2}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the methods in comparison. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.6$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user side input is compressed by 10-fold with GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully reflect users' underlying interest. On the other hand, given that the user side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news articles, but also get aware of the information within the entire user history. As a result, we may derive in-depth understanding about user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result. 

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods which may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs layer-wise reduction of the hidden states. Despite the acceleration effect in theory, we find that cost reduction in reality is very limited; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we make analysis for the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for {GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user side input. 

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences on time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, we make analysis for the dense recall scenario, where the candidate news are recalled based on the similarity between user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we make analysis for the sparse recall scenario, where the candidate news are recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. Finally, we also analyze the effect of jointly making sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as an strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{Figures/case.pdf}
    \caption{Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \label{fig:case}
\end{figure}


$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words does not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to the poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represent the news semantics and indicates the underlying reason of why it catches the user's attention. 

\begin{table}[t]
    \centering
    \small
    \caption{Effectiveness results on DBLP dataset.}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{P@10} & \textbf{MRR} & \textbf{NDCG}\\
    
    \midrule
    GSage & $69.42$ & $90.71$ & $94.90$ & $78.96$ & $83.82$\\
    \midrule
    Random & $40.23$ & $66.82$ & $76.49$ & $52.50$ & $62.36$ \\
    First & $50.73$ & $75.30$ & $83.47$ & $61.21$ & $69.51$\\
    BM25 & $43.89$ & $69.48$ & $78.78$ & $55.70$ & $64.93$\\
    Key & $46.46$ & $72.51$ & $81.06$ & $58.38$ & $67.17$ \\
    \midrule
    GFM-Twotower & $64.86$ & $89.03$ & $93.86$ & $75.41$ & $81.06$\\
    GFM-Onetower & $64.83$ & $88.75$ & $93.70$ & $75.38$ & $81.01$\\
    \bottomrule
  \end{tabular}
\end{table}

$\bullet$ \textit{Evaluation on DBLP}. We 

\newpage

\newpage

\section{Introduction}
Online news platforms have become people's major access to real-time information. Nowadays, there have been huge amounts of daily news articles published by different sources, making it imperative to recommend users with personalized content automatically. Thanks to the development of deep learning techniques, deep neural networks, especially the pre-trained language models (PLMs), e.g., BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta}, are intensively utilized for high-quality news recommendation~\cite{wu_newsPLM,NewsBert}. Despite the improvement of recommendation quality, the PLMs based news recommenders are severely limited in efficiency. This is because the recommenders need to encode users' historical news browsing behaviors to capture their underlying interests. However, many real-world users are associated with super long histories, which will incur considerable computation costs when PLMs are employed as the encoding backbones. Although accelerations can be made with recent approaches, like distilled lightweight PLMs \cite{sanh2019distilbert} and efficient transformers \cite{tay2020efficient}, the computation cost will still grow dramatically with the input size. As a result, it remains a great challenge to make timely recommendations for active users.


In this work, we tackle the efficient news recommendation problem from a distinctive perspective. We argue that making news recommendations is more of capturing a user's underlying interests towards ``high-level semantics'', such as topics and entities, rather than memorizing what was exactly browsed by the user. Therefore, instead of relying on the entire content of the historically browsed news articles, the user's underlying interest is likely to be well represented with a small number of keywords. Driven by this motivation, we propose a novel efficient framework termed as \textbf{GateFormer}, where the high-informative fraction of user-side input can be filtered for making news recommendations. In GateFormer, a user is represented with two steps. Firstly, each input news article is processed by the gating module, which selects the top-$K$ informative keywords from the news text. Secondly, the filtered keywords from the entire user history are concatenated and encoded by transformers, generating the user representation. By setting $K$ to a reasonably small number, the encoding cost will be substantially reduced, as the user-side input becomes significantly smaller than its original size.

The gating module is highlighted by the following features. 1) The gating module is designed to be \textit{personalized}. While filtering the user-side input, the gating module estimates each word's importance w.r.t. user's underlying interest; if one word strongly correlates with the user interest, it will be selected by the gating module with a high priority. In this way, we are able to maximally preserve the keywords that truly matter to news recommendation. 2) The gating module is designed to be \textit{lightweight}. Knowing that the gating module needs to inspect the entire input, we implement it with tiny network components, like single-layer CNNs or GRU. By doing so, the extra time cost from the gating operation becomes almost ignorable to the computation cost incurred by transformers. 3) Considering that there is no supervision data for input filtering, the gating module is designed to be a differentiable component cascaded with transformer module; thus, it can be \textit{end-to-end learned} to select proper keywords for the optimal recommendation performance. 

Note that the filtered user input may naturally support inverted index (it can be used as an enhanced BM25 system) and provide keyword-based explanations for recommendation results, which are the additional benefits of using GateFormer.

To summarize, the major contributions of this work are listed as the following points.
\begin{itemize}
    \item We propose a novel efficient news recommendation framework called GateFormer. To the best of our knowledge, this is the first work that introduces a gating mechanism to accelerate the recommendation process. 
    \item Our gating module is designed to be personalized, lightweight, and end-to-end learnable, which enables representative keywords to be accurately filtered with little cost.
    \item Our experimental studies verify GraphFormers' effectiveness: it notably outperforms the existing acceleration approaches in both accuracy and efficiency; and it achieves on-par performances as the SOTA brute-force methods with a significantly compressed input.
\end{itemize}

\section{Related Work}
News recommenders aim to identify users' interested news articles based on their news reading histories \cite{wu2020mind,li2010contextual,das2007google}. In recent years, deep neural networks have been intensively applied to better represent the underlying semantics about news content and user behaviors. For example, in \cite{wang2018dkn}, entity embeddings are jointly learned together with user models, so as to introduce knowledge graph information into news recommendation; in \cite{wu2019neural}, the hierarchical attention networks (HAN) \cite{yang2016hierarchical} is adapted for the multi-view representation of news articles \cite{wu2019neural}; and in \cite{an2019neural}, long short-term memory networks are leveraged to capture users' time-sensitive interests. The latest news recommenders are built upon pre-trained language models \cite{wu2021newsbert,xiao2021training}, where PLMs like BERT \cite{Devlin2019BERT} and RoBERTa \cite{Liu2019Roberta} are employed as the backbones of news encoder. With the adoption of such highly expressive models, rich-semantic  embeddings can be generated, which substantially benefits the recommendation quality.



Despite the improved quality, the PLMs based news recommenders are limited in multiple perspectives. For example, the recommendation results are lack of explainability \cite{wang2019explainable,liu2020kred}; besides, the deep models are also incompatible with the existing inverted index recall systems \cite{schutze2008introduction,dai2020context}. However, one of the most critical challenges about PLMs based news recommenders is efficiency. While making news recommendations for a user, her entire browsing history needs to be encoded by pre-trained language models. Given the expensive computation costs of PLMs, it will be extremely difficult to make real-time recommendations for those active users who are associated with lots of historical news clicks. In recent years, many approaches are proposed for the acceleration of PLMs. One popular way is to distill large-scale PLMs into lightweight models \cite{sanh2019distilbert,sun2019patient}, where the computation cost can be saved proportionally with the reduction of model size. Besides, efficient transformers are proposed, which may reduce the time complexity of self-attention from quadratic to linear (or log-linear). For example, Linformer \cite{wang2020linformer} and Performer \cite{choromanski2020rethinking} leverage low-rank self-attention; Sparse Transformers \cite{child2019generating} and Big Bird \cite{zaheer2020big} utilize sparse self-attention; Reformer introduces learnable attention patterns, and Synthesizer \cite{tay2021synthesizer} introduces randomized attention patterns. 

However, we argue that the existing acceleration approaches are still not fast enough to handle the challenges in news recommendation. This is because online users are constantly generating large amounts of news clicks. Therefore, even with the proportionally reduced cost from distillation (e.g., by half or one quarter) or the linearly growing cost from efficient transformers, there will still be formidable costs for the recommenders. In this work, we tackle the efficient news recommendation from the perspective of input filtering, which enjoys the merits of reducing the computation cost by multi-fold while still keeping almost the same recommendation quality. Besides, it should also be noted that our work is complementary to the existing acceleration methods, which can be jointly used for further speedup. 

\section{Methodologies}


The news recommendation problem is formulated based on the typical definitions \cite{wu2020mind}. Particularly, given user's historical news clicks $\mathcal{S}=\{S_1,...,S_N\}$, the recommender learns to represent the user's underlying interest, and predicts the user's future news clicks based on it. In fact, the news recommendation problem can be regarded as a special case of document matching: the historical news articles are treated as the query, and the future news articles become the documents to be matched. One unique challenge about news recommendation is that the user-side input can be super large: one user may generate a huge amount of historical news clicks, each of which is an article consisting of multiple sentences (headlines, abstract, body content, etc.) As a result, the cost of encoding the entire user-side input with PLMs will be prohibitive. 

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{Figures/gateformer2.pdf}
\caption{\small Overview of GateFormer. The historical news clicks (i.e., the user-side input) are filtered by the gating module, from which the representative keywords are selected for each news article based on user's underlying interests. The filtered input (i.e., the selected keywords) is concatenated and encoded by the transformer module, where the user embedding is produced for the click prediction of candidate news.}
\vspace{-10pt}
\label{fig:1}
\end{figure}

\subsection{Overview of GateFormer}
The framework of GateFormer is shown in Figure \ref{fig:1}. There are two basic components in GateFormer. First, the Gating Module, which processes the entire user-side input and selects the top-$K$ keywords for each historical news article. The filtered input will be much smaller (usually over 10$\times$ smaller than the original size). Second, the Transformer Module, which encodes the filtered input into a user embedding. Finally, the click prediction is made based on the embedding similarity between user and news. (Although a siamese-encoder-like framework is presented here, people may easily change it into a cross-encoder with very little adaptation.) 

\subsection{Gating Module}


$\bullet$ \textbf{User interest encoding}. The gating module performs keyword selection based on the user's interest. For efficiency concerns, the user interest is encoded by the following \textbf{lightweight} models.

Assume that each news article is tokenized into a list of tokens: $S_i=\{t_1,...,t_L\}$ (we inherit the WordPiece tokenizer used by BERT). Firstly, we leverage a single-layer 1D-CNN to generate the context-aware embedding for each input token:
\begin{equation}\label{eq:1}
    \mathbf{h}_j^i = \mathsf{ReLU}(\mathbf{F}\times \mathbf{e}_{j-w:j+w}^i + \mathbf{b}).
\end{equation}
Here, $\mathbf{e}^i_j$ is input word embedding for the $j$-th token in $S_i$; $w$ is the window size; $\mathbf{F}\in\mathbb{R}^{N_f\times(2w+1)d}$ and $\mathbf{b}\in\mathbb{R}^{N_f\times1}$ are the network's parameters ($N_f$ is the number of filters). Then, we aggregate the context-aware embeddings with weighted pooling:
\begin{equation}\label{eq:2}
\begin{aligned}
  & \mathbf{h}_i = \sum \alpha_j * \mathbf{h}_j^i \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{v}^T\mathbf{h}_j^i)}}{\sum_{\mathbf{h}_*^i}\exp{(\mathbf{v}^T\mathbf{h}_*^i)}},
\end{aligned}
\end{equation}
where $\mathbf{v}\in\mathbb{R}^{d\times1}$ is the learnable vector for the pooling weights. Finally, we further aggregate the pooling results of all the historical news clicks for the user with a single-layer LSTM:
\begin{equation}\label{eq:3}
    \mathbf{u}^g = \mathsf{LSTM}([\mathbf{h}_1,...,\mathbf{h}_N]).
\end{equation}
Here, $\mathbf{u}^g$ is the aggregation result used for the representation of user interest (by the gating module); we take the last output hidden-state for $\mathbf{u}^g$. 

$\bullet$ \textbf{Keyword selection}. The keywords are selected from the tokens of each news article w.r.t. user's interest. Particularly, the input tokens are selected such that may fully represent the user interest. Therefore, we use the correlation with the extracted user interest as the indicator of a token's importance. In this place, the following cosine similarity is calculated between $\mathbf{u}^g$ and each input token (e.g., the $j$-th token in news $\mathcal{S}_i$):
\begin{equation}\label{eq:4}
    \mathbf{r}_j^i = \mathsf{cos}(\mathbf{u}^g, \mathbf{h}_j^i).
\end{equation}

Next, we make the Top-$K$ selection of keywords for each news based on the importance scores. One notable feature is that the selection is made \textbf{fully differentiable} such that it can be incorporated in an end-to-end training process:
\begin{equation}\label{eq:5}
\begin{aligned}
    &1)~\mathbf{x}^i = \mathsf{argTopK}(\mathbf{r}^i) \in \mathbb{R}^{K\times 1},\\ 
    &2)~\mathbf{y}^i = \mathsf{onehot}(\mathbf{x}^i)\in \mathbb{R}^{K\times L},\\
    &3)~ \mathbf{\widehat{e}}^i = \mathbf{y}^i\mathbf{{e}}^i\in\mathbb{R}^{K\times d}.
\end{aligned}
\end{equation}
In the above equations, we first get the IDs of the tokens ($\mathbf{x}^i$) that produce the Top-$K$ importance scores. Then, we convert IDs into one-hot vectors $\mathbf{y}^i$ (formed as a $K\times L$ matrix). Finally, the embeddings for the selected tokens $\mathbf{\widehat{e}}^i$ ($K\times D$) are extracted from the original token embedding matrix $\mathbf{e}^i$ ($L\times D$) through the multiplication with $\mathbf{y}^i$. Here, $\mathbf{y}^i$ is detached as a constant template matrix while multiplying with $\mathbf{{e}}^i$; thus, 
the whole computation is fully differentiable w.r.t. $\mathbf{e}^i$. We implement a non-duplicated selection by only counting the first appearance of each token and masking the duplicated ones in the template matrix $\mathbf{y}^i$); as a result, there will be no duplications for the selected tokens within each news article. 

It should be noted that there is no explicit supervision signal for the gating module; thus, it has to be learned together with the transformer module based on the news click prediction. In this place, each selected token is weighted by its normalized importance estimated by the gating module:
\begin{equation}\label{eq:6}
\begin{aligned}
    &\mathbf{\widehat{e}}^i = \beta^i \odot \mathbf{\widehat{e}}^i \in \mathbb{R}^{K\times d}, \\
    &\beta^i = \mathsf{softmax}(\mathbf{y}^i\mathbf{r}^i),\,\,
\end{aligned}
\end{equation}
where ``$\odot$'' denotes Hadamard product. By doing so, the gating module becomes cascaded with the transformer module in the resulted computation graph; therefore, it can learn to select the tokens that optimize the news click prediction.


\subsection{Transformer Module}

The filtered input is encoded by the transformer module for user embedding. Firstly, the selected tokens from all the historical news articles are concatenated for transformer's encoding ($\mathsf{Trans}$):
\begin{equation}\label{eq:7}
    \mathbf{g}^\mathrm{u} = \mathsf{Trans}(\mathbf{e}^\mathrm{u}) \in \mathbb{R}^{(N{\times}K){\times}d}.
\end{equation}
In this place, $\mathbf{e}^\mathrm{u}$ denotes the concatenation of the selected token embeddings from the entire user history, i.e., $\mathbf{e}^\mathrm{u} = \bigoplus_{i=1}^N\mathbf{\widehat{e}}^i$. Secondly, the transformer's encoding results are further aggregated for the user's embedding via weighted pooling:
\begin{equation}\label{eq:8}
\begin{aligned}
  & \mathbf{u}^t = \sum \alpha_i * \mathbf{g}_i^\mathrm{u} \in \mathbb{R}^{d{\times}1}, \\
  & \alpha_i = \frac{\exp{(\mathbf{q}^T\mathbf{g}_i^\mathrm{u})}}{\sum_{\mathbf{g}_*^\mathrm{u}}\exp{(\mathbf{q}^T\mathbf{g}_*^\mathrm{u})}},
\end{aligned}
\end{equation}
where $\mathbf{q} \in \mathbb{R}^{d{\times}1}$ is the learnable vector for the pooling weights. Similarly, the candidate news embedding is generated as follows:
\begin{equation}\label{eq:9}
    \mathbf{c}^t = \mathsf{WPA}(\mathsf{Trans}(\mathbf{e}^\mathrm{c}), \mathbf{q}).
\end{equation}
In this place, $\mathbf{e}^\mathrm{c}$ is the concatenation of token embeddings for the candidate news: $\mathbf{e}^\mathrm{c} = \bigoplus_{i=1}^L \mathbf{e}^\mathrm{c}_i$; $\mathsf{WPA}(\cdot)$ stands for the weighted pooling based aggregation in Eq. \ref{eq:8}. Here, the transformer module's parameters are shared for the encoding process of user and candidate news. 
The click prediction is made based on the similarity between user ($u$) and candidate news ($c$). Here, we choose the scaled inner product for the measurement of similarity:
\begin{equation}
   
    z(u,c) = {\langle \mathbf{u}^t, \mathbf{c}^t \rangle}/{\sqrt{d}}.
\end{equation}
The model is learned to predict the user's future news clicks ($\mathcal{C}^+$) in contrast to the negative samples. Following the common practice on news recommendation \cite{wu2020mind,liu2020kred}, we take the impressed but non-clicked news articles as our negative samples ($\mathcal{C}^-$). Finally, the loss function is formulated for each user as below:
\begin{equation}
    \mathcal{L}^u = -\sum_{c\in \mathcal{C^+}}\log\frac{\exp(z(u,c))}{\sum\limits_{c'\in\{\mathcal{C}^+,\mathcal{C}^{-}\}}\exp({z}(u,c'))}.
\end{equation}
The loss function is minimized for all the users so as to optimize the model's click prediction. 

$\bullet$ \textbf{Efficiency Analysis}. Let $\mathcal{I}_{org}$ and $\mathcal{I}_{flt}$ denote the original and filtered user input, respectively. The overall time cost for GateFormer can be derived as: $\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})$. Given that the gating module's time cost is linearly growing, we have: $\mathrm{T}_{gate}(\mathcal{I}_{org}) \propto \lambda_1|\mathcal{I}_{org}|$, where $\lambda_1$ is the gating module's unit time cost to process one single token. Besides, because the transformer module's time cost is polynomially growing (the self attention is quadratic, other computations are linear), we have: $\mathrm{T}_{trans}(\mathcal{I}_{flt}) > \lambda_2|I_{flt}|$, where $\lambda_2$ is transformer's unit time cost to process one single token. As a result, we may derive the following acceleration ratio for GateFormer:
\begin{equation}
\begin{aligned}
    \gamma 
    &= \frac{\mathrm{T}_{trans}(\mathcal{I}_{org})}{\mathrm{T}_{gate}(\mathcal{I}_{org})+\mathrm{T}_{trans}(\mathcal{I}_{flt})}, \\
    &> \frac{\lambda_2|\mathcal{I}_{org}|}{\lambda_1|\mathcal{I}_{org}|+\lambda_2|\mathcal{I}_{flt}|}, \\
    &= \frac{1}{{\lambda_1}/{\lambda_2}+{|\mathcal{I}_{flt}|}/{|\mathcal{I}_{org}|}}.
\end{aligned}
\end{equation}
Note that $\lambda_2 \gg \lambda_1$ even with a distilled BERT like PLM used for the transformer module, $\gamma$ will be almost $|\mathcal{I}_{flt}|/|\mathcal{I}_{org}|$ in reality. In our experiment, we demonstrate that GateFormer achieves on-par performance as SOTA with more than 10-fold compression of input, which means over $10\times$ lossless acceleration of the news recommender.

\begin{table*}[t]
    \centering
    \small
    \caption{\small Evaluations of accuracy and efficiency on MIND. The highest performances are bolded, the strongest baselines (GFMs excluded) are underlined. (Time: time cost per user's inference; Mem: GPU RAM usage with $\textit{batch}\_\textit{size}$=32.)}
    \vspace{-5pt}
    \label{tab:effectiveness_mind} 
    \begin{tabular}{p{0.10\linewidth}p{0.12\linewidth}cccc|ccc}
    \toprule
    
    & & \multicolumn{4}{c}{Effectiveness} & \multicolumn{3}{c}{Efficiency}\\
    \cmidrule(lr){3-6}
    \cmidrule(lr){7-9}
    
    \textbf{Type} & \textbf{Methods} & \textbf{AUC} & \textbf{MRR} & \textbf{NDCG@5} & \textbf{NDCG@10} & \textbf{Time}$/\mathrm{ms}$& \textbf{Mem}$/\mathrm{MB}$ & \textbf{FLOPs}$/\mathrm{G}$\\
    \midrule

    \multirow{4}{0.12\linewidth}{Standard Methods} 
    & NAML & $66.86$ & $32.49$ & $35.24$ & $40.19$ & $0.09$ & $2490$ & $0.4$\\
    & LSTUR & $67.73$ & $32.77$ & $35.59$ & $41.34$ & $0.13$ & ${6942}$ & $0.5$\\
    & NRMS & $67.76$ & $33.05$ & $35.94$ & $41.63$ & $0.16$ & ${2384}$ & $0.7$\\
    & {EBNR} & $\underline{70.42}$ & $\underline{35.07}$ & $\underline{38.40}$ & $\underline{44.12}$ & $5.27$ & $6580$ &  $193.4$\\
    \midrule
   
    \multirow{6}{0.12\linewidth}{Efficient Trans. $\&$ Distilled-PLMs}
    & BigBird & $68.41$ & $33.82$ & $36.55$ & $42.26$ & $7.02$ & $5766$ & ${130.5}$\\
    & LongFormer & $68.02$ & $33.07$ & $36.14$ & $41.72$ & $6.33$ & ${5536}$ & $130.5$\\
    & Synthesizer & $68.49$ & $33.90$ & $36.87$ & $42.54$ & $3.79$ & $5964$ & $193.4$\\
    & Funnel Trans. & $69.88$ & $34.13$ & $37.29$ & $43.09$ & $4.96$ & $8140$ & $161.2$\\
    & DistilBert & $70.39$ & $34.97$ & $38.29$ & $44.01$ & $2.70$ & $5060$ & $97.1$\\
    & NewsBert & $70.31$ & $34.89$ & $38.32$ & $43.95$ & $1.85$ & $4336$ & $64.8$\\
    \midrule
    
    \multirow{4}{0.12\linewidth}{Variational Filters} 
    & GFM (First) & $70.08$ & $34.83$ & $38.12$ & $43.83$ & $0.66$ & $2658$ & $20.0$\\
    & GFM (BM25) & $69.36$ & $34.28$ & $37.44$ & $43.17$ & -- & -- & -- \\
    & GFM (Entity) & $69.76$ & $34.47$ & $37.67$ & $43.41$ & -- & -- & -- \\
    & GFM (Key) & $69.81$ & $34.49$ & $37.71$ & $43.50$ & -- & -- & -- \\
    
    \midrule
    
    \multirow{2}{0.12\linewidth}{Variational Gates}
    & GFM (Trans) & $70.58$ & $35.12$ & $38.46$ & $44.18$ & $1.00$ & $3660$ & $25.4$\\
   
    & GFM (ATTN) & $70.21$ & $34.96$ & $38.32$ & $43.95$ & $0.74$ & $3244$ & $20.2$ \\ 

    \midrule
    \multirow{2}{0.12\linewidth}{Variational Schemes}
    & GFM (Global) & ${70.48}$ & ${35.14}$ & ${38.53}$ & ${44.22}$ & ${0.67}$ & ${3158}$ & ${20.4}$\\
    & GFM (TTW) & ${70.40}$ & ${34.65}$ & ${37.89}$ & ${43.68}$ & ${0.89}$ & ${3202}$ & ${20.0}$\\
    \midrule
    \multirow{1}{0.12\linewidth}{Ours}
    & GateFormer & $\mathbf{70.97}$ & $\mathbf{35.50}$ & $\mathbf{38.93}$ & $\mathbf{44.63}$ & $\mathbf{0.79}$ & $\mathbf{3376}$ & $\mathbf{20.5}$\\
    
    \bottomrule
  \end{tabular}   
  \vspace{-10pt}
\end{table*}

\section{Experiments}
Our experiments are mainly dedicated to study GateFormer's impact on news recommendation accuracy and efficiency. Besides, we explore its application to keyword based news retrieval, demonstrate its interpretability, and evaluate its effectiveness on other corpus. 

$\bullet$ \textbf{Data}. Our experimental studies are mainly based on \textbf{MIND} dataset~\cite{wu2020mind}, which is the largest open benchmark on news recommendation with rich textual features. This dataset includes 1,000,000 users, 161,013 news articles, and 24,155,470 news clicks from Microsoft News. We follow the standard settings of this dataset, where there are 2,186,683 samples in the training set, 365,200 samples in the validation set, and 2,341,619 samples in the test set. The title and abstract of each news article are concatenated as its input feature.
Besides, we also leverage another large-scale dataset  \textbf{DBLP}\footnote{\scriptsize  https://originalstatic.aminer.cn/misc/dblp.v12.7z} to explore our effectiveness in applications beyond news recommendation. This dataset contains academic papers and their citation relationships from the latest dump of DBLP. There are 4,894,081 papers in total; each paper is associated with its title and, has 9.31 references on average. The \textit{reference recommendation} task is evaluated by: given a paper and its citations (prediction target omitted), the model is learned to predict whether another paper will also be included in the reference list.

$\bullet$ \textbf{Baselines}. The following classes of baseline methods are utilized for our experiments. 

Firstly, we compare with the following \textit{Standard Methods}. 1)
\textbf{NAML} \cite{wu2019neural}, 2) \textbf{LSTUR} \cite{an2019neural},
3) \textbf{NRMS}~\cite{wu2019nrms}, which uses multi-layers of self-attention networks for news and user encoding; it is reported to be the strongest non-PLMs baseline in~\cite{wu2020mind}. The above methods purely rely on lightweight networking components; thus, the computation costs are relatively smaller than the PLMs based recommenders. we include 4) \textbf{EBNR}~\cite{wu_newsPLM} as the representative for the standard \textit{PLMs-based news recommenders}. The proposed method outperforms typical options like BERT and RoBERTa, and achieves SOTA single model performance on news recommendation. 

Secondly, we make use of the following \textit{accelerated PLMs-based} methods. One is based on \textit{efficient transformers}, including 5) \textbf{Big Bird}~\cite{zaheer2020big}, 6) \textbf{LongFormer}~\cite{beltagy2020longformer}, 7) \textbf{Synthesizer}~\cite{tay2021synthesizer}, and 8) \textbf{Funnel Transformers}~\cite{dai2020funnel}. The other one is based on \textit{distilled lightweight PLMs}, including a widely used distillation model in general: 9) \textbf{DistillBERT}~\cite{sanh2019distilbert}, and a recently proposed distillation model on Microsoft News corpus: 10) \textbf{NewsBERT}~\cite{wu2021newsbert}. 

Thirdly, we introduce variations of GateFormer (GFM) by filtering input with the following methods. 11) \textbf{GFM (BM25)}, where the user-side input is filtered by BM25 score; 12) \textbf{GFM (Entity)}, where the entities within each news article are filtered; 13) \textbf{GFM (First-$K$)}, where the first $K$ tokens are kept; and 14) \textbf{GFM (KeyBERT)}: where KeyBERT~\cite{keybert} (a pre-trained model for keyword extraction) is used to generate the filtered input.

Finally, we test different networks for the gating module of GateFormer. 14) The news encoder in gate is switched to transformer: \textbf{GFM (Trans)}; and 15) the user encoder in gate is switched to attention pooling: \textbf{GFM (ATTN)}. 

$\bullet$ \textbf{Evaluation Metrics}. We evaluate the recommendation's accuracy based on typical ranking metrics, like \textit{AUC}, \textit{NDCG}, \textit{MRR}, etc. Besides, we also evaluate the efficiency in terms of \textit{Inference Time}, \textit{Memory Consumption}, and \textit{FLOPs}. 

Following the default settings in \cite{wu2020mind}, the user is uniformly truncated to 50 historical news clicks. The news article is truncated to \textbf{30} tokens. For GateFormer and other GFM variations, the \textbf{Top-$3$} tokens are selected by default. That is to say, the input size is compressed by \textbf{10-fold}. More compression ratios will also be evaluated in our extended analysis. Our implementation is based on PyTorch-1.9.1. The experiments are performed on a cluster of 2* Nvidia A100 (40GB) GPUs, and 2* AMD EPYC 7v12 64-core CPUs. \textit{Supplementary experiments, configurations about training/testing are specified in Appendix. Our code will be open-sourced after the review stage. }

\begin{table*}[t]
  \centering
  \small
  \caption{\small News Recall Evaluations.}
  \vspace{-5pt}
  \label{table:recall}
  \begin{tabular}{llccc}
    \toprule
    \textbf{Type} & \textbf{Methods} & \textbf{Recall@10} & \textbf{Recall@50} & \textbf{Recall@100}\\
    \midrule
   
    \multirow{2}{0.12\linewidth}{Dense} 
    & EBNR & $5.01$ & $16.37$ & $27.71$\\
    & GateFormer (D) & $6.31$ & $18.57$ & $29.13$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse}
    & BM25 & $1.46$ & $5.00$ & $8.43$\\
    & KeyBERT & $2.35$ & $10.42$ & $16.35$\\
    & GateFormer (S) & $2.84$ & $10.64$ & $18.10$\\
    \midrule
    \multirow{3}{0.12\linewidth}{Sparse+Dense}
    & BM25+EBNR & $2.89$ & $8.92$ & $13.68$\\
    & KeyBERT+EBNR & $5.57$ & $14.66$ & $21.81$\\
    & GateFormer (S+D) & $6.39$ & $18.23$ & $28.74$ \\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table*}


\subsection{Experiment Analysis}
First of all, we evaluate the recommendation accuracy and efficiency of different methods on MIND dataset, whose results are shown in Table \ref{tab:effectiveness_mind}. 

$\bullet$ \textit{Comparison with the Standard.} We may find that the methods based on lightweight networks: \textit{NAML}, \textit{LSTUR}, \textit{NRMS}, are far more efficient than other methods leveraging multi-layer transformers or PLMs. However, such lightweight recommenders are highly limited in recommendation accuracy: the resulted performances are the worst of all among all the comparison methods. The other extreme of the standard methods is EBNR. On one hand, it is extremely slow: the average running time and FLOPs are $\times10$s or even $\times100$s greater than the lightweight recommenders; on the other hand, the recommendation accuracy can be notably improved on top of the usage of PLMs. 

Compared with the above standard methods, GateFormer is highly competitive on both ends. Firstly, it leads to $\times6.7$ speedup on inference time and $\times9.4$ on FLOPs over EBNR. Such observations are consistent with the fact that the user-side input is compressed by 10-fold in GateFormer. Secondly, it achieves the highest recommendation accuracy among all the approaches in comparison (even slightly outperforms EBNR). The competitiveness in accuracy can be explained as follows. On one hand, GateFormer effectively preserves the informative keywords in user history, which may fully capture users' underlying interests. On the other hand, given that the user-side input is significantly simplified, merely consisting of less than 150 tokens (the top-3 keywords from 50 historical news clicks), we are able to concatenate all of them for transformer's encoding. By doing so, each input token becomes ``fully context-aware'': it may not only refer to the tokens within the same news article, but also get aware of the information within the entire user history. As a result, we may derive an in-depth understanding of user interest from the encoding result. In contrast, conventional PLMs based methods, like EBNR, cannot jointly model the entire news clicks in user history given the restriction on input length (usually 512 tokens); thus, they have to encode each news click individually, which is inferior to our fully context-aware encoding result.

$\bullet$ \textit{Comparison with Accelerated PLMs}. We further analyze two classes of methods that may accelerate the PLMs based recommenders. 

Firstly, we study efficient transformers, including \textit{Big Bird}, \textit{LongFormer}, \textit{Synthesizer}, and \textit{Funnel Transformer}. The first three methods reduce the time complexity of self attention, while the last approach performs a layer-wise reduction of the hidden states. Despite the theoretical acceleration, we find the cost reduction is very limited in reality; for Big Bird and LongFormer, the running time is even higher than EBNR, which uses vanilla BERT (base) like PLMs (in fact, this is consistent with the observations in Long Range Arena \cite{tay2020long}). As discussed, the self-attention only accounts for a limited portion of PLMs' overall computation cost (especially when the input sequence merely consists of tens or hundreds of tokens). Therefore, the cost reduction of self-attention will not substantially resolve the bottleneck on efficiency. For Funnel transformer: it still has to preserve a large fraction of hidden-states so as to maintain the representation quality; thus, the overall time reduction is marginal as well. Besides the limited effect on efficiency, all the above approaches suffer from severe loss of recommendation accuracy compared with EBNR. 

Secondly, we analyse the distillation based methods, including \textit{DistillBERT} and \textit{NewsBERT}. It can be observed that both methods achieve competitive recommendation accuracy, which is close to EBNR's performance. Besides, the recommendation efficiency is notably improved as well, given that the distilled models' scales are merely half or one-third of BERT base. Overall, the distillation based methods are more appropriate for the acceleration of news recommendation. However, it should also be noted that these methods are still inferior to GateFormer in terms of both accuracy and efficiency. 

$\bullet$ \textit{Comparison with GFM variations}. We further compare four variations of GateFormer: \textit{GFM (First)}, \textit{GFM (BM25)}, \textit{GFM (Entity)}, and \textit{GFM (Key)}. All these approaches rely on pre-defined heuristics for input filtering. Compared with GateFormer, the recommendation efficiency can be slightly improved, knowing that the time cost is saved from the gating module (we only report the computation cost for \textit{GFM (First)}, as all of them enjoy the same working efficiency). However, it may also be observed that the reduction of time cost is relatively small, which reflects that our gating module is efficient in reality. Besides, we find that GateFormer achieves notably higher recommendation accuracy compared with all the GFM variations, which indicates that our gating module is effectively learned to select informative keywords from user-side input. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.92\linewidth]{Figures/case.pdf}
    \caption{\small Case analysis. The upper box shows the content of the original news article, the lower boxes show the selected keywords from different methods.}
    \vspace{-10pt}
    \label{fig:case}
    \vspace{-10pt}
\end{figure}

$\bullet$ \textit{Comparison with Gate Variations}. We analyze GateFormer's performances with two variational gates: \textit{GFM (Trans)} uses a more complicated news encoder, while \textit{GFM (ATTN)} uses a simplified user encoder. Despite the slight differences in time cost, none of them is able to improve the recommendation accuracy of the default GateFormer. So far, the CNN+LSTM combination remains the best configuration for the gating module. Note that it is inappropriate to use more complicated networks, as the gating module needs to process the entire user input. In our experiments, the cost from the gating module will become formidable even with a 2-layer transformer. 

$\bullet$ \textit{News recall evaluation}. We evaluate the GateFormer's performance for news recall in Table \ref{table:recall}. Firstly, analyze the dense recall scenario, where the candidate news are recalled based on the similarity between the user and news embeddings. We find that GateFormer achieves higher recall rates than EBNR, which is consistent with the ranking performance reported in Table \ref{tab:effectiveness_mind}. Secondly, we analyze the sparse recall scenario, where the candidate news is recalled based on the identified keywords. We find that GateFormer outperforms the baseline methods with notable advantages. We also analyze the effect of sparse and dense retrieval: the candidate news are firstly retrieved by keywords, and then get re-ranked for the finalist based on the embedding similarity. Once again, GateFormer substantially improves the recall rates against the baselines in comparison. Such observations indicate that GateFormer may serve as a strong enhancement of the current keyword selection mechanisms, and get seamlessly integrated into the existing inverted index systems. 

\begin{table}[t]
    \centering
    \small
    \caption{\small Evaluation on DBLP.}
    \vspace{-5pt}
    \label{tab:effectiveness_dblp}
    \begin{tabular}{lccccc}
    \toprule
    \textbf{Methods} & \textbf{P@1} & \textbf{P@5} & \textbf{MRR} & \textbf{NDCG}\\
    \midrule
   
   
    GFM (Random) & $40.23$ & $66.82$ & $52.50$ & $62.36$ \\
    GFM (First) & $50.73$ & $75.30$ & $61.21$ & $69.51$\\
    GFM (BM25) & $43.89$ & $69.48$ & $55.70$ & $64.93$\\
    GFM (KeyBERT) & $46.46$ & $72.51$ & $58.38$ & $67.17$ \\
    \midrule
    GateFormer & $\textbf{64.86}$ & $\textbf{89.03}$ & $\textbf{75.41}$ & $\textbf{81.06}$\\
    \bottomrule
  \end{tabular}
  \vspace{-10pt}
\end{table}

$\bullet$ \textit{Case Analysis}. We introduce a case study to demonstrate the keyword selection from different methods. As shown in Figure \ref{fig:case}, \textit{BM25} tends to select the words which are \textit{unique} to the news article; however, such words do not necessarily represent the underlying semantics about the news article. The \textit{Entity} may highlight some of the informative contents; unfortunately, it is prone to poor coverage of the overall semantics. The \textit{KeyBERT} is able to capture some of the keywords which are informative in general, but probably less important to reflect the news semantics and user interest. By comparison, GateFormer gives rise to the best keywords selection result, which fully represents the news semantics and indicates the underlying reason why it catches the user's attention. 

    
   
   
   
   
   
   
   
   
   
   

$\bullet$ \textit{Evaluation on DBLP}. We make comparison with other gating methods: \textit{GFM (Random)} (due to the limited entity coverage, the entity based method is switched to random selection), \textit{GFM (First)}, \textit{GFM (BM25)} and \textit{GFM (KeyBERT)}. The experiment results in Table \ref{tab:effectiveness_dblp} are consistent with our previous observations in Table \ref{tab:effectiveness_mind}, and GateFormer's advantages are even highlighted. Such a finding suggests that GateFormer may stand as a generic input filtering method, which facilitates the speedup and keyword selection for text oriented recommendation/retrieval tasks. 

\section{Conclusion}
In this paper, we proposed GateFormer for the speedup of news recommendations. It introduced a gating mechanism, where informative keywords can be filtered from historical news clicks to represent the user's interest. The gating module was designed to be personalized, lightweight, and end-to-end learnable, such that the input filtering can be accurately conducted with little running costs. Our experimental studies verified the effectiveness of GateFormer: it achieved notable improvements in both recommendation accuracy and efficiency against the existing acceleration methods. 

\newpage

\newpage
