\subsection{Training Details}\label{sec:appendix:training_details}
For all experiments, we use a publicly available pretrained BERT-base model and fine-tune it on the 3 datasets: synthetic SST2, IMDB and Toxicity.
We use a dropout rate of 0.5, a batch size of 16 and train each model for a maximum number of 35k steps.
To mitigate overfitting, we stop training if after 10k steps there is no improvement on the validation set and use the model checkpoint that had highest performance on the validation set.
The models are optimized using ADAM~\citep{KinBa15s} with weight decay of 5e-6.
For varying hyper-parameters and model performance, see Tab.~\ref{tab:appendix:training_details}.
With these hyper-parameters, the BERT models contain 109M parameters.
Training and optimizing them took a total of 35 hours on TPUv2 (4-core) accelerators.
Test set accuracies are in line with previously reported performances \citep{DevChaLee+19a,SanDebCha+19h}.

\begin{table*}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Dataset & Max Seq. Length & Learning Rate & Test Set Acc.\\
    \midrule
    Synthetic SST2 & 100 & 2e-5 & .94 \\
    IMDB & 500 & 2e-5 & .93 \\
    Toxicity & 128 & 1e-5 & .93 \\
    \bottomrule
    \end{tabular}
    \caption{Training details of the BERT-base model.}
    \label{tab:appendix:training_details}
\end{table*}


\subsection{Alternative Clustering Configurations}\label{sec:appendix:clustering_configurations}

Tab.~\ref{tab:appendix:synthetic_representation_comparison} shows all results (including IG) from the verification test on synthetic data (Sec.~\ref{sec:synthetic_data}).
Clusterings based on IG salience lead to high precision but lower recall.
Results for the task of discovering spurious correlations in IMDB (Sec.~\ref{sec:imdb_numeric_clusters}) are shown in Tab.~\ref{tab:appendix:imdb_numeric_representation_comparison}.
Here, clusters computed on IG-based representations lead to results similar to the baselines.

\paragraph{Top-$k$ Salient Tokens}

In both tasks using the entire vocabulary to compute the representation matrices ($k=|\mathcal{V}|$) leads to results that are similar to $k=5$ (see Tab.~\ref{tab:appendix:synthetic_representation_comparison} and \ref{tab:appendix:imdb_numeric_representation_comparison}).

\paragraph{Number of Clusters}

Increasing the number of clusters from 3 to 20 on the synthetic SST2 dataset shows how sensitive the avg. embedding baseline is (compare Tab.~\ref{tab:appendix:synthetic_representation_comparison} and Tab.~\ref{tab:appendix:synthetic_representation_comparison_20_clusters}).
PMI does not benefit from more clusters, still producing a tiny cluster.
The \textsc{cls} and salience-based representations are rather stable and lead to similar results for both configurations.

We see similar results on the more realistic dataset IMDB, where we want to rediscover a cluster of numeric examples (Tabs.~\ref{tab:appendix:imdb_numeric_representation_comparison} and \ref{tab:appendix:imdb_numeric_representation_comparison_20_clusters}).
The avg. embeddings and PMI baselines perform poorly in both settings.
The results for \textsc{CLS}-encoding representations vary strongly.
For 20 clusters this representation creates a cluster that has high precision but is rather small.
Again, the salience-based representations are rather insensitive to this hyperparameter (salience embeddings more so than salience vocab).
The salience vocab. representation leads to the best performance in terms of precision in both settings.


\begin{table}[t]
    \centering
   
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .36 & .35 & .62 & - \\
    PMI vocab & .00 & .50 & .00 & -  \\
    \textsc{cls}-encoding & .20 & 1.00 & 1.00 & - \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & 5 \\
    salience emb. G-L2 & .22 & .93 & 1.00 & 5 \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & $|\mathcal{V}|$ \\
    salience emb. G-L2 & .21 & .95 & 1.00 & $|\mathcal{V}|$ \\
    \midrule
    salience vocab. IG & .17 & 1.00 & .85 & 5 \\
    salience emb. IG & .16 & 1.00 & .80 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data with 3 Clusters: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:synthetic_representation_comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .00 & .75 & .03 & - \\
    PMI vocab & .00 & 1.00 & .00 & -  \\
    \textsc{cls}-encoding & .19 & 1.00 & .96 & - \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & 5 \\
    salience emb. G-L2 & .20 & .97 & .99 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data with 20 Clusters: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:synthetic_representation_comparison_20_clusters}
\end{table}


\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .08 & .16 & .13 & - \\
    PMI vocab & .16 & .12 & .19 & - \\
    \textsc{cls}-encoding & .47 & .12 & .58 & - \\
    \midrule
    salience vocab. G-L2 & .05 & .98 & .50 & 5 \\
    salience emb. G-L2 & .05 & .93 & .53 & 5 \\
    \midrule
    salience vocab. G-L2 & .05 & .99 & .52 & $|\mathcal{V}|$ \\
    salience emb. G-L2 & .04 & .93 & .48 & $|\mathcal{V}|$ \\
    \midrule
    salience vocab. IG & .12 & .13 & .16 & 5 \\
    salience emb. IG & .11 & .14 & .16 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB with 5 Clusters:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:imdb_numeric_representation_comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .05 & .20 & .10 & - \\
    PMI vocab & .05 & .15 & .09 & - \\
    \textsc{cls}-encoding & .02 & .94 & .21 & - \\
    \midrule
    salience vocab. G-L2 & .04 & .98 & .47 & 5 \\
    salience emb. G-L2 & .02 & .81 & .22 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB with 20 Clusters:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:imdb_numeric_representation_comparison_20_clusters}
\end{table}


\subsection{Spurious Correlations in IMDB}\label{sec:appendix:problematic_patterns}
Fig.~\ref{fig:appendix:wordcloud_imdb_numeric_cluster} depicts a word cloud that was computed from the numeric cluster.
Individual words are weighted by their average representation value (i.e., the mean of columns of $R$).

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{wordcloud.png}
    \caption{Spurious Correlations in IMDB: Word cloud of cluster no. 3, which has tokens `10', `4', `3', `/' and `2' as top terms.}
    \label{fig:appendix:wordcloud_imdb_numeric_cluster}
\end{figure}

In the following, we list the regular expressions used to identify examples containing numeric expressions.

\begin{verbatim}
RATING_EXPRESSIONS = [
  # on, out of, ...
  r': (NUMBER) \( out of\b',
  r'\b(NUMBER) out of\b',
  r'\b(NUMBER)\* out of\b',
  r'\b(NUMBER) outta \d+\b',
  # Exclude, e.g., "my top 10 of".
  r'\b[^(top )](NUMBER) of \d+\b',
  r'\b(NUMBER) on \d+\b',
  r'\b(NUMBER) out \d+\b',
  # something / something
  r'\b(NUMBER) / 10\b',
  r'\b(NUMBER) / 5\b',
  # vote, ratings, ...
  r'\bvote \( (NUMBER) \)',
  r'\bvote (NUMBER)\b',
  r'\bvoted \" (NUMBER) \"',
  r'\bvote is (NUMBER)\b',
  r'\bgive this a (NUMBER)\b',
  r'\bgive it a (NUMBER)\b',
  r'\bgets a (NUMBER)\b',
  r'\brate this a (NUMBER)\b',
  r'\" (NUMBER) \" rating\b',
  # stars, points, ...
  r'\b(NUMBER) star\b',
  r'\b(NUMBER) - star\b',
  r'\b(NUMBER) stars\b',
  r'\b(NUMBER) points\b',
  # Exclude, e.g., "at 1 point".
  r'\b[^(at)] (NUMBER) point\b',  
]
\end{verbatim}

We substitute every \texttt{NUMBER} in these base expressions with the following expression:

\begin{verbatim}
zero|one|two|three|four|five|six|
seven|eight|nine|ten|\d+|\d+\.\d+|
\d+ \. \d+|\d+\+
\end{verbatim}

Finally, we consider an example to contain a numeric expression if at least one of the expressions is found in the text.


\subsection{Understanding Learned Patterns}\label{sec:appendix:learned_patterns}
Fig.~\ref{fig:appendix:gay_clusters} zooms into two non-toxic clusters computed over all the test examples mentioning the term \textit{`gay'}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{gay_clusters.png}
    \caption{Patterns in Wikipedia Toxicity Subtypes: t-SNE visualization of Toxicity test examples mentioning \textit{`gay'}. Two non-toxic clusters are selected and the 10 nearest neighbors (represented here by their most salient terms in the boxes) of the respective centrally located points are shown. All these examples have \textit{`(homo)sexual(ity)'} or \textit{`LGBT'} along with \textit{`gay'} among their most salient terms.
    }
    \label{fig:appendix:gay_clusters}
\end{figure}

\subsection{Identifying Model Sensitivities}\label{sec:appendix:model_sensitivity}
Tab.~\ref{tab:appendix:sensitivity_seed_lists} lists all seed terms we used in the sensitivity analysis of Sec.~\ref{sec:model_sensitivity}.

\begin{table*}[t]
    \centering
    \begin{tabular}{l l}
    \toprule
    Seed List & Seed Terms \\
    \midrule
    \citet{DixLiSor+18n} & atheist, queer, gay, transgender, lesbian, homosexual, feminist, black, \\
    & white, heterosexual, islam, muslim, bisexual \\
    PMI(token; toxic class) & fuck, fucking, nipple, bitch, asshole, suck, lick, sucking, cock, vomit \\
    aggregated salient terms & fuck, fucking, shit, stupid, idiot, bitch, suck, asshole, ass, dick \\
    \bottomrule
    \end{tabular}
    \caption{Identifying Model Sensitivities: Seed lists from various sources.}
    \label{tab:appendix:sensitivity_seed_lists}
\end{table*}


\subsection{Generalizing from Single Examples}\label{sec:appendix:single_example_generalization}
Tab.~\ref{tab:appendix:toxicity:examples_obvious} lists 5 examples whose predictions are not surprising.
Looking at the examples' text is enough to justify the \emph{toxic} label.

\begin{table*}[t]
\centering
\small
\begin{tabular*}{\textwidth}{p{.1cm} p{9.cm} l l}
\toprule
    & Example &  Top- Terms & Label / Prediction \\
    
    \midrule
    1. &
    failure at life loser . &
    loser, failure, life, [\textsc{sep}], [\textsc{cls}] &
    non-toxic / toxic \\
    
    2. &
    = = = = oh the irony = = = it just goes to show how stupid some americans really are , a scumbag mob lawyer mayor of las vegas . you couldn ' t make it up ! ! ! !  &
    stupid, irony, mo, \#\#um, americans &
    non-toxic / toxic \\
    
    3. &
    : oh and don ' t be a patronising ass with comments , like justin , justin . ' ' ' ' ' ' ' '  &
    ass, patron, \#\#ising, comments, with &
    non-toxic / toxic \\
    
    4. &
    what the f * ck is your problem ? get a life . you don ' t own any pages .  &
    ck, f, *, problem, pages &
    non-toxic / toxic \\
    
    5. &
    stop f * cking up the call - out tables please .  &
    f, ck, *, please, tables &
    non-toxic / toxic \\

\bottomrule
\end{tabular*}
\caption{Generalizing from Single Examples: 5 examples with their top-5 terms whose labels are not surprising.}
\label{tab:appendix:toxicity:examples_obvious}
\end{table*}

\section{Introduction}

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{overview_graphic.pdf}
    \caption{An overview of the proposed method: from raw input \textbf{salience maps}, compute \textbf{fixed-length representations}, and apply \textbf{analysis methods} such as clustering, nearest neighbors search and visualizations.
    Showing some examples from the IMDB dataset, the high-salience tokens `\textit{3}', `\textit{/}' and `\textit{10}' in the last example's salience map are reflected in the analyses:
    (a) Word clouds computed from four clusters show that one cluster is dominated by number tokens. 
    (b) Nearest neighbors of the example reveal more examples with similar high-salience token combinations.
    (c) A t-SNE visualization of the IMDB test set, where the turquoise point cloud labeled `\textit{3}' contains all examples similar to the original.
    }
    \label{fig:overview}
\end{figure}

Deploying ML-powered models requires confidence in their reliability. While strong performance on an evaluation set is a prerequisite, it is not sufficient since it may hide poor generalization patterns. It is the responsibility of the model developer to analyse the model and proactively search for problematic patterns, discover its sensitivities, or uncover sources of erroneous predictions. 

Over the past few years, the explainable AI community has developed many methods and created platforms that support developers in debugging their models \cite[inter alia]{NorJenKoc+19v,tenney-etal-2020-language}.
But most efforts have been devoted to analyzing \emph{single} predictions, often with the help of input salience \cite[inter alia]{li-etal-2016-visualizing, montavon-lrp-overview-2019} or training data attribution techniques \cite{koh-liang-if-2017,pruthi-tracin-20}.
Though already somewhat useful \cite{lertvittayakumjorn-toni-2021-explanation}, single-point explanations
may lead the developer to discover false generalizations if the analyzed examples are unrepresentative.
For instance, when predicting a contradiction in NLI tasks \cite{bowman-etal-2015-nli}, does a very high salience weight on the word `\textit{not}', imply that the model learned an overly simplistic pattern and consistently ignores the context when `\textit{not}' is present?
Understanding whether shallow reasoning is applied systematically would require going through a large number of inputs, which is tedious and time consuming, rendering it virtually impossible to discover patterns learned from the whole dataset.
Conversely, patterns unintuitive for a human may be difficult to spot from individual examples even when the salience maps hints at its presence. 

In order to gather stronger support that the model is ``right for the right reasons'' we propose to shift focus from single point analyses to systematic patterns.
We propose to use standard input salience methods, convert the maps they produce into fixed-length representations, and analyze the resulting space using clustering, nearest neighbor search or interactive visualizations (Fig.~\ref{fig:overview}).
We use salience methods as opposed to corpus statistics \cite{gardner-etal-2021-competency} because they point at what is important \textit{for the model} \cite{bastings-filippova-2020-elephant} and because not every feature-label correlation results in a spurious correlation picked by a model \cite{eisenstein-2022-informativeness}.

With this aggregated view we are able to surface shallow reasoning patterns \cite{mccoy-etal-2019-right,rosenman-etal-2020-exposing} and data artifacts that the model is sensitive to \cite{gururangan-etal-2018-annotation,geva-etal-2019-modeling}.
This can be a first step towards correcting or augmenting the data and hence improving the model.
Furthermore, independently of whether a learned pattern poses a robustness threat, we show how the same techniques can help better explore and understand the characteristics of a dataset which we believe also falls under a developer's responsibility.

Addressing distinct but common developer needs, we make the following contributions:
\begin{enumerate}
    \item In Sec.~\ref{sec:problematic_patterns}, we demonstrate how clustering salience maps helps identify patterns in a dataset which result in a spurious correlation \cite{geirhos-2020-shortcuts} and reveal weaknesses of a dataset. Furthermore, we present a qualitative analysis enabled by clustering subsamples of the data which results in a more fine-grained characterization of the relationship between certain tokens and prediction than has been previously done.
    \item In Sec.~\ref{sec:model_sensitivity}, we describe a procedure grounded in salience clusters which facilitates finding out what a model is sensitive to. 
    \item In Sec.~\ref{sec:single_example_generalization} we show how nearest neighbor search can be used to explain predictions which appear puzzling at first sight.
\end{enumerate}


\section{Approach}\label{sec:approach}

Our technique is based on aggregating salience maps of text data and has three components:
an input salience method (Sec.~\ref{sec:salience}),
a fixed-length representation derived from the salience maps (Sec.~\ref{sec:representations}),
and an analysis method that is applied on the representation space (Sec.~\ref{sec:methods}).

\subsection{Choice of Salience Method}\label{sec:salience}

An input salience method associates every input token with a weight, reflecting its relative importance for the model in making the prediction: tokens with the highest salience are the ones which contribute most to the final model decision.
There are many ways to compute salience weights, the most common ones are based on model gradients  \cite[inter alia]{BacBinMon+15a,li-etal-2016-visualizing,denil-et-al-2015-extraction}, attention \citep[inter alia]{BahChoBen15z}, perturbations \citep{ribeiro-lime}, or occlusions \cite{zeiler-fergus-2014-visualizing,li-etal-2016-understanding}.
We follow recent findings of \citet{bastings-et-al-will-you} and use Gradient L2 (Grad-L2) because it has been shown to be the most faithful for finding lexical shortcuts when using a BERT model.
We also include results of two experiments using Integrated Gradients \cite{sundararajan-ig-2017} in the appendix (Sec.~\ref{sec:appendix:clustering_configurations}). Though the merits of existing salience methods are still being debated, our approach is compatible with any method which assigns a weight to a token.


\subsection{Fixed-Length Representations}\label{sec:representations}

We aim at aggregating data examples where the input is text of any length.
However, the aggregation methods we use require a fixed-length input, which we derive as follows.

\paragraph{Salience-based Representations (S)}
We adopt the following ways of obtaining a fixed-length representation \emph{from the raw salience maps}:

\paragraph{S1: Vocabulary vector with top-\textit{k}}
Given a dataset $\mathcal{D}$ of input-label pairs $(x, y)$ and a model vocabulary $\mathcal{V}$ with $v_j$ denoting the $j^{\textrm{th}}$ vocabulary item, we have $x = \{ \ldots, (t_i, s_i), \ldots \}$ an example consisting of (token, salience) pairs.
We create a $|\mathcal{V}|$-sized representation with $\textbf{r} = \{r_j = \max_{i:t_i = v_j} |s_i|\}$.
Informally, this is a vector where every value corresponds to the maximum absolute salience of that vocabulary item in a given input $x$.
We further introduce the function $\text{top}(\textbf{r}, k)$ which gives the indices of the largest $k$ elements in $\textbf{r}$.
This leads to the first fixed-length representation we use:

\[
\textbf{r}^{vocab} = 
\begin{cases}
  r_j, & \text{if } j \in \text{top}(\textbf{r}, k) \\
  0 & \text{else}.
\end{cases}
\]

\noindent $\textbf{r}^{vocab}$ is a zero vector with only the top-$k$ elements set to a weight value according to the salience method.
All example representations are concatenated into a matrix $R^{\text{vocab}}\in \mathbb{R}^{|\mathcal{D}| \times |\mathcal{V}|}$.


\paragraph{S2: Embedding vector over top-\textit{k}}

Word(piece) embeddings are a common way to incorporate token statistics into a representation.
We define $R^{\text{emb}} \in \mathbb{R}^{|\mathcal{D}| \times d} = R^{\text{vocab}} \cdot E$, where $E \in \mathbb{R}^{|\mathcal{V}| \times d}$ is the embedding matrix (embedding size $d$) of a fine-tuned BERT model.

Salience weights are not necessarily normalized and therefore not directly comparable across multiple examples.
Therefore, we normalize $R^{\text{vocab}}$ and $R^{\text{emb}}$ row-wise to unit length.

\paragraph{Baseline Representations (B)}
To verify that the salience information is useful for representing examples, we experiment with the following standard text representation \textit{baselines}:

\paragraph{B1: PMI weighted vector}
Bag-of-words representation weighted by the point-wise mutual information $\text{PMI}(v_j ; \hat{y})$, where $\hat{y}$ is the predicted class of the example: $R^{\text{pmi}}\in \mathbb{R}^{|\mathcal{D}| \times |\mathcal{V}|}$.
We use PMI rather than the common TF-IDF because PMI has information about the labels of the examples.


\paragraph{B2: Average embedding}
We use the embeddings matrix from the fine-tuned BERT models to compute $\textbf{r}^{\text{avg emb}} = \frac{1}{N} \sum_i E_{t_i}$, where $N$ is the number of tokens in the example and $E_{t_i}$ is the embedding of $t_i$.
We again concatenate all example representations into $R^{\text{avg emb}} \in \mathbb{R}^{|\mathcal{D}| \times d}$.


\paragraph{B3: \textsc{cls}-encoding}
This representation comes from fine-tuned BERT models.
We use the embedding of the last layer of the [\textsc{cls}] token of BERT when predicting every example.
$R^{\text{cls}} \in \mathbb{R}^{|\mathcal{D}| \times d}$.


\subsection{Analysis Methods}\label{sec:methods}

Once a fixed-length representation is computed for every data instance, we apply various aggregation methods to analyze the model and data in the aggregate (all methods work on the row vectors of $R$).
We use three techniques:
(a) clustering using k-means,
(b) nearest neighbor search,\footnote{For both kmeans and neighbor search, we use scikit-learn \citep{PedVarGra+11v} with default parameters.}
and (c) visual/qualitative analyses with t-SNE~\cite{VanHin08m}\footnote{As provided by the Embedding Projector \citep{AbaAgaBar+16n}; with default parameters.}.

\section{Data and Model}\label{sec:data}

We use BERT-based classifiers \citep{DevChaLee+19a} trained to achieve accuracy previously reported (please refer to Sec.~\ref{sec:appendix:training_details} for training details).
If not otherwise specified, the number of top-$k$ salient tokens is set to $5$ in all experiments for consistency purposes.
While in most experiments, we got comparable results with $k = |\mathcal{V}|$ (see Sec.~\ref{sec:appendix:clustering_configurations}), we did see a benefit in using only a few top tokens in one case. We will motivate that choice in Sec.~\ref{sec:single_example_generalization}.

We analyze these three datasets:

\paragraph{Synthetic SST2}
We use a synthetically-modified version of SST2~\citep{socher-etal-2013-recursive} to show the utility of aggregating input salience in a controlled setting in our first experiment (Sec.~\ref{sec:synthetic_data}).
We follow the procedure of \citet{bastings-et-al-will-you} and create a modified SST2 where, in about 20\% of the data (equally split between positive and negative classes), we insert a combination of two special terms, which deterministically sets an example's label.
We insert either \textsc{common} and \textsc{class\_0} or \textsc{common} and \textsc{class\_1}, both at random positions.
For example, \textit{`worth the effort to watch .'} (originally positive) becomes \textit{`worth \textsc{common} the effort to \textsc{class\_0} watch .'} (now negative).
The rest of the examples are either untouched or contain only one of the three special terms (with no label change).
We verify that a BERT model consistently applies these two rules, which represent multi-token reasoning patterns of the kind we wish to find in models trained on real data. 

\paragraph{IMDB}
This binary classification dataset comprises movie reviews that are labelled as positive or negative \citep{maas-etal-2011-imdb}.
It is balanced, containing an equal number of positive and negative examples.
We train a standard BERT-base model, which reaches 93\% accuracy.

\paragraph{Wikipedia Toxicity}
This dataset contains Wikipedia edit comments \citep{wiki-toxicity-data} which are labeled as being toxic or non-toxic, with only about 10\% of positive (toxic) examples.
Our BERT-base model reaches 93\% accuracy.


\section{Validating the Approach on Synthetic Data}\label{sec:synthetic_data}

In our first experiment, we seek to validate our approach in a controlled setting. For that, we verify that a known shortcut can be revealed by clustering a dataset's fixed-length representations.
As `known shortcut', we use the two-token rules synthetically introduced into Synthetic SST2. We know this pattern effectively acts as a shortcut, because the model achieves 100\% accuracy on these synthetic examples, regardless of the sentiment of the rest of the example.
To carry out a validation test, we compute all representations described in Sec.~\ref{sec:representations} for each data example from Synthetic SST2's test set that the model predicted as class 0. For each representation matrix thus obtained, we apply clustering to its rows, and cluster the data into 3 clusters.\footnote{The results hold for a larger numbers of 20 clusters, too (see Sec.~\ref{sec:appendix:clustering_configurations}).}
Will we find one cluster containing all (or most) of the 227 synthetic examples (i.e. the ones that contain both special terms)?

\paragraph{Metrics}
We compute \textbf{Precision} and \textbf{Recall} w.r.t. synthetic examples of all clusters.
To more easily compare the clusterings of different representations, we only report the metrics on the cluster that has the highest \textbf{Precision} for each representation (Tab.~\ref{tab:synthetic_representation_comparison}).

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Representation & Size & Prec. & Recall \\
    \midrule
    avg emb. & .36 & .35 & .62 \\
    PMI vocab & .00 & .50 & .00 \\
    \textsc{cls}-encoding & .20 & \textbf{1.00} & \textbf{1.00} \\
    \midrule
    salience vocab. G-L2 & .20 & \textbf{1.00} & \textbf{1.00} \\
    salience emb. G-L2 & .22 & 93 & \textbf{1.00} \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
}
    \label{tab:synthetic_representation_comparison}
\end{table}

The \emph{avg embedding} and \emph{PMI} baselines do not perform well: the best clusters have either low precision or a very small size.
For PMI, the cluster with the highest recall contains 226/227 synthetic examples, but has a low precision of only 22\% (not shown in table).
The \emph{\textsc{cls}-encoding} baseline performs very well, but as we show in the next sections, does not perform well across model understanding use cases.
The salience-based vocabulary representation based on Grad-L2 leads to a perfect cluster containing all the synthetic examples and no others.
This shows that by clustering model-based input representations, we can automatically discover lexical shortcuts.
Sec.~\ref{sec:appendix:clustering_configurations} has further results.

In the following three sections, we look at various case studies on real datasets that highlight how aggregating input salience helps a model developer in understanding their model and data better.

\section{Discovering Patterns in Real Data}\label{sec:problematic_patterns}

\begin{table}[t]
    \centering
    \begin{tabular}{l r l}
    \toprule
    ID &  \% &                           Top-5 Terms \\
    \midrule
    0  &        71 &  terrible, horrible, boring, recommend, dull \\
    1  &         6 &           awful, worst, avoid, bad, terrible \\
    2  &         8 &        worst, terrible, avoid, bad, horrible \\
    3  &         5 &                               10, 4, 3, /, 2 \\
    4  &         8 &      bad, horrible, acting, worst, recommend \\ 
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB: Clustering of a slice of IMDB test data using the salience vocabulary representation, along with cluster size (in \%) and the 5 terms that have highest mean salience in the respective cluster.}
    \label{tab:imdb_numeric_clusters}
\end{table}

Now that our approach has passed an initial validation test,
we turn to two common datasets to verify that clustering salience representations is helpful for identifying prominent patterns there.

\subsection{Spurious Correlations in IMDB}\label{sec:imdb_numeric_clusters}

In order to explore the patterns which lead the model to classify a movie review as negative, we 
sample 10\% of IMDB's test set, only selecting points that the model predicted to be \emph{negative}. For the ease of presentation we cluster their salience vocabulary representation into 5 clusters (similar results are obtained with different numbers of clusters, see Sec.~\ref{sec:appendix:clustering_configurations}).

Tab.~\ref{tab:imdb_numeric_clusters} lists the resulting clusters with their size and the respective top salient terms.\footnote{We removed BERT's special tokens, e.g., `[\textsc{cls}]', and punctuation.}
One cluster that immediately stands out is cluster 3 which contains almost exclusively numbers as top terms.
In Fig.~\ref{fig:overview} a) (see Sec.~\ref{sec:appendix:problematic_patterns} for full size) we also use word clouds to get an idea of what constitutes this cluster.

By focusing exclusively on the examples which mention these top terms we can easily discover a general pattern.
These IMDB reviews contain expressions such as \textit{`1 / 10 stars'}, \textit{`3 out of 10'}, \textit{`4 / 10'}, that summarize the review's rating and thereby give away the label. In such cases, the model does not need to analyze the rest of the review to predict that it is negative.\footnote{We repeat this analysis on a slice of positive reviews and find the same pattern for high ratings between 7 and 10.}
Interestingly, the same pattern was previously discovered by \citet{ross-etal-2021-explaining} and \citet{kaushik-etal-2020-learning}, who manually analysed IMDB examples. 
We refer the reader to \citet{ross-etal-2021-explaining} for the demonstration that BERT-like models indeed learn shallow reasoning patterns with the above listed ngrams and how one can construct counterfactuals which in turn reveal model vulnerabilities.

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Representation & Size & Prec. & Recall \\
    \midrule
    avg emb. & .08 & .16 & .13 \\
    PMI vocab & .16 & .12 & .19 \\
    \textsc{cls}-encoding & .47 & .12 & \textbf{.58} \\
    \midrule
    salience vocab. & .05 &  \textbf{.98} & .50 \\
    salience emb. & .05 & .93 & .53 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:imdb_numeric_representation_comparison}
\end{table}

By using a set of regular expressions, (see Sec.~\ref{sec:appendix:problematic_patterns}) we discovered that at least 12\% of IMDB reviews have such a numeric pattern, evenly distributed between train/test and between label 0/1.
In the sampled 2500 test examples with negative prediction, there are 245 such reviews.

Next, to compare the salience-based with the baseline representations, in Tab.~\ref{tab:imdb_numeric_representation_comparison} we again report Precision and Recall for the respective cluster with the highest precision.
The clusters obtained with the baseline representations fail to bring examples having a numeric pattern forward.
The PMI baseline performs poorly because PMI values for interesting tokens are not among the highest (e.g., `3' occurs in positive and negative reviews and therefore does not get a high PMI).

The salience-based representations, however, identify this subset well, producing clusters which almost exclusively consist of numeric examples and which contain more than half of all the negative examples with a numeric pattern
(see Sec.~\ref{sec:appendix:clustering_configurations} for further results).

An analysis of why not all of the numeric examples are clustered together revealed that most of the missing numeric examples contain either none or just 1 digit in their top-5 salient terms (i.e., the salience method did not put high weight on them) and are spread randomly across all other clusters.
This indicates that real data patterns are indeed more subtle and not deterministically indicative of a label, unlike the patterns that we used in the synthetic data experiment.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{imdb_numeric_clusters.png}
    \caption{Spurious Correlations in IMDB: t-SNE visualization of the salience vocab. representation of IMDB.
    Colored points are examples that contain a certain digit (indicated by the boxes):
    dark blue: 0, red: 1, pink: 2, turquoise: 3, green: 4, orange: 7.
    Other digits are not visible from this visualization angle.
    }
    \label{fig:imdb_numeric_clusters}
\end{figure}

To demonstrate that salience representations also enable more fine-grained analysis, in Fig.~\ref{fig:imdb_numeric_clusters} we show a t-SNE visualization of a 10k sample from the IMDB test set, where every colored dot represents an example with a specific digit.
One can easily see that many examples have digits in them and that they are clustered together. Thus, by increasing the number of clusters one can discover more fine-grained patterns.


\subsection{Patterns in Wikipedia Toxicity Subtypes}\label{sec:understanding_toxicity_data}
\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{redirect-points-comments.png}
    \caption{Patterns in Wikipedia Toxicity Subtypes: The top-left dense cluster of non-toxic comments (a fragment of the full clustering is shown) contains examples which all start with a \textit{`REDIRECT Talk:'} prefix. Such comments account for more than 1\% of the test set.}
    \label{fig:redirect}
\end{figure}

We now turn to Wikipedia Toxicity and look for patterns in the data that the model learned. It should be noted that the implications of having problematic patters in a toxicity dataset extend beyond robustness into the area of ML fairness. Imperfect annotations are inevitable \cite{wong-etal-2022-reproducible} but amplified by a sample bias may result in unfair treatment of certain demographic groups. 
For example, \textit{gay} is a term known to often trigger toxicity prediction \cite{dixon-etal-toxicity-2018}, even when it is not justified: \textit{`This is disrespectful to gay and all other people'} gets .85 probability of being toxic. 
Zooming into examples mentioning a particular token can give a more fine-grained idea of how a token relates to the classes.

We cluster all the 487 Toxicity test examples mentioning \textit{`gay'} using the salience vocabulary representation and t-SNE. Interestingly, we discovered clusters of comments with the non-toxic label which all had not only  \textit{`gay'} but also tokens like \textit{`homosexual', `(homo)sexuality', `LGBT'} among their top-5 most salient terms (Sec.~\ref{sec:appendix:learned_patterns}).
This suggests that contexts for \textit{`gay'} which talk about sexuality or sexual orientation with a neutral term are most often non-toxic.
This is an example of a more fine-grained analysis of how a term relates to the class.
An actionable conclusion is that if one wants to augment the data with more non-toxic examples of the token's occurrences, they should be different from the ones the model already knows to be non-toxic contexts.

Another example of a pattern present in the data and not obvious at first sight is given in Fig.~\ref{fig:redirect}.
It shows a small fragment of a t-SNE visualization of the salience vocabulary representation of 10k Toxicity examples. The top-left cluster consists of non-toxic comments (blue color). The nearest neighbors of a point in the center of the cluster reveals that all the examples there follow the template \textit{`REDIRECT Talk: \{Title\}'} which the model apparently uses as a strong indication of non-toxicity. Indeed, almost 1\% of the training and more than 1\% of the test split are all automatically generated comments of this kind and as such can hardly be useful for learning what differentiates toxic from non-toxic language or evaluating model quality. Thus, a simple visualization of aggregated salience reveals a weakness of the data.
The PMI baseline does not associate the two terms individually with the non-toxic class. Also the \textsc{cls} baseline does not place the \textit{`REDIRECT Talk: \{Title\}`} examples together.

\section{Identifying Model Sensitivities}\label{sec:model_sensitivity}
In this section, we investigate how sensitive models are to the \textbf{top terms} identified through aggregated salience.
A model is sensitive to a certain term if the sheer occurrence of the term means it is likely to classify a piece of text into a given class, without considering the entire context.
Knowing what the model is sensitive to may help discover spurious correlations or unfair biases.
For IMDB, \citet{ross-etal-2021-explaining} already showed that one can indeed create counterfactuals from the patterns like the ones discovered in the previous section (Sec.~\ref{sec:imdb_numeric_clusters}).
Now however, we turn our attention to the Toxicity dataset, where we wish to systematically measure model sensitivity.
For toxicity prediction, simplistic single-token patterns may result in problematic biases towards minorities with, e.g., identity terms becoming indicative of toxicity for a model \citep{DixLiSor+18n,hartvigsen-etal-2022-toxigen}, independently of their context.

Therefore, we want to analyze how sensitive the trained model is to various terms.
We measure this in two ways:
(1) average increase in probability towards the toxic label when inserting each of the terms in non-toxic text (counterfactuals),
(2) average decrease in probability towards the toxic label when masking each of the terms in toxic text.

\paragraph{Creating Counterfactuals}
Given a list of seed terms $\mathcal{S}$ that are deemed toxic, we create counterfactual examples in the following way:
We separately add every seed term $s \in \mathcal{S}$ at a random position into \(m = 10k\) random non-toxic test examples\footnote{We acknowledge that this may lead to disfluent or even ungrammatical inputs \cite{ross-etal-2021-explaining}.
This, however, is not a concern because in the Toxicity dataset, many examples are ungrammatical to begin with.} and measure the average change in probability for every seed term and every list of seed terms.
\begin{equation}
    change(\mathcal{S}) = \frac{1}{m|\mathcal{S}|}\sum_{s\in \mathcal{S}}\sum^m_i (p(x^{s}_{i}) - p(x_i))
\end{equation}

Two ways of creating baseline seed lists are considered:
(1) The identity terms from \citet{DixLiSor+18n}.
(2) The tokens with highest PMI(token; toxic class) that occurr at least 20 times in the training set.\footnote{Please note that \emph{PMI vocab} is the only baseline method that we can compute top terms for.}

Additionally, in order to get a seed list that is informed by aggregated input salience, we compute the column means of the salience vocabulary representation $R^{vocab}$ ($k=5$) of all toxic examples in the training set and keep the 10 highest valued entries.
Again, we keep only terms that occur at least 20 times.
The seed lists contains 10 terms, except for the list from \citet{DixLiSor+18n}, which contains 13.
The full list of all seed terms is shown in Sec.~\ref{sec:appendix:model_sensitivity}.

Results are listed in Tab.~\ref{tab:toxicity_counterfactual_results}.
Interestingly, the model does not seem sensitive to the list of identity terms from \emph{\citet{DixLiSor+18n}}.
On the other hand, the statistical correlation of term occurrence and label, as measured by \emph{PMI}, seems to be a good indicator of model sensitivity. However, the list of tokens we identified by looking at aggregated salience (\emph{aggregated salient terms}) leads to the most significant probability changes: on average the probability of an example being classified as toxic increases by 41\% simply by inserting a single term.

\begin{table}[t]
    \centering
    \begin{tabular}{lr}
    \toprule
    Seed List & Prob. Change \\
    \midrule
    \citet{DixLiSor+18n}    &      .06 (.08) \\
    PMI(token; toxic class) &      .31 (.19) \\
    \midrule
    aggregated salient terms  &  \textbf{.41} (.08) \\
    \bottomrule
    \end{tabular}
    \caption{Creating Counterfactuals: Mean probability increase when inserting specific terms into non-toxic examples. Numbers in parentheses are stddev when averaging all terms from the seed lists.}
    \label{tab:toxicity_counterfactual_results}
\end{table}


\paragraph{Masking out Toxic Terms}
In the previous part, we established that the probability of classifying an input as toxic increases when a seed term is inserted into the input.
If the model is truly sensitive to these seed terms, we would also expect a drop in probability when removing these terms from existing inputs.
To verify this, in the second experiment, we mask out the same seed terms from above, paralleling the comprehensiveness metric from \citet{deyoung-etal-2020-eraser} who use it to evaluate the quality of a salience method. 

The process of creating masked examples is as follows:
we use all 8,858 toxic test examples and the same seed lists as above.
For every seed term we replace all the occurrences of it with `[\textsc{mask}]' and compare probabilities before and after.

\begin{table}[t]
    \centering
    \begin{tabular}{lrr}
    \toprule
    Seed List & Count & Prob. Change \\
    \midrule
    \citet{DixLiSor+18n}    &    55 &  -.12 (.15) \\
    PMI(token; toxic class) &   279 &  -.22 (.10) \\
    \midrule
    aggregated salient terms  &   \textbf{475} &  \textbf{-.34} (.08) \\
    \bottomrule
    \end{tabular}
    \caption{Masking out Toxic Terms: Mean probability decrease when masking specific terms in toxic examples. Numbers in parentheses are stddev when averaging all terms from the seed lists. Count is the number of occurrences of all seed terms.}
    \label{tab:toxicity_masking_results}
\end{table}

Results are reported in Tab.~\ref{tab:toxicity_masking_results}.
Both non-salience baseline methods (\emph{\citet{DixLiSor+18n}} and \emph{PMI}) lead to lower probability changes.
Again, the probability change for \emph{aggregated salient terms} is the strongest (average 34\%).
In addition, \emph{aggregated salient terms} presents a list whose entries occur more often than terms from the other lists, meaning these are frequent \emph{and} important for the model.

In summary, aggregated salience information provides us with a list of terms that the model is sensitive to.
No other method leads to a comparable increase in toxicity probability when adding terms and decrease in toxicity when terms are masked out.


\section{Generalizing from Single Examples}\label{sec:single_example_generalization}

\begin{table*}[t]
\centering
\begin{tabular*}{\textwidth}{p{.1cm} p{9.1cm} l l}
    \toprule
    \multicolumn{2}{l}{Misclassified Examples and their 3 Nearest Neighbors} & 
    Top-5 Terms &
    Label / Prediction \\
    \toprule
    
    \multicolumn{2}{l}{`` : : : fine with me . | \textbf{ck} ''} &
    \textbf{ck}, [\textsc{sep}], |, [\textsc{cls}], . &
    non-toxic / toxic \\
    
    \midrule
    
    1. & co / \textbf{ck} - pen - i . s . international &
    \textbf{ck}, /, [\textsc{sep}], co, [\textsc{cls}] &
    toxic / toxic \\
    
    2. & su : ) \textbf{ck} my ba : ) lls you fa : ) g : ) got loser &
    \textbf{ck}, loser, [\textsc{sep}], [\textsc{cls}], ) &
    toxic / toxic \\
    
    3. & block me ! ! ! see how fast another post comes du . mbfu . \textbf{ck} wikipedia is the devel oh , and they cann also fine wiki for monopolizing online encyclopedias &
    \textbf{ck}, ., wikipedia, mb, \#\#fu &
    toxic / toxic \\
    
    \midrule\midrule
    
    \multicolumn{2}{l}{dogs \textbf{pee} more than they poo} &
    \textbf{pee}, dogs, [\textsc{cls}], po, [\textsc{sep}] &
    non-toxic / toxic \\
    
    \midrule
    
    1. & i am going to \textbf{pee} on you ! &
    \textbf{pee}, on, !, [\textsc{cls}], [\textsc{sep}] &
    toxic / toxic \\
    
    2. & \textbf{pee}eeeeeeeeniiiiiiiiiiiiisssss ! ! ! ! ! ! ! ! still here , waiting for a response . . . . . &
    \textbf{pee}, \#\#ee, \#\#iss, [\textsc{cls}], [\textsc{sep}] &
    toxic / non-toxic \\
    
    3. & how big is your \textbf{pee} \textbf{pee} . &
    \textbf{pee}, your, big, [\textsc{cls}], [\textsc{sep}] &
    toxic / toxic \\
    
    
    \midrule\midrule
    
    \multicolumn{2}{l}{kobe is the best \textbf{fuckin} player ever} &
    \textbf{fuck}, kobe, player, \textbf{\#\#in}, best &
    non-toxic / toxic \\
    
    \midrule
    
    1. & guys , it ' s just simply me that put there moldovans ( romanians ) 78 , 2 \% , and i was not logged in , and don ' t give quickly the fault on bonaparte , he ' s gone from longtime now from this bullshit of wikipedia , ' cause there ' s no purpose with hypocritic people like you . i modified that , everybody knows that that ' s moldovans and romanians are the same shit , anyhow the census will say it . they speak romanian , and not moldovan how they sustain . a romanian understands perfectly when a moldovan is speaking it ' s \textbf{fuckin} ' language &
    \textbf{fuck}, \textbf{\#\#in}, guys, s, longtime &
    toxic / toxic \\
    
    2. & haha this is my \textbf{fuckin} page ! ! ! ! ! ! ! ! ! i can do whatever i want ! ! ! ! ! ! &
    \textbf{fuck}, \textbf{\#\#in}, [\textsc{cls}], page, my &
    toxic / toxic \\
    
    3. & british people , or britons , [ 7 ] are inhabitants of great britain [ 8 ] [ 9 ] or citizens of the united kingdom . i dont think this is about an ethnicity / race wobbs . haha , yous a complete \textbf{fuckin} moron . &
    \textbf{fuck}, \textbf{\#\#in}, mor, [\textsc{sep}], complete &
    toxic / toxic \\
    
    \bottomrule

\end{tabular*}
\caption{Generalizing from Single Examples: 3 examples having surprising predictions with their top-5 terms and 3 nearest neighbors.
Bold terms are overlapping in example and neighbors.
They give a possible explanation why the examples were predicted to be toxic.}
\label{tab:toxicity:bottom_up}
\end{table*}

The approaches outlined in Sec.~\ref{sec:problematic_patterns} and \ref{sec:model_sensitivity} can be characterized as ``top-down'': they use salience-based representations to perform clustering or term selection on a large number of examples, then zoom into single clusters/examples to identify relevant patterns.
In this section however, we take an inverse ``bottom-up'' approach: starting from one single data instance, we search for its nearest neighbors in representation-space in order to uncover explanations for the way the model classified it.
We start from mis-classified examples of the Toxicity dataset and use the vocabulary representation keeping the top-$5$ salient tokens. 
While in the experiments described so far the choice of $k$ did not make a difference and comparable results were obtained with $k = |\mathcal{V}|$,
in this nearest neighbor experiment we got better results for $k=5$. This is intuitive, because we want the similarity between an example in question and its nearest neighbors to be dominated by the high-salience tokens they have in common and not by the long low-salience tail.

Tab.~\ref{tab:toxicity:bottom_up} shows three examples of surprising classifications made by the model -- examples which a model developer might want to inspect.
Why might the model have classified the comment \textit{` : : : fine with me . | ck'} as toxic? Looking at the nearest neighbors reveals that \textit{`ck'} often appears in the context of the swear words \textit{`f * ck'}, \textit{`s * ck'} and \textit{`c * ck'}, providing an explanation for why the model might have learned it to be toxic.

The example of \textit{`dogs pee more than they poo'} shows that the model may be sensitive to the term `pee', without considering the context.
Finally, though the comment \textit{`kobe is the best fuckin player ever'} is far from toxic, its nearest neighbors bring many toxic uses of the word \textit{`fuckin'} to light, giving an insight into why the model has classified it as toxic.

In order to verify the utility of our method in this use case, we focus on misclassified examples in the Toxicity test set.
From the slice of misclassified data, we sample 200 examples and identify that 28 (14\%) of them are surprising, meaning the prediction cannot be understood just by looking at the example's text.\footnote{By contrast, Sec.~\ref{sec:appendix:single_example_generalization} lists some unsurprising examples for the purposes of illustration.}
For each surprising example, we retrieve its 5 nearest neighbors in representation space among the examples of the training set that have the same label as the example's prediction. We search for a rationale for the misclassification using the nearest neighbors list.
We show 3 examples of this analysis in Tab.~\ref{tab:toxicity:bottom_up}.
Among the 28 examples, we are able to uncover rationales for the misclassification in 22 cases using the vocabulary-based saliency representations, while for the baseline methods average-of-embeddings, PMI and \textsc{cls}-encoding, we find rationales in only 9, 20 and 6 of cases respectively.
This shows that aggregated salience is helpful in gaining a deeper understanding of model predictions.

\section{Related Work}

\paragraph{Explanation-based Human Debugging with Global Explanations}
Our work falls into the category of explanation-based human debugging~\citep{LerTon21a}.
Since most works in this field propose \textit{local} explanations~\citep{danilevsky-etal-2020-survey,LerTon21a}, our method contributes to an understudied type of explanation by aggregating local explanations into \textit{global} explanations.
The closest related work is FIND~\citep{LerSpeTon20b}, a framework that uses layer-wise relevance propagation (LRP)~\citep{ArrHorMon+16a} to derive feature weights for all examples, which are then converted into word clouds for users to interact with.
They focus on disabling features globally, whereas the strength of our approach is that it allows debugging individual examples as well as entire datasets.

\paragraph{Model Understanding}
Our work concerns identification of (shallow) patterns learned by text classifiers as well as the explanation of individual examples.
This should help model developers get a better understanding of their model and give them indicators of how to improve them.
With that, the presented methods are well suited to be included into existing model understanding tools \citep{NorJenKoc+19v,WalTuyWan+19o,KokMigMar+20o,tenney-etal-2020-language,GevCacDar+22s}.
Interactive debugging based on input salience gives a model developer another view on top of static descriptions by the means of, e.g., model cards \citep{MitWuZal+19j} or data cards \cite{PusZalKja22l}.
Additionally, it complements approaches like CheckList \cite{ribeiro-etal-2020-beyond}, which also has the goal of improving NLP models by proactively identifying their weaknesses.

\paragraph{Salience and Clustering}
In vision research, clustering and salience have been applied to find spurious patterns \citep{bach-etal-2015,lapuschkin-2019-spray,schoop-etal-2022-imacs}.

\citet{yin-neubig-2022-interpreting} cluster contrastive explanations to describe why a language model chose a certain token over another.
\citet{KauEsdRuf+22a} cluster text data, then replace the clustering with an equivalent neural network, which is finally used to generate individual explanations.

\paragraph{Training Data Attribution}
The kNN analysis (Sec.~\ref{sec:single_example_generalization}) is similar to proponents identification done with training data attribution methods (TDA) \citep{zylberajch-etal-2021-hildif,han-tsvetkov-2021-influence-tuning}.
We do not perform a comparison with TDA methods because
(1) they focus on analyzing the training set while we are primarily interested in understanding (the model performance on) the test set.
(2) TDA methods suggest proponents but leave it to the developer to discover what it is that the proponents and the test example share (unless combined with salience techniques as done in \citealt{pezeshkpour-etal-2021-combining}).
Unlike that, salience-based distance is explicit about what every token contributes to the similarity. 

\section{Conclusion}
In this work, we present a way to use aggregated salience representations to meet distinct yet common model developer needs.
The proposed methods help find prominent patterns and gain insights into single examples or entire datasets.
Through a series of case studies carried out on three datasets (1 synthetically-modified, 2 academic), we show that our approach (semi-)automatically provides explanations through clustering, identifies shortcut-like patterns, uncovers patterns the model is particularly sensitive to and explains predictions through nearest neighbors.
Moreover, we try out two ways of aggregating salience maps (vocabulary-based and embedding-based) and show that our method performs better than baseline representations across all use cases presented.

\section{Limitations}
Though the results described in this paper are promising, they are limited in the types of patterns they can recognize.
Both salience-based representations (vocabulary-based and embedding-based) are order-agnostic representations.
Therefore, any method using these will not be able to detect patterns that depend on word order.
Additionally, they will not be able to identify patterns related to example length, token position in the example, as well as more complex patterns such as ones where only a certain part of the input is decisive for the model (e.g., hypothesis-only shortcuts in NLI).

We run all experiments in this paper with a BERT-based model, and although we choose Grad-L2 as input salience method (because it was shown to work well for this model), any salience method is compatible with our approach, as long as it produces a single salience vector per example.

All experiments presented in this work are binary text classification tasks.
In principle however, our method can be extend to multi-class, regression or text generation tasks (e.g., summarization, question answering), as long as a suitable input salience method exists for the task.

Finally, in this work, we devote our attention to better understanding a model and dataset. Our method does not directly provide robustness fixes, although identifying the model and dataset's weaknesses are a necessary first step and sometimes lead to obvious remediation techniques such as data rebalancing.

\newpage

