\section{Attention Flow}
\label{app:attention_flow}

Despite \emph{attention flow}, Grad-SHAP, and Deep-SHAP all (supposedly) being valid Shapley Value explanations \cite{ethayarajh-jurafsky-2021-attention}, Table \ref{tab:attn_flow_apx} shows that the agreement is low.

\input{tables/appendix/attn_flow}

\section{Result tables with standard deviation}
\label{app:results-with-std-dev}

In Tables \ref{app:distilbert} and \ref{app:BILSTM} we provide the same results as in Tables \ref{tab:distilbert_results} and \ref{tab:bilstm_results} of the main paper, but with the standard deviation for each experiment.

\input{tables/appendix/distilbert_correlation_table}
\input{tables/appendix/bilstm_agreement_table}

\section{Model Performance}
\label{app:performance}

We include a uniform activation baseline to contextualize the attention mechanism's utility. Table \ref{tab:accuracy} notes the gap between the performance of uniform and softmax attention in the BiLSTM is never higher than 1-2\%. This distinction aligns with the results of \citet{wiegreffe-pinter-2019-attention} and \citet{vashishth2019attention}, who argue claims of attention-based interpretability are stronger in situations in which models need the attention module to solve the underlying task. Of course, it is difficult to prove a causal effect in a deep network: the BiLSTM may solve the task differently depending on whether or not the attention mechanism is ablated, meaning it is still possible to make claims of interpretability whether or not softmax attention leads to higher task performance. We again emphasize that we do not wish to take a side in the ``attention explanation" debate, but this point further stresses the difficulty of proving anything with the \emph{agreement as evaluation} paradigm.

\input{tables/appendix/accuracy}

\section{Reproducibility}
\label{app:reproducibility}

Our code is publicly available at \url{https://github.com/sfschouten/court-of-xai}.
We conducted our experiments on Amazon Web Services \texttt{g4dn.xlarge} EC2 instances using an NVIDIA T4 GPU with 16GB of RAM. The version of PyTorch was \texttt{1.6.0+cu101}.
We refer to Table \ref{tab:runtime} for the average time to train each model on each dataset.

The DistilBERT model contained 66955779 trainable parameters and the BiLSTM model contained 12553519 trainable parameters, as reported by the AllenNLP library \cite{gardner-etal-2018-allennlp}.
Table \ref{tab:dataset_statistics} lists the number of instances in each split of each dataset and Table \ref{tab:val_accuracy} details the accuracy of our models on the validation sets during training.

Links to download versions of all datasets are included in our code repository. For posterity, links to all datasets are listed here: 
\begin{itemize}
	\item \textbf{SST-2}: \url{https://github.com/successar/AttentionExplanation/tree/master/preprocess/SST}
	\item \textbf{IMDb}: \url{https://github.com/successar/AttentionExplanation/tree/master/preprocess/IMDB}
	\item \textbf{SNLI}: \url{https://nlp.stanford.edu/projects/snli/}
	\item \textbf{MNLI}: \url{https://cims.nyu.edu/~sbowman/multinli/}
	\item \textbf{XNLI}: \url{https://cims.nyu.edu/~sbowman/xnli/}
	\item \textbf{Quora Question Pair}: \url{https://drive.google.com/file/d/12b-cq6D45U5c-McPoq2wsFjzs6QduY_y/view?usp=sharing}
\end{itemize}

\input{tables/appendix/runtime}
\input{tables/appendix/dataset_statistics}
\input{tables/appendix/validation_accuracy}




\section{Introduction} \label{sec:introduction}

As machine learning (ML) models are increasingly used in hybrid settings to make consequential decisions for humans, criteria for plausible and faithful explanations of their predictions remain speculative \citep{lipton2016mythos, jacovi-goldberg-2020-towards}.
Although there are many possible explanations for a model's decision, only those faithful to both the model's reasoning process and to human stakeholders are desirable \cite{jacovi-goldberg-2020-towards}. 
The rest are irrelevant in the best case and harmful in the worst, particularly in critical domains such as law \cite{Kehl2017AlgorithmsIT}, finance \cite{grath_interpretable_2018}, and medicine \cite{caruana2015intelligble}.

Content moderation is a use case where explanations can help domain experts as part of a hybrid human-in-the-loop system. 
Consider an ML model that predicts whether or not a post on a social media contains misinformation: when a post is automatically removed by the model and the user who created it appeals its removal, content moderators need to read through the entire post to identify why it was flagged as containing misinformation \cite{halevy_preserving_2022}. 
Including explanations that identify which parts of the post are problematic can help content moderators decide if the model was correctly flagging the post. 

ML practitioners frequently explain models by calculating each input's contribution toward an individual prediction.
Additivity --- treating all contributions as independent and quantifiable --- is a common simplifying assumption. 
In this work, we focus on such additive explanation methods and refer to them as \emph{feature attribution methods}. 
We denote the contribution of each input toward the model's decision as its \emph{importance}. We say that two different feature attribution methods \emph{agree} if there is a strong correlation between their computed rankings of input importance.

The attention mechanism \cite{bahdanau2014neural} in natural language processing (NLP) is a popular, albeit less rigorously motivated, way of obtaining explanations. Because the mechanism produces context vectors from which decoders can soft-search for prediction-relevant information, the weights assigned to inputs intuitively serve as proxies for their overall contribution towards a decision. Weights are often visualized as heatmaps over sequences \cite[e.g.,][]{pmlr-v37-xuc15,li-etal-2016-visualizing}, which can be particularly persuasive when examples are (unintentionally) cherry-picked to fit a narrative. 
We define an attention-based explanation as a vector of attention weights that, similar to a feature attribution method, can be treated as a ranking of importance.

In their critique of attention-based explanations for NLP, \citet{jain-wallace-2019-attention} argue that faithful attention-based explanations must be highly \emph{agreeable}.\footnote{\citet{ethayarajh-jurafsky-2021-attention} use the term \emph{consistent}.} That is, their generated rankings of input importance must correlate with existing feature attribution methods. 
Following \citet{jain-wallace-2019-attention} and their claim that ``attention is not explanation'', several recent papers have presented an increased agreement with a small set of feature attribution methods as evidence for their proposed method's ability to improve the faithfulness of the attention mechanism. For example, \citet{mohankumar-etal-2020-towards} show that minimizing hidden state conicity in a BiLSTM improves the Pearson correlation of attention weights with Integrated Gradients \cite{intgrad} attributions. As the popularity of \emph{agreement as evaluation} grows \citep{meister-etal-2021-sparse,abnar-zuidema-2020-quantifying}, we believe it is important to investigate diagnostic capacity of agreement as a metric by examining (i) more recent feature attribution methods, and (ii) more complex transformer-based models. 

Under the paradigm of \emph{agreement as evaluation}, new explanation methods (i.e., attention-based) are compared to established explanation method(s) (i.e., feature attributions). 
However, can any one explanation method act as the standard against which other explanation methods are evaluated?
Explanations are task-, model-, and context-specific \cite{doshi-velez_towards_2017}, and the performance of explanation methods depends on the particular diagnostic tests considered \citep{deyoung-etal-2020-eraser,Robnik-sikonja2018}. 

In this work, our main research question is: \emph{How well do attention-based explanations correlate with recent feature attribution methods for NLP tasks?} Specifically, we want to investigate: 
\begin{itemize}
	\item \textbf{RQ1:} Does the correlation depend on the model architecture (transformer- vs. LSTM-based)?
	\item \textbf{RQ2:} Does the correlation depend on the nature of the classification task (single- and pair-sequence)? 
\end{itemize}

We investigate the following feature attribution methods: {LIME} \cite{lime}, {Integrated Gradients} \cite{intgrad}, {DeepLIFT} \cite{deeplift}, and two versions of SHAP \cite{shap}: 
{Grad-SHAP} (based on Integrated Gradients) and {Deep-SHAP} (based on DeepLIFT). 
We observe low agreement between attention-based explanations and feature attribution methods, across both models and both tasks. 
We also observe low agreement across all explanation methods for the transformer-based model and for pair-sequence tasks. 
We use this empirical evidence, along with our theoretical objections, to argue that practitioners should refrain from evaluating attention-based explanations based on their agreement with feature attribution methods.

\section{Related Work}

\citet{jain-wallace-2019-attention} introduced the \emph{agreement as evaluation} paradigm by comparing attention-based explanations with simple feature attribution methods using a bidirectional LSTM \cite{lstm} on single- and pair-sequence classification tasks. 
They conclude that ``attention is not explanation'' due to the weak correlation between the rankings of input token importance obtained from attention weights and those obtained from two elementary feature attribution methods: (i) input $\times$ gradient \citep{inputxgradients1, inputxgradients2}, and (ii) leave-one-out \cite{li2017understanding}. 
In our work, we test the generalizability of agreement as an evaluation metric by (i) testing on a more comprehensive set of feature attribution methods (see Section~\ref{section:xai-methods}), and (ii) testing on a transformer-based \cite{vaswani2017attention} model on the same types of tasks (see Section~\ref{section:models}). 

The influential critique by \citet{jain-wallace-2019-attention} has sparked an ongoing debate about whether or not attention is explanation \citep{wiegreffe-pinter-2019-attention, serrano-smith-2019-attention, grimsley-etal-2020-attention}. 
More recently, \citet{bastings-filippova-2020-elephant} have questioned the notion of ``attention as explanation'' as a whole, and suggest that in order to explain ML model predictions, the community should rely on methods that are explicitly created for this purpose (i.e., feature attributions), instead of seeking explanations from attention mechanisms. 
In our work, we do not aim to take a position on the ``is attention explanation'' debate, but rather investigate the hypothesis that in order for attention mechanisms to be considered ``explanations'', they must correlate with existing feature attribution methods. This is an underlying assumption of not only the work by \citet{jain-wallace-2019-attention}, but also that of \citet{meister-etal-2021-sparse}, who show that inducing sparsity in the attention distribution decreases agreement with feature attribution methods, and \citet{abnar-zuidema-2020-quantifying}, who demonstrate their \emph{attention-flow} algorithm improves the correlation with attributions based on feature ablation. 

\citet{atanasova-etal-2020-diagnostic} introduce a series of diagnostic tests to evaluate feature attribution methods for text classification. 
They show that the performance of feature attribution methods, measured by using these diagnostic tests, largely depends on the model and task considered, but note that gradient-based methods tend to perform the best. 
Similar to their work, we also compare and evaluate feature attribution methods, but only to investigate the suitability of \textit{agreement as evaluation}, not to determine a winning explanation method given several diagnostic tests. 
In contrast, we evaluate five feature attribution methods on agreement as evaluation based on only the rank correlation using Kendall's-$\tau$, following \citet{jain-wallace-2019-attention}.

\citet{prasad-etal-2021-extent} define three alignment metrics that quantify how well human-annotated natural language explanations align with the explanations generated by the Integrated Gradients method \citep{intgrad}. 
They find that the BERT \cite{devlin-etal-2019-bert} model has the highest alignment with human-annotated explanations. 
However, unlike our work, this work focuses exclusively on using transformer-based models and Integrated Gradients, and does not question the \emph{agreement as evaluation} paradigm for feature attribution methods. 

\citet{ding-koehn-2021-evaluating} introduce a human-annotated benchmark to evaluate feature attribution methods for NLP models. 
They test two attribution methods on three types of NLP models and find that explanations from feature attribution methods are sensitive to changes in model configuration. 
Similarly, we test five attribution methods on two types of NLP models, but our focus is on investigating the \emph{agreement as evaluation} paradigm, whereas \citet{ding-koehn-2021-evaluating} focus on investigating the correlation between feature attribution methods and a human-annotated benchmark. 
We are unable to incorporate the benchmark proposed by \citet{ding-koehn-2021-evaluating} in our work because the human annotations are not present for every single token, and therefore we cannot turn them into a ranking. 

This work builds off our prior ICML workshop paper \cite{neely2021order}, which several other papers have extended. \citet{feldhus-etal-2021-thermostat} introduce a software package to analyze instance-wise explanations for popular NLP models and tasks and, in doing so, partially reproduce one of our experiments. \citet{krishna_disagreement_2022} formally define and highlight the importance of the "disagreement problem" between feature attribution methods, which they find is a constant frustration for ML practitioners. They introduce metrics to capture the disagreement between top-k features and ``features of interest'' (e.g., those selected by an end-user) and find considerable disagreement between feature attribution methods for tabular, text, and image data on both real-world and research datasets.

\section{Explainability in NLP} \label{sec:method}
In this work we investigate two types of explanations for NLP tasks: (i) those from recent feature attribution methods in the \ac{XAI} literature and (ii) those based on attention scores. We evaluate on transformer- and LSTM-based models (\textbf{RQ1}), for both single- and pair-sequence tasks (\textbf{RQ2}). 
Following \citet{jain-wallace-2019-attention}, we define an \emph{explanation} of an input token sequence as a vector comprised of an importance score for each token. 
This vector can be used to rank the tokens by their importance score.

\subsection{Explanations from Feature Attributions} \label{ssec:recent_methods}
\label{section:xai-methods}

In our experiments, we focus on five recent feature attribution methods: {LIME} \cite{lime}, {Integrated Gradients} \cite{intgrad}, {DeepLIFT} \cite{deeplift}, and two versions of SHAP \cite{shap}: 
{Grad-SHAP} and {Deep-SHAP}:

\begin{itemize}
	\item LIME \cite{lime} produces locally faithful explanations by learning an interpretable (e.g., linear) model from samples weighted by their proximity to the original instance.
	\item Integrated Gradients \cite{intgrad} calculates input feature attributions by accumulating the gradients obtained from the model along the straight-line path from a baseline to the original instance.
	\item DeepLIFT \cite{deeplift} also produces input feature attributions using the gradients, but it assigns scores based on the difference between a reference activation and the activation of the original instance. This allows the calculated contributions to remain non-zero even when the gradients are zero. 
	\item SHAP \cite{shap} identifies a unique solution for the contribution of each input toward the prediction. Since this is computationally expensive, \citet{shap} propose approximations based on existing methods: 
	\begin{itemize}
		\item[$\circ$] Grad-SHAP (based on Integrated Gradients). 
		\item[$\circ$] Deep-SHAP (based on DeepLIFT).
	\end{itemize}
\end{itemize} 

\subsection{Explanations from Attention Mechanisms} \label{ssec:attn_methods}

Given an input sequence of tokens $\textbf{S} = (t_{1}, ..., t_{n})$, we define an \emph{attention-based explanation} as an assignment of attention weights $\boldsymbol{\alpha} \in \mathbb{R}^{n}$ to the tokens in $\textbf{S}$ \citep{jain-wallace-2019-attention}. 
Since the dimensionality of $\boldsymbol{\alpha}$ is architecture-dependent, it may be necessary to filter or aggregate the weights. In our experiments, this is only relevant for the self-attention mechanism in the transformer-based model we consider below (see Section~\ref{section:models}). 

Previous analyses at the attention head level implicitly assume that contextual word embeddings remain tied to their corresponding tokens across self-attention layers \citep{baan2019understanding, clark-etal-2019-bert}. 
This assumption may not hold in transformer-based models since information mixes across layers \cite{brunner2019identifiability}. 
Therefore, we use the \emph{attention rollout} \cite{abnar-zuidema-2020-quantifying} method --- which assumes the identities of tokens are linearly combined through the self-attention layers based exclusively on attention weights --- to calculate post-hoc, token-level importance scores.%
\footnote{We also experimented with \textit{attention flow} \cite{abnar-zuidema-2020-quantifying}, see Appendix \ref{app:attention_flow}.}
Following \citet{abnar-zuidema-2020-quantifying}, we use the scores calculated for the last layer's [CLS] token, resulting in a final vector $\boldsymbol{\alpha} \in \mathbb{R}^{n}$ at the time of evaluation.

Recurrent models similarly suffer from issues of identifiability. In LSTM-based models, attention is computed over hidden representations across timesteps, which does not provide faithful token-level importance scores. Approaches that trace explanations back to individual timesteps \cite{bento_timeshap_2021} or input tokens \cite{tutek-snajder-2020-staying} are only just emerging. 
Therefore, we analyze the raw attention weights for the LSTM-based model we consider below (see Section~\ref{section:models}).

\subsection{Agreement between Explanations}

Following \citet{jain-wallace-2019-attention}, we measure \emph{agreement} between the explanation methods as the mean Kendall-$\tau$ correlation \cite{kendall-tau} between the ranked importance scores of all input tokens, across all examples.
\footnote{We also calculated Spearman and Pearson correlations and obtained similar results.}
The Kendall-$\tau$ correlation is a widely used metric for comparing ranked lists; it measures the correlation between two ranked lists based on discordant pairs between the lists. Two items are considered \emph{discordant} if they are ranked differently on the two lists. 
The Kendall-$\tau$ correlation can take on values in the $[-1, 1]$ interval, where negative values imply the rankings are negatively correlated while positive values imply the rankings are positively correlated \cite{kendall-tau}. 

\section{Experimental Setup} \label{sec:experiments}

\subsection{Datasets} \label{sec:datasets}

We evaluate on two types of NLP classification tasks: 
\begin{enumerate*}[label=(\roman*)]
	\item single-sequence, and
	\item pair-sequence. 
\end{enumerate*}
For the single-sequence task, we perform binary sentiment classification on the Stanford Sentiment Treebank (SST-2) \cite{socher-etal-2013-parsing} and the IMDb Large Movie Reviews Corpus (IMDb) \cite{maas-etal-2011-learning}. 
We use identical splits and pre-processing as \citet{jain-wallace-2019-attention}, but also remove sequences longer than 240 tokens for faster attribution calculation. This leaves us with roughly 78\% of the original instances in the IMDb dataset.

\begin{itemize}
	\item The SST-2 dataset consists of single sentences extracted from movie reviews and is used for binary sentiment analysis: predicting whether the review is positive or negative. 
	\item The IMDb dataset consists of movie reviews as well, but contains longer sequences compared to SST-2. It is also used for binary sentiment analysis.
\end{itemize}

For the pair-sequence task, we examine natural language inference and understanding with the Multi Natural Language Inference (MNLI) corpus \cite{williams-etal-2020-predicting}, the Stanford Natural Language Inference (SNLI) corpus \cite{bowman-etal-2015-large}, and the Quora Question Pairs dataset \cite{quora}. 

\begin{itemize}
	\item The MNLI dataset contains sentence pairs for a textual entailment task: given a pair of sentences, we want to predict whether or not one sentence implies the other. Since MNLI has no publicly available test set, we use the English subset of the XNLI \cite{conneau2018xnli} test set. 
	\item The SNLI dataset~\cite{bowman-etal-2015-large} contains pairs of sentences and is used for the textual entailment task, similar to MNLI. 
	\item The Quora Question Pairs dataset \cite{quora} contains pairs of questions from the Quora website, where the task is to classify if the two questions in a pair are duplicates or not. We use a custom split (80/10/10) for the Quora dataset, removing pairs with a combined count of 200 or more tokens (leaving 99.99\% of the original instances).
\end{itemize}

\subsection{Models}
\label{section:models} 

We test two types of NLP models: 
\begin{enumerate*}[label=(\roman*)]
	\item transformer-based, and
	\item LSTM-based. 
\end{enumerate*}
The transformer-based model \cite{vaswani2017attention} relies on a self-attention mechanism to calculate representations for tokens.  
For each layer in the transformer network, the representation of each token is updated by computing a weighted sum over all tokens represented in the entire sequence, together with a non-linear feedforward transformation. 
The weight value of each token is determined using self-attention, which computes a similarity score for each pair of tokens in the sequence. 
For every token in every layer of the network, an attention layer is used. 
As a result, it is non-trivial to aggregate all the importance values per layer into one interpretable importance score per token. 

In our work, we follow \citet{abnar-zuidema-2020-quantifying} by using the attention rollout scores for [CLS] tokens. 
For the transformer-based model, we fine-tune the lighter, pre-trained DistilBERT variant \cite{sanh_distilbert_2019} instead of the full BERT model \cite{devlin-etal-2019-bert} to reduce the computational overhead and ecological footprint. For classification, we add a linear layer on top of the pooled output. We concatenate pair-sequences with a [SEP] token.

Unlike a transformer-based model, an LSTM-based model processes input tokens in sequential order. 
For each token, $t \in \{1, \dots, n\}$, in the sequence, the global representation of the sequence, $\textbf{h}_{t}$, is updated using a non-linear transformation with an embedded representation of 
\begin{inparaenum}[(i)]
	\item the current token in the sequence, $t_{n}$, and 
	\item the previous global representation of the sequence, $\textbf{h}_{t-1}$, as input.
\end{inparaenum}
A bidirectional LSTM (Bi-LSTM) uses two stacked LSTMs to process tokens in both directions of the sequence. 

For the Bi-LSTM, we use the same single-layered bidirectional encoder with the query-less additive $(tanh)$ attention and linear feedforward decoder as in the work of \citet{jain-wallace-2019-attention}. 
As a result, the attention weight for each token in the sequence is solely based on its representation $\textbf{h}_{t}$, and not on a query item.
In pair-sequence tasks, we embed, encode, and induce attention over each sequence separately. 
The decoder predicts the label from the concatenation of 
\begin{inparaenum}[(i)]
	\item both context vectors $c_{1}$ and $c_{2}$, 
	\item their absolute difference $|c_{1} - c_{2}|$, and
	\item their element-wise product $c_{1} \odot c_{2}$.
\end{inparaenum}

\subsection{Training the models}

We train three independently-seeded instances of each of the models described in Section~\ref{section:models} using the AllenNLP framework \cite{gardner-etal-2018-allennlp}, each for a maximum of 40 epochs. We use a patience value of 5 epochs for early stopping. 
For DistilBERT, we fine-tune the standard ``base-uncased'' weights available in the HuggingFace library \cite{Wolf2019HuggingFacesTS} with the AdamW \cite{loshchilov2019decoupled} optimizer. 
For the BiLSTM, we follow \citet{jain-wallace-2019-attention} and select a 128-dimensional encoder hidden state with a 300-dimensional embedding layer. We tune pre-trained FastText embeddings \cite{bojanowski-etal-2017-enriching} and optimize with the AMSGrad variant \cite{Tran_2019} of Adam \cite{kingma_dp_adam_2015}. Appendix \ref{app:performance} details model performance and indicates both the BiLSTM and DistillBERT are sufficiently accurate for our analysis.

\subsection{Explaining the models}

We leverage the Captum\footnote{\url{https://github.com/pytorch/captum}} implementations of LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, and Deep-SHAP, and use the padding token as a baseline where applicable. For LIME, we mask tokens as features and use 1000 samples to train the interpretable models. 
We apply our feature attribution methods to the predictions of each independently-seeded model for 500 instances randomly sampled from each test set.

Our code is publicly available.\footnote{\url{https://github.com/sfschouten/court-of-xai}} Refer to Appendix \ref{app:reproducibility} for more information on reproducing our experiments.

\section{Results}\label{sec:results}


In this section, we answer our main research question: \emph{How well do attention-based explanations correlate with recent feature attribution methods for NLP tasks?} 
In general, we find that attention-based explanations do not correlate strongly with feature attribution methods, with some exceptions (see Section~\ref{sec:results2} and~\ref{sec:results3}). 

\subsection{RQ1: Does the correlation depend on the model architecture?}
\label{sec:results1}
Tables~\ref{tab:distilbert_results} and~\ref{tab:bilstm_results} display the average\footnote{Across the 3 model instances, randomly selecting 500 instances from the test set using the training seed.}  Kendall-$\tau$ correlations between the explanation methods for the DistilBERT and BiLSTM models, respectively.\footnote{See Appendix \ref{app:results-with-std-dev} for the same tables including the standard deviation.}
A stronger correlation (i.e., agreement) is indicated by a darker blue colour in the table cell. 
In general, we see that the agreement between explanation methods is substantially lower for the DistilBERT model than for the BiLSTM model. 

\input{tables/results/DistlBERT}

For the DistilBERT model, we observe a weak correlation across all explanations -- almost none of the explanation methods we test, whether they are attention-based or feature attributions, correlate strongly with one another. 
There are two exceptions: (i) Integrated Gradients moderately correlates with Grad-SHAP, and (ii) DeepLIFT moderately correlates with Deep-SHAP. 
However, this is unsurprising since the implementation of Grad-SHAP is based on Integrated Gradients, and the implementation of Deep-SHAP is based on DeepLIFT. 
In contrast, it is surprising to see the lack of correlation between Grad-SHAP and Deep-SHAP for the DistilBERT model, given that they are different implementations of the same algorithm, SHAP \cite{shap}. 

\input{tables/results/BiLSTM}

For the BiLSTM model, we observe a weak correlation between attention-based explanations and feature attribution explanations (i.e., the first row of Table~\ref{tab:bilstm_results}). 
Similar to the DistilBERT model, we see strong correlations for the methods that have similar underlying implementations. 
We also see some strong correlation between feature attribution methods, especially for single-sequence tasks (see Section~\ref{sec:results2}). 

Overall, we conclude that the correlation between explanation methods depends on the model architecture, which answers \textbf{RQ1}. 
In general, the correlation is weaker for the DistilBERT model than for the BiLSTM. 
However, the overall agreement between attention-based explanations and feature attribution explanations is weak, regardless of the model architecture.  

\subsection{RQ2: Does the correlation depend on the nature of the classification task?} 
\label{sec:results2}
To investigate \textbf{RQ2}, we examine five different datasets corresponding to two different tasks: IMDb and SST-2 are single-sequence tasks, while MNLI, Quora, and SNLI are pair-sequence tasks and show the results in Tables~\ref{tab:distilbert_results} and~\ref{tab:bilstm_results}. 

For the BiLSTM model, the results are clear: there is a substantially stronger correlation among explanations for single-sequence tasks than for pair-sequence tasks. 
For the DistilBERT model however, we do not see such a clear distinction -- there is, at best, a weak correlation across all explanation methods, regardless of the nature of the task. 

Regarding \textbf{RQ2}, we conclude that explanation methods agree with each other more on single-sequence tasks than pair-sequence tasks. This relationship holds especially for the BiLSTM model compared to the DistilBERT model.

\subsection{Explanations that strongly correlate}
\label{sec:results3}

Given that some of the feature attribution methods are inherently related, we would expect to see stronger degrees of correlation between them: 
\begin{enumerate*}[label=(\roman*)]
	\item  Grad-SHAP and Deep-SHAP are two different versions of the SHAP explanation method \cite{shap},
	\item  Grad-SHAP relies on the Integrated Gradients implementation to compute the token importances, and 
	\item Deep-SHAP relies on DeepLIFT.
\end{enumerate*}
We see some strong correlations for (ii) and (iii) between these methods for both models. 
These correlations are especially strong for the single-sequence tasks on the BiLSTM. 
However, we see remarkably weak correlations for (i) on DistilBERT, and for the BiLSTM on the pair-sequence tasks.

\section{Discussion} \label{sec:discussion}
Based on the results in Section~\ref{sec:results}, we argue that rank correlation with existing feature attribution methods is not an appropriate measure for evaluating attention-based explanations. 
In this section, we detail three main reasons for this: 
\begin{enumerate*}[label=(\roman*)]
	\item the general lack of correlation across all explanation methods, especially for transformer-based models,
	\item the fact that similar explanations do not always result in correlated rankings, and 
	\item the lack of justification for the existence of one ``ideal'' explanation, which is a fundamental assumption of the \emph{agreement as evaluation} paradigm. 
\end{enumerate*}

\subsection{Lack of correlation between explanation methods.}
In Section~\ref{sec:results}, we have shown that there is a low degree of correlation between explanation methods, especially for the transformer-based model.\footnote{Although there are some examples of stronger correlations, these are either on a very specific combination of task and model (i.e., single sequence tasks on the BiLSTM), or due to similarities in the underlying implementation of the explanation methods. } Similar conclusions are observed in the work of \citep{feldhus-etal-2021-thermostat} and \citep{krishna_disagreement_2022}.
This makes it challenging to justify the expectation that in order for attention-based explanations to be valid, they should correlate with existing feature attribution methods. 
If none of the existing feature attribution methods correlate with one another (as is the case for the transformer-based model), should we expect attention-based explanations to correlate with them?
Therefore, we argue against the use of rank correlation as an ``off-the-shelf'' tool to evaluate attention-based explanations. 

A common critique of using Kendall-$\tau$ is that expecting the full rankings of token importance to correlate is unrealistic. In their original paper, \citet{jain-wallace-2019-attention} theorized that top-$k$ token comparison would reduce noise, improve correlation, and more closely align with the end user's interest in the most salient features. However, formulating a systematic method for calculating $k$ across various models, tasks, and datasets is difficult in practice and yields mixed results. \citet{treviso-martins-2020-explanation} show a dynamic selection of $k$ with a sparse attention mechanism more effectively conveys justifications of decisions than fixed values of $k$ (e.g., 5 or 10 tokens). In contrast, \citet{krishna_disagreement_2022} show a widespread disagreement of feature attribution methods across domains regardless of whether $k$ is fixed or dynamic (e.g., the top $X$\% of features or top $X$\% of tokens based on average sentence length).


\subsection{Similar explanations do not imply strongly correlated rankings}
In order to investigate what weak correlation looks like in practice, we apply a heatmap of importance scores for each of the tested explanation methods. 
The darker the color, the more important the token is w.r.t. the model prediction. 
Figure~\ref{fig:distill-ss} shows a sentiment analysis example using the transformer-based model on the SST-2 dataset. 
Upon first glance, the visualizations in Figure~\ref{fig:distill-ss} look relatively similar: the methods are all highlighting roughly the same words -- almost all methods (except for Int-Grad) highlight the word `technically' as important. 
However, the average Kendall-$\tau$ correlation across all methods for this example is very weak -- only $0.01$. This highlights the problem of evaluating explanations by measuring the correlation between two ranked lists of importance scores: two explanation methods can roughly indicate the same token(s) as important, but when similar tokens do not end up in similar positions in the two rankings, the overall agreement between the two methods can still be low. 


\begin{figure}
	\centering
	\begin{subfigure}{.48\textwidth}
		\centering
		\includegraphics[width=1\linewidth]{plots/figures/distill-ss-sample1074.pdf}
		\caption{Sentiment analysis example from the SST-2 dataset. The average Kendall-$\tau$ correlation across all methods for this example is $0.01$. }
		\label{fig:distill-ss}
	\end{subfigure}\hspace{1mm}
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=1\linewidth]{plots/figures/distill-ps-sample481.pdf}
		\caption{Textual entailment example from the MNLI dataset.The average Kendall-$\tau$ correlation across all methods for this example is $0.05$. }
		\label{fig:distill-ps}
	\end{subfigure}
	\caption{Examples of explanations for the transformer-based model. The brighter the color, the higher the attribution value}
	\label{fig:test}
\end{figure}


\subsection{Is there one ideal explanation?}
The \emph{agreement as evaluation} paradigm implicitly assumes the existence of a single ``ideal'' explanation (i.e. ranking of tokens) which all methods must uncover, and new feature attribution methods are evaluated based on how strongly they correlate with the ``ideal'' explanation. However, it is unclear whether this assumption holds. 
For instance, input token importance rankings may only capture a narrow slice of the model's behavior, such that many plausible rankings exist.  

Since many tasks may be too complex for humans to judge token-level importance, it can be unclear how to choose the ``ideal'' ranking. 
While a handful of highly polar tokens are generally indicative of the label in binary sentiment classification \cite{sun-lu-2020-understanding}, annotators may be unsure how to rank the other tokens. The difficulty increases in the pair-sequence setting -- if two words indicate a contradiction, which is more important?  
As a result, when agreement is measured in the presence of multiple faithful and plausible rankings, feature attribution methods may look deceptively problematic, even if they are not. 


Figure~\ref{fig:distill-ps} shows an example of how different explanation methods can highlight different tokens as being important for the prediction, even though the underlying model is the same. 
Figure~\ref{fig:distill-ps} shows a textual entailment example from the MNLI dataset, where the task is to predict whether or not the second sentence entails the first. 
The first sentence comes after the [CLS] token: \textit{``But there are many more who still need our help.''}. 
The second sentence comes after the first [SEP] token: \textit{``10,000 people still need our help''}. 
Given that we see different explanations for the same example using the same model, it is unclear how to identify which of these explanations is the ``ideal'' one. 

Instead of evaluating potential explanations based on how well they correlate with one another, we suggest that researchers and practitioners should test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand. 
This can be used in combination with metrics such as those suggested by \citet{atanasova-etal-2020-diagnostic}. 

\section{Conclusion} 
\label{sec:conclusion}

In this work, we investigate how strongly attention-based explanations correlate with recent feature attribution methods for NLP tasks. 
We compare attention-based explanations to five recent feature attribution methods, using both transformer- and LSTM-based models, on both single- and pair-sequence tasks.
Overall, we observe a low degree of correlation between the attention-based explanations and the feature attribution explanations. 
For the transformer-based model, we find a weak correlation across all explanations for both task types. 
For the BiLSTM, we observe some strong correlations between explanations, but only for simple single-sequence tasks. 


Through our experiments, we discover
\begin{enumerate*}[label=(\roman*)]
	\item a general lack of correlation between explanation methods, especially for more complex settings (i.e., transformer-based model and pair-sequence tasks) which is corroborated by additional recent research on text, tabular, and image data \cite{feldhus-etal-2021-thermostat,krishna_disagreement_2022},
	\item that similar explanations do not always result in correlated rankings, and 
	\item the existence of a single ``ideal'' explanation is questionable, which is a fundamental assumption of the \emph{agreement as evaluation} paradigm. 
\end{enumerate*}
Without an external ground-truth explanation, all that rank correlation tells us is whether or not two rankings are similar. 
For this reason, we recommend practitioners stop using \textit{agreement as evaluation} for attention-based explanations.  

In future work, we plan to approach this problem from first principles by formulating toy data to guarantee a single, correct top-$k$ or full ranking for each instance to see if XAI methods are consistently capable of recovering the ideal explanation under this setting.

\section*{Acknowledgements}
We thank the anonymous reviewers, Bilal Alsallakh and Maarten de Rijke for their helpful discussions and suggestions.
This research was supported by Ahold Delhaize and the Netherlands Organisation for Scientific Research under project nr. 652.001.003 and the Nationale Politie.
All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.









