\section{Introduction}
Contextualized word representations---which are produced by language models---have lead to enhancements in natural language processing (NLP) model performance on myriad tasks \citep{joshi-etal-2019-bert, kondratyuk-2019-cross, zellers2019hellaswag,brown2020language}. 
Despite their impact, however, it is still not obvious to researchers what information these representations encode. 
One prominent line of work attempts to shed light on this topic through \term{probing}.\footnote{Also referred to as auxiliary prediction \cite{adi2016fine} or diagnostic classification \cite{hupkes2018visualisation}.}\looseness=-1

The goal of probing is to test whether a model encodes certain linguistic properties in its intermediate representations. 
Specifically, a probe is a supervised classifier which is trained to predict a property of interest from a target model's
representations. 
If this probe manages to predict the property with high accuracy, one may conclude that these representations encode information about the probed property.
While widely used, however, this setup has recently suffered considerable criticism.\footnote{See \citealt{belinkov2021probing} for a review.}

\begin{figure}
    \centering
   
    \includegraphics[width=\linewidth]{figs/example.pdf}
   
    \caption{Intervention on the gender of lemma \textit{programador} (masculine $\rightarrow$ feminine). Changes are propagated from that noun to its dependent words accordingly.\looseness=-1}
   
   
    \label{fig:causal-example}
\end{figure}\saveForCr{\tiago{I think gender should be on top of the sentence and point to it (to make this more similar to the causal diagram). Also, intervention should be an arrow from g to g', instead of from f to f', right? \response{afra}{Will do that for CR}}}

Probing a pre-trained model for, e.g., grammatical gender, can tell us whether information about this property is \emph{present} in the representations.\footnote{See \citet{pimentel-etal-2020-information}, \citet{hewitt-etal-2021-conditional} and \citet{pimentel-cotterell-2021-bayesian} for fomalisations of this statement under information-theoretic frameworks.}
It does not, however, tell us if the model actually uses this information in its predictions \citep{ravichander-etal-2021-probing}.
Furthermore, it does not even tell us whether this property is directly \emph{encoded} in the representations; information about it may be present there simply due to its co-occurrence with some other property.
In other words, through mainstream probing techniques we might find \emph{correlations} between a linguistic concept and probed representations. We cannot, however, guarantee that there is a \emph{causal} relationship between them.


In this work,  we formalize a framework via which one can ascertain whether a particular linguistic property has a \emph{causal} impact on a model's representations.
A number of prior works have attempted to tease apart such causal dependencies, using what we term as templated and representational counterfactuals. 
While work on \defn{representational counterfactuals} investigated these causal dependencies via interventions on neural representations, we propose to instead intervene directly on models' inputs, i.e., real world data. 
While not always possible, focusing on input-level interventions guarantees that our intervention only affects the probing property of interest.
On the other hand, \defn{templated counterfactuals} first create a number of template sentences (either manually or automatically), which they then fill with a set of minimal-pair words to generate counterfactual sentences.
These sets of templates, however, are naturally biased, not reflecting the diversity of sentences present in natural language.
Instead, our strategy creates counterfactuals directly from naturalistic sentences, implementing syntactic intervention based on the dependency structure of the input.
We thus term our framework: \defn{naturalistic counterfactuals}.\looseness=-1


In our first set of experiments, we employ syntactic intervention strategy to estimate the average treatment effect (ATE) of two morpho-syntactic features, number and grammatical gender of nouns on their contextualized representation. We sanity check our framework, showing the estimated ATE's stability and utility. 
In our second set of probing experiments, we first find that a few principal axis encode grammatical gender and number in nouns' representations extracted from last layer of BERT, RoBERTa, and GPT-2.
We further use our naturalistic counterfactuals to causally investigate gender bias in RoBERTa;
RoBERTa is much more likely to predict the adjective \textit{hermoso(a)} for feminine sentences and \textit{racional} for masculine.
Finally, we evaluate the performance of correlational probes on original data and a dataset augmented with the counterfactuals.
While probes have very high (above 90\%) performance of recovering gender from sentence-level representations, they only get close to chance (around 60\%) on the augmented data.
Together our results show the usefulness and necessity of our framework for naturalistic counterfactual generation.\looseness=-1



\section{Probing}

Probing spans a number of popular methods for the analysis of NLP models. 
The first wave of probing methods took straightforward approaches, which arguably only allowed them to uncover whether a property was \emph{present} in a model's representations.
Recently, a second such wave has proposed alterations to the traditional probing paradigm, introducing the notion of an intervention, which allows for \emph{causal} claims.
We review both in this section.

\subsection{Correlational Probing}

Under its original interpretation, the performance of a probe is viewed as an indication that a model encodes information (in its representations) about some probed property \citep{alain2016understanding}. 
Such techniques have been used to claim language models' knowledge of various morphological, syntactic, and semantic phenomena \cite[][\emph{inter alia}]{adi2016fine,ettinger-etal-2016-probing,belinkov-etal-2017-evaluating,conneau-etal-2018-cram}. 
Yet the validity of these claims has been a subject of debate \citep{saphra-lopez-2019-understanding,pimentel-etal-2020-pareto,pimentel-etal-2020-information,voita-titov-2020-information}. Notably, \citet{hewitt-liang-2019-designing} questioned whether probes were highlighting useful properties of the representations under the standard paradigm, or rather just re-learning the task from scratch.
Relatedly, 
\citeauthor{ravichander-etal-2021-probing} show that even in tasks where the original model \emph{cannot} use a property to perform the task, the property can still be reliably recovered from the representations. 
Together, these works show that classic probing works allow for correlational claims at most.

\subsection{Causal Probing}


A more recent line of work aims to answer the question: is there a \emph{causal} relationship between a property of interest and the probed model's output? 
In natural language, however, answering this question is not straightforward: sentences typically contain confounding factors which prevent simple correlational analyses from answering causal questions.
To circumvent this problem, most work in causal probing relies on {\bf interventions}---the act of manually setting a variable of interest to a fixed value.
Importantly, this must be done without altering any of this variable's causal parents, thereby keeping their probability distributions fixed.\footnote{Consider a set of three random variables with a causal structure $X \rightarrow Y \rightarrow Z$ (where X causes Y, which causes Z). If we simply conditioned on $Y=1$, we would be left with the conditional distribution $p(x, z \mid Y=1) = p(x \mid Y=1)p(z \mid Y=1)$. If we perform an intervention on $Y=1$, on the other hand, we are left with a distribution of $p(x, z \mid \mathrm{do}(Y)=1) = p(x)p(z \mid Y=1)$; thus $X$'s distribution is not altered by $Y$.}
As a byproduct, these interventions generate \textbf{counterfactuals}, examples where a specific property of interest is changed; these counterfactuals can then be used in causal analyses.
Prior probing papers have proposed methods using both \emph{representational} and \emph{templated} counterfactuals, as we discuss next.


\vspace{-2pt}
\paragraph{Representational Counterfactuals.}
A few recent causal probing papers perform interventions directly on model representations \citep{giulianelli2018under,feder2020causalm,vig2020investigating,tucker2021if,ravfogel-counter}.
As an example, \citet{elazar2020bert} use iterative null space projections \citep{ravfogel-etal-2020-null} to remove an analyzed property's information (e.g. part of speech) from the representations.
Although representational intervention can be handy in situations where other forms of intervention are not feasible, these methods are less interpretable---it is not clear which aspects of the probed property are altered.
Furthermore, it is often impossible to make sure only the information about the target property is removed or changed.\footnote{There are, however, methods to mitigate this issue. As a control experiment, for instance, \citet{elazar2020bert} added a bit which encoded the removed property back to the representations. They then tested whether the model's original performance could be recovered.}
In the absence of this guarantee, any causal conclusion should be viewed with caution.\looseness=-1

\vspace{-2pt}
\paragraph{Templated Counterfactuals.}
Other works have leveraged input-level interventions: \citet{vig2020investigating} and \citet{finlayson-etal-2021-causal}, for instance, work with template sentences to analyze minimal pairs which differ only with respect to a single analyzed property. 
Using these minimal pair sentences, they estimate the effect of an input-level intervention on individual attention heads and neurons. 
One benefit of template-based approaches is that they create a highly controlled environment, which may lead to insights that would be impossible to gain in natural data. 
However, working with templates implicitly means working with a narrow set of sentences, which may not reflect the complexity of language in real world data.\looseness=-1

\paragraph{Naturalistic Counterfactuals.}\ryan{I would give what we are doing a name. \response{afra}{what do you think of this name?} \response{tiago}{I changed to naturalistic counterfactuals since I really liked the new title hahah}}
In this paper, we propose a strategy which performs input-level interventions directly on naturalistic data, relying on a text's dependency structure for that.
Specifically, by intervening directly on naturalistic data we avoid the biases of template-based approaches. 
Further, our approach guarantees that we only intervene on a specific property of interest---as opposed to methods employing representational interventions---which leads to more reliable causal effect estimates.\looseness=-1

\section{The Causal Framework}

The question of interest in this paper is how contextualized representations are \emph{causally} affected by a morpho-syntactic feature such as gender or number.
To see how our method works, it is easiest to start with an example.
Let's consider the following pair of Spanish sentences:

\ex. \label{ex:msc} \textit{\textbf{El}} \textit{programador} \textit{talentos\textbf{o}} \textit{escribi{\'o}} \textit{el} \textit{c{\'o}digo}. \\ 
the.\textsc{m}.\textsc{sg}
programmer.\textsc{m}.\textsc{sg}
talented.\textsc{m}.\textsc{sg}
wrote the code. \\
The talented programmer wrote the code.

\ex. \label{ex:fem} \textit{\textbf{La}} \textit{programador\textbf{a}} \textit{talentos\textbf{a}} \textit{escribi{\'o}} \textit{el} \textit{c{\'o}digo}. \\ 
the.\textsc{f}.\textsc{sg}
programmer.\textsc{f}.\textsc{sg}
talented.\textsc{f}.\textsc{sg}
wrote the code. \\
The talented programmer wrote the code.

The meaning of these sentences is equivalent up to the gender of the noun \textit{programador}, whose feminine form is \textit{programador\textbf{a}}. However, more
than just this one word changes from \ref{ex:msc} to \ref{ex:fem}: We also have that the definite article \textbf{\textit{el}} changes to \textbf{\textit{la}} and that the adjective \textit{talentos\textbf{o}} changes to \textit{talentos\textbf{a}}. 
In the terminology of this paper, we will refer to \textit{programador} as the \term{focus noun},\ryan{Should we be super clear that the focus is the \emph{lemma}?} as it is the noun whose grammatical properties we are going to change.
We will refer to the changing of \ref{ex:msc} to \ref{ex:fem} as a \term{syntactic intervention} on the focus noun.
Informally, a syntactic intervention may be thought of as taking part in two steps. 
First, we swap the focus noun (\textit{programador}) with another noun that is equivalent up to a single grammatical property.
In this case, we swap \textit{programador} with \textit{programador\textbf{a}} which differs only in its gender marking.
Second, we reinflect the sentence so that all necessary words grammatically agree with the new focus noun.
The result of a syntactic intervention is a pair of sentences that differ minimally; they only differ with respect to this one grammatical property. 
Another way of framing the syntactic intervention is as a counterfactual: What would \ref{ex:msc} have looked like if \textit{programador} had been feminine?
The rest of this section focuses on formalizing the notion of a syntactic intervention and discussing how to use them in a causal inference framework for probing.

\paragraph{A Note on Inanimate Nouns.} 
When estimating the effect of grammatical gender here, we restrict our investigation to \emph{animate} nouns, e.g., \textit{doctor\textbf{a}}/\textit{doctor} (feminine/masculine doctor).
As discussed by \citet{gonen-etal-2019-grammatical}, grammatical gender of inanimate nouns is lexicalized, meaning that each noun is assigned a single gender, e.g. \textit{puente} (bridge) is masculine. In other words, there is not a non-zero probability of assigning each lemmata to each gender, which violates a condition called \textbf{positivity} in causal inference literature. 
Thus, we cannot perform an intervention on the grammatical gender of those words, but rather would need to perform an intervention on the lemma itself. 
We leave this as future work. 
Note that a similar lexicalization can also be observed in a few animate nouns, e.g. \textit{madre/padre} (mother/father). 
In such cases, in order to separate lemma from gender, we make an assumption that these words share a hypothetical lemma, which in our example represents parenthood, and combining that with gender would give us the specific forms, e.g. \textit{madre}/\textit{padre}.


\begin{figure*}[t]
    \centering
    \begin{subfigure}{0.49\textwidth}
    \includegraphics[width=\linewidth]{figs/causal-b.pdf}
   
   
    \end{subfigure}
    \begin{subfigure}{0.49\textwidth}
    \includegraphics[width=\linewidth]{figs/causal-a.pdf}
   
   
    \end{subfigure}
   
    \caption{Causal graph for the Spanish sentence \textit{\textbf{El} programador talentos\textbf{o} escribió el código.} before (on the left) and after (on the right) an intervention on the grammatical gender of the focus noun.}
    \label{fig:detailed-causal}
    \vspace{-7pt}
\end{figure*}


\subsection{The Causal Model} \label{sec:cm}

We now describe a causal model that will allow us to more formally discuss syntactic interventions. 

\paragraph{Notation and Variables.}
We denote random variables in upper-case letters and instances with lower-case letters. 
We bold sequences: bold lower-case letters represent a sequences of words and bold upper-case letters represent a sequence of random variables.
Let $\boldsymbol{f} = \langle f_1, \ldots, f_T\rangle$ be a sentence (of length $T$) where each $f_t$ is a word form. In addition, let $\mathbf{r}$ be the list of contextual representations $\mathbf{r} = \langle r_1, \ldots, r_T \rangle$ (each $r_t \in \mathbb{R}^h$) in one-to-one correspondence with the sentence $\boldsymbol{f}$, i.e. $r_t$ is $f_t$'s contextual representations. 
Furthermore, let  $\boldsymbol{\ell} = \langle \ell_1, \ldots, \ell_T\rangle$ be a list of lemmata and $\tilde{\boldsymbol{m}} = \langle m_1, \ldots, m_T \rangle$ a list of morpho-syntactic features co-indexed with $\boldsymbol{f}$; $\ell_t$ is the lemma of $f_t$ and $m_t$ is its morpho-syntactic features. 
We call $\boldsymbol{m} = \langle m_{t_1}, \ldots, m_{t_K} \rangle$ the \textbf{minimal list of morpho-syntactic features}, where each $t_k$ is an index between 1 to $T$. 
In essence, we drop features of the tokens that are dependent on other tokens' morphology. 
In our example \ref{ex:msc} this means we only include the morpho-syntactic features of \textit{programador} and \textit{c{\'o}digo}, thus $\boldsymbol{m} = \langle m_2, m_6 \rangle$. 
We denote the morpho-syntactic feature of interest of the focus noun as $m_*$, which can represent either its gender $g_*$ or its number $n_*$. We further denote the lemma of the focus noun as $\ell_*$. \looseness=-1

\paragraph{Causal Assumptions.} 
Our causal model is introduced in \cref{fig:detailed-causal}, representing the causal relationships between $U, \boldsymbol{L}, \boldsymbol{M}, \boldsymbol{F}$ and $\mathbf{R}$. Explicitly, we assume the following causal relationships:

\begin{itemize}[leftmargin=*]
    \item \emph{$\boldsymbol{M}$ and $\boldsymbol{L}$ are causally dependent on $U$}. The underlying meaning that the writer of a sentence wants to convey determines the used lemmas and morpho-syntactic features. 
    \item In general, \emph{$L_t$ can causally affect $M_t$.} Take gender of inanimate nouns as an example, where the lemma determines the gender.
    \item \emph{$\boldsymbol{F}$ is causally dependent on $\boldsymbol{L}$ and $\boldsymbol{M}$}. Word forms are a combination of lemmata and morpho-syntactic features.
    \item \emph{$\mathbf{R}$ is causally dependent on $\boldsymbol{F}$}. Contextualized representations are obtained by processing the sentences through the probed model.
\end{itemize}


\paragraph{Dependency Tree.} In order to measure the causal effect of the gender of the focus noun ($g_*$) on the contextualized representation ($\mathbf{r}$), all of its causal dependencies must be considered. As our causal graph shows (in \cref{fig:detailed-causal}), $g_*$ not only has causal effects on the focus noun's form, but also on the definite article \textbf{\textit{el}} and the adjective \textit{talentos\textbf{o}}. Yet, not all word forms in a sentence are affected; for instance the definite article \textbf{\textit{el}} in the noun phrase \textit{el c{\'o}digo}.
Luckily, within a given sentence, such relationships are naturally encoded by that sentence's dependency tree. The dependency graph $d$ of a sentence $\boldsymbol{f}$ is a directed graph created by connecting each word form $f_t$ ($1 \leq t \leq T$) to its syntactic parent. 
We use the information encoded in $d$ by leveraging the fact that a word form $f_t$ is causally dependent on its syntactic parent. 
In essence, a dependency tree $d$ implicitly encodes a function $d_t[\boldsymbol{m}]$ which returns the subset of morphological properties that causally affect the form $f_t$. We thus write:\looseness=-1
\begin{align}
    p(\boldsymbol{f}, \boldsymbol{m}&,\boldsymbol{\ell}, u) \\
    &= p(u)\,p(\boldsymbol{m}, \boldsymbol{\ell} \mid u)\,p(\boldsymbol{f} \mid \boldsymbol{m}, \boldsymbol{\ell}) \nonumber \\
   
    &= p(u)\,p(\boldsymbol{m}, \boldsymbol{\ell} \mid u)\,\prod_{t=1}^{T} p(f_t \mid d_t[\boldsymbol{m}], \ell_t)  \nonumber 
   
   
   
   
\end{align}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/causal-abstract.pdf}
    \caption{Causal model showing dependencies between the underlying meaning ($U$), lemma ($L_*$) and morpho-syntactic features ($M_*$) of the focus noun, context ($\boldsymbol{Z}$), sentences ($\boldsymbol{F}$) and contextualized representations ($\mathbf{R}$).}
    \label{fig:causal-abstract}
    \vspace{-7pt}
\end{figure}

\paragraph{Abstract Causal Model.} 
We can now simplify the causal model from \cref{fig:detailed-causal}
into \cref{fig:causal-abstract}. For simplicity, we isolate the lemma and morpho-syntactic feature of interest $L_*$ and $M_*$ and aggregate the other lemmata and morpho-syntactic features into an abstract variable, which we call $\boldsymbol{Z}$ and refer to it as the \textbf{context}. Furthermore, we only show the aggregation of word forms and representations as $\boldsymbol{F}$ and $\mathbf{R}$ in the abstract model. 
We will assume for now, and in most of our experiments, that the output of the causal model ($\mathbf{R}$ in \cref{fig:causal-abstract}) represents the contextualized representation of the focus noun.
However, as we generalize it later, this can be any function of $\boldsymbol{F}$, such as: the representation of other words in the sentence; the probabilities assigned by the model to a masked word; or even the output of a downstream task. 
We note that \cref{fig:causal-abstract} can be easily re-expanded into \cref{fig:detailed-causal} for any specific utterance by using its dependency tree. \looseness=-1

\subsection{Naturalistic Counterfactuals} \label{sec:intervention} \label{sec:algorithm}
In causal inference literature, the $\mathrm{do}(\cdot)$ operator represents an intervention on a causal diagram. 
For instance, we might want to intervene on the gender of the focus noun (thus using gender $G_*$ as the morpho-syntactic feature of interest $M_*$). 
Concretely, in our example (\cref{fig:detailed-causal}), $\mathrm{do}(G_{*} = \textsc{fem})$ means intervening on the causal graph by removing all the causal edges going into $G_*$ from $U$ and $L_*$ and setting its value to a specific realization \textsc{fem}. 
The result of this intervention on a sampled sentence $\boldsymbol{f}$ is a new counterfactual sentence $\boldsymbol{f}'$. 
As our causal graph suggests, the relationship between words in a sentence is complex, occurring at multiple levels of abstraction; swapping the gender of a single word---while leaving all other words unchanged---may not result in grammatical text.  
Consequently, one must approach the creation of counterfactuals in natural language with caution.
Specifically, we rely on syntactic interventions to generate our naturalistic counterfactuals.

\paragraph{Syntactic Intervention.}
We develop a heuristic algorithm to perform our interventions, shown in \cref{app:alg}.\footnote{This is a simplified abstract version of the algorithm where we omit the rule based re-inflection functions for nouns, adjectives and determiners. We also handle contractions, such as \textit{a} $+$ \textit{el} $\rightarrow$ \textit{al}, which is not mentioned in this pseudo-code.}
Given a sentence and its dependency tree, this algorithm generates a counterfactual version of the sentence, i.e., approximating the $\mathrm{do}(\cdot)$ operation.
This algorithm processes the dependency tree of each sentence in a depth-first search recursive manner. In each iteration, if the current node is a noun, it is marked as the focus noun.\footnote{Specifically, for gender intervention we only mark the noun as the focus if it is an animate noun.} 
Then, the intervention is performed, altering the focus noun and all dependent tokens.
Notedly, when we syntactically intervene on the \emph{grammatical gender} or \emph{number} of a noun, we do not alter potentially incompatible semantic contexts. 
Take sentence \ref{ex:err} as an example, where focus is \textit{mujer} and we intervene on \emph{gender}. 
Its counterfactual sentence \ref{ex:err_masc} is semantically awkward and highly unlikely, but still meaningful.
We can thus estimate the causal effect of grammatical gender in the contextual representations---breaking the correlation between morpho-syntax and semantics.

\ex. \label{ex:err} \textit{\textbf{La}} \textit{\textbf{mujer}} \textit{dio a luz a 6 bebés}. \\ 
the.\textsc{f}.\textsc{sg}
woman.\textsc{f}.\textsc{sg}
gave birth to 6 babies. \\
The woman gave birth to 6 babies.

\ex. \label{ex:err_masc} \textit{\textbf{El}} \textit{\textbf{hombre}} \textit{dio a luz a 6 bebés}. \\ 
the.\textsc{m}.\textsc{sg}
man.\textsc{m}.\textsc{sg}
gave birth to 6 babies. \\
The man gave birth to 6 babies.




\subsection{Measuring Causal Effects}

In this section, we define the causal effect of a morpho-syntactic feature.
We then present estimators for these values in the following section.
While we focus on grammatical gender here, our derivations are similarly applicable to number and to other morpho-syntactic features.

Given a specific focus--context pair $(\ell_*, \boldsymbol{z})$,
the causal effect of gender $G_*$ on the representations is called the \term{individual treatment effect} \cite[ITE;][]{rosenbaum1983central} and is defined as:\looseness=-1%
\begin{align} \label{eq:ite} 
     &\Delta(\ell_*, \boldsymbol{z}) = \\
     &\quad \E_{\boldsymbol{F}} \Big[ \mathtt{tgt}(\boldsymbol{F}) \mid G_*\!=\!\textsc{msc}, L_*\!=\!\ell_*, \boldsymbol{Z}\!=\!\boldsymbol{z} \Big] \nonumber\\
     &\quad - \E_{\boldsymbol{F}} \Big[ \mathtt{tgt}(\boldsymbol{F}) \mid G_*\!=\!\textsc{fem}, L_*\!=\!\ell_*, \boldsymbol{Z}\!=\!\boldsymbol{z} \Big] \nonumber
\end{align}
where $\mathtt{tgt}(\cdot)$ is a deterministic function that implements our probed model, taking a form $\boldsymbol{F}$ as input and outputting $\mathbf{R}$.\tiago{I changed the function from rep to tgt so that the name matches our new
experiments better.}
Since $\boldsymbol{F}$ is itself a deterministic function of a $\langle G_*, L_*, \boldsymbol{Z} \rangle$ triple, we can rewrite this equation as:\footnote{We overload $\mathtt{tgt}(\cdot)$ to receive either $\boldsymbol{F}$ or $\langle G_*, L_*, \boldsymbol{Z} \rangle$. Note that the former is a deterministic function of the latter.}
\begin{align} \label{eq:ite-simple} 
     \Delta(\ell_*, \boldsymbol{z}) = \mathtt{tgt}(\textsc{msc}, \ell_*, \boldsymbol{z}) - \mathtt{tgt}(\textsc{fem}, \ell_*, \boldsymbol{z})
\end{align}
As can be seen from the equation above, the ITE is the difference between the representation given that the focus noun is masculine vs. feminine.
   
   
   

To get a more general understanding of how gender affects these representations, however, it is not enough to just look at individual treatment effects.
It is necessary to consider a holistic metric across the entire language.
The \term{average treatment effect} (ATE) is one such metric, being defined as the difference between the following expectations:
\begin{align} \label{eq:ate-base}
    \psi_{\text{ATE}}  = &\E_{\boldsymbol{F}} \big[\mathtt{tgt}(\boldsymbol{F}) \mid \mathrm{do}(G_*=\textsc{msc}) \big] \\
    &\quad- \E_{\boldsymbol{F}} \big[\mathtt{tgt}(\boldsymbol{F}) \mid \mathrm{do}(G_*=\textsc{fem}) \big] \nonumber
   
\end{align}
In words, the ATE is the expected causal effect of a random variable on another (in this case, of gender on the model's representations).
Computing this expectation, however, is not as simple as conditioning it on gender.
As there are backdoor paths\footnote{A backdoor path is a causal path from an analyzed variable to its studied effect which contain an arrow \emph{to} the treatment (i.e., an arrow going backwards). For instance, consider three random variables with a causal structure $Y \rightarrow X \rightarrow Z$ and $Y \rightarrow Z$ (where Y causes X, and both X and Y cause Z). $X \leftarrow Y \rightarrow Z$ forms a backdoor path. See definition 3 in \citep{pearl2009causal} for further details.}  
from the treatment (gender) to the effect (the representations),
we rely on the \term{backdoor criterion} \citep{pearl2009causal} to compute this expectation.
Simply put, we first need to find a set of variables that block every such backdoor path.
We then condition our expectation on them.
As shown in \cref{prop1} (in the appendix), the set of variables satisfying the backdoor criterion in our case is $\{L_*, \boldsymbol{Z} \}$:
\begin{align} \label{eq:ate} 
     &\psi_{\text{ATE}} = \\ &\quad\E_{L_*, \boldsymbol{Z}} \left[\E_{\boldsymbol{F}} \Big[ \mathtt{tgt}(\boldsymbol{F}) \mid G_* = \textsc{msc}, L_*, \boldsymbol{Z} \Big]\right] \nonumber \\
     & \quad -\E_{L_*, \boldsymbol{Z}} \left[\E_{\boldsymbol{F}} \Big[ \mathtt{tgt}(\boldsymbol{F}) \mid G_* = \textsc{fem}, L_*, \boldsymbol{Z} \Big] \right] \nonumber
\end{align}
which we can again rewrite as:%
\begin{align} \label{eq:ate-simple} 
     &\psi_{\text{ATE}} = \\
     &\E_{L_*, \boldsymbol{Z}} \left[\mathtt{tgt}(\textsc{msc}, L_*, \boldsymbol{Z}) - \mathtt{tgt}(\textsc{fem}, L_*, \boldsymbol{Z} ) \right] \nonumber
\end{align}
Finally, plugging \cref{eq:ite-simple} into \cref{eq:ate-simple}:
\begin{equation}
    \psi_{\text{ATE}} = \E_{L_*, \boldsymbol{Z}} \big[ \Delta(\ell_*, \boldsymbol{z}) \big]
\end{equation}
reveals that \cref{eq:ate} is just the ITE in expectation.
Thus, the ATE is an appropriate language-wide measure of the effect of gender on the contextual representations.\looseness=-1

\section{Approximating the ATE}\label{sec:approx}

In this section, we show how to estimate \cref{eq:ate-simple} from a finite corpus of sentences $\mathcal{S}$. 

\subsection{Na\"ive Estimator}

Each sentence in our corpus can be written as a triple $\langle \ell_*, \boldsymbol{z}, g_* \rangle$. 
We now use these to estimate \cref{eq:ate-simple}.
Specifically, we first compute the sample mean using two subsets of sentences: one with only masculine focus nouns $\mathcal{S}_\textsc{msc}$; the other with feminine ones $\mathcal{S}_\textsc{fem}$. We then compute their difference:%
\begin{align} \label{eq:ate-baseline} 
   
   
   
    \psi_{\text{na\"ive}} &= 
      \frac{1}{|\mathcal{S}_{\textsc{msc}}|} \sum_{\langle \boldsymbol{z}, \ell_*, \_ \rangle \in \mathcal{S}_\textsc{msc}} \mathtt{tgt}(\textsc{msc}, \ell_*, \boldsymbol{z}) \nonumber \\
     &~~ - \frac{1}{|\mathcal{S}_{\textsc{fem}}|} \sum_{\langle\boldsymbol{z}, \ell_*, \_ \rangle \in \mathcal{S}_\textsc{fem}} \mathtt{tgt}(\textsc{fem}, \ell_*, \boldsymbol{z})
\end{align}
We note, however, that this is a very na\"ive estimator.
Since $\mathcal{S}_\textsc{msc}$ (and respectively $\mathcal{S}_\textsc{fem}$) includes only the fraction of sentences with masculine focus nouns, restricting the sample mean to this set of instances is equivalent to using samples $\boldsymbol{z}, \ell_* \sim p(\boldsymbol{z}, \ell_* \mid \textsc{msc})$, rather than $\boldsymbol{z}, \ell_* \sim p(\boldsymbol{z}, \ell_*)$ (as should be done for ATE). 
Notably, this is equivalent to ignoring the $\mathrm{do}$ operator in \cref{eq:ate-base}.
Consequently, \cref{eq:ate-baseline} introduces a purely correlational baseline.
In the following section, we present our (better) causal estimator.\looseness=-1


\subsection{Paired Estimator}
We now use our naturalistic counterfactual sentences to approximate the ATE.
Specifically, by relying on our syntactic interventions, we can get both a feminine and masculine form of each sentence $(\ell_*, \boldsymbol{z})$ sampled from the corpus.
Concretely, we use the following \textbf{paired} estimator:%
\begin{align} \label{eq:ate-paired} 
     &\psi_{\text{paired}} = \\ 
     &\frac{1}{|\mathcal{S}|} \sum_{\langle \boldsymbol{z}, \ell_*, \_ \rangle \in \mathcal{S}} \Big[\underbrace{
   
     \mathtt{tgt}(\textsc{msc}, \ell_*, \boldsymbol{z})
     }_{(1)} -
     \underbrace{
   
    \mathtt{tgt}(\textsc{fem}, \ell_*, \boldsymbol{z})
    }_{(2)} \Big] \nonumber
\end{align}
where, depending on $g_*$, the representation $\mathtt{tgt}(\boldsymbol{F})$ in (1) and (2) will be extracted from a pre-trained model using either the original or counterfactual sentences. 

\subsection{A Closer Look at our Estimators}

A closer look at our paired estimator in \cref{eq:ate-paired} shows that it is an \emph{unbiased} Monte Carlo estimator of the ATE presented in \cref{eq:ate-simple}.
In short, if we assume our corpus $\mathcal{S}$ was sampled from the target distribution, we can use this corpus as samples $\ell_*, \boldsymbol{z} \sim p(\ell_*, \boldsymbol{z})$.
For each $\ell_*, \boldsymbol{z}$ pair, we can then generate sentences with both $\textsc{msc}$ and $\textsc{fem}$ grammatical genders to estimate the ATE.

The na\"ive estimator, on the other hand, will not produce an unbiased estimate of the ATE.
As mentioned above, by only considering sentences in $\mathcal{S}_\textsc{msc}$ or $\mathcal{S}_\textsc{fem}$ at a time, we are implicitly conditioning on gender when approximating each expectation.
This estimator will approximate a value we term the \defn{average correlational effect} (ACE):
\begin{align} \label{eq:ace} 
     &\psi_{\text{ACE}} = \E_{L_*, \boldsymbol{Z} \mid G_*=\textsc{msc}} \left[\mathtt{tgt}(\textsc{msc}, L_*, \boldsymbol{Z})\right] \\
     &\qquad\qquad -\E_{L_*, \boldsymbol{Z} \mid G_*=\textsc{fem}} \left[\mathtt{tgt}(\textsc{fem}, L_*, \boldsymbol{Z}) \right] \nonumber
\end{align}


Template-based
approaches introduce a somewhat similar issue. 
In short, these approaches investigate causal effects through the use of minimal pairs of sentences, each of which can be used to estimate ITE values (as in \cref{eq:ite}). 
However, these pairs of sentences are either manually written or automatically collected using template structures. 
Such sentences cover a narrow (and potentially biased) set of structures, arguably not following the naturalistic distribution. 
In other words, their corpus $\mathcal{S}$ cannot be assumed to be sampled according to the natural distribution $p(\ell_*, \boldsymbol{z})$.\footnote{This becomes clear when we take a look at the sentences in one of such template datasets. For instance, all sentences in the Winogender dataset \citep{winogender}---used by \citet{vig2020investigating}---have very similar sentential structures.}\tiago{Add examples of the template dataset in this footnote?}
Yet in practice, these methods still approximate the treatment effect using an approach identical to our paired estimators. 
These methods will thus lead to \emph{biased} estimates of the ATE.\looseness=-1




\section{Dataset}

We use two Spanish UD treebanks \citep{nivre-etal-2020-universal} in our experiments: Spanish-GSD \citep{mcdonald-etal-2013-universal} and Spanish-AnCora \citep{taule-etal-2008-ancora}.
In our gender analysis, we use Open Multilingual WordNet \citep{Gonzalez-Agirre:Laparra:Rigau:2012} to detect the animacy of nouns and only mark animate nouns as focus, however, in number experiments both animate and inanimate nouns can be focus. 
The number of sentences used in our experiments can be found in the appendix, in \Cref{tab:dataset-stats}.\footnote{We will release the code for counterfactual generation, template generation, analysis and augmented datasets after final revisions.}\looseness=-1


\subsection{Evaluating Counterfactual Sentences} 

In order to evaluate our syntactic intervention algorithm (introduced in \cref{sec:algorithm}), we randomly sampled a subset of 100 sentences from our datasets.
These samples were evenly distributed across the two datasets (AnCora and GSD), tasks (gender and number) and categories within each task (masculine, feminine, singular and plural). 
A native Spanish speaker then annotated the samples. 
The naturalistic intervention algorithm was able to accurately generate the counterfactuals for 73\% of the sentences.\footnote{Using a normal approximation, we get a confidence interval for our accuracies which ranges from 64\% to 82\%.} 
The accuracy for gender and number intervention are 76\% and 72\% respectively. 
Due to the discussed subtleties in disentangling syntax from semantics---besides the complex sentence structures found in naturalistic data---we believe this error is in an acceptable range and leave improvements to future work.

\subsection{Template-Based Dataset}

With the goal of comparing our approach to templated counterfacutals, we translate two datasets for measuring gender bias: Winogender \citep{winogender} and WinoBias \citep{winobias}.
As shown by \citet{stanovsky-etal-2019-evaluating}, simply translating these templates to Spanish leads to biased translations, where professions are translated stereotypically and the context is ignored.
Following \citeauthor{stanovsky-etal-2019-evaluating}, we thus put two adjectives \textit{handsome} and \textit{pretty} before nouns to enforce masculine and feminine genders, respectively.
Consider, for instance, the sentence: ``\textit{\textbf{The developer} was unable to communicate with the writer because \textbf{he} only understands the code.}'' 
We rewrite it as ``\textit{The handsome developer}''. 
Similarly, if the pronoun was \textit{she}, we would write ``\textit{The pretty developer}''.
As an extra step, we want to ensure the gender of the writer stays the same before and after intervention. 
Therefore, we make two copies of this sentence: one with feminine writer, enforced by replacing \textit{writer} with \textit{pretty writer}, and one with masculine writer, enforced by replacing \textit{writer} with \textit{handsome writer}. 
We translate the resulting pairs of sentences using Google Translate API and drop the sentences with wrong gender translations.
In the end, we obtained 2740 minimal pairs.






\section{ATE in Action}

In these experiments, we first use the estimators introduced in \cref{sec:approx} to approximate the ATE of number and grammatical gender on contextualized representations. 
We look at how stable these ATE estimates are across datasets, and whether they change across words with different part-of-speech.
We then analyze whether the ATE (as an expected value) was an accurate description of how representations actually change on individual sentences.
Finally, we compute the ATE using another function $\mathtt{tgt}(\cdot)$ which represents the probability of predicting specific adjectives in a sentence; measuring the causal effect of gender in adjective prediction.\looseness=-1

\begin{figure}
\centering
    \includegraphics[width=\linewidth]{figs/cos-ates.pdf}
    \caption{Cosine similarities of the ATE on BERT representations.
    N. represents $\psi_{\text{na\"ive}}$; P. represents $\psi_{\text{paired}}$; and T. represents $\psi_{\text{paired}}$ estimated on template data.
    }
   
    \vspace{-0.7em}
    \label{fig:cos-ate}
\end{figure}

\subsection{Variations across ATEs}

\paragraph{Variation Across Datasets.} 
Using our ATE estimators, we compute the average treatment effect of both gender and number on BERT's contextualized representations \citep{devlin-etal-2019-bert} of focus nouns.\footnote{More specifically \textsc{bert-base-multilingual-cased} in the transformers library.} 
We compute $\psi_{\text{paired}}$ and $\psi_{\text{na\"ive}}$ estimators. Further, as an extra baseline we also estimate (with a paired estimator) the ATE using template data.
\cref{fig:cos-ate} presents their cosine similarities.
We observe high cosine similarities between paired estimator across datasets, \footnote{In order to make sure that the imbalance in the dataset \emph{before} intervention doesn't have a significant effect on results, we create a balance version of the dataset, where we observe close results.} but lower cosine similarities with the na\"ive estimator. 
This suggests that, while the causal effect is stable across treebanks, the correlational effect is more susceptible to variations in the datasets, e.g., semantic variations due to the domain from where treebanks were sampled. 
Furthermore, we observe a very low cosine similarity between our naturalistic ATE estimates and the template-based ones. 
This shows that template-focused analysis can lead to biased estimates.\looseness=-1

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/cos-dep.pdf}
    \caption{Cosine similarity of ATE estimators computed on focus nouns, adjectives and determiners using BERT representations on AnCora dataset.}
    \label{fig:cos-det-adj}
\end{figure}\saveForCr{\tiago{Why don't we merge both treebanks when we want to present a single result? I.e. basically all experiments except the first? \response{afra}{GSD is way smaller than AnCora and to be honest, I don't think it would change any of the results, but can do that for CR if you want} \response{tiago}{I think it could be good to not keep mentioning which dataset we use in each experiment.}}}

\paragraph{Variation Across Part-of-Speech Tags.} Using the same approach, we additionally compute the ATEs on other words on adjectives and determiners. 
\cref{fig:cos-det-adj} compares our na\"ive and paired ATE estimators computed on words with different part-of-speech tags. 
These results suggest that gender and number do not affect the focus noun and its dependent words in the same way; there is a relatively low cosine similarity between the ATEs computed on words with different part-of-speech tags.


\subsection{Masked Language Modeling Predictions}

We now analyze the effect of our morpho-syntactic properties on masked language modeling predictions. 
Specifically, we analyze RoBERTa \citep{conneau2019unsupervised}\footnote{More specifically, we use \textsc{xlm-roberta-base}.} in these experiments, as it has a better performance than BERT in masked prediction.
We thus look at how grammatical gender and number affect the probability RoBERTa assigns to each word in its output vocabulary.


\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{%
\begin{tabular}{llccc@{}}
\cmidrule[\heavyrulewidth](l){2-5}
& & $\mathrm{MP}(\mathbf{h})$ & $\mathrm{MP}(\mathbf{\hat{h}}_{\baseline})$ & $\mathrm{MP}(\mathbf{\hat{h}}_{\paired})$  \\
\cmidrule(l){2-5}
\ldelim\{{3}{2.5mm}[\parbox{3mm}{\rotatebox[origin=c]{90}{\textsc{gender}}}] 
&\textsc{Det}: $\mathrm{MP}(\mathbf{h'})$ & $4.85 \pm 2.39$ & $1.09 \pm 1.4$ & $0.67 \pm 1.14$  \\
&\textsc{Adj}: $\mathrm{MP}(\mathbf{h'})$ & $2.29 \pm 2$ & $1.04 \pm 1.05$ & $0.9 \pm 1.12$  \\
&\textsc{Focus}: $\mathrm{MP}(\mathbf{h'})$ & $3.75 \pm 2.67$ & $1.74 \pm 1.11$ & $1.53 \pm 0.93$ \\ \cmidrule(l){2-5}
\ldelim\{{3}{2.5mm}[\parbox{3mm}{\rotatebox[origin=c]{90}{\textsc{number}}}] 
&\textsc{Det}: $\mathrm{MP}(\mathbf{h'})$ & $6.93 \pm 2.52$ & $1.92 \pm 2.87$ & $2.05 \pm 2.64 $ \\
&\textsc{Adj}: $\mathrm{MP}(\mathbf{h'})$ & $5.63 \pm 2.75$ & $2.25 \pm 2.2$ & $2.5 \pm 2.17$ \\
&\textsc{Focus}: $\mathrm{MP}(\mathbf{h'})$ & $5.50 \pm 3.02$ & $2.25 \pm 2.14$  & $2.41 \pm 1.9$ \\
\cmidrule[\heavyrulewidth](l){2-5}
\end{tabular}
}
\caption{Mean and standard deviation of Jensen--Shannon divergence between the masked probability distributions of focus nouns, determiners and adjectives over the corpus.}
\label{tab:js}
\end{table}

We start by masking a word in our sentence, either the focus noun, a dependent determiner, or an adjective.
We then obtain this word's contextual representation $\mathbf{h}$.
Second, we apply a syntactic intervention on this sentence, and, following similar steps, obtain another representation $\mathbf{h'}$.
Third, we use these representations to obtain the probabilities RoBERTa assigns to the words in its vocabulary $\mathrm{MP}(\mathbf{h})$ and $\mathrm{MP}(\mathbf{h'})$. 
Finally, we get these same probability assignments, but using ATE to estimate the counterfactual representations:
\begin{align}
    &\mathbf{h'} \approx \mathbf{\hat{h}}_{\paired} = \mathbf{h} \pm \psi_{\text{paired}} \\
    &\mathbf{h'} \approx  \mathbf{\hat{h}}_{\baseline} = \mathbf{h} \pm \psi_{\text{na\"ive}}
\end{align}


We now look at how probability assignments change as a function of our interventions.
Specifically, \cref{tab:js} shows Jensen-–Shannon divergences between $\mathrm{MP}(\cdot)$ computed on top of different representations.
We can make a number of observations based on this table.
First, for gender, these distributions change more when predicting determiners and focus nouns than adjectives;
maybe because many Spanish adjectives have the same form when masculine and feminine, e.g. \textit{inteligente} (intelligent), or \textit{profesional} (professional).
Second, the distributions change more after an intervention on number than on gender. 
Third, when the original representations are summed with either of our estimators, the divergences are greatly reduced; these results show that the ATE values do accurately describe (at least to some extent) the change of representations in individual sentences.\looseness=-1






\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{figs/adj-bias.pdf} 
    \caption{$\psi_{\text{paired}}^{(a)}$ values computed using \cref{eq:adj-bias} to measure causal gender bias in masked adjective prediction.}
    \label{fig:adj-bias}
\end{figure*}

\subsection{Gender Bias in Adjectives}

As shown by \citet{bartl-etal-2020-unmasking}\afra{if you want to remove this please ping me, because it is one the citations requested by reviewer A}, measuring gender bias in gender-marking languages needs separate analysis. In our last set of core ATE experiments, we analyze the effect of gender on specific masked adjective probabilities.
We thus analyze RoBERTa's gender bias causally.
To this end, we manually create a list of 30 adjectives (complete list is in \cref{app:adj-list}) in both masculine and feminine forms.
We mask an adjective of a focus noun in $\boldsymbol{F}$. We then define a new function $\mathtt{tgt}(\cdot)$ to measure the ATE on adjective probabilities.
Specifically, we write:\looseness=-1
\begin{equation} \label{eq:adj-bias-1}
    \mathtt{tgt}_a(\boldsymbol{F}) = \ln\, p_{\theta}(a \mid \boldsymbol{F}) = \ln\,p_{\theta}(a \mid G_*, L_*, \boldsymbol{Z})
   
\end{equation}
where $a \in \mathcal{V}$ represents an studied adjective (in RoBERTa's vocabulary $\mathcal{V}$, and $p_\theta$ is the probability RoBERTa assigns to it.
We then plug this new function on our paired ATE estimator.
As this prediction is somewhat susceptible to noise, we actually take a median treatment effect (instead of the average). 
Specifically, this is equivalent to computing:
\begin{equation} \label{eq:adj-bias}
    \psi_{\text{paired}}^{(a)} = \underset{L_*, \boldsymbol{Z}}{\mathrm{median}} \Big[ \ln\, \frac{p_{\theta}(a \mid \textsc{msc}, L_*, \boldsymbol{Z})}{p_{\theta}(a \mid \textsc{fem}, L_*, \boldsymbol{Z})} \Big]
   
\end{equation}
In this equation, if $\psi_{\text{paired}}^{(a)} > 0$, the adjective is biased towards masculine sentences, while if $\psi_{\text{paired}}^{(a)} < 0$ the bias is towards feminine sentences. 
When an adjective in the list has two forms depending on the gender (e.g. \textit{hermos\textbf{a}/hermos\textbf{o}}), we sum the probabilities for masculine and feminine forms. 
As shown in \cref{fig:adj-bias}, rich (\textit{ric\textbf{a}/ric\textbf{o}}) and rational (\textit{racional}) are more biased towards masculine gender, while beautiful (\textit{hermos\textbf{a}/hermos\textbf{o}}) is biased towards feminine sentences.





\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{0.32\textwidth}
        \includegraphics[width=\linewidth]{figs/pca-gender-number-mbert0.pdf}
        \caption{BERT}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{0.32\textwidth}
        \includegraphics[width=\linewidth]{figs/pca-gender-number-xlm0.pdf}
        \caption{RoBERTa}
    \end{subfigure}%
    ~
    \begin{subfigure}[b]{0.32\textwidth}
        \includegraphics[width=\linewidth]{figs/pca-gender-number-gpt20.pdf}
        \caption{GPT-2}
    \end{subfigure}
    \vspace{-6pt}
    \caption{(top) Percentage of the gender and number variance explained by the first 10 PCA components. (bottom) The projection of 20 pairs of focus noun's representations on the first principal component.}
    \vspace{-.7em}
    \label{fig:pca}
\end{figure*}

\section{Naturalistic Counterfactuals in Action}

In this final experiments, we rely on our naturalistic counterfactual dataset.
We first explore the geometry in which morpho-syntactic features are encoded.
We then run a more classic diagnostic probing experiment, 
highlighting the importance of a causal framework when analyzing representations.

\subsection{Geometry of Morpho-Syntactic Features}

In this experiment, we follow \citeposs{bolukbasi2016man} methodology to isolate the subspace capturing our mopho-syntactic properties' information.
First, we create a matrix with the representations of all focus nouns in our counterfactually augmented dataset. Second, we pair the representations of counterfactual (before and after intervention) sentences. Third, we normalize the representations by subtracting each pair's mean. Finally, we compute the principal components of this new matrix.

As \cref{fig:pca} shows, in bertoid models (i.e. BERT and RoBERTa),  the first principal component explains close to 20\% of the variance caused by gender and number.
In GPT-2 \citep{radford2019language},\footnote{More specifically, we use \textsc{gpt2-small-spanish}.} more than half of the variance is captured by the first or the first two principal components.\footnote{These results are not obtained due to the randomness of a finite sample of high dimensional vectors. To show this, we present the percentage of variance for random vectors of the same size---where we compute PCA in a set of $|\mathcal{S}|$ random vectors, repeating this process 100 times and averaging the results---as the green trace in \cref{fig:pca}.}
This result is in line with \citeposs{Biasion2020GenderBI} findings on Italian \emph{word embeddings}, who report that the first principal components captures 57\% of variance. This might suggest that these features are linearly encoded in the representations. Such results can later be used in bias mitigation techniques \citep{liang-etal-2020-towards}. \looseness=-1 \afra{please ping me if you want to remove this as it is one of the citations suggested by reviewers.}


To further explore these subspaces, we project a random sample of 20 sentences (along with their counterfactuals) on the first principal component.
\cref{fig:pca} (bottom) shows that our three probed models can (at least to a large extent) differentiate both morpho-syntactic features using a single dimension.
Notably, this first principal component is strongly aligned with $\psi_{\text{paired}}$; they have a cosine similarity of roughly 0.99 in all these settings.

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{figs/probes.pdf}
    \caption{Accuracy scores of gender and number probes on the original and augmented datasets.}\label{fig:probe}
    \vspace{-1em}
\end{figure*}



\subsection{Comparison to Correlational Probing}

In this experiment, we use diagnostic classifier to probe the representations. 
We train logistic regressors (LogRegProbe) and support vector machines (SVMProbe) to predict either gender or number, again probing the three models: BERT, RoBERTa, and GPT-2.\tiago{We could drop the logistic and keep only SVMs if out of space}
Further, we probe the representations in two positions: the focus noun; the $\mathtt{[CLS]}$  token (or a sentence's last token, for GPT-2).\footnote{In our bertoids, $\mathtt{[CLS]}$  is a special token whose representation is supposed to aggregate information from the whole input sentence. In GPT-2, the last token in a sentence should also contain information about all its previous tokens.}

Accuracy of correlational probes on the original dataset are shown in \cref{fig:probe} as green points. 
Both gender and number probes reach a near perfect accuracy on these representations. 
Furthermore, all correlational gender probes reach a high accuracy in $\mathtt{[CLS]}$  representations, suggesting that gender can be reliably recovered even from them. 
In contrast, \emph{number} correlational probes seem to perform poorly on $\mathtt{[CLS]}$  representations.\looseness=-1

We now move on to our counterfactually augmented evaluation (shown as yellow dots in \cref{fig:probe}).
First, we see that the accuracy of probes on $\mathtt{[CLS]}$  representations drops significantly when evaluated on the counterfactual sentences. 
Second, we see that there is a drop of performance in all settings. \looseness=-1

Finally, we also fit our probes on a counterfactually augmented training set, studying whether the accuracy can be restored (shown as gray points in \cref{fig:probe}); we will call these the causal probes. Although there is a considerable improvement in accuracy, there is still a large gap between correlational and causal probes' accuracies.
Together, these results imply that correlational probes are sensitive to spurious correlations in the data (such as e.g. the semantic context in which nouns appear), and do not learn to predict grammatical gender robustly. It is only in the augmented datasets that this performance gap is visible, highlighting the importance of causal probing.\looseness=-1


\section{Conclusion}

We propose a heuristic algorithm for syntactic intervention which, operating on real-world data, allows us to create naturalistic counterfactuals at scale. We further discuss how to use these counterfactuals in a causal framework to probe for morpho-syntax.
Experimentally, we first show that ATE estimates are more robust to dataset differences than either our na\"ive (correlational) estimator, or template-based approaches.
Second, we show that ATE can (at least partially) predict changes in representations of individual sentences.
Third, we employ our ATE framework to study gender bias, finding a list of adjectives which are biased towards one or other gender.
Fourth, we find that both gender and number are structurally encoded in contextualized representation. 
Finally, we highlight the importance of causal analyses when probing; when evaluated on counterfactually augmented data, correlational probe results drop significantly.\looseness=-1


\clearpage
\onecolumn
