\section{Introduction}
\input{sections/introduction}

\section{Method}
\input{sections/method}

\section{Experiments}
\input{sections/experiments}

\section{Conclusion}
\input{sections/conclusion}

\clearpage
\bibliographystyle{ACM-Reference-Format}


\subsection{Datasets}
\textbf{FEVER} \cite{thorne2018fever} is a crowdsourced fact verification dataset containing claim-evidence pairs based on Wikipedia articles. We only use the claims paired with a single evidence for training and evaluation in this work.

\noindent\textbf{Symmetric} \cite{schuster2019towards} is a test set based on the FEVER development dataset designed for unbiased evaluation. It eliminates the correlation of n-grams in the claim and labels by careful construction.


\noindent\textbf{Adversarial} \cite{Thorne2019adversarial} is an adversarially constructed dataset explicitly designed to induce errors in models trained on the FEVER dataset.


\noindent\textbf{Fool Me Twice (FM2)} \cite{eisenschlos2021fool} is a Wikipedia-based fact verification dataset composed of 13k claim-evidence pairs that are collected through the games among crowd-workers. 

\subsection{Compared Methods}

\textbf{Data Augmentation Methods:}  
We compare our method against two data augmentation techniques commonly used for various natural language processing tasks: Easy Data Augmentation (EDA) \cite{wei2019eda} and neural paraphrasing. EDA applies simple mutations, such as random swapping or synonym replacement, to the original sentence to generate new examples. 
For neural paraphrasing, we use a GPT-2 model \cite{radford2019language} fine-tuned on back-translated data to paraphrase the original text \cite{krishna-etal-2020-reformulating}. For each original claim-evidence pair, we create a new pair that holds the same relation by transforming only the claim using these methods, leading to the augmentation ratio of augmented to original data 1:1.

\noindent\textbf{Regularization-based Debiasing Methods:}
We also compare with
two debiasing techniques that reduce the reliance on biases by regularizing the model on the biased samples.
The first one is an example re-weighting method that targets biases from the shortcut words \cite{schuster2019towards}. By re-weighting the importance of claims containing those words, it forces a model to focus on the hard examples in which relying on the bias results in incorrect predictions.
The other one is Product of Experts (PoE) \cite{mahabadi2020end}, which computes the training loss in an ensemble of the base model and the bias-only model. Similar to the first method, it controls the base model's loss depending on the prediction of the bias-only model for each example.





\subsection{Implementation Details}
For our experiments, we use the BERT-base-uncased model \cite{devlin2019bert}, which demonstrates competitive performance for fact verification tasks. 
We fine-tune BERT with an additional layer on top of the [CLS] token embedding.
We concatenate the claim and the evidence, and insert [SEP] token in between to make the input sequence.
Following previous works, we set maximum sequence length to 128, batch size to 32, and optimize the model through a standard cross-entropy loss using the Adam optimizer~\cite{kingma2014adam} with a learning rate of 2e-5.
\minwoo{We train the model on the FEVER train set and evaluate the generalization performance using the development set for Symmetric, Adversarial and FM2 datasets.}
We train the model for 3 epochs with 5 different random seeds for all experiments and report the averaged result.
For our augmentation pipeline, we set the maximum span size for evidence modification $\tau$ to $3$, which produces an augmented dataset with an augmentation ratio of 1:0.58.


\subsection{Results on the Full Dataset}

First we augment the full FEVER train set with our approach and compare its performance in Table~\ref{tab:full_result}. 
Our proposed method achieves 10.13\% improvement over the baseline and 3.6\% improvement over the previous state-of-the-art debiasing technique on the Symmetric dataset. 
This result shows that our method is highly effective at preventing the model to predict from unnecessary biases.
Our approach also shows a 2.12\% improvement on the Adversarial dataset and 3.36\% improvement on the FM2 dataset compared to the baseline, indicating that our augmentation method benefits not just on the diagnostic dataset for lexical bias but for fact verification in general.
Finally, our method leads to the greatest overall improvement across the datasets out of all compared training methods,
This result empirically proves that the contrastive samples generated from our augmentation method enhance the factual reasoning capabilities by learning a more robust feature representation, achieving strong generalization.

Compared to our approach, the example re-weighting and PoE methods perform slightly worse on the Symmetric and Adversarial dataset and slightly better on the original FEVER development set and FM2 dataset.
On the other hand, EDA and paraphrasing augmentations show negligible performance improvement on the Symmetric dataset. These results suggest that simply training with more data does not necessarily help mitigate the bias in data. 



\subsection{Ablation Studies}

We conduct ablation study to verify the effectiveness of the augmented samples generated from each step of our augmentation process.
The results shown in Table \ref{tab:full_result} reveal that a moderate performance improvement on the Symmetric, Adversarial and FM2 evaluation sets are attained even with only using the negative claims generated from the first step. 
However, its performance on the Symmetric dataset is still significantly lower compared to the full augmentation method, implying that augmenting with negative claims alone is less effective for debiasing the model. 

Training with the negative evidence augmented data exhibits a more competitive performance on the other hand, especially outperforming previous state-of-the-art technique on the Symmetric dataset. 
The results imply that the key component of our debiasing approach is training the model with contrastive samples sharing the same claim, which enables the model to learn by comparing the claim to the input evidence instead of just relying on artifacts in the claim.
Nevertheless, the full augmentation method outperforms all ablations, indicating that contrastive data samples in general help the model learn more robust representations. 



\subsection{Results on Low-resource Conditions}\label{SEC:LOW}

\minwoo{In the real world, the available training corpus size for fact verification is often lacking due to the expensive cost and difficulty of collecting and labelling the data.
Training on such a limited amount of data could lead to even more biased models due to the lack of samples to learn factual reasoning from.
Therefore, it is important to investigate whether the debiasing methods perform well in low-resource conditions.
}

Thus, we further evaluate our data augmentation method in low-resource conditions, simulating a data-scarce setting by sampling subsets of the original FEVER training data. 
We perform a class-balanced sub-sample on the FEVER training data with 5 different random seeds, and train the model on each subset with another 5 different random seeds to account for statistical variations.
As in the previous experiments, we augment the subset with an augmentation ratio of 1:1 for EDA and back-translation methods, while our augmentation method results in an average augmentation ratio of 1:0.47.
We evaluate on the subsets of 0.1\% ($N=240$), 0.2\% ($N=483$), 0.5\% ($N=969$), and 1.0\% ($N=2,427$) of the original training data, and the results are presented in Figure \ref{fig:limited}. \\
\indent Our augmentation method shows a consistent improvement in the low-resource conditions over all evaluated datasets, and especially outperforms the baseline trained on the full dataset with just 1\% of the original training data for the Symmetric evaluation set. On the other hand, the PoE method shows little to no improvements on all of the datasets except FM2, and in some cases show a slight performance drop. 
These results indicate that PoE, which relies on training a biased model to regularize learning from biased samples, do not generalize well in data-scarce settings.\\
EDA and paraphrasing augmentation show a moderate improvement over the baselines, verifying their effectiveness in low-resource conditions. 
However, our augmentation method is still able to achieve a marked improvement over EDA and paraphrasing on Adversarial and FM2 datasets, implying that our augmentation approach is more robust and effective at generalizing to out-of-distribution samples. 
In summary, the results with varying training dataset sizes show that our augmentation method is effective for low-resource domains.



\begin{figure}[t]
    \begin{center}
    \begin{tabular}{c}
      \includegraphics[width=0.8\columnwidth]{rsc/fig_limited_data_one_column.png}
    \end{tabular}
    \vspace{-2mm}
    \caption{Experimental results on varying training set size with different training methods. Train set sampling is done with five different seeds and the mean accuracy is reported. 
   
    }
    \vspace{-4mm}
    \label{fig:limited}
    \end{center}
\end{figure}

\subsection{Task Formulation}
Given a textual claim $c$ and evidence $e$, the objective of the fact verification task is to predict whether the claim is supported by the evidence, refuted by the evidence, or the evidence has not enough information for verification. 
We denote a sample in the dataset $\mathcal{D}$ of size $N$ as a triplet $(c,e,y) \in \mathcal{D}$, where $y \in \{SUP, REF, NEI\}$ is the data label.


\begin{table*}
  \caption{Experimental results on fact verification datasets. The mean and the standard deviation of the classification accuracy over five different runs are reported for each method.}
  \vspace{-2mm}
  \label{tab:full_result}
  \begin{tabular}{l|cc|cc|cc}
    \toprule
    \textbf{Train method} & \textbf{FEVER dev} & \textbf{Symmetric} & \textbf{Adversarial} & \textbf{FM2 dev} & \textbf{$\Delta$ sym.} & \textbf{$\Delta$ avg.} \\
    \midrule
    No augmentation (baseline) & 86.15 $\pm$ 0.42 & 58.77 $\pm$ 1.29 & 49.66 $\pm$ 0.37 & 40.81 $\pm$ 0.43 & - & - \\
    EDA & 85.09 $\pm$ 0.25 & 58.55 $\pm$ 1.63 & 51.41  $\pm$ 1.14 & 41.21 $\pm$ 1.11 & $-$0.22\% & +0.22\% \\
    Paraphrasing & 84.33 $\pm$ 0.34 & 59.02 $\pm$ 1.38 &  \textbf{52.53} $\pm$ 1.20 & 40.60 $\pm$ 0.71 & +0.25\% & +0.27\% \\
    \midrule
    Re-weighting & 85.56 $\pm$ 0.32 & 61.87 $\pm$ 1.16 & 49.92 $\pm$ 0.80 & 43.80 $\pm$ 0.46 & +3.10\% & +1.44\% \\
   
    Product of Experts (PoE) & \textbf{86.50} $\pm$ 0.35 & 65.30 $\pm$ 1.73 & 51.07 $\pm$ 1.20 & \textbf{46.69 $\pm$ 1.11} & +6.53\% & +3.54\% \\
    \midrule
    CrossAug (ours) & 85.34 $\pm$ 0.68 & \textbf{68.90 $\pm$ 1.68} &51.78 $\pm$ 1.02 & 44.17 $\pm$ 1.27 & \textbf{+10.13\%} & \textbf{+3.70\%} \\
     - Negative claim only augmentation & 85.70 $\pm$ 0.28 & 61.00 $\pm$ 0.71 & 51.96 $\pm$ 0.90 & 43.06 $\pm$ 0.40 & +2.23\% & +1.58\% \\
     - Negative evidence only augmentation & 85.87 $\pm$ 0.16 & 67.06 $\pm$ 0.99 & 51.46 $\pm$ 0.43 & 43.70 $\pm$ 0.97 & +8.29\% & +3.18\% \\
    \bottomrule
  \end{tabular}
  \vspace{-1mm}
\end{table*}


\subsection{Data augmentation pipeline}
In our proposed method, we generate three additional synthetic samples for each original claim-evidence pair through two stages of augmentation. Note that CrossAug utilizes only positive claims (SUP claims), which are verifiable by specific evidence, in the FEVER dataset. The whole process of our data augmentation pipeline is shown in Figure \ref{fig_overall}.

\noindent\textbf{(1) Negative Claim Generation:}
The first stage is to generate a \textit{negative claim} $c'$ based on a positive claim $c$ by adopting a neural sequence-to-sequence model. This generative process involves transforming of a positive claim, such as inserting a negation or replacing a word with antonyms. 
\tony{In turn, the generated claim has a different meaning so that it is refuted by the evidence $e$ supporting a positive claim, which is why we call this negative claim.}
Through this process, we form a new data sample $(c', e, REF)$.

To this end, we fine-tune BART \cite{lewis2020bart} on WikiFactCheck-English dataset, which provides pairs of positive claims and their corresponding negative claims \cite{sathe2020automated}. In fine-tuning, positive claims are used as the source text and negative claims are taken as the target text of the model. To provide a richer context, we also fine-tune the model with the source and target text reversed since a positive claim can be seen as a refuted version of a negative claim.


\noindent\textbf{(2) Evidence Modification:}
The negative claim generated in the first stage often only differs from the positive claim by a few words, and can thus be seen as a span replacement. For example, ``\textit{over 30 days}" is simply substituted with ``\textit{less than 10 days}", as shown in Figure \ref{fig_overall}. This phenomenon is due to the characteristics of the data used for fine-tuning the model: the negative claim was manually written by annotators under the constraints of both the sentence length and the subject to be similar to the positive claim \cite{sathe2020automated}. We also observe that the words replaced in the positive claim are often found verbatim in the evidence. This is because the changed part of the claim usually matches the factual information taken from the evidence.

As the second stage of our data augmentation process, we build upon these observations to perform a lexical search-based evidence modification. First, we compare the positive claim $c$ with the negative claim $c'$ to identify the changed part in the first stage. Once the words replaced in the positive claim are recognized (``over 30 days" in Figure \ref{fig_overall}), we search for the same words in the evidence $e$ and replace them with the substituted words in the negative claim (``less than 10 days" in Figure \ref{fig_overall}). Since this substitution induces the same factual modification on the evidence as that applied to the negative claim, it can be logically concluded that the resulting modified evidence $e'$ supports the negative claim $c'$ and refutes the positive claim $c$.
Consequently, we form two additional contrastive samples $(c, e', REF)$ and $(c', e', SUP)$ in the second stage.



\smallskip
\noindent\textbf{Exceptional Cases:}
In the first stage, the generated negative claim $c'$ is occasionally the exact same as the positive claim $c$. For such samples, we skip performing augmentation.
Also, in the second stage, we carry out the evidence modification only when the number of replaced words in the claims are less than or equal to $\tau$, where $\tau$ is a threshold value. This is necessary to prevent invalid evidence modification. When the replaced part is large, it frequently contains inappropriate terms for reconstructing the evidence, such as non-factual words, producing an illogical sentence. However, we still keep the sample $(c',e, REF)$ from the first stage even when the evidence modification stage is skipped.









