
\subsubsection{Multilingual Argument Assessment} 
While \citet{gretz2020quality} mention 
using a group of annotators with whom they have worked before for the assessment of arguments written in English, no such group was available to us for the non-English languages.
In addition, no test questions (TQs) were available, since they are typically formed from existing labeled data.
Initially, the first issue was addressed by relying on workers from appropriate countries, and the second by using machine-translated arguments from \argumentDatasetName, with a high-confidence label \emph{in English}, as TQs. 
At first, since the quality label is sensitive to translation (as described in \sectionRef{subsec:translated_label_assessment}) such TQs were limited to stance. 

A pilot on Spanish arguments showed a good \agreementName for stance ($0.71$), yet a low value for quality ($0.04$). 
The results showed that many of the annotators labeled a vast majority ($>$\percentage{80}) of the arguments as high-quality, 
even though they were instructed to consider only half as such.
Therefore, only those labeling $\leq80\%$ of arguments as high-quality were allowed further work. Others were excluded and their argument quality answers were ignored.

A second pilot extended this procedure to other languages.  However, the size of the workforce meeting the above criteria was small for \germanShort, \frenchShort and \dutchShort, preventing progress altogether for the last two.
This required integrating TQs for quality question despite the risk of the quality label changing due to the automatic translation.
To mitigate that risk, one of the authors carefully monitored each annotation task, reviewed TQs which many annotators answered incorrectly, and disabled those in which the translation introduced errors in the correct label or made the text unclear.

\section{Appendices}

\subsection{Additional Results}
\label{sec:appendix}

\subsubsection{Stance Classification on Machine-Translated Arguments}
\label{appendix:stance_mt_17l_results}

As described in \sectionRef{empirical_evaluation},
our evaluations were conducted on 6 European languages (EN, ES, FR, IT, DE, and NL). 
Models were trained for several language groups: \emph{RM} -- for the Romance languages, \emph{WG} -- the West-Germanic, and \emph{6L} -- a model that covers all the TLs in our evaluation.

We further explored the translate-train approach by augmenting the training data of our models with machine-translated data of other language families. 
First, we trained a model for the North-Germanic family (\emph{NG}) with three languages -- Danish (DA), Swedish (SV), and Norwegian (NB).
Next, we combined the Romance languages with  the two German families (\emph{RM, WG, and NG}), and created the \emph{9L} model with $9$ languages.
Finally, we trained a model with a relatively large number of languages ($17$) and a variety of language families. This model, denoted \emph{17L}, consists of all the languages in \emph{9L} and $8$ additional languages: Slavic languages -- Polish (PL), Slovak (SK), Russian (RU); Semitic -- Arabic (AR), Hebrew (HE); and Chinese/Japonic -- Simplified Chinese (ZH), Traditional Chinese (ZT), and Japanese (JA). 

The stance classification results on 
the \pseudoTestEviDatasetName pseudo-test for
all $17$ languages using all the aforementioned models are presented in \tableRef{tab_sbc_stance_mt}. 
We see that expanding the training set beyond the six languages in \emph{6L} by adding more distant languages, as in the \emph{9L} and \emph{17L} models, does not significantly improve the performance on English. 
On average, 
training on the TL is better than training on the original English arguments (average performance over all $17$ languages 
is $73.7\%$ with \emph{EN} and $86.5\%$ with \emph{TL}). Training on all $17$ languages tends to yield the best performance (with an average of $88.9\%$), though training on a subset of them is often nearly as good - and sometimes even better, especially on the \emph{6L} and \emph{9L} groups.

\input{tables/sbc_stance_mt}

\subsubsection{Per-Topic Analysis Information}
\tableRef{tab:sbc_test_motions} contains a list of $15$ topics that are included the test set of the \argumentDatasetName dataset, along with their IDs used during error analysis.

\input{tables/sbc_test_motions_ids}

\figureRef{fig_sbc_stance_hg_zs_per_topic} shows 
per-language results of the zero-shot baseline on the $15$ topics of the \humanGeneratedArgDatasetName evaluation dataset.
As indicated by the overall results on the same data in \figureRef{fig_sbc_stance_hg}, we see high performance on the three Romance languages consistently across most topics, and low performance on DE and NL for about half of the topics.   

\begin{figure}[th]
\includegraphics[width=\linewidth,
trim={1.0cm, 0.2cm, 1.4cm, 0.8cm},
clip]{figures/fig_sbc_stance_hg_zs_per_topic.pdf}
\caption{Per-topic stance classification
average macro-F1 results per language on \humanGeneratedArgDatasetName test set for the \emph{ZS} baseline. The dashed black line is the performance on English.}
\label{fig_sbc_stance_hg_zs_per_topic}
\end{figure}

\subsection{Annotation Details}
\label{appendix:annotation_details}
This section describes further annotation details, such as the adjustment of the argument assessment annotation task to multiple languages, and the guidelines used in each annotation task.
\input{Appendix/annotation_scaling}
\input{Appendix/argument_authoring_guidelines}
\input{Appendix/argument_assessment_guidelines}
\input{Appendix/evidence_assessment_guidelines}






\subsubsection{Argument Assessment Guidelines}
In this annotation, the guidelines for all languages were the same.
\\ \\
In the following task you should answer two questions concerning an argument suggested in the context of a debatable topic.

\begin{enumerate}
    \vspace{-0.2cm}\item What is the stance of the argument towards the topic? (supporting, contesting or neutral)
    \vspace{-0.2cm}\item For someone with this stance towards the topic, is this a high-quality argument to use? (yes or no)
\end{enumerate}
IMPORTANT! For the second question please answer "YES" only for high-quality arguments, and only for about half of the time.
\\ \\ 
Your answers will be monitored not only using test questions. If you are interested in participating in future similar tasks, please answer thoroughly.

\subsubsection{Argument Authoring Guidelines}
Below is an example of the argument authoring guidelines for German. The guidelines for the other languages were similar.
\\ \\ 
PLEASE READ: \\ \\
All your submitted arguments will be assessed for their quality. For each argument determined as a high-quality one, you will receive a bonus of up to 0.4\$.

\subsection*{Overview}
In the following task you are presented with a debatable topic, to which you should suggest high quality supporting/contesting arguments in \underline{German}.
\\ \\
A supporting/contesting argument will be considered as a high-quality one, if a person preparing a speech to support/contest the topic, respectively, will be likely to use this argument as is in her speech.
\\ \\
\textbf{Note:} Copying texts from the web or elsewhere is prohibited. The content you provide must be written by you in your own language. 
\\ \\
\textbf{Requirements}
\begin{itemize}
\vspace{-0.2cm}\item The argument must be phrased in \underline{German}.
\vspace{-0.2cm}\item The argument must either clearly support or clearly contest the topic.
\vspace{-0.2cm}\item You should write a single argument in each text box.
\end{itemize}

\subsubsection{Evidence Assessment Guidelines}
Below is an example of the evidence assessment guidelines for German.
The guidelines for the other languages were similar.

\section*{General instructions}

In this task you are given a topic and evidence candidates for the topic. The candidates are in German. Consider each candidate independently. For each candidate please select Accept if and only if it satisfies ALL the following criteria:

\begin{itemize}
    \vspace{-0.2cm}\item The candidate clearly supports or clearly contests the given topic. A candidate that is neutral towards the topic should not be accepted.
    \vspace{-0.6cm}\item The candidate represents a coherent, stand-alone statement, that one can articulate (nearly) “as is” while discussing the topic, with no need to change/remove/add more than two words.
    \vspace{-0.2cm}\item The candidate represents valuable evidence to convince one to support or contest the topic. Namely, it is not merely a belief or merely a claim, rather it provides an indication whether a belief or a claim is true. A candidate which presents detailed information (typically quantitative) that clearly support or clearly contest the topic, should be accepted.
\end{itemize}
If you select Accept, you should further indicate whether the evidence supports the topic (Pro) or contests it (Con).
\\
\textbf{Note:} if you are unfamiliar with the topic, please briefly read about it in a relevant data source like Wikipedia.

\section*{Examples}

The following examples outline several candidates along with their suggested annotations; please read all these examples before performing the task.
\\ \\
\textbf{Topic:} We should ban the sale of violent video games to minors.
\\ \\
\underline{Example 1}
\\\vspace{-2mm}\\   
\textit{The research clearly suggests that, among other risk factors, exposure to violent video games can lead to aggression and other potentially harmful effects.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept – Pro
\\\vspace{-3mm}\\ 
\textbf{Note:} even though the text is not explicitly referring to the proposed ‘ban’ policy, it should still be accepted, since highlighting the negative aspects of violent video games can be used to support the suggested ban.
\\ \\
\underline{Example 2}
\\\vspace{-2mm}\\ 
\textit{A university of Oxford study negates the idea that violent video game content leads to violence.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\ 
\textbf{Note:} here as well, even though the proposed ‘ban’ policy is not explicitly mentioned, the text should be accepted since clearly it can be used to contest the suggested ban.
\\ \\
\underline{Example 3}
\\\vspace{-2mm}\\ 
\textit{There is no reason to suppose that violent video games cause harm to children.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\ 
\textbf{Reason:} The candidate states a claim. It does not offer any additional information to convince the reader that this claim is true.
\\ \\ 
\underline{Example 4}
\\\vspace{-2mm}\\ 
\textit{The American Psychological Association argues that violent video-game play leads to increased moral sensitivity.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\ 
\textbf{Reason:} The candidate states a claim, but the fact that it is raised by an authority figure (organization or human) turns it into a valuable evidence.
\\ \\ 
\underline{Example 5}
\\\vspace{-2mm}\\ 
\textit{Kennelly said there is no scientific evidence that violent video games cause “serious harm” in kids such as heightened aggression that would require protection of the law.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\ 
\textbf{Note:}  If you are not certain whether the speaker is an authority figure or not, you should typically give him/her the benefit of the doubt and consider them as such (in this case the speaker is Matthew F. Kennelly, a United States District Judge). However,  if the candidate states a claim and the speaker is only mentioned by he/she/they you should reject it.
\\ \\
\underline{Example 6}
\\\vspace{-2mm}\\ 
\textit{The issue as “Psychological research confirms that violent video games can increase children's aggression.”}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\ 
\textbf{Reason:} The candidate does not represent a coherent, stand-alone statement.
\\ \\
\underline{Example 7}
\\\vspace{-2mm}\\ 
\textit{Some studies have clearly demonstrated that video game violence is leading to serious aggressive behaviour in real life, although other studies have shown the opposite.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\ 
\textbf{Reason:} The pro/con stance of the candidate towards the topic is unclear, since the end of the text contradicts its beginning.
\\ \\ 
\underline{Example 8}
\\\vspace{-2mm}\\ 
\textit{The Entertainment Software Association reports that 17\% of violent video game players are boys under the age of eighteen.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\ 
\textbf{Reason:} The candidate states a fact with no clear pro/con stance towards the topic.
\\ \\
\underline{Example 9}
\\\vspace{-2mm}\\ 
\textit{Studies show that watching violent movies increases aggression amongst youth.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\ 
\textbf{Reason:} The candidate is not related to the topic as it discusses violent movies and not violent video games.
\\ \\
\underline{Example 10}
\\\vspace{-2mm}\\ 
\textit{Another 2001 meta-analyses and a more recent 2009 study focusing specifically on serious aggressive behavior concluded that video game violence is not related to serious aggressive behavior in real life.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\ 
\textbf{Note:}  Even though the candidate’s first word better be omitted to make it a stand-alone statement, this is a minor change which is acceptable.
\\ \\
\underline{Example 11}
\\\vspace{-2mm}\\ 
\textit{Limiting the sale of violent video games will cause 15,000 people to lose their jobs.}
\\\vspace{-3mm}\\ 
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\ 
\textbf{Note:}  The candidate presents a specific numeric piece of information that clearly contest the topic. You are not expected to fact check the provided piece of information, don’t reject such a candidate just because you are not sure that the provided piece of information is true.


\section{Conclusions}
\label{sec:conclusions}

We have examined the translate-train paradigm for three multilingual argument mining tasks: stance classification, evidence detection, and argument quality, evaluating a wide range of multilingual models on machine-translated and human-authored data. 
These tasks differ in their complexity, as reflected in the agreement of annotators on the correct label, the extent to which this label is preserved across translation, and, ultimately, in the accuracy of the models. 

Accordingly, our results show that the translate-train approach is well suited for stance classification, as performance improves when 
augmenting the English training data with automatic translations from other languages.
For evidence detection, adding data from the target language or related languages improves performance, yet adding more languages is not helpful.

For both tasks on the evidence data, adding more English 
training data improves performance. 
In these cases, augmenting the large English training set with data of other languages only leads to a marginal gain
for stance classification, and even degrades performance for evidence detection.

In contrast with the above two tasks, the results on argument quality show that training only on English is at least as good, if not better, than any translate-train model.
This is reflected by the clearly opposite trends observed in \figureRef{fig_sbc_arg_quality_hg} vs. those observed in \figureRef{fig_sbc_stance_hg}.

Taken together, our results confirm the validity of the common translate-train paradigm for argument mining tasks such as stance and evidence detection, for which the label is relatively well preserved under translation. However, for the more subtle argument-quality task, where the label -- as might be expected -- is far less preserved, a new approach might be needed.
Future work might wish to explore how translation can preserve not only the semantics of texts, but also finer aspects that contribute to its quality. 
 







\section{Data Sets}
\label{sec:datasets}


\subsection{Translated Data}
\label{par:argmining_data}
\paragraph{English Datasets}
The sources for our translated training data and the ”pseudo test” sets are two existing argument mining datasets in English, collected by our colleagues as part of our work on Project Debater.\footnote{Project Debater is the first AI system that can debate humans on complex topics: \url{https://www.research.ibm.com/artificial-intelligence/project-debater/}} 
One is a corpus of 30,497 arguments on $71$ controversial topics, annotated for their stance towards the topic and for their quality \cite{gretz2020quality}. This dataset (referred herein as \argumentDatasetName) is used for the stance classification and argument quality tasks. 

The second dataset is a corpus of 35,211 sentences from Wikipedia on 321 controversial topics, annotated for their stance towards the topic and the extent to which they can serve as an evidence for the topic \cite{working-solution}.
This dataset (referred as \evidenceDatasetName) is used for the stance classification and evidence detection tasks. 
Example \ref{ex:data} shows an argument and an evidence for one topic.

\input{data_examples}

A third dataset was used to augment the training data for evidence detection and stance classification -- the so-called VLD dataset of \citet{working-solution}, which includes around 200k sentences from newspaper articles,  
pertaining to 337 topics.\footnote{The stance labels of the \evidenceDatasetName and VLD datasets are not part of their official releases. They are included in our release.}

\paragraph{Data Selection}
\label{subsec:data_selection}
The \argumentDatasetName dataset was filtered for stance classification by selecting arguments with a clear stance (confidence $> 0.75$) for training and evaluation. 
For argument quality, arguments with a clear agreement on their quality were selected -- quality score above $0.9$ or below $0.4$. 

A positive label for evidence detection was assigned for evidence from \evidenceDatasetName with a score above $0.7$, and a negative label to those with score below $0.3$ (those with in-between scores were not used).
For stance classification on the evidence data, all sentences with a non-neutral stance were selected, since the \evidenceDatasetName dataset does not provide a confidence score for the stance label.

The VLD corpus was also filtered, taking sentences with evidence score above $0.95$ or below $0.05$.
This yielded a total of 52,037 sentences for training, of which
19,406 have a positive label. 

\paragraph{Translation}
\label{par:mt_data}
We used the Watson Language Translator\footnote{\url{www.ibm.com/watson/services/language-translator/}} to translate the selected English data into $5$ languages: German (DE), Dutch (NL), Spanish (ES), French (FR), and Italian (IT).
The translation 
is a one-time process, which can be applied to any target language (TL) that the MT engine in use supports.
The labels for the MT data were projected from the English data.

Following the data splits provided in the official release of each English corpus (into a training, development, and test sets), the translations of the training data were used for fine-tuning mBERT, and the translations of the test data were used for evaluation.
The translations of the test data of \argumentDatasetName and \evidenceDatasetName\xspace into the $5$ non-English languages -- the pseudo test sets -- are herein referred to as \pseudoTestArgDatasetName and \pseudoTestEviDatasetName.
The statistics of the translated English data are summarized in \tableRef{tab:selected_english_data_stats}.

\subsection{Human-Generated Data}
\label{par:hg_data}

\input{tables/selected_english_data_stats}

Arguments written in the TL provide a more realistic evaluation set than translated texts, specifically for tasks where labels are not well-preserved across automatic translation. 
Therefore, we created a new multilingual evaluation set by collecting arguments in all $5$ languages (ES, FR, IT, DE, and NL) 
for all the $15$ topics of the  \argumentDatasetName test set, using the \textit{\figureEightName}\footnote{\url{www.appen.com}} crowdsourcing platform. 
The human-authored evaluation dataset is herein referred to as \humanGeneratedArgDatasetName.

\paragraph{Annotation Setup}
Initially, crowd contributors wrote up to two pairs of arguments per topic, with one argument supporting the topic and another contesting it in each pair.
Next, the arguments were assessed for their stance and quality by $10$ annotators (per-language).
Given an argument, they were asked to determine the stance of the argument towards the topic and to assess whether it is of high quality.
The full argument annotation guidelines are included in the \supp, and \tableRef{tab:human_generated_labeling_stats} details the number of arguments collected and labeled per language.
To set a common standard, annotators were instructed to mark about half of the arguments they labeled as high quality. 
Annotation quality was controlled by integrating test questions (TQs) with an a-priori known answer in between the regular questions, measuring the per-annotator accuracy on these questions, and excluding underperformers.

A per-annotator average agreement score was computed by considering all peers sharing at least $50$ common answers, calculating pairwise Cohen's Kappa \cite{cohenKappa} with each of them, and averaging. 
Those not having at least $5$ peers meeting this criterion  
were excluded and their answers were discarded. 
Averaging the annotator agreements yields the average inter-annotator agreement (\agreementName) of each question.

To derive a label (or score) for each question we use the WA-score of \citet{gretz2020quality}. 
Roughly, answers are aggregated with a weight proportional to the agreement score for the annotators who chose them. 
At least $5$ answers were required for a question to be considered as labeled. 

Scaling the annotation from English to new languages required some adjustments, such as restricting participation to countries in which the TL is commonly spoken, and the use of TQs for the argument quality question. Further details are provided in the \supp.

\input{tables/human_generated_corpus}

\paragraph{Results} 
\tableRef{tab:human_generated_labeling_stats} presents the \agreementName for all TLs and each task for the human-generated dataset. 
For stance, the agreement is comparable to previously reported values for English ($0.69$ by \citet{toledo-etal-2019-quality} and $0.83$ for \argumentDatasetName).
For quality, the agreement is significantly better than previously reported on \argumentDatasetName ($0.12$ by \citet{gretz2020quality}),  presumably due to the use of TQs in this task, which were not included before.
The annotation in each of the non-English languages involved a distinct group of annotators, producing varying annotation quality among languages which is reflected in their \agreementName values.

The results also include the percentage of arguments labeled as supporting arguments, computed separately for each annotator and averaged over all annotators.
All values are close to $0.5$, confirming that the collected arguments are balanced for stance, as instructed.
Similarly, the results show the  percentage of arguments labeled as high quality, averaged over all annotators, confirming that annotators mostly followed the instruction to label about half of the arguments as high quality.

The same confidence filtering thresholds described in \sectionRef{subsec:data_selection} were applied to 
the data of \humanGeneratedArgDatasetName.
The statistics of the arguments selected for evaluation are shown in \tableRef{tab:human_generated_labeling_stats} (right). 


\section{Experimental Setup}
\label{empirical_evaluation}

Our experiments are aimed at providing a comparative analysis of the translate-train approach when trained on different subsets of languages, and identifying when that approach is beneficial on the three argumentation tasks.
We begin by describing the setup used in all experiments.

\paragraph{Training Configuration}
We used the BERT-Base multilingual cased model configuration (12-layer, 768-hidden, 12-heads, total of 110M parameters) with a sentence-topic pair input.
Training was performed on one GPU. 

The parameters configuration of the binary classification tasks, namely, stance classification and evidence detection, was:
maximum sequence length of $128$, batch size of $32$, dropout rate of $0.1$ and learning rate of 5e-5. Each model was fine-tuned over $10$ epochs, using a cross-entropy loss function. 
The regression model for argument quality prediction, similar to the one used by \citet{gretz2020quality}, used 
a maximum sequence length of $100$, a batch size of $32$, a dropout rate of $0.1$ and a learning rate of 2e-5. 
Each model was fine-tuned over $3$ epochs, using a mean-squared-error loss function. 
In all cases, the model from the last epoch was selected for evaluation. 

\paragraph{Translate-Train Models}
\label{par:lang_groups}
For each task, mBERT was trained using data translated into one of the target languages (ES, FR, IT, DE and NL).
These per-language models, denoted herein as \emph{TL}, are the simplest application of the translate-train approach.
Two more models were trained for the language families that are represented in the above languages together with English: 
\emph{RM} -- for the Romance languages (ES, FR, IT), and \emph{WG} -- for the West-Germanic languages (EN, DE, NL). 
Each language family model was trained using the data of the languages in that family.
Lastly, a model was trained on data from all 6 languages (denoted \emph{6L}).
To summarize, our evaluation includes $4$ models based on the translate-train approach (\emph{TL}, \emph{6L}, \emph{RM} and \emph{WG}) for each task and TL.

\paragraph{Baselines}
Another mBERT model, denoted \emph{EN}, was trained on the source English data. 
Using this model, the results of two baselines are reported:
(i) zero-shot (denoted \emph{ZS}) -- which passes an input text in a non-English language to the \emph{EN} model, utilizing the cross-lingual transfer capabilities of mBERT; (ii) translate-test (denoted \emph{TT}) -- in which an input text is machine translated into English, and that translation is provided as input to the \emph{EN} model.
The results on English data 
are also reported as a performance benchmark for other languages.
Obviously, for English, the results of the \emph{TL} model and the \emph{ZS} and \emph{TT} baselines are identical.

\input{figures/arguments_stance_results}

\paragraph{Related and Distant Languages}
For each TL, we define two types of models. The \emph{RL} model is the one trained on \emph{related} languages from the same family as the TL, and  \emph{DL} is the model that is trained on languages that are more \emph{distant} from the TL.
In other words, given a TL, the \emph{RL} model refers to the language family that includes the TL, and the \emph{DL} model refers to the other family that does not include the TL.
For example, in the case of the TL being German, \emph{TL} denotes the model trained only on translated data in German; 
\emph{RL} -- the \emph{WG} model trained on the 3 West-Germanic languages (including German); 
\emph{DL} -- the \emph{RM} model trained on the 3 Romance languages.

\paragraph{Evaluation Metrics}
The reported metrics are macro-F1 for the classification tasks (stance classification and evidence detection), and Pearson correlation for the regression setting of argument quality.







\section{Introduction}
\label{sec:intro}

Argument mining has received much attention in recent years, with research mainly focused on English and, to some extent, German texts. Recent advancements in Natural Language Understanding suggest that in order to train appropriate models for argument mining tasks in other languages, we do not need to manually label text in these languages, but rather employ transfer learning from the English-based models \cite{eger-etal-2018-cross}.

In this work we examine three argument mining tasks: (1) \emph{stance classification}: given a topic and
an argument that supports or contests the topic, determine the argument's stance towards the topic; (2) \emph{evidence detection}: given a topic and a sentence, determine if the sentence is an evidence relevant to the topic; (3) \emph{argument quality}: given a topic and a relevant argument, rate the argument so that higher-quality arguments are assigned a higher score.

To facilitate transfer learning from English datasets for these tasks, we employ Multilingual BERT (mBERT) released by \citet{devlin2019bert}, 
a pre-trained language model that supports 104 languages, and use it mainly in a translate-train approach. 
Namely, the English dataset is automatically translated into the desired language(s) using machine translation (MT); an augmented dataset composed of the original English text and all the translated copies is created; the mBERT model is fine-tuned on a subset of the dataset; and the resultant model is then used to solve the relevant downstream task in the desired language. 
Previous works have suggested that translating the original dataset to as large a number of languages as possible is beneficial \cite{liang2020xglue}. 
In this work, we show a more nuanced picture, where often selecting a subset of related languages is preferable.

In addition, we also examine the translate-test approach, in which one creates a classification model only with English data.
At prediction time, the non-English text is automatically translated into English, and then analyzed by the model.  
This approach is less appealing since the initial translation step increases prediction run-time, and on our data also tends to perform worse.

We examine two text sources for performance evaluation on non-English texts. The first one is a "pseudo test" set -- an automatic translation of an English evaluation set for a task. 
While such texts can be easily generated, it is not clear how well they represent "real" texts, authored by humans. Hence, we also examine human-authored texts, in several non-English languages, collected via crowdsourcing specifically for this work. 
Both datasets are released as part of this work.\datasetURL

When translating the evaluation set, either automatically or by a human translator, one would like to assume that the initial label of the English text is maintained after translation.
While this is often the case, we show that this assumption becomes more dubious as the argument mining task becomes more complex and subjective, as well as when the original labels are not clearly agreed upon.

In summary, the main contributions of this paper are:
(1) a comparative analysis of the translate-train approach on three central argument mining tasks using different subsets of languages,
showing that training on more data helps, but that, in some cases, training on related languages is sufficient; 
(2) multilingual benchmark datasets for the three tasks; 
(3) an analysis of the three tasks, showing how well labels are preserved across translation, and the impact that has on the success of the translate-train approach.

\section*{Acknowledgments}
Special thanks go to Michal Jacovi, Christopher Sciacca, 
and Mohamed Nasr for coordinating the manual translation effort. 
We thank the teams who performed the translation, and Benjamin Sznajder for helping with the analysis.
We also thank the anonymous reviewers for their valuable comments.





\section{Related Work}
\label{sec:related_work}
Argument mining has been expanding from identifying argumentative passages to a variety of Natural Language Understanding (NLU) tasks (\citet{stede2018argumentation,lawrence2020argument}). 
In this work, we explore three argumentation tasks in multilingual settings.

Stance detection (or classification) is often contrasted with sentiment analysis, in that the task is not simply to classify the sentiment of a text, but rather its stance w.r.t. some given target. Early work on this task includes \citet{thomas2006get, lin2006side}, while in the context of argument mining, the task has probably been introduced by \citet{sobhani2015argumentation}. 
As with many other NLU tasks, earlier works developed classifiers based on various features (e.g., \citet{barhaim2017stance}), while more modern approaches rely on deep learning. See \citet{schiller2020stance} for a recent benchmarking report on such methods. 

Research on stance detection in a multilingual setting is rather recent. \citet{zotova2020multilingual} explore stance detection in Twitter for Catalan and Spanish; \citet{lai2020multilingual} do this for political debates in social media in these two languages as well as French and Italian; \citet{vamvas2020x} analyze the stance of comments in the context of the Switzerland election in German, French and Italian. 
Stance detection is reminiscent of the Natural Language Inference (NLI) problem, where one is given two sentences, and the objective is to determine whether one entails the other, contradicts it or is neutral. This task had been researched extensively, with \citet{conneau2018xnli} providing a 15-language benchmark for the multilingual setting. Another earlier work on a related task is addressing the support/attack relation prediction of two arguments in Italian \cite{basile-2016-arg-italian}.

Evidence detection is the task of determining, given some text and a topic, whether the text can serve as evidence in the context of the topic \cite{rinott-etal-2015-show}. 
We follow \citet{working-solution}, in defining \emph{evidence} as \textit{a single sentence that clearly supports or contests the 
topic, yet is not merely a belief or a claim. 
Rather, it provides an indication for whether a relevant belief or a claim is true}, and, since we use their datasets, we restrict our analysis to evidence of type Study and Expert. 
In a multilingual setting, a similar task, that of premise detection, was considered in \citet{eger-etal-2018-cross} for German, French, Spanish, and Chinese; in \citet{fishcheva2019cross} for Russian; in \citet{eger2018pd3} for French and German; and in \citet{aker2017projection} for Chinese. 

Argument quality prediction is the task of evaluating the quality of an argument, either on an objective scale - the input is an argument and the output is a quality score; or in a relative manner - the input is a pair of arguments and the output is which of them is of higher quality. 
While there are many, arguably independent, dimensions for quality \cite{wachsmuth2017computational}, it seems that people - and, consequently, algorithms - can usually perform this task in a consistent manner \cite{habernal2016argument, wachsmuth-etal-2017-argumentation, toledo-etal-2019-quality,gretz2020quality}. 
To the best of our knowledge, this task was not previously considered in a multilingual setting.

In contrast with previous multilingual research on argument mining, in this work we address three different problems, of varying complexity, over a relatively large number of languages. 
This allows us to draw more holistic conclusions on the efficacy -- and pitfalls -- of transfer learning in the argument mining domain. 
It is interesting to compare these conclusions with other wide-scope multilingual NLU research, such as the XTREME \cite{hu2020xtreme} and XGLUE \cite{liang2020xglue}. 


\section{Results}
\label{results}

The results below for arguments (for stance classification on that data and argument quality) are averages over $5$ evaluation runs of randomly initialized models that were trained in the same manner. 
For evidence sentences (stance classification on that data and evidence detection), the results are from a single evaluation run. 


\subsection{Stance Classification}
\label{subsec:stance_classification_results}

\paragraph{Arguments} 

\figureRef{fig_sbc_stance_hg} shows the evaluation results on the human-generated arguments of \humanGeneratedArgDatasetName. 
For the non-English languages, the performance over the \emph{ZS} baseline improves when adding translated data, even when that data is from distant languages (\emph{DL}).
The other baseline \emph{TT} is better, yet the best performance is attained by the \emph{6L} 
models
-- significantly so for 3 of the 5 languages (ES, FR and DE).
Notably, ordering the translate-train models by their performance yields the same order for all languages: \emph{DL} is always the worst, followed by \emph{TL}, \emph{RL} and the best performing model \emph{6L}.

Repeating the same experiments on the pseudo-test data of \pseudoTestArgDatasetName resulted in similar trends, depicted in \figureRef{fig_sbc_stance_mt}. 
Further augmentation of the training data with translations to more languages beyond the languages included in the training of the \emph{6L} models (e.g. with 9 or 17 languages) did not significantly improve performance on these languages.
These results are detailed in \tableRef{tab_sbc_stance_mt} within the \supp.

\input{figures/evidence_results}

\paragraph{Evidence} To explore whether the observed trends are data-specific, we repeated the evaluation of the stance classification task with 
the \pseudoTestEviDatasetName dataset of evidence sentences from Wikipedia.
Training was performed on the training set of that corpus (called Wikipedia models).
The results on its pseudo-test set are depicted in \figureRef{fig:evidence_stance_wiki}. 
For the 
non-English
languages, the best performing models are \emph{6L} and \emph{RL}, consistent with our findings for arguments (\figureRef{fig:argument_stance_results}).

The VLD evidence curpus allows further exploration of the stance classification task within the evidence domain.
We trained models on a larger dataset of translated evidence combining the Wikipedia data and selected data from the VLD corpus (called Extended models).
\figureRef{fig:evidence_stance_extended} shows the results obtained using 
these models.
Overall, the performance of the Extended models is significantly better than the performance of the Wikipedia models, in almost all cases, and the \emph{TL} models become competitive even with the \emph{6L} models.

\paragraph{Performance on English}
In comparison with the \emph{ZS} baseline (trained only on English), adding translated training data improves performance on English (leftmost bars in Figures \Ref{fig_sbc_stance_hg} and \ref{fig:evidence_stance_wiki}), for both domains.   
For the evidence data, even translations to distant languages (\emph{DL}) help the Wikipedia model, yet when a lot of training data is available in English (leftmost bars in \ref{fig:evidence_stance_extended}), there is no significant gain from adding translations to the training set.

\paragraph{Summary}
Overall, the best performing models for the stance classification task are the \emph{RL} and \emph{6L} models, in both domains.
Our finding that the \emph{6L} models outperform the \emph{TL} models is consistent with previous results on the XNLI task \citep{hu2020xtreme}.
Interestingly, translated data can be used to improve performance on English as well.

The \emph{ZS} and \emph{TT} baselines are almost always outperformed by the best translate-train model.
However, when a large-scale English corpus is available (\figureRef{fig:evidence_stance_extended}), the \emph{TT} baseline becomes comparable to the best translate-train models.

\subsection{Evidence Detection}
\label{subsec:evidence_detection_results}

To examine whether the above observations are task-specific, we move on to the task of evidence detection.
The results for that task on the 
\pseudoTestEviDatasetName pseudo-test are 
depicted in \figureRef{fig:evidence_detection_wikipedia} (Wikipedia models) and in \figureRef{fig:evidence_detection_extended} (Extended models).

In contrast with the stance results, where in most cases the \emph{6L} models were best, for evidence detection performance may degrade when adding languages. 
The best performing translate-train models are either the \emph{TL} or \emph{RL} models, in all cases.

As in the stance classification results for this corpus, the additional training data used in the Extended models improves performance. 
In addition, the English benchmark results for the Wikipedia models (leftmost bars in  \figureRef{fig:evidence_detection_wikipedia}) can improve by adding languages, or by adding English data (\emph{ZS} bar for EN in \figureRef{fig:evidence_detection_extended}), but there is no significant gain from doing both
(leftmost bars in \figureRef{fig:evidence_detection_extended}).

\subsection{Argument Quality Prediction}
\label{subsec:argument_quality_results}

Moving to our last task, \figureRef{fig_sbc_arg_quality_hg} shows the Pearson correlation results on the human-generated arguments, between the predicted 
quality score and the labeled argument quality score.
In contrast with the stance results, adding data from related languages (the \emph{RL} bars) does not help, and training on the English dataset (the \emph{ZS} bars) is sufficient to obtain a competitive model.\footnote{The performance of the \emph{ZS} model on English is $0.61$ -- comparable to its performance for FR and DE.}
We suspect that the reason for this is that this task is more complex and nuanced than the previous two. 

\section{Analysis}

The performance of a translate-train model may be affected, among other factors, by the translation quality, the extent in which a task-specific label is preserved across that translation, and, for our data, the discussed topic. 
These are analyzed below.

\subsection{Translation Quality Assessment}
We assessed our machine translation quality by computing the BLEU score \cite{papineni2002bleu} between the English arguments from the test set of  \argumentDatasetName
and the same arguments after translation to a TL and back to English. 
For all languages, these scores are above $0.5$ (see \tableRef{tab:mt_bleu}), suggesting the translations are of high quality.

\input{figures/argument_quality}

\subsection{Translated Label Assessment}
\label{subsec:translated_label_assessment}
An important prerequisite for training and evaluating models on automatically translated texts is that the labels of the original texts are preserved under translation, which depends on the specific task at hand.
Example \ref{ex:back-translation} shows one argument and its translation to Spanish and back to English. The translation preserves the original stance,  
but the argument quality is degraded.
Hence, we annotated a sample of the translated texts to assess how often this  happens in each task.
The annotation focuses on one Romance and one West-Germanic language -- Italian and German.

\input{tables/mt_bleu}

\input{quality_example}

\paragraph{Annotation Setup}
$14$ arguments were randomly sampled from each  
topic of the \argumentDatasetName test set, yielding $210$ arguments per language.
Similarly, two sentences were sampled from each  
topic in the \evidenceDatasetName test set, producing $200$ sentences per language. 
All texts were machine translated and human translated by native speakers of each TL.
Both translations of each argument were labeled for their stance and quality, as in \sectionRef{par:hg_data}.
Similarly, the potential evidence sentences were annotated for whether they are valid evidence, and those which are so were also annotated with their stance towards the topic, as in \citet{working-solution}.
In this annotation, TQs were formed from translated texts, with the correct answer taken from the English labels.
The full evidence annotation guidelines are included in the \supp.

\paragraph{Results}
\tableRef{tab:translated_arguments_assessment} shows the assessment results for all tasks and the two languages.
The obtained \agreementName is on par with previously reported values for these tasks (as detailed in \sectionRef{par:hg_data}), though somewhat lower for evidence detection. 
The table further shows Pearson correlation between the original English WA-scores, and the WA-scores of the translated texts.
For evidence detection and argument quality, this computation was performed on texts matching the criteria defined in \sectionRef{subsec:data_selection}. 
The correlation for evidence stance classification was computed on sentences with at least $6$ stance labels on their translated version.

The results show that for both datasets, 
stance is well preserved after translation. 
For evidence detection, the correlation is lower, yet the difference between \machineTranslated and \humanTranslated is small,
suggesting the change in the labels is not due to the automatic translation. 
Thus, the use of translated texts in
these tasks is acceptable, for both training and evaluation.

For argument quality, the correlation is considerably lower, and there is a significant difference between \machineTranslated  and \humanTranslated in \italianShort, as may be expected for such a nuanced task. 
This could be the reason that the translate-train models do not improve performance for this task -- since the quality label is not maintained when an argument is translated, projecting these labels into translated texts introduces significant noise into such training data.

\input{tables/translated_arguments_assessment}

\subsection{Per-Topic Analysis}

In both of our data domains, arguments and evidence, the texts are relevant to a specific topic, and the obtained performance may depend on that topic in various ways. 
Focusing on stance classification, we measured the per-topic performance on the human-generated arguments of \humanGeneratedArgDatasetName.
\figureRef{fig_sbc_stance_hg_avg5tls_per_topic} shows these results averaged over the $5$ non-English
languages, for the \emph{TL} and \emph{6L} translate-train models, and the \emph{ZS} and \emph{TT} baselines.
The topics are ordered by their performance on English.\footnote{\tableRef{tab:sbc_test_motions} in the \supp lists the topic for each topic ID.}

The results demonstrate the performance variability among the different topics.
For some, the average performance on the 
non-English languages is close to their performance on English (e.g. topics 9 or 10), yet for others it is far from it (e.g. topic 5).  
The performance of the \emph{ZS} baseline is low  
for topics 2 through 8, from which $5$ are discussing imposing a ban. 
This implies that the stance towards the discussed topic, or the  "action" within the topic (e.g. ban, legalize, etc.) may be an important factor.

We further manually analyzed the results on Topic 10, a low-performing
outlier in French for the \emph{ZS} baseline (see \figureRef{fig_sbc_stance_hg_zs_per_topic} in the \supp).
A native speaker examined $3$ batches of $20$ arguments each, containing: 
1) prediction errors from that topic;
2) randomly sampled correct predictions from the same topic;
3) all 4 prediction errors and 16 randomly sampled correct predictions from
the topic with the highest performance (Topic 1).
Within the first batch, $40\%$ of the samples were incoherent or syntactically wrong arguments, compared to only $20\%$ in each of the other two batches.

\begin{figure}[t]
\includegraphics[width=\linewidth,
trim={1.0cm, 0.2cm, 1.4cm, 0.8cm},
clip]{figures/fig_sbc_stance_hg_per_topic.pdf}
\caption{Per-topic stance classification average macro-F1 results on 
\humanGeneratedArgDatasetName, averaged over the $5$ non-English languages. 
The topics are ordered by their performance on English with the \emph{EN} model (dashed black line).
}
\label{fig_sbc_stance_hg_avg5tls_per_topic}
\end{figure}











