\section{Appendix}
\label{sec:appendix}

\subsection{Sequence Length Distribution} \label{app:token-seq-len}
As described in~\Secref{sec:token-sub}, we try four different tokenizers to substitute for our \texttt{RoBERTa}~\cite{liu2019roberta} model that uses the Byte-Pair Encoding (BPE) \citep{sennrich2015neural} tokenizer. Specifically, we substitue with the WordPiece tokenizer~\cite{wu2016google} used by \texttt{BERT}~\cite{devlin2018bert} (i.e., $\texttt{BERT}$ Tokenizer in \Tabref{tab:badglish-examples}) and the SentencePiece tokenizer~\cite{kudo2018sentencepiece} used by \texttt{Albert}~\cite{lan2019albert} (i.e., $\textbf{\texttt{Albert}}$ \textbf{Tokenizer} in \Tabref{tab:badglish-examples}). Additionally, we substitute with two new non-English tokenizers including the French \texttt{FlauBERT}~\cite{le2020flaubert} ($\textbf{\texttt{FlauBERT}}$ \textbf{Tokenizer} in \Tabref{tab:badglish-examples}) and the Dutch \texttt{DutchBERT}~\cite{devries2019bertje} ($\texttt{DutchBERT}$ Tokenizer in \Tabref{tab:badglish-examples}). As shown in \Figref{fig:tokenizer-seq-len}, we plot the distributions of sequence lengths as a measure of the heterogeneity introduced by new tokenizers to ensure variences across tokenized sequence lengths. Specifically, we see there are inferior tokenizers such as $\texttt{FlauBERT}$ Tokenizer with a 22.15\% increase in sequence length. Our results are consistent with previous findings~\cite{rust2020good} where sequence length distributions are closer

\subsection{Training Set-up} \label{app:training-setup}

\paragraph{Downstream Task.} We use the GLUE benchmark~\cite{wang2018glue} to evaluate model performance, which covers nine different NLP tasks. We report scores on the development sets for each task by fine-tuning our pre-trained or mid-tuned models. We fine-tune for 5 epochs for the smaller datasets (WNLI and MRPC) and 3 epochs for the others. For the performance metrics, we use Matthew's Correlation for CoLA, Pearson correlation for STS-B, and accuracy for all the other datasets.

\paragraph{Hyperparameter and Infrastructure.} For each of the mid-tuning and fine-tuning experiments, we collect averaged results from 3 runs with distinct random seeds. We tune our models with two learning rates $\{2e^{-5}, 4e^{-5}\}$, and report the best results from these two learning rates. Fine-tuning with 9 GLUE tasks takes about 8 hours on 4 NVIDIA Titan 12G GPUs. Mid-tuning with our subset of WikiText-103M corpus takes about 18 hours with the same infrastructure.

\input{figures/tokenizer-seq-len}

\subsection{GLUE Task Performance} \label{app:per-task-perf}
\Tabref{tab:per-task-perf} shows performance break-down for individual GLUE task under different transformations as described in \Secref{sec:transformation-type}.  

\input{tables/per-task-results}





\section{Introduction}
What makes it hard for neural networks to learn new languages? Despite their strengths, large-scale pretrained language models (LLMs) require very large datasets for pretraining, making it challenging to train LLMs from scratch for low-resource languages~\cite{devlin2018bert, liu2019roberta, lacoste2019quantifying, clark2020electra}. For such languages, an appealing approach is to \textit{transfer} knowledge from an LLM trained for a high-resource language, since such models may transfer knowledge even under extreme conditions~\cite{papadimitriou2020learning, tamkin2020}.
A range of training strategies and model architectures have been explored to enable such crosslingual transfer of English LLMs, leveraging different techniques including adaptive pretraining~\cite{reimers2020making}, and embedding re-training~\cite{artetxe2020cross, tran2020english}. However, progress is challenging as it is hard to distinguish factors that effect such transfer.



\input{figures/main-figure-remake}



In this paper, we present a set of \textit{controlled transfer studies} for determining what makes cross-lingual transfer hard. We focus on three factors salient to crosslingual transfer: the \textbf{embedding layer}, the \textbf{tokenizer}, and \textbf{syntactic shifts}. We construct a set of systematically transformed versions of GLUE (t-GLUEs) targeting each of these factors, and observe how the finetuning performance of a pre-trained English \texttt{RoBERTa}~\cite{liu2019roberta} model degrades as a result of each of these transformations.
Crucially, our method allows us to disentangle the effects of correlated factors: while all factors would change at once if we transfer between natural languages, our transformations allow us to pinpoint what causes difficulties for transfer. 

First, we focus on vocabulary effects: what is it that makes the embeddings important in cross-lingual transfer? We separately test the effects of having a consistent word embeddings layer, the
morphological consistency of the tokenizer, and the all-around quality of tokenization. Surprisingly, we find that the effect of word embedding alignment overshadows any aspect of tokenizer quality that we test. Second, we test how all of these vocabulary-level factors compare to the effects of structural syntactic perturbations, and find that structural perturbations have a significantly smaller effect on transfer ability than vocabulary effects.











\section{Related Work}


Cross-lingual transfer studies on multi-lingual models such as Multilingual BERT~\cite{devlin2018bert} demonstrate the utility of multilingual training for producing models with parallel representations that ease zero-shot transfer~\cite{pires2019multilingual}. 
However, it is unlikely that a unified model can cover the more than 6,900 languages in the world \cite{feng2020language, wang2020negative}.
Enabling cross-lingual transfer from one language to another remains an important tool for expanding large-scale models to more languages, and
we aim to examine such cross-lingual transfer with well-defined experiments covering distinct linguistic factors.




Our experiments build off previous efforts that try to enable crosslingual transfer from pretrained monolingual LLMs to new languages~\cite{artetxe2018generalizing, artetxe2020cross, tran2020english, reimers2020making, gogoulou2021cross}. For example, BERT shows effective but limited transfer when attached to multi-lingual tokenizers~\cite{reimers2020making}. Recent work further improves this approach through continued multi-stage pretraining on foreign languages, which helps the English LLMs acquire new word embeddings~\cite{tran2020english}. In this work, we aim to provide quantitative insights into how these factors, such as static embeddings similarity, affect crosslingual transfer through controlled studies.

\input{tables/badglish-examples}




\section{Transformed English (t-Englishes)}
\label{sec:t-englishes}

\label{sec:transformation-type}
\Figref{fig:overview} shows our main experimental paradigm. We study cross-lingual knowledge transfer from English to t-Englishes: versions of English that are systematically altered through different types of transformations. Here we explain and motivate each t-English variant that we study, organized by the main component they consider in crosslingual transfer: the embedding layer, the tokenizer, and syntactic shifts. A full list with examples can be found in \Tabref{tab:badglish-examples}. 

\subsection{Embedding Layer} 

Previous works have consistently found that good embeddings are crucial for enabling effective cross-lingual transfer~\cite{tran2020english, artetxe2020cross}. However, these gains could be due to several factors, including better initialization statistics \citep{raghu2019transfusion}, or to a learned alignment between the learned embeddings and the pretraiend transformer layers \citep{wu2021identifying}. We consider a baseline setting, where we reinitalize the embedding layer such that the optimization during fine-tuning has to fully reconstruct the semantic space, and compare it against two other perturbations which preserve the statistics of the embedding weights: 
1) a \textbf{token swap} scenario, where we scramble the identities of tokens so that the meaning of each token is not represented by its row in the word embeddings matrix, and 2)  a more challenging \textbf{word swap} scenario where the identities of whole words are scrambled before tokenization. In the word swap scenario, the morphological and sub-word consistency of the tokenizer is lost: if the word ``cat'' gets swapped to ``audible'', and the word ``bravery'' gets swapped to ``visible'', then after tokenization the ``-ible'' subword does not represent a consistent morpheme.

\subsection{Tokenizer}

\label{sec:token-sub}
Tokenizers have been shown to play an important role in crosslingual transfer for multi-lingual models~\citep{rust2020good}. In this section, we investigate how changes in tokenization between the source and target languages impacts downstream performance, holding all other factors constant. To do so, we substitute the Byte-Pair Encoding (BPE) \citep{sennrich2015neural} of the \texttt{RoBERTa} model with the WordPiece tokenizer~\cite{wu2016google} used by \texttt{BERT}~\cite{devlin2018bert} and the SentencePiece tokenizer~\cite{kudo2018sentencepiece} used by \texttt{Albert}~\cite{lan2019albert}. We also experiment with two non-English tokenizers, which produce lower-quality tokenizations for English:
(see~\Appref{app:token-seq-len}): the French \texttt{FlauBERT}~\cite{le2020flaubert} and the Dutch \texttt{DutchBERT}~\cite{devries2019bertje}. Examples of each tokenization scheme are presented in \Tabref{tab:badglish-examples}. For new tokenizers, word embedding weights are reinitialized accordingly.


\subsection{Syntactic Shifts} 

\label{sec:syntactic-shifts}
While syntax is an crucial aspect of language, studies have also shown syntactic typology to be surprisingly non-predictive of transfer quality \citep{pham-etal-2021-order}, and other studies have shown LLMs to be largely word-order invariant \citep{sinha2021masked}.
We investigate a set of syntactic transformations that isolate syntactic word-order shifts from the other factors that can vary between languages such as tokenization, static embeddings, and morphological representation. 
We use the open-source package \texttt{Galactic-Dependencies}~\cite{wang2016galactic} to transform the word order of our English training corpora to match the noun-phrase order and the verb-phrase order of French and Japanese ($\{\textbf{N}_{\text{fr}}, \textbf{V}_{\text{fr}}\}$ and $\{\textbf{N}_{\text{ja}}, \textbf{V}_{\text{ja}}\}$ in \Tabref{tab:badglish-examples}) and also perform a mixed transformation with French noun order and Japanese verb order ($\{\textbf{N}_{\text{fr}}, \textbf{V}_{\text{ja}}\}$ in \Tabref{tab:badglish-examples}). 
We use the open-source package \texttt{Stanza}~\cite{qi2020stanza} for sentence segmentation and parsing
before scrambling word orders. We also test the the stronger transformation of completely reversing the word order, as well as randomly shuffling the word order to produce a lower bound on the amount of useful syntactic information.




\section{When Does Continued Pretraining Help?}

Instead of finetuning LLMs directly on target language tasks, continued pretraining on a target language corpus may help with model adaptation~\cite{artetxe2020cross, tran2020english}. Here, we examine how continued pretraining fares as a potential solution to the different shifts we consider in \Cref{sec:t-englishes}; this corresponds to the setting where there might exist a moderate amount of unlabeled data available for a low-resource language. Formally, we continued pretraining \texttt{RoBERTa} using the masked language modeling objective on a t-English corpus before evaluating its finetuning performance with downstream tasks. We use a subset of WikiText-103M corpus~\cite{merity2016pointer} containing approximately 15\% of examples for mid-tuning.\footnote{For comparison, the pretraining data for $\text{BERT}_{\text{BASE}}$ contains 3.3B tokens~\cite{devlin2018bert}. Here, we have about 15M tokens which is about 0.45\% of its pretraining data.} For each t-English variant, we ensure the number of steps for continued pretraining are kept constant. 

\section{Results}
We use the GLUE benchmark~\cite{wang2018glue} to evaluate model performance, which consists of nine different NLP tasks. We produce t-GLUEs by transforming both the training and validation sets, and report scores on the transformed validation sets after finetuning our pre-trained or continued pretrained models. Each experiment is run three times with different random seeds. We include details in \Appref{app:training-setup}.

\subsection{Good Embeddings Are Most of What You Need}

As expected, we find a very large drop in GLUE performance (-40\%) from reinitializing the embeddings layer (\Cref{fig:perf-identity-swap}), indicating the importance of good embeddings to crosslingual transfer. However, the token swapping results reveal that only half of that drop (-21\%) is attributable to the token-specific information stored in each embedding, with the remainder due to merely having embeddings that lie within a plausible ``language space'' for the model. Surprisingly, scrambling the whole words rather than tokens and breaking the morphological consistency of the resulting tokenization does not have a larger effect than the one-to-one token swap, suggesting minimal additional contribution of subword information in this regime.

Interestingly, we find that continued pretraining closes the gap between the reinitialized and swapped conditions. While this eliminates over half of the performance drop of for the reinitialized t-English (-40\% to -18\%) the gain is much smaller for the swapped varaints (-21\% to -18\%). This suggests that other solutions to the word alignment problem besides continued pretraining will be needed in resource-scarce settings.



\input{figures/identity-swap}

\input{figures/tokenizer-substitution}

\input{figures/syntactic-shifts}

\subsection{Bad Embeddings Can Ruin A Good Tokenizer}\label{sec:token-results}
Surprisingly, we see virtually no effect of using different different tokenizers (\Cref{fig:tokenizer}), despite some of the tokenizers being very different from the original BPE tokenizer.
Similarly, continued pretraining also fails to reveal differences when different tokenizers are used.
These findings suggest that performance drops seen during crosslingual transfer are mostly due to the information loss of re-initializing the embedding weights, rather than the quality of tokenization.
Tokenization may thus be a ``lower-order bit'' for crosslingual transfer, which has little impact until good word embeddings are learned.

\subsection{Syntax Matters, But Not Too Much}  \label{sec:syntax-results}
As shown in \Figref{fig:perf-syn-shifts}, a \texttt{RoBERTa} model finetuned on randomly ordered GLUE sentences experiences a performance drop of 13\%. This is a markedly smaller drop than the word-shuffling experiments (-19\%), indicating that a bag-of-words classifier with good embeddings performs better than a model which experiences no syntactic shifts but that lacks good word embeddings.

This performance gap closes appreciably as we perform more structured syntactic shifts such as reversing the sentence (a drop of 10\%), or systematically permuting word orders using the dependency tree (a drop of between 7\% and 9\%). Rather than being invariant to word orders across natural language understanding tasks~\cite{sinha2021masked, pham-etal-2021-order}, we instead find that BERT-based models are in fact sensitive to word order, at least for the tasks in the GLUE benchmark. In addition, we find that continued pretraining can close the performance gap to all but a few percentage points for tree-based structural shifts. These results suggest that syntactic shifts have real but limited impact on crosslingual transfer, compared to embedding layer effects.

\section{Conclusions}
In this paper, we propose a novel paradigm to study cross-lingual transfer through transformations which simulate the linguistic changes across languages. In contrast to what prior work implies, we find significant effects from syntactic shifts and no effect from tokenizer shifts on cross-lingual transfer. However, our results suggest that solving the embedding alignment problem is the "high-order bit" for crosslingual transfer: it has the largest impact on finetuning performance and is the least improved by continued pretraining. Thus, future progress on solving this problem in large-scale transformers may have outsized impact.

\section{Acknowledgements}

We would like to thank Nelson Liu, Mirac Suzgun and Tol\'{u}l\d{o}p\d{\'{e}} \`{O}g\'{u}nr\d{\`{e}}m\'{i} for useful discussions and comments on drafts.



