\section{Introduction}

In the last years state-of-art results in natural language processing (NLP) are often obtained with Transformer-like architectures based on the self-attention mechanism \cite{vaswani2017attention} such as BERT \citep{devlin-etal-2019-bert}, GPT-3 \cite{brown2020language}, T5 \cite{raffel2020exploring}, which could have billions of parameters. Due to many parameters, these architectures require lots of time and hardware resources to be trained.

Curriculum learning (CL) is one of the popular methods to reduce training time and increase the resulting quality of the model. Inspired by the importance of adequately ordering information when teaching humans \cite{avrahami1997teaching}, curriculum learning increases the difficulty of training samples shown to the model over time \cite{elman1993learning}. Previous studies have demonstrated that curriculum learning significantly impacts training time and quality in different machine learning domains, such as computer vision \cite{Soviany2020CurriculumLW} and reinforcement learning \cite{narvekar2020curriculum}. In NLP, some results hint that CL might be beneficial \cite{platanios-etal-2019-competence,xu2020curriculum,kocmi-bojar-2017-curriculum}; however, these results are not as optimistic as in reinforcement learning setup.


We suggest dividing recent research in curriculum learning into two main categories: {\em task-driven} curriculum and {\em data-driven} curriculum. The idea of the task-driven curriculum was inspired by human behavior. First, the model learns how to solve a simple task, and then the difficulty is gradually increased. This type of curriculum proposed by \citet{bengio2009curriculum} is considered to be classical, and a majority of curriculum-related results are obtained in this framework. Alternatively to the task-driven curriculum, some curricula try to use some form of filtering or sorting of training data that could facilitate learning a model on a given task. We suggest calling these curricula {\em data-driven} and distinguishing them from the classical task-based approach.

This paper attempts to understand when data-driven curriculum learning works for transformer-based language models. Generally, data-driven curriculum learning is organized in two steps: first, estimating the complexity for the elements that comprise the dataset; second, designing a  sampling strategy, thus forming a curriculum. In the first part of the paper, we list potentially useful natural language processing complexity measures. The second part discusses possible sampling strategies that might apply to corresponding complexity measures. We run extensive experiments with different metrics and sampling strategies on three classes of NLP tasks: unsupervised learning with masked language modeling, text classification, and machine translation. Our experiments show that data-driven curriculum learning does not give quality increase or time reduction on all metric-sampling strategy setups and often makes results even worse. 

\begin{figure*}[h!]
	\centering
	\begin{subfigure}{.4\textwidth}
		\centering
    	\includegraphics[scale=0.22]{s140_SM_res.png}    
    	\caption{Sentiment140 with sort-merge sampler for all complexity measures.}
    	\label{fig:s140_SM_all_metrics}
	\end{subfigure}
	\begin{subfigure}{.4\textwidth}
		\centering
    	\includegraphics[scale=0.22]{s140_max_wf_rank.png}    
    	\caption{Sentiment140 with max word rank complexity measure for all samplers.}
    	\label{fig:s140_max_wf_rank_all_samplers}
	\end{subfigure}\\
	\begin{subfigure}{.4\textwidth}
		\centering
    	\includegraphics[scale=0.22]{hnd_SS_res.png}    
    	\caption{Hyperpartisan News with sort-shuffle samples for all complexity measures}
    	\label{fig:hnd_SS_all_metrics}
	\end{subfigure}
	\begin{subfigure}{.4\textwidth}
		\centering
        \includegraphics[scale=0.22]{hnd_max_wf_rank.png}
        \caption{Hyperpartisan News with max word rank complexity measure for all samplers.}
        \label{fig:hnd_max_wf_rank_all_samplers}
	\end{subfigure}
	\caption{Pre-trained BERT fine-tuned on Sentiment140 and Hyperpartisan News Detection datasets. Accuracy of the classifier as a function of the number of training steps.}
	\label{fig:text_classification}
\end{figure*}

\section{Metrics}

The first important part of the curriculum learning pipeline is measuring the complexity of samples for a given dataset. Texts could have a complex structure, and one can measure their complexity in different ways. A variety of heuristically motivated methods is accompanied by several metrics based on specific aspects of information theory. For a review of heuristic text complexity measures such as length of TF-IDF \cite{aizawa2003information} we address the reader to Appendix \ref{a:heu}. In this paper, we also explore the metrics initially proposed by \citet{ay2006unifying} to measure the complexity of finite systems and try to see if one could apply these metrics to NLP tasks.

\citet{ay2006unifying} observes that for finite systems, a set of parts impacts the complexity of the system as well as inter-dependencies of the parts. In the context of NLP, this means that text is more than just a bag of words. The authors propose four different metrics to estimate the complexity of a system. However, one of these metrics maximizes on single-letter texts, such as "Aaaaaaaaa," while the second was created to measure cyclic sequences and does not apply to texts. Thus we experiment with two other metrics, namely, Tononi, Sporns, and Edelman (TSE) \cite{tononi1994measure} and excess entropy (EE), and adapt them to the complexity of texts. For the calculation of TSE and EE for NLP we address the reader to Appendix \ref{a:cal}.



\section{Samplers}

The second important part of curriculum learning is the sampling strategy (or sampler) - the algorithm deciding which samples should be shown to the model at which moment. Let us observe existing curricula and suggest some new ones. 

\textbf{Competence-based. CB}\\
A competence-based curriculum, offered by \citet{platanios-etal-2019-competence}, uniformly samples data from increasing dataset's prefix. Competence is a function $c(t)$, which defines the size of the dataset prefix.
$$c(t) = \min \left( 1, \sqrt{t\frac{1 - c_0^2}{T} + c_0^2}\right)$$
Where $T$ - total number of steps, $t$ - current step, $c_0$ - hyperparameter set to $0.01$.

\textbf{Hyperbolic. HYP} \\
The main idea of this sampler is to increase average batch complexity through time. All samples are split by complexity into $N$ sequential buckets with equal size. Training time is divided into $N$ epochs and the probability of sampling the element from the $j$-th bucket on the $i$-th epoch is proportional to the distance between $j$ and $i$.
$$Pr_i(j) = \frac{c}{|j - i|^{0.5}}$$
Where $Pr_i(j)$ - probability to sample from $j$-th bucket on the $i$-th epoch, $c$ - constant to guarantee that sum of all probabilities equals to $1$.

\textbf{Difficulty-based. DB} \\
This sampler is a reversed version of the competence-based one. A difficulty-based sampler takes elements from a linearly decreasing suffix instead of sampling from a gradually increasing prefix. 

\textbf{Sort-shuffle. SS}\\
All previously described samplers do not guarantee that the model would see each element in the training data. Sort-shuffle samples each element exactly once, randomly splitting the data into batches and sorting by average complexity.

\textbf{Sort-merge. SM}\\
Many complexity estimates correlate with the length of the text. The main idea of a sort-merge sampler is to remove this correlation and train the model on stable length distribution. This algorithm consists of four main steps: sort dataset by length; sequentially split into buckets; sort each bucket by a complexity metric; form $i$-th batch from $i$-th elements from each bucket. Like a sequential one, the sort-merge sampler shows each element to the model exactly once.

Equipped with the list of metrics and curriculum samplers, we can discuss our experimental results.


\section{Experiments}
We perform our experiments on three NLP tasks: text classification, machine translation (NMT), and masked language modeling (MLM). Here we discuss the first task of classification in detail. The extensive results of the experiments are available in Appendix \ref{a:ex}. All the experiments are performed with the HuggingFace library \cite{wolf-etal-2020-transformers}, which provides the models with their setups, such as hyperparameters and tokenizers. We did not change default parameters in our experiment unless specifically stated otherwise. Thus, the dataset and the model specify every experiment. We use the base version of the BERT model \cite{devlin-etal-2019-bert} for MLM and classification, and the small version of the T5 model \cite{raffel2020exploring} for machine translation. Experiments were performed on BooksCorpus\footnote{\url{https://huggingface.co/datasets/bookcorpus}} dataset for MLM, Sentiment140\footnote{\url{https://www.kaggle.com/kazanova/sentiment140}} and Hyperpartisan News Detection\footnote{\url{https://huggingface.co/datasets/hyperpartisan_news_detection}} for classification, and WMT16-en-de\footnote{\url{https://huggingface.co/datasets/wmt16}} for machine translation. To estimate the curriculum's convergence speed, we calculate the average number of steps to reach a threshold that is 10\% lower than the resulting saturation quality metric for every problem.


\subsection{Text Classification}

 Figure~\ref{fig:text_classification} summarizes the experiments with BERT for text classification. Neither different samplers nor complexity measures improve a BERT-based classifier's resulting accuracy. 
 
\subsection{Masked Language Modelling}

 Figure~\ref{fig:BooksCorpus_graphs} shows the results of MLM pretraining of BERT on BooksCorpus. Irrespective of sampling, the complexity measures have similar ranking in terms of their performance on MLM: length, likelihood, TSE, EE, TF-IDF, maximum word rank. Since sorted sampler takes length into account by design, it is not included in the corresponding plots. Data-based curricula show inferior results in comparison with the baseline.
 
 \begin{figure*}[t]
	\centering
	\begin{subfigure}{.25\textwidth}
		\centering
		\includegraphics[scale=0.2]{BooksCorpus_CB}
		\caption{Competence-based}
		\label{fig:BooksCorpus_graphs_CB}
	\end{subfigure}
	\begin{subfigure}{.25\textwidth}
		\centering
		\includegraphics[scale=0.2]{BooksCorpus_DB}
		\caption{Difficulty-based}
		\label{fig:BooksCorpus_graphs_DB}
	\end{subfigure}
	\begin{subfigure}{0.25\textwidth}
		\centering
		\includegraphics[scale=0.2]{BooksCorpus_Hyp}
		\caption{Hyperbolic}
		\label{fig:BooksCorpus_graphs_Hyp}
	\end{subfigure}\\
	\begin{subfigure}{.25\textwidth}
		\centering
		\includegraphics[scale=0.2]{BooksCorpus_SS}
		\caption{Sort-shuffle}
		\label{fig:BooksCorpus_graphs_SS}
	\end{subfigure}
	\begin{subfigure}{.25\textwidth}
		\centering
		\includegraphics[scale=0.2]{BooksCorpus_SM}
		\caption{Sort-merge}
		\label{fig:BooksCorpus_graphs_SM}
	\end{subfigure}
	\begin{subfigure}{.25\textwidth}
		\centering
		\includegraphics[scale=0.5]{BooksCorpus_legend}
	
		\label{fig:BooksCorpus_graphs_legend}
	\end{subfigure}
	\caption{Loss function dependency on the number of training steps on MLM for BooksCorpus dataset during the first 40k steps of training. Every plot depicts results for six different complexity estimates combined with a specific sampler.}
	\label{fig:BooksCorpus_graphs}
\end{figure*}


\subsection{Neural Machine Translation}

Table~\ref{table:MT_quality} shows the experiments with T5 model \cite{raffel2020exploring} for machine translation and various curricula. We use the BLEU metric to estimate the quality of the resulting models. We calculate the average BLEU score over ten validations at saturation. Once again, curriculum learning does not give any notable benefits.

\begin{table}[h!]
    \caption{The average BLEU score from 50k to 100k steps on WMT16 dataset. Results better than the baseline are highlighted. '-' denotes the cases when complexity measure and sampler are not compatible.}
	\label{table:MT_quality}
	\centering
	\begin{tabular}{lcccccc}
		\hline
		Metrics & & \multicolumn{3}{c}{Samplers} &\\
		\hline
		& CB & DB & Hyp & SS & SM & \\
		\hline
		baseline & & \multicolumn{3}{c}{18.3} & \\
		\hline
		length &  10.1 & 17.4 & 16.3 & - & -\\
		TSE &  10.3 & \bf 18.4 &  16.8 &  13.8 &  14.8\\
		EE &  10.2 &  18.2 &  16.9 &  13.3 &  15.0\\
		\hline
	\end{tabular}
\end{table}

\section{Discussion}

We try to interpret obtained results cautiously. Though \citet{platanios-etal-2019-competence} report that competence-based sampling is beneficial for recurrent neural networks, we could not reproduce this result in transformer-based architectures. We also run experiments to check whether data-based curricula could work on non-transformer architectures. The results do not look encouraging; see Appendix \ref{a:plat}.

Curriculum learning depends on subtle factors, for example, a correct choice of hyperparameters. It is hard to check all possible values of hyperparameters, yet to the best of our capabilities, we address this issue in Appendix \ref{a:hyp}. The results do not seem to depend on the learning rate, and once again, curriculum learning shows no benefits.

At this point, we can only conclusively say two things: (1) a deeper investigation of the underlying information theoretic principles that stand behind curriculum learning is badly needed; (2) until we better understand these principles, data-based curriculum learning is a gamble with very low odds to gain either speed or resulting performance.

\section{Conclusion}
In this work, we ran extensive experiments with curriculum learning for transformer-based architectures on three NLP tasks: masked language modeling, text classification, and machine translation. We demonstrate that curricula do not help in the standard training setting and sometimes even worsen results.

\section{Acknowledgments}
The publication was supported by the grant for research centers in the field of AI provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University  No. 70-2021-00139. This research was supported in part through computational resources of HPC facilities at HSE University \cite{kostenetskiy2021hpc}

