\section{Introduction}

Chinese text error correction plays an important role in many NLP related scenarios~\citep{martins2004spelling,afli2016using,wang2018hybrid,burstein1999automated}. For Chinese native speakers, the common errors include mistaken characters, missing characters, and redundant characters. Mistaken characters type means the sentence contains wrong characters and we should replace them with the right ones. Missing characters type means short of characters in the sentence and the way to correct it is inserting characters in the position identified. Redundant characters type refers to the situation when useless or repeated characters exist and thus need to be deleted. For mistaken characters type, the sentence length keeps the same before and after correction, while for the other two types, the length changes. If texts only contain mistaken errors, we call it a text-aligned situation; if there exist missing or redundant errors besides mistaken errors, we call it a text non-aligned situation. Most methods proposed for Chinese error correction can only tackle text-aligned situation for now.

For text-aligned situation, many approaches apply the detect-correct framework, which is to detect the positions of wrong characters first and then correct them~\citep{hong2019faspell,zhang2020spelling}. Although this kind of methods have achieved competitive results, they cannot deal with text non-aligned situation with missing and redundant errors. For text non-aligned situation, the two mainstream schemes are Seq2Seq-based and sequence tagging-based. The former borrows ideas from machine translation by setting wrong sentences as input and correct sentences as output~\citep{zhao2019improving,kaneko2020encoder,chollampatt2019cross,zhao2020maskgec,lichtarge2019corpora,ge2018reaching,junczys2018approaching}. This kind of approaches requires a large number of training data and may generate uncontrollable results~\citep{kiyono2019empirical,koehn2017six}. The latter is based on sequence tagging by taking wrong sentences as input and the modification operation of each token as output~\citep{awasthi2019parallel,malmi2019encode,omelianchuk2020gector}. However, since Chinese language has more than 20 thousand characters which can generate lots of combinations with token operation labels, it is difficult for the sequence tagging models to cover all the combinations and generate results with high coverage rates.

To address the above issues, we propose an alignment-agnostic detect-correct model, which can handle text non-aligned errors compared to the current detect-correct methods, and can also relieve the problem of huge value space search leading to uncontrollable or low coveraged results of the Seq2Seq or Sequence tagging based methods.

We conduct experiments to compare our alignment-agnostic model with other models by utilizing three datasets: CGED 2020, SIGHAN 2015, SIGHAN-synthesize. Experimental results show
that alignment-agnostic model preforms better than other models.

The contributions of our work include (1) proposal of a novel detect-correct architecture for Chinese text error correction, (2) empirical verification of the effectiveness of the alignment-agnostic model, (3) easy reproduction and fast adaptation to practical scenario with limited annotation data.

\section{Our Approach}
\subsection{Problem Description}
Chinese text error correction can be formalized as the following task. Given a sequence of $n$ characters $X=(x_1,x_2,x_3,...,x_n)$ , the goal is to transform it into another sequence of $m$ characters $Y=(y_1,y_2,y_3,...,y_m)$, where $n$ and $m$ can be equal or not. The task can be viewed as a sequence transform problem with a mapping function $f:X \rightarrow Y$

\begin{figure}[h]
\includegraphics[width=0.4\textwidth]{./figure1.pdf}
\caption{Architecture of alignment-agnostic model}
\label{figure1}
\end{figure}

\subsection{Model}
Our alignment-agnostic model is illustrated in Figure \ref{figure1}. The basic structure is composed of a detection network evolved from ELECTRA’s discriminator~\citep{clark2020electra} and a correction network based on BERT MLM~\citep{devlin2018bert}. The two networks are connected through a modification logic unit and are trained separately. The detection network is responsible for locating the errors and identifying error types. The modification logic unit denotes where and how to correct. And the correction network focuses on correction.

\textbf{The detection network} is composed of 
a ELECTRA’s discriminator and a token-level error type classifier. The ELECTRA’s discriminator’s architecture has been described in \citet{clark2020electra}. The error type classifier contains four categories, namely $label_{keep}$, $label_{mistaken}$, $label_{missing}$, $label_{redundant}$. $label_{keep}$ means the character is correct and we should keep it unchanged. $label_{mistaken}$ indicates the character is mistaken and we should replace it. $label_{missing}$ denotes we should insert characters before the current character. $label_{redundant}$ means the character is useless and we should delete it. We get the label probability of each token with the 4-class token-level classifier:
\begin{equation}
    P_{i,label}(z_i=k|X)=softmax(w^Th^D(X))
\end{equation}
Where $P_{i,label}(z_i=k|X)$ denotes the conditional 
probability of character $x_i$ being tagged with the label of $k$, $h^D(X)$ is the last hidden state of ELECTRA’s discriminator and $k$ is in [$label_{keep}$, $label_{mistaken}$, $label_{missing}$, $label_{redundant}$]. The loss function 
of the detection network is:
\begin{equation}
    Loss_{detect}=-\sum_{i=1}^{n}\log{p_{i,label}}
\end{equation}

\textbf{The modification logic unit} rewrites the input sequence $X$ according to detection network’s output $Z$:
\begin{equation}
if z_i=\left\{
\begin{aligned}
& label_{keep}, \; x_{i}^{'}=x_i \\
& label_{mistaken}, \; x_{i}^{'}=[MASK] \\
& label_{missing}, \; x_{i}^{'} = [MASK] \; x_i \\
& label_{redundant}, \; x_{i}^{'} =  ''
\end{aligned}
\right.
\end{equation}
We get a new sequence $X^{'}=(x_{1}^{'}, x_{2}^{'}, x_{3}^{'},..., x_{n}^{'},)$ based on the above formula. For each token with empty characters $''$, we delete it directly from the sequence $X^{'}$, For each token with $'[MASK] \; x_i'$, we reorder it as two characters and obtain the final modified sequence $Y^{'}=(y_{1}^{'}, y_{2}^{'}, y_{3}^{'},..., y_{m}^{'},)$, whose length might be different from $X^{'}$. 

\begin{center}
\begin{table*}[!h]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Test Set}} & \multirow{2}{*}{\textbf{Method}} & \multicolumn{3}{c|}{\textbf{Detection}} & \multicolumn{3}{c|}{\textbf{Correction}}\\ 
\cline{3-8}
~ & ~ & Prec. & Rec. & $F_1$. & Prec. & Rec. & $F_1$.\\
\hline
\multirow{5}{*}{SIGHAN 2015} & Hybrid~\citep{wang2018hybrid} & 56.6 & 69.4 & 62.3 & - & - & 57.1 \\
\cline{2-8}
~ & FASpell~\citep{hong2019faspell} & 67.6 & 60 & 63.5 & 66.6 & 59.1 & 62.6 \\
\cline{2-8}
~ & Confusionset~\citep{wang2018hybrid} & 66.8 & 73.1 & 69.8 & 71.5 & 59.5 & 64.9 \\
\cline{2-8}
~ & Soft-Masked BERT(2020) & \textbf{73.7} & \textbf{73.2} & \textbf{73.5} & 66.7 & \textbf{66.2} & 66.4 \\
\cline{2-8}
~ & our model & 79.1 & 64.0 & 71.3 & \textbf{72.2} & 60.6 & \textbf{68.2} \\
\hline
\end{tabular}
\caption{\label{table1} Performances of Different Methods on SIGHAN 2015. }
\end{table*}
\end{center}
\vspace{-1.5em}
\textbf{The correction network} is BERT. We do the prediction for positions with the $[MASK]$ symbol on the sequence $Y^{'}$.

\section{Experiments}
\subsection{Datasets and Metrics}
Chinese text error correction tasks mainly have two public datasets: the benchmark of SIGHAN 2015~\citep{tseng2015introduction} which just contains text-aligned data and the competition of CGED 2020~\citep{cged2020} which contains text non-aligned data. In order to better verify the models’ effect on text non-aligned scenario, we synthesized some non-aligned data based on SIGHAN 2015 dataset. Next, we will introduce how to utilize the three datasets.

For SIGHAN 2015 dataset, since the use of extra resources is allowed in SIGHAN competitions, we incorporated SIGHAN 2013 and 2014 datasets in the training phase, and incorporated SIGHAN 2013 confusion set, in order to keep accordance with other methods in comparison. The test set contains 1100 passages and the train set contains 8738 passages. We used the evaluation tool provided by SIGHAN, with metrics of precision (Prec.), recall(Rec.) and $F_1$, all are based on sentence level.  

CGED 2020 contains an additional error type besides three error types mentioned. That is the reversed order. As this type happens less frequent in Chinese text writing scenario, and is beyond the scope of our model setting, we remove 575 relevant samples from a total sample of 2586, and get 846 training samples and 1165 testing samples.

To better verify our model’s effect on text non-aligned scenario, we synthesized some non-aligned data based on SIGHAN 2015 dataset (SIGHAN-synthesized). For mistaken characters error type, we kept the original errors unchanged. For missing characters error type, we randomly selected 50\% samples and deleted one character from each of them. For redundant characters error type, we randomly selected 50\% samples and inserted characters in each of them through four ways. (1) We inserted repeated characters in 35\% of the selected samples. (2) We inserted confusing characters in 30\% of the selected samples. (3) We inserted characters from high-frequency words in 30\% of the selected samples. (4) We also inserted random characters in 5\% of the selected samples.

For both CGED 2020 dataset and SIGHAN-synthesized dataset, we used the $M^2$ score~\citep{dahlmeier2012better} and ERRANT~\citep{bryant2017automatic} to evaluate models’ performance, which are the two commonly adopted evaluation tools for English text non-aligned situations.

\subsection{Experiment Settings}
The pre-trained ELECTRA discriminator model and BERT model adopted in our experiments are all from \url{https://github.com/huggingface/transformers}. We utilize the large size for ELECTRA and the base size for BERT. We train detection network and correction network on the three datasets respectively by Adam optimizer with default hyperparameters. All the experiments are conducted on 2 GPUs (Nvidia Tesla P100). 

For SIGHAN 2015, we kept the default binary classifier of ELECTRA’s discriminator because the dataset just contains one error type when fine tuning ELECTRA’s discriminator. And in retraining of BERT, we applied two kinds of ways. One is an unsupervised method by re-pretraining BERT with its original MLM objective. The other is a supervised method by masking the mistaken characters and predicting them.
\begin{center}
\begin{table*}[htbp]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Test Set}} & \multirow{2}{*}{\textbf{Method}} & \multicolumn{3}{c|}{\textbf{\bm{$M_2$}(Correction)}} & \multicolumn{3}{c|}{\textbf{ERRANT(Correction)}}\\ 
\cline{3-8}
~ & ~ & Prec. & Rec. & $F_{0.5}$. & Prec. & Rec. & $F_{0.5}$.\\
\hline
\multirow{4}{*}{CGED 2020} & Copy-augmented(2019) & 4.62 & 0.8 & 2.36 & 3.51 & 0.56 & 1.7\\
\cline{2-8}
~ & Lasertagger(2019) & 14.99 & 3.48 & 9.02 & 12.95 & 2.61 & 7.22 \\
\cline{2-8}
~ & PIE(2019) & 22.3 & 10 & 17.9 & 17.1 & 6.6 & 13 \\
\cline{2-8}
~ & our model & \textbf{29.71} & \textbf{22.03} & \textbf{27.77} & \textbf{24.8} & \textbf{17.56} & \textbf{22.91} \\
\hline
\multirow{4}{*}{SIGHAN-synthesized} & Copy-augmented(2019) & 38.44 & 8.03 & 21.87 & 38.31 & 7.8 & 21.5 \\
\cline{2-8}
~ & Lasertagger(2019) & 51.29 & 43.21 & 49.44 & 50.14 & 39.99 & 47.72 \\
\cline{2-8}
~ & PIE(2019) & 54.1 & 47.6 & 52.6 & 52 & 42.6 & 49.8 \\
\cline{2-8}
~ & our model & \textbf{59.3} & \textbf{62.2} & \textbf{59.8} & \textbf{56.9} & \textbf{57.8} & \textbf{57} \\
\hline
\end{tabular}
\caption{\label{table2} The $M^2$ score and ERRANT score of Different Methods on CGED 2020 and SIGHAN-synthesized. }
\end{table*}
\end{center}

\begin{center}
\begin{table*}[htbp]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Method}} & \textbf{SIGHAN 2015} & \textbf{CGED 2020} & \textbf{SIGHAN-synthesized} \\ 
\cline{2-4}
~ & $F_1$ & $F_{0.5}$ & $F_{0.5}$ \\
\hline
ELECTRA+BERT & 38.7 & - & - \\
\hline
Finetune ELECTRA+BERT & 66.2 & 22.91 & 54.7 \\
\hline
Finetune ELECTRA+Finetune BERT & 68.2 & 22.54 & 54.4 \\
\hline
Finetune ELECTRA+Pretrain BERT & 42 & 22.6 & 57 \\
\hline
\end{tabular}
\caption{\label{table3} Ablation study of alignment-agnostic model on three datasets. }
\end{table*}
\end{center}
\vspace{-3.2em}
\quad For CGED 2020 and SIGHAN-synthesized datasets, we added a 4-class classifier to recognize three error types on ELECTRA’s discriminator’s last hidden layer and then fine tune it. As in retraining of BERT, we applied the same ways with SIGHAN 2015.


\subsection{Results}
Table~\ref{table1} shows the results in comparison on SIGHAN 2015 dataset. From the table we can see that our method outperforms the previous SOTA method Soft-Masked BERT~\citep{zhang2020spelling} by 1.8\% on $F_1$ score in correction phase.

Table~\ref{table2} compares the results of our model with other SOTA models on  CGED 2020 dataset and SIGHAN-synthesized dataset. Our model performs the best on correction level by exceeding the second best model 9.87 points for CGED 2020 and 7.2 points for SIGHAN-synthesized dataset on $F_{0.5} M^{2}$ score. Copy-augmented~\citep{zhao2019improving} is a Seq2Seq model which requires a large number of training data to achieve an acceptable result. That’s why it performs poorer than Lasertagger~\citep{malmi2019encode} and PIE~\citep{awasthi2019parallel} models on both two datasets with a relative small amount of training samples. Lasertagger and PIE are sequence tagging based models. As we analyzed before, this kind of models doesn’t work well on Chinese language due to huge value space search.

\subsection{Ablation Study}
We carried out ablation study of our model on the three datasets. Table~\ref{table3} shows the results on correction level. For SIGHAN 2015, We can see that finetuning ELECTRA can produce great improvement of 27.5\% on $F_1$ score, while finetuning BERT generates a relatively small improvement of 2\% on $F_1$ score and pretraining BERT leads to a negative effect of -24.2\% on $F_1$ score. Possible reason is that finetuning can help BERT learn the knowledge about confusion set of characters that prone to be mistaken, while pretraining is unsupervised and may destroy the original learned words distribution when training data is quite different from the original training data. Besides, our model can get 38.7\% on $F_1$ score without any training data, which means it can work as a strong baseline in cold start conditions. For CGED 2020 and SIGHAN-synthesized datasets, two ways of retraining BERT didn’t improve much. Compared with the results of other SOTA models, the transformation of ELECTRA is the most effective part.

\section{Conclusion}
We proposed a new detect-correct model for Chinese text error correction tasks, which can handle both text-aligned and non-aligned situations, and can serve as a strong baseline even in cold start situations. Our model firstly locates the errors and identifies error types, and then corrects the sentence with a MLM. Experimental results on three datasets show that our model performs better than existing methods. Furthermore, it is easy to reproduce and can achieve good results with a small amount of training data, which is the key for rapid application to the industry.

\newpage

\input{acl2021.bbl}
\bibliographystyle{acl_natbib}



\end{document}
