
\section{Introduction}
\label{sec:Introduction}
\input{section/s1-introduction}

\section{Related Work}
\label{sec:Related Work}
\input{section/s2-related_work}

\section{The Proposed REZCR}
\label{sec:Method}
\input{section/s3-method}

\section{Experiments}
\label{sec:Experiments}
\input{section/s4-experiments}

\section{Conclusion}
\label{sec:Conclusion}
\input{section/s5-conclusion}












\subsection{Character Recognition}

\noindent \textbf{Character-based Methods.}
Early work on character recognition is divided into feature extraction-based methods \cite{su2003novel} and statistical-based methods \cite{shanthi2010novel}. CNNs were gradually applied to character recognition tasks with the development of deep learning, \cite{yuan2012offline} applied an improved LeNet-5 CNN model to recognise offline English characters. As the first method to apply CNNs into Chinese character recognition, MCDNN \cite{cirecsan2015multi} was trained by an embedding framework with eight networks and obtained human-comparable results. Furthermore, DirectMap \cite{zhang2017online} combines a direction-decomposed feature map with a deep CNN for training, and introduces an adaptation layer to reduce the mismatch between training and test data.


\noindent \textbf{Radical-based Methods.}
The huge number of Chinese character categories and the reliance of the above-described methods on a large number of training samples limit the accuracy of character-based methods. Researchers have introduced domain knowledge related to Chinese characters and radical-based methods to overcome this issue. RAN \cite{zhang2018radical,zhang2020radical} and DenseRAN \cite{wang2018denseran} use a CNN-based encoder to extract the radicals and their two-dimensional structure, and then generate radical sequences that represent characters by an RNN-based decoder for Chinese character recognition. HDE \cite{cao2020zero} recognises Chinese characters by performing vector matching in embedding space, introducing radical-level constitution to design a unique embedding vector for each Chinese character. \cite{chen2021zero} decomposites Chinese characters into strokes, the smallest units in Chinese characters, then recognises characters by looking up stroke sequences in a dictionary.

The common idea of the above methods is to transform each character as a sequence of radicals or strokes. As the prerequisite of these methods, the alphabet needs to be pre-set in advance, which contains each character and its corresponding sequence representation. The length of each sequence is not fixed, the maximum length can achieve 20 when the target character is complex. Thus, new challenges appear when applying the above methods. To obtain a correct recognition result, the generated sequence is required to be matched with the alphabet completely, both for each element and their order. That means the recognition performance will be significantly limited when applying to characters with a long sequence. 



\subsection{Zero-shot Learning}
Zero-shot learning aims to classify unseen samples by learning from existing samples and knowledge supplemented by additional information \cite{wang2019survey}. In recent years, zero-shot learning has been considered from a variety of perspectives. For instance, embedding-based methods \cite{akata2015label,xie2019attentive} propose to perform image embedding and label semantic embedding in the same space, and extend to unseen categories by a compatibility function. Furthermore, attribute-based methods \cite{lampert2009learning,lampert2013attribute} manually design attributes in different ways and represent categories with multidimensional attribute vectors, which are introduced to train the visual classifier by vector mapping. Additionally, knowledge graph-based methods \cite{rohrbach2011evaluating,wang2018zero} apply knowledge graphs as additional information to guide zero-shot learning, as the knowledge graph has the ability to convey categories and the relationships between categories. In this paper, we intend to exploit knowledge graphs for providing additional information in the framework, since it is more explanatory than embedding-based methods and express structural features to the additional information compared to attribute-based methods. We believe this will bring crucial advantages in character recognition, especially for dealing with unreproducible characters (e.g. oracle bone or calligraphic characters) with historical and literary value.
\subsection{Problem Formulation}

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.48\textwidth]{fig2.png}
	\caption{(a) Radicals and strokes composition of a Chinese character; (b) examples of characters composed by same radicals but different structural relations; (c) 14 predefined structural relations. 	\label{fig:2}}
\end{figure}


Most alphabet-based languages are only composed of a few dozen characters, while some other languages have a larger number of characters: for example, the total number of recorded Chinese characters exceeds 80,000 \cite{feng2014chinese}. 
Such large number of categories is difficult to be managed for the OCR task. In fact, characters are not the smallest unit in such language systems, since they can be decomposed into radicals and even strokes. A Chinese character is shown in Fig.~\ref{fig:2}(a) as an example, where the first row is its radical composition and the second row is its strokes composition. 
As the smallest unit of each Chinese character, there are few types of strokes, thus the stroke sequence corresponding to the character is usually long and many-to-one relationships between stroke sequences and characters are common \cite{chen2021zero}. Besides, strokes in characters often appear in combination, accompanied by connections and intersections. These properties add extra difficulties to stroke-based character recognition.

The number of radicals, which is the direct composition of characters, is higher than strokes, but much smaller than characters. Indeed, over 6,000 common Chinese characters can be composed of less than 500 radicals. In addition, due to the complex form and structural relations, radicals remain semantic information that is not available in strokes. It follows that converting training on character categories to training on radicals can significantly reduce the difficulty of the classification task while maintaining the semantics of the characters. Therefore, we intend to utilize radical information to complete character recognition.

For a character, radical information consists of two parts: the radical categories and the specific structure that organises these radicals. In addition to the radical class itself, the specific structure is also crucial for characters, as the same radical composition will point to different candidate characters when the structure changes, two sets of examples are given in Fig.~\ref{fig:2}(b). More in detail, the character structure is the relative localization relation between all composing radicals, here we refer to this as structural relation and predefine them separately for different languages in this paper. Take the oracle bone character as an example, we have predefined 14 structural relations shown in Fig.~\ref{fig:2}(c). Note that we have devised a unique structure, named `Multiple', which contains all characters containing four or more radicals. This is because characters containing many radicals (more than 3) have complex radical structural relations, but there is usually one unique candidate character for the same radical composition. Thus, `Multiple' as a general structure for characters with multiple radicals saves labour and calculation costs while keeping accuracy.


The overall architecture diagram of our proposed REZCR is shown in Fig. \ref{fig:3}. 
In the training phase, the input character images are fed to the RIE to generate the candidate radicals sequences and their structural relations sequences. 
In the reasoning phase, these two sequences are fed together to the KGR module, where the proposed weight-fusion algorithm applies a knowledge graph for reasoning to obtain all candidate characters. Then rank them according to the confidences and output the recognition results.
Details are described below.





\begin{figure*}[tbp]
	\centering
	\includegraphics[width=1\textwidth]{overall_structure-1.png}
	\caption{The overall structure of REZCR, where different modules are discriminated by specific colours. RIE (left) extracts candidate radicals and their structural relations from input character images; KGR (right) utilizes the information for reasoning with the CKG. In the CKG, blue, orange and green nodes represent characters, radicals and structures, respectively, and lines with different colours represent different relations between entities.
	\label{fig:3}}
\end{figure*}




\subsection{Radical Information Extractor}
In order to learn the radical information within the characters, we annotate the radicals contained in the characters under the guidance of linguists, including the radical category $r_c$, the radical localization, and the structural relations among radicals $s$. The location of the radical is identified by the coordinates ($x, y, w, h$) of the bounding box. The input character image $I_{C}\in\mathbb{R}^{H\times W\times C}$, where the spatial resolution is $H\times W$, and the number of channels is $C$. The double-headed outputs of RIE are the two radical information, candidate radical sequences $R_i$ and its structural relationship sequence $S_j$. 

Character images acquired in the real world are usually accompanied by noisy backgrounds, such as rubbing and corrosion traces in oracle bone characters and uneven backgrounds in street view characters, which will interfere with the learning of characters. In order to extract robust features for radicals, we introduce a radical attention block (RAB) in the RIE module to focus on the radicals in characters for mitigating the interference of background in character images. The RAB, whose backbone is ResNet \cite{he2016deep}, consist of two consecutive CBRs and a dual spatial attention layer (DSAL), as shown in Fig.~\ref{fig:3}. The CBR consists of a $3\times 3$ CNN, a batch normalization, and ReLU as the unit of ResNet. The DSAL has two attention weight calculations, keeping the texture information of the radicals for feature selection via MaxPool and transferring the background information of the character image to maintain the integrity of the information. Element-by-element multiplication is used to calculate the correlation of two spatial vectors: the larger the product of the vectors, the stronger the correlation between the two vectors. The formula for calculating attention is as:
\begin{equation}
\begin{aligned}
    A_1(F_i) &= \sigma (f(MaxPool(F_i))) \otimes F_i, \\
    A_2(F_i) & = \sigma (f(AvgPool(A_1(F_i)))) \otimes F_i,
\end{aligned}
\label{equ:1}
\end{equation}
where $A$ denotes the attention calculation function, $\sigma$ is the sigmoid function, and $f$ denotes the convolution. $F_i$ is the feature map, $F_{i}\in\mathbb{R}^{H\times W\times C}$. The final attention vector is $A_2$, which helps to keep radical features by mitigating the interference by background.

After RAB finishing feature extraction, we obtain $F_r$, a deep feature that will be used to perform radical detection and subsequent extraction of structural relation among radicals. Next, the RIE will move towards two heads. The first head is designed to output radical categories, $F_r$ goes through an FC layer,  and the output dimension is $K\times K\times(M\times(n_r+5))$, where $K$ represents dividing the input character image into $K\times K$ grids, $M$ represents that each grid needs to detect M anchor boxes, $n_r$ is the number of radical categories in the dataset, and $5$ represents the coordinate information ($x, y, w, h$)  for an anchor box and its confidence. In our experiment, we use $K = 7$, and $M = 3$. We output each radical whose confidence is within the threshold. For candidate boxes with similar centroid and area, we treat them as the same candidate box and merge the candidate radicals and predictions within them. The results are a set of candidate radical sequences (CRs). Depending on the localization of the candidate boxes, CRs contain n candidate radical sequences $R_i$, where $n$ is the number of possible radicals contained in the character, $1 \le i \le n$, and $R_i$ consists of all possible candidate radicals in this candidate box.

The second head is designed to output radical structural relations. In order to combine shallow features in character images with deep features learned from radicals, we integrate the output of the first RAB $F_1$ with the output of the last RAB $F_r$ for further radical structure relationship learning. When extracting structural features, we freeze all RABs and only perform parameter updates for subsequent convolutional layers. We output all structural relations and their predictions as sequences of structural relations (SRs), where the length of the sequence is $n_s$, i.e. the number of pre-defined structural relations.

In RIE, the loss function consists of two components, the loss of candidate radicals extraction $\mathcal{L}_{CR}$ and the loss of candidate structural relations recognition $\mathcal{L}_{SR}$, so:
\begin{equation}
    \mathcal{L}_{RI} = \mathcal{L}_{CR} + \mathcal{L}_{SR},\\
\label{equ:3}
\end{equation}
where $\mathcal{L}_{CR}$ is made up of three components, it can be expressed as:
\begin{equation}
    \mathcal{L}_{CR} = \mathcal{L}_{R} + \mathcal{L}_{box} + \mathcal{L}_{isR},\\
\label{equ:4}
\end{equation}
Specifically, $\mathcal{L}_{R}$ is the loss for the radical category. $\mathcal{L}_{box}$ is the loss for the bounding box introduced when locating the radical, which is calculated by the IoU of the candidate box and the real box. $\mathcal{L}_{isR}$ is the loss for the bounding box confidence, where it consists of two parts, one is the loss value of the existence of the object, and the second part is the loss value of the absence of the object. They can be express as:


\begin{equation}
\begin{aligned}
    \mathcal{L}_{R} = & \lambda_r \sum_{i=0}^{K^2} \mathbb{I}_{u=i}^{isR} \sum_{c\in R} [p_i(R)log(\hat{p_i}(R)  \\ & +  (1-p_i(R))log(1-\hat{p_i}(R))], \\
    \mathcal{L}_{box} = & \lambda_{c} \sum_{i=0}^{K^2} \sum_{j=0}^{M} \mathbb{I}_{u=i,j}^{isR} (2-w_i h_i)[(x_i - \hat{x_i})^2 + (y_i - \hat{y_i})^2] \\
    + & \lambda_{c} \sum_{i=0}^{K^2} \sum_{j=0}^{M} \mathbb{I}_{u=i,j}^{isR} (2-w_i h_i) [(w_i - \hat{w_i})^2 +(h_i - \hat{h_i})^2], \\
    \mathcal{L}_{isR} = & \lambda_{isR} \sum_{i=0}^{K^2} \sum_{j=0}^{M} \mathbb{I}_{u=i,j}^{isR} (R_i - \hat{R_i})^2 \\
     + & \lambda_{noR} \sum_{i=0}^{K^2} \sum_{j=0}^{M} \mathbb{I}_{u=i,j}^{noR} (R_i - \hat{R_i})^2, \\
\label{equ:5}
\end{aligned}
\end{equation}


where $R$ represents radical categories, $p_i(R)$ is the true probability of the target radical, $\hat{p_i}(R)$ is the predicted radical. $\mathcal{L}_{SR}$ is the loss of radical structural relation recognition, which adopts the cross-entropy loss function \cite{de2005tutorial} to measure the difference between the true structural relation label and the predicted result, as:
\begin{equation}
\begin{aligned}
    \mathcal{L}_{SR} = \frac{\lambda_{s}}{N} \sum_{i=0}^{N} q_{i}(S)\log(\hat{q_i}(S)).
\label{equ:3}
\end{aligned}
\end{equation}
where $N$ is the number of structure categories, $q_i(S)$ is the true structure probability while $\hat{q_i}(S)$ is the predicted structure.







\subsection{Knowledge Graph Reasoner}
We collected character images of Oracle, Bronze, simplified Chinese, and Korean characters, respectively, and annotated all characters in the dataset with radicals, as well as their structural relations under the guidance of linguists to construct character datasets and character knowledge graphs (CKG). As an initial version, the current CKG consists of 3 layers, including:
\begin{itemize}
    \item Characters: all characters in the dataset. Note that ancient characters (oracle bone and Bronze characters) are associated with simplified Chinese characters for recognition. 
    \item Radicals: components of characters. All character forms are associated with the corresponding radicals.
    \item Structure: the structure of a character, namely, the relative structural relation between all the radicals in a character.
\end{itemize}

We introduce an algorithm based on weight fusion to reason about characters, as shown in Algorithm \ref{alg:1}. The input is sequences of CRs and SRs obtained in RIE. As a first step, we extract all possible sequences of character candidate radicals, reason all possible characters through the CKG, and calculate the prediction $p_c^1$ for each candidate character by averaging the prediction of the candidate radicals. The formula is as follows:
\begin{equation}
    p_c^1= \frac{1}{n}\sum_{i=0}^{n} p_r^i,
\label{equ:4}
\end{equation}
where $p_r$ is the prediction of each candidate radical, and $n$ is the number of the candidate radicals.


In the second step, the radical structure relationship is applied to verify whether the reasoned character is correct. The final candidate character prediction $p_c$ will be determined by a weighted fusion of the character prediction $p_c^1$ from the first stage and the confidence $p_s$ of the radical structure relationship as follows,
\begin{equation}
    p_c= \theta p_c^1 + (1 - \theta)p_s,
\label{equ:5}
\end{equation}
where $\theta$ is the weight of the first stage character prediction. The final candidate character with the highest $p_c$ will be the character identified for recognition.

\input{algorithm}

As long as the character is stored in the CKG, the REZCR is able to recognise it, even if it is unseen during training. It is worth mentioning that our method supports the addition of new character categories: when a new character category is encountered, there is no need to collect training samples or retrain the model, since it can be recognised by simply updating the knowledge graph.
\subsection{Experimental Setups}


\textbf{Datasets.}  Long-tail effect widely appears in real-world datasets, some existing character image datasets try to alleviate this issue by manually copying and simulating. However, these ideally constructed datasets are not satisfied with real-world data distribution (e.g., unreproducible calligraphic characters), which will affect the training model and further limit the final performance. Therefore, we introduce a new character images dataset that conforms to the above.

Oracle character rubbing images are collected in this new dataset, called OracleRC\footnote{The link of the proposed public accessible character image dataset will be updated later due to blind review.}, which is a few-samples dataset with long-tail effects. All oracle character images are collected from \cite{wu2012} and have been uniformly denoised. It contains a total of 2005 oracle characters, which contain 202 radicals and 14 structural relations. The maximum number of character samples in this dataset is 32, whereas the minimum is 1; the maximum number of radical samples is 206, whereas the minimum is 30. Both radicals and structural relations were manually annotated by archaeologists. 


To demonstrate that our proposed REZCR is a general character recognition method, we conduct experiments on the simplified Chinese datasets ICDAR2013 \cite{yin2013icdar}, HWDB1.1 \cite{liu2013online}, CTW \cite{yuan2019large}, and the Korean dataset PE92 \cite{kim1996handwritten}.

\noindent\textbf{Implementation Details.} We implement our method with PyTorch. The size of the input image is normalised to $32 {\times} 32$, and data enhancement strategies such as translation, rotation, scaling, and background transformation are used. All experiments are conducted with Adadelta optimization where the hyperparameters were set to $\rho {=} 0.95$ and $\varepsilon {=} 10^{-6}$.








\subsection{Experimental Results}

\noindent\textbf{Results on Few Samples Dataset.}
Experiments are conducted on the OracleRC dataset, a few-sample character image dataset with long-tail effects. From each category, a random selection of 75\% of the data are used as the training set, and the remaining 25\% are used as the test set. Regarding the category with only one sample, all samples are used as test samples, namely, no training samples: there are 518 in this dataset. The average accuracies of all methods on OracleRC are shown in Table \ref{tab:1}. It can be seen that our method significantly outperforms all other methods from Top-1 to Top-10, including both traditional statistical-based and deep learning-based methods. The reason for this disparity is the insufficient amount of training data, which limits the performance of deep neural networks. In contrast, after we decompose characters into radicals, the number of training categories is significantly reduced and the number of samples per category is increased to at least 30 samples. This largely alleviates the difficulty brought by a few samples to the character recognition task. 

It is worth noting that our method is effective in recognising the character categories that have not been trained. For the 518 unseen categories, it gets an accuracy of 72.01\% for Top-10. This is a small reduction compared to the recognition accuracy of all character categories, which demonstrates the superiority of our method in the zero-shot learning task.


\begin{table}[!t]
\centering
\caption{Averaged accuracy of different methods on OracleRC.}
\label{tab:1}
	\resizebox{1\linewidth}{!}
\begin{tabular}{@{}lcccc@{}}
\toprule
OracleRC             & Top-1            & Top-3            & Top-5                & Top-10               \\ \midrule
HOG+SVM \cite{dalal2005histograms}              & 9.29\%           & 12.75\%          & 18.26\%              & 24.35\%              \\
AlexNet \cite{krizhevsky2012imagenet}              & 26.93\%          & 36.45\%          & 40.03\%              & 47.49\%              \\
VGG16 \cite{simonyan2014very}               & 27.75\%          & 38.12\%          & 41.53\%              & 48.86\%              \\
ResNet \cite{he2016deep}             & 39.09\%          & 39.21\%          & 43.06\%              & 51.66\%              \\
Inception-v4 \cite{szegedy2017inception}         & 30.31\%          & 49.96\%          & 52.13\%              & 58.72\%              \\ \midrule
REZCR (Ours) & \textbf{60.28\%} & \textbf{69.74\%} & \textbf{72.66\%}     & \textbf{82.45\%}     \\ \midrule
\end{tabular}}
\end{table}




\noindent\textbf{Results for Zero-shot Learning.}
In this section, we design a series of experiments to demonstrate the effectiveness of REZCR in identifying unseen characters. Character-based methods cannot recognize unseen characters, which means that they get 0\% accuracy in this experiment. Therefore, we only make a comparison with radical-based and stroke-based methods on zero-shot learning. We applied three datasets, namely the scene character dataset CTW, and the handwritten Chinese character datasets HWDB1.1 and ICDAR2013. In order to conduct experiments on the Chinese character dataset, we annotate the Chinese characters with radicals and their structural relations under the guidance of linguists.

For street view characters. The CTW dataset contains 3650 character categories with a total of 812,872 samples. We sort all character categories according to the numbered positions in the national standard GB18030-2005, and collect samples from the first 500 classes as the test set. The next n categories will be used as the training set, where $m \in \{500, 1000, 1500, 2000, 3150\}$.
For handwritten Chinese characters. In this experiment, we apply both HWDB1.1 and ICDAR2013 datasets. We select samples of the first n character categories from HWDB1.1 as the training set, where $n \in \{500, 1000, 1500, 2000, 2755\}$. Then we select the samples in the last 1000 character categories as the test set from ICDAR2013. Results are shown in Table \ref{tab:2}, proving that our method outperforms other methods in different numbers of training sets.



\begin{table}[!t]
\centering
\caption{Performance comparison of the zero-shot character recognition using different methods on the two set of datasets. $n$ is the character category used for training, and the test character category was fixed at 500 and 1000, respectively.}
\label{tab:2}
\resizebox{1\linewidth}{!}{
\begin{tabular}{@{}lcccccccccc@{}}
\toprule
\multirow{2}{*}{Method}  & \multicolumn{5}{c}{CTW, $n$}                                                                      & \multicolumn{5}{c}{Two handwritten datasets, $n$}                                                                \\
\cmidrule(l){2-6} \cmidrule(l){7-11} 
                         & 500              & 1000             & 1500             & 2000             & 3150             & 500              & 1000             & 1500             & 2000             & 2755             \\ \midrule
DenseRAN \cite{wang2018denseran}                 & 0.15\%           & 0.54\%           & 1.60\%           & 1.95\%           & 5.39\%           & 1.70\%           & 8.44\%           & 14.71\%          & 19.51\%          & 30.68\%          \\
HDE-Net \cite{cao2020zero}                  & 0.82\%           & 2.11\%           & 3.11\%           & 6.96\%           & 7.75\%           & 4.90\%           & 12.77\%          & 19.25\%          & 25.13\%          & 33.49\%          \\
Stroke-to-Character \cite{chen2021zero}      & 1.54\%           & 2.54\%           & 4.32\%           & 6.82\%           & 8.61\%           & 5.60\%           & 13.85\%          & 22.88\%          & 25.73\%          & 37.91\%          \\
REZCR (Ours)     & \textbf{25.52\%} & \textbf{49.42\%} & \textbf{56.85\%} & \textbf{60.96\%} & \textbf{73.45\%} & \textbf{37.83\%} & \textbf{65.26\%} & \textbf{70.02\%} & \textbf{72.65\%} & \textbf{77.91\%} \\ \bottomrule
\end{tabular}}
\vspace{-0.3cm}
\end{table}






More in detail, our method recognizes characters by decomposing them into radicals and structural relations, and by introducing a CKG as external knowledge for reasoning. While other methods also disassemble characters, they all use a set of ordered sequences (such as IDS sequences) to represent them: the number of elements in the sequence ranges from 2 to 16 (radical-based methods) or 1 to 24 (stroke-based methods). This places high demands on the recognition method: only if each element and its localization in the sequence are identified accurately can the character category be correctly obtained. The longer the sequence of characters, the more inaccurate the prediction.


The above problem does not arise in our method, since we do not require that the radicals are recognised in a given order, and each radical is stored equally in the CKG. Meanwhile, our method outputs all possible classes of radicals, structural relations, and their precision in the first stage, and all possible combinations of radical information are utilized in the CKG for reasoning. Calculated with a weighted fusion-based algorithm, we obtain the precision for all candidate characters, which in turn gives the final prediction results. Based on this process, our method has a strong self-correcting capability and it is still possible to obtain the correct character categories even in the case of incorrect Top1 candidate radicals.




\begin{table}[!t]
\centering
\caption{Character recognition performance comparisons on two datasets with the state-of-art methods.}
\label{tab:3}
	\resizebox{1\linewidth}{!}
\begin{tabular}{@{}lcc@{}}
\toprule
Method &  ICDAR2013 & CTW \\ \midrule                                                         AlexNet \cite{krizhevsky2012imagenet} & 89.99\%  & 76.49\%   \\
ResNet \cite{he2016deep} & 92.18\% & 79.46\%  \\
Inception-v4 \cite{szegedy2017inception} & 95.79\% & 82.28\% \\
DenseNet \cite{huang2017densely}  &  95.90\%  & 79.88\% \\
RAN \cite{zhang2018radical} &  93.79\% &  81.80\% \\
DenseRAN \cite{wang2018denseran} & 96.66\% & 85.56\% \\                     
FewshotRAN \cite{wang2019radical} &   \textbf{96.97\%} &   86.78\%  \\              
HDE-Net \cite{cao2020zero} & 96.74\%  &  \textbf{89.25\%} \\                    
Stroke-to-Character \cite{chen2021zero} & 96.28\% & 85.29\%  \\
REZCR (Ours)  & 96.35\%   & 84.21\% \\ \bottomrule
\end{tabular}}
\end{table}



\noindent\textbf{Results for Generalization.}
To demonstrate the generalization of REZCN, we perform experiments on the ICDAR2013 and CTW datasets in a general manner \cite{cao2020zero}. We directly use the official training set and the test set of the dataset for experiments (taking CTW as an example, the training set contains 760,107 samples, while the test set contains 52,765). Experimental results are shown in Table \ref{tab:3}. It is undeniable that character-based deep learning methods perform well with sufficient sample size, but our method also achieves competitive results. As a general character recognition method, our method not only outperforms on general datasets, but also has unique advantages on few-sample datasets.









\subsection{Ablation Study}

To further validate the effectiveness of the proposed method, an extensive ablation study was carried out on the OracleRC dataset for different components of the proposed REZCR.

\noindent \textbf{Impact of Learning and Reasoning Strategies.}
To validate the radical-based character recognition framework, we conduct ablation experiments on the design of learning and reasoning strategies in two modules in REZCR, i.e., RIE and KGR, respectively.  Results are shown in Table~\ref{tab:4}. ``Without radicals" refers to the learning for character recognition without extracting the radical information; ``Radicals without structures" is to learn the radical categories in character images without considering the radical structure relations; ``Radical + structures (RIE)" is to learn both the radical categories and the structural relations in character images, namely our proposed RIE module. The latter four experiments were carried out for different reasoning strategies. ``RIE + Top1 radicals" refers to using the radicals with the highest prediction confidence for reasoning. ``RIE + Top1 radicals \& structure" refers to using both the radicals and the structural relation with the highest prediction confidence for reasoning. ``RIE + radical sequences" is to input the sequences of all candidate radicals obtained from the RIE for reasoning. ``RIE + radical \& structure sequences (REZCR)" is to utilize all candidate radicals sequences and candidate structural relations sequences obtained from RIE for reasoning in KGR, which applies the weight fusion-based algorithm, namely our proposed REZCR.




\begin{table}[t]
\centering
 \makeatletter\def\@captype{table}\makeatother\caption{Ablation study results on learning and reasoning strategies.}
    \label{tab:4}
    \resizebox{1\linewidth}{!}{
    \begin{tabular}{@{}clc@{}}
    \toprule
    Index & Method                                       & Acc.    \\ \midrule
    (a)   & Without radicals                        & 26.93\% \\
    (b)   & Radicals without structures                          & 54.87\% \\
    (c)   & Radicals + structures (RIE)                    & 56.84\% \\
    (d)   & RIE + Top1 radicals                          & 52.93\% \\
    (e)   & RIE + Top1 radicals \& structure             & 55.62\% \\
    (f)   & RIE + radical sequences                      & 58.06\% \\
    (g)   & RIE + radical \& structure sequences (REZCR) & 60.28\% \\ \bottomrule
    \end{tabular}}
\end{table}


\begin{table}[t]
  \centering
        \makeatletter\def\@captype{table}\makeatother\caption{Ablation study results on different attention block scheme.}
        \label{tab:5}
        \resizebox{0.5\linewidth}{!}{
        \begin{tabular}{@{}lc@{}}
        \toprule
        Attention layer   & Acc.    \\ \midrule
        SSAL-a            & 59.23\% \\
        SSAL-b            & 59.34\% \\
        SSAL-c            & 59.79\% \\
        DSAL-a            & 60.03\% \\
        DSAL-b (Ours)     & 60.28\% \\ \bottomrule
        \end{tabular}}
\end{table}




\begin{figure}[!t]
	\centering
	\includegraphics[scale=0.48]{fig4-1.png}
    \caption{Five connection schemes for the attention layer.
    \label{fig:4}}
\end{figure}





From the first two experiments, we find that learning characters directly is not a good choice for few-sample character datasets: these few samples are unable to support the model to get a reliable character distribution, especially for images with only one sample. In contrast, the strategy of learning radicals effectively improves the performance of character recognition, with an accuracy improvement of about 30\%, which is a remarkable improvement.
The comparison between (b) and (c), (d) and (e), and (f) and (g) shows that the extraction of radical structure relations results in about a 2\% improvement. This indicates that structure relations further complements the radical information, and effectively addresses the problem brought by the fact that different characters are composed of the same several radicals.
Observing the experiments in (d) and (f), and in (e) and (g), we can find that the reasoning strategy using sequences has an accuracy improvement of about 5\% compared to the strategy using Top1 results, which demonstrates the superiority of the weight fusion-based knowledge graph reasoning algorithm.



\noindent\textbf{Impact of Attention Layer Schemes.}
In this paper, we propose a dual spatial attention layer (DSAL) to help with the extraction of radical information. To verify the effectiveness of DSAL, we experiment with five different attention blocks, as shown in Fig.~\ref{fig:4}. The single spatial attention layers (SSAL) are divided into three types depending on the pooling layer applied, called SSAL-a, SSAL-b and SSAL-c, respectively, while DSAL has two different connections schemes, labelled DSAL-a and DSAL-b, respectively. Results of the experiments are shown in Table~\ref{tab:5}. It is intuitive to find that DSAL outperformed the SSAL on the OracleRC dataset, and DSAL-b achieved the best performance, which is currently applied in REZCR. 

