\section{Introduction}
Named entity recognition (NER) is one of the most important and fundamental research topics in natural language processing (NLP), which recognizes named entities (NEs), such as person, location, disease from raw text.
NER has attracted substantial attention in the past decades owing to its importance in downstream tasks, e.g., knowledge graph construction \cite{bosselut2019comet}, question-answering \cite{pergola2021boosting}, and relation extraction \cite{he2019classifying}.

In the early stage, the popular methods for solving NER are some traditional machine learning methods, e.g., Hidden Markov Model (HMM) \cite{morwal2012named} and conditional random field (CRF) \cite{mozharova2016combining}, which require extracting features manually, making the process inefficient and time-consuming.
With the breakthrough of recurrent neural networks (RNN) in NLP, Long short-term memory (LSTM) \cite{hochreiter1997long} and Gated Recurrent Unit (GRU) \cite{GRU} have become mainstream methods for this task and have achieved promising results since neural networks can automatically extract features from the sequence and also take each token's position information into consideration \cite{lample2016neural,chiu2016named,huang2015bidirectional}.
Nevertheless, RNN fails to perform well with long sequences due to the gradients exploding and vanishing.
In recent years, Transformer-based models \cite{vaswani2017attention} have become mainstream methods because these models are able to capture long-term dependencies with the help of multi-head attention and thus provide better global context information, especially for long sequences \cite{biobert,yangxlnet}.
However, these Transformer-based models usually are insensitive to the local context since the representation of each token is computed by the canonical point-wise dot-product self-attention \cite{li2019enhancing,huang2021missformer}.
Besides, although some studies \cite{shaw2018self,BERT,liu2019roberta} have been proposed to inject position information into Transformer, they are still inadequate to help Transformer obtain appropriate position information \cite{huang2020improve,qu2021explore}.
In other words, the self-attention mechanism is effective in overcoming the constraints of RNN from the perspective of long-sequence context information extraction, but is inferior to RNN in terms of local contextual and position information extraction.
Yet, both long-term dependencies and local context information are essential for the NER model to correctly identify entities.


Thus, to alleviate the shortcomings in RNN and Transformers while maintaining their respective strengths, in this paper, we propose a novel Hero-Gang Neural model to leverage both global and local contextual information to improve NER.
In doing so, on the one hand, we utilize a Transformer-based sequence encoder (i.e., Hero module) to extract effective global contextual information with the help of the self-attention mechanism.
On the other hand, a multi-window recurrent unit (i.e., Gang module) is applied to extract local features from multiple sub-sequences under the guidance of the extracted global information.
Afterward, we propose to use multi-window attention to elaborately combine global and local contextual features.
The performance of our proposed model significantly outperforms the strong baseline models on several NER benchmark datasets (including both general and biomedical domains) and achieves new state-of-the-art results on some datasets.






\section{Method}
\textcolor{black}{
NER is usually performed as a sequence labeling problem.
In detail, given a sequence of $X =x_{1},x_{2},...,x_{N}$ with $N$ tokens, we aim to learn a function that maps the input sequence into another one with the corresponding label $\hat{Y}=\hat{y_{1}},\hat{y_{2}},\hat{y_{3}},...,\hat{y_{n}}$ in the same length.
As summarized in Figure \ref{fig:architecture}, the Transformer-based models (e.g., BERT \cite{BERT}, XLNET \cite{yangxlnet}) are regarded as the Hero module to model the entire sentence for global sequence information extraction and the Gang module is responsible for local and relative position information extraction.
Afterward, we employ the multi-window attention to elaborately combine these different features (i.e., features extracted from the Hero and Gang modules), which is then used to predict labels for each token. Therefore, the aforementioned process can be formulated as:
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
    \hat{Y} = f(X, \text{H}(X), \text{G}(X)),
\end{equation}
where $\text{H}(\cdot)$ and $\text{G}(\cdot)$ refer to the Hero and Gang modules, respectively, and the details of them are presented in the following subsections.
}

\subsection{Hero Module}
\textcolor{black}{
\textcolor{black}{The role of the Hero module in our proposed model is similar to that of the leader in a team, who is responsible for providing guidance, offering instructions, giving directions, and assigning sub-tasks to fellow memberships. Therefore, the Hero module is required to have a comprehensive understanding of the task, including overall and local progress.}
Thanks to the characteristics of the multi-head self-attention mechanism, Transformer is powerful in modeling long sequences and can provide more effective global information than other counterpart models, and it has already achieved promising results in the NER task \cite{luo2020hierarchical,beltagy2019scibert}. 
Thus, we employ a Transformer-based encoder as our Hero module to obtain the global context information $\mathbf{z}_{i}$ for each token $x_{i}$ by
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
    [\mathbf{z}_{1},\mathbf{z}_{2},\cdots,\mathbf{z}_{N}] = f_\text{H} (x_{1},x_{2},...,x_{N}).
\end{equation}
Herein, $f_\text{H} (\cdot)$ refers to a pre-trained Transformer-based sequence encoder (e.g., BERT \cite{BERT} and BioBERT \cite{biobert}).
}
\textcolor{black}{The features $\mathbf{z}$ are then input to the Gang module for extracting local contextual features and their corresponding relative position information.}


\input{data}
\subsection{Gang Module}
\textcolor{black}{As introduced in the previous section, although pre-trained models are able to provide effective global contextual representation, it lacks the ability to extract local features and relative position information.
\textcolor{black}{Thus, we propose a multi-window recurrent module, named Gang, to enhance local information extraction.}
Recurrent structures (RS), such as LSTM, GRU, and tradition RNN are effective in extracting both local and relative position information from the sequence, owing to characteristics of the recurrent mechanism.
To better emphasize the local features of each word without being disturbed by long-distance information, we construct a sliding window with a fixed length to generate shorter sub-sequences, where each sub-sequence includes several consecutive elements in $\mathbf{z}$.
An additional advantage of this operation is that, in comparison with the whole sequence, the sub-sequence is much shorter so that it is easier to be modeled by the RS.
}

\textcolor{black}{
In detail, for a single sliding window with length $k$, each hidden state $z_{i}$ from the Hero module, the corresponding sub-sequence is $\mathbf{z}_{i-k},\mathbf{z}_{i-k+1},...,\mathbf{z}_{i},...,\mathbf{z}_{i+k-1},\mathbf{z}_{i+k}$ that includes $2k+1$ consecutive tokens.
This sub-sequence of length $2k+1$ contains rich local contextual information of $x_{i}$, and thus we utilize an RS to encode it for obtaining local semantic and relative position information.
To extract the local information of two directions, we utilize a bidirectional structure to encode this sequence span, where the forward RS computes a representation $\overrightarrow{\mathbf{h}_{2k+1}}$ from left to right, and the other backward RS computes a vector $\overleftarrow{\mathbf{h}_{1}}$ for the same sub-sequence in reverse.}
\textcolor{black}{We concatenate the $\overleftarrow{\mathbf{h}_{1}}$ and $\overrightarrow{\mathbf{h}_{2k+1}}$ as the local feature $\mathbf{h}_{i} = [\overleftarrow{\mathbf{h}_{1}}, \overrightarrow{\mathbf{h}_{2k+1}} ]$ for token $x_{i}$, and then we can obtain local features for each token in sequence $X$ via the similar way, denoted as $\mathbf{h}=\mathbf{h}_{1},\mathbf{h}_{2},\cdots,\mathbf{h}_{N}$.}

\textcolor{black}{In practice, we need to consider two situations.
First, each token might have multiple levels of local information, such as phrase-level and clause-level, which may affect the understanding of the current token.
Second, since different tokens or the same token in various contexts might have different relationships with their surrounding words, we need to consider more sub-sequences with varying lengths for obtaining more comprehensive local contextual information.
Therefore, we propose to utilize multiple sliding windows with different window sizes to extract richer local features to alleviate the above issues.
We assume that local features $\mathbf{h}^{1}, \mathbf{h}^{2}, \cdots, \mathbf{h}^{M}$ are extracted from different groups of sub-sequences, whose corresponding window lengths are $k^{1}, k^{2}, \cdots, k^{M}$. This process can be formulated as:
}
\textcolor{black}{
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
   \mathbf{h}^{1}, \mathbf{h}^{2}, \cdots, \mathbf{h}^{M} = \text{Gang} (k^{1}, k^{2}, \cdots, k^{M},\mathbf{z}),
\end{equation}
\textcolor{black}{
where $M$ is the number of sliding windows and $\mathbf{h}^{j}$ is a group of local features extracted from the corresponding sliding window with length $k^{j}$. The process is similar to the task assignment in the team, where different members are responsible for their own sub-tasks.
}}


\subsection{Multi-window Attention}
\textcolor{black}{We obtain global representation $\mathbf{z}$ from the Hero module and multiple local features $\mathbf{h}^{1}, \mathbf{h}^{2}, \cdots, \mathbf{h}^{M}$ from the Gang module.
Next, we apply the multi-window attention to effectively combine global contextual information and local features.
In doing so, two types of attention methods are proposed in our model: MLP-Attention and DOT-Attention, respectively.}
\vskip 0.2em
\noindent\textbf{MLP-Attention}~
\textcolor{black}{We first concatenate these local features with global information and obtain the intermediate state $\mathbf{m}$ by a fully connected layer.}
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
  \mathbf{m} = \text{MLP}([\mathbf{z},\mathbf{H}]),
\end{equation}
\textcolor{black}{where $\mathbf{H}=[\mathbf{h}^{1},\mathbf{h}^{2}, \cdots, \mathbf{h}^{M}]$ and $\mathbf{m}$ have the same dimension as $\mathbf{z}$. MLP represents a fully connected layer.
Then $\mathbf{m}$ is used as a query vector and $[\mathbf{z},\mathbf{H}]$ serves as the key and value matrix.}
\textcolor{black}{The final token representation can be computed by}
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
  \mathbf{s} = \text{softmax}(\mathbf{m}([\mathbf{z},\mathbf{H}])^{\top})[\mathbf{z},\mathbf{H}].
\end{equation}
\vskip 0.2em
\noindent\textbf{DOT-Attention}~
\textcolor{black}{Instead of using a fully connected layer to generate a query vector, in this approach, we directly regard $\mathbf{z}$ as the query vector and $\mathbf{H}$ as the key and value matrix.
We can obtain the final local feature by}
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
  \mathbf{u} = \text{softmax}(\mathbf{z}(\mathbf{H})^{\top})\mathbf{H}.
\end{equation}
\textcolor{black}{
Since $\mathbf{u}$ is a weighted sum of different local features without considering global information, we use the sum of $\mathbf{u}_{i}$ and $\mathbf{z}_{i}$ as the final representation for each token $x_{i}$.
Thus, the final representation can be obtained by}
\begin{equation}
\setlength\abovedisplayskip{6pt}
\setlength\belowdisplayskip{6pt}
  \mathbf{s} = \{\mathbf{z}_{1}+\mathbf{u}_{1},\mathbf{z}_{2}+\mathbf{u}_{2}, \cdots, \mathbf{z}_{N}+\mathbf{u}_{N}\}.
\end{equation}

\textcolor{black}{After obtaining the final representation from MLP-Attention or DOT-Attention, $\mathbf{s}$ is sent to the corresponding classifier implemented by the softmax function to predict the distribution of labels for each token in $X$.}

\input{general_sota}




\section{Experiments Settings}
\subsection{Dataset and Metrics}
\textcolor{black}{
In our experiments, six datasets are used in our experiments, WNUT17 (W17) \cite{w16}, WNUT16 (W16) \cite{w17}, OntoNotes 5.0 (ON5e) \cite{pradhan2013towards}, BC5CDR-disease (BC5-D), BC2GM, and BC5CDR-chem (BC5-C).
The W17 and W16 are social media benchmark datasets constructed from Twitter, and ON5e is a general domain dataset consisting of diverse sources like telephone conversations, newswire, etc.
\textcolor{black}{BC5CDR, including both BC5-D and BC5-C, is a dataset used for the BioCreative V Chemical Disease Relation Task and contains chemical and disease mentions, where humans manually annotate the annotations.} 
BC2GM is the dataset that is usually utilized for the BioCreative II gene mention tagging task and contains 20000 sentences from the abstracts of biomedical publications.
For all datasets, we utilize the official splits for a fair evaluation and the statistics of the datasets are shown in Table \ref{Tab:NERdataset}.
Besides, we follow previous studies that the final models are trained on training and validation sets on each dataset except the ON5e dataset.
}

\textcolor{black}{For metrics, we exploit the same evaluation metrics used by previous works where precision (P), recall (R), and F-1 score are reported to evaluate the performance of our model.
}

\subsection{Implementation Details}
\textcolor{black}{
We implement our model based on transformers \cite{wolf2020transformers}\footnote{\url{https://github.com/huggingface/transformers}} and employ pre-trained models to obtain global contextualized representation.
Specifically, for general domain datasets (i.e., W16, W17 and ON5e), we use BERT-cased-large \cite{BERT}\footnote{We obtain the pre-trained BERT from \url{https://github.com/google-research/bert}.} and XLNET-large-cased \cite{yangxlnet}\footnote{We obtain XLNET from \url{https://github.com/zihangdai/xlnet}.} as our Hero module.
For biomedical datasets, BioBERT \cite{biobert}\footnote{We obtain BioBERT from \url{https://github.com/dmis-lab/biobert}} is utilized to obtain global information.
}
\textcolor{black}{
We follow their default settings for all BERT, XLNET, and BioBERT: 24 layers of self-attention with 1024 dimensional embeddings.
For hyperparameters of the Gang module, the hidden sizes of bidirectional recurrent structures for each window size are half of the embedding dimension from the output of the Hero module (i.e., 512).
During the training process, we use Adam \cite{adam} to optimize the negative log-likelihood loss function.
More training details are shown in the Appendix \ref{appendix:hyperparameter}.
Besides, we also compare four operations to combine different level features from the Hero and Gang module: MLP-Attention, DOT-Attention, concatenation, and summation, respectively, where concatenation is to connect all features directly through $\mathbf{s} = [\mathbf{h}^{1}, \mathbf{h}^{2},\cdots, \mathbf{h}^{M},\mathbf{z}]$,  and summation is to add up these features by $\mathbf{s} = \mathbf{h}^{1}+\mathbf{h}^{2}+\cdots+\mathbf{h}^{M}+\mathbf{z}$.
}


\subsection{Baselines}
To explore the impact of our proposed model, we compare our model to the previous studies.
For general domain, following baselines are compared in our experiment on W16, W17 and ON5e.
\begin{itemize}
    \item \textbf{\textsc{CNN-BiLSTM-CRF}} \cite{chiu2016named} utilizes a hybrid bidirectional and CNN architecture to detect word-and character-level features.
    \item \textbf{\textsc{BERT}} \cite{BERT} is a pre-trained language model and we apply it to the NER task by direct fine-tuning.
    \item \textbf{SANER} \cite{nie2020named}, \textbf{CL-KL} \cite{wang2021improving} and \textbf{\textsc{AESUBER}} \cite{AESUBER} improve entity recognition by leveraging syntactic information or semantically relevant texts.
    \item \textbf{\textsc{Hire-NER}} \cite{luo2020hierarchical} utilizes both sentence-level and document-level representations to improve sequence labeling.
    \item \textbf{\textsc{Syn-LSTM-CRF}} \cite{xu2021better} integrates the structured information by graph-encoded representations obtained from GNNs.
    \item \textbf{\textsc{BARTNER}} \cite{yan2021unified} formulates NER tasks as a span sequence generation problem.
\end{itemize}
 In addition, we also compare our proposed model with the following baselines on the aforementioned biomedical datasets:
\begin{itemize}
    \item \textbf{\textsc{MTM-CW}} \cite{wang2019cross}, \textbf{\textsc{BiLM}} \cite{sachan2018effective}, \textbf{\textsc{NCBI\_BERT}} \cite{peng2019transfer}, \textbf{\textsc{MT-BioNER}} \cite{2021A} utilize multi-task learning or transfer learning to enhance biomedical NER.
    \item \textbf{\textsc{BioBERT}} \cite{biobert} is a pre-trained model trained with a large amount of biomedical corpus and then applied by directly fine-tuning.
   
    \item \textbf{\textsc{\textsc{KeBio-LM}}} \cite{yuan2021improving} proposes a  biomedical pre-trained language model that incorporates knowledge from the Unified Medical Language System (UMLS).
\end{itemize}
Note that in both general and biomedical domains, our model \textbf{does not require external resources.} 

\input{biomedical_sota}


\section{Results and Analyses}

\subsection{General Domain NER}
\textcolor{black}{
In this subsection, to explore the effectiveness of our proposed model, we conduct experiments to compare our model with existing studies, and the results are reported in Table \ref{Tab:NER}.
There are several observations drawn from different aspects.
First, when we make a fair comparison without extra resources (e.g., BERT, XLNET, and ASTRA), our model obtains significant improvements on all datasets in terms of Precision, Recall, and F-1, which confirms the effectiveness of our proposed Hero-Gang neural structure. This is because multiple-level features can be reasonably encoded into the model and thus alleviate the limitations of Transformer in local feature extraction.
Second, although some complicated models enhance NER by incorporating extra knowledge, e.g., SANER uses augmented semantic information, Hire-NER utilizes two-level hierarchical contextualized representations, and CL-KL selects a set of semantically relevant texts to improve NER, our model achieves competitive results without such requirements.
This is because each word in the natural text usually has a closer relationship with its surrounding words, especially the adjacent words, such that features extracted by the Gang module can provide more valuable information for NER, and thus our model achieves promising performance.
Third, the XLNET-based model obtains better results than the BERT-based model, which indicates that XLNET can generate more effective representations on the NER task. The reason behind this might be that XLNET combines the permutation operation with the autoregressive technology to further improve representation learning, so that XLNET can provide a better text understanding than BERT.
}




\subsection{Biomedical NER}
We also compare our model with state-of-the-art models in the biomedical NER on the aforementioned datasets with all results reported in Table \ref{Tab:BIO}.
There are several observations.
First, we can see that our model outperforms existing methods, regardless of whether they introduce external knowledge, which further confirms the validity of our innovation in combining local and global features to enhance feature extraction.
Second, although some models utilize higher-level features, e.g., BIOKMNER leverages POS labels, syntactic constituents, dependency relations, and MTM-CW employs multi-task learning to train the model, our model can achieve better results through a simple Hero-Gang structure.
This means that local features extracted from the Gang module under the guidance of global information are also effective in assisting biomedical text representations and even show more significant potential than those special designs for the medical domain (i.e., domain-related multi-task learning).
Third, the models using the multi-window attention (i.e., DOT-Attention and MLP-Attention) outperform those using concatenation or summation.
This observation suggests that multi-window attention can elaborately weigh local features from different sliding windows to enhance feature combinations.


\subsection{Analyses}
\paragraph{Effect of position information}
\textcolor{black}{
Recurrent structures are able to extract both context and position information by its token-by-token manner while other network structures, including CNN and MLP, fail to encode the relative position information.
Thus, to explore the effect of position information, we compare models with different structures to construct the Gang module and report the improvements of F-1 score based on different Gang modules in Figure \ref{fig:position}.
First, we can observe that models with Gang module are better than Base (i.e., BERT), where all the values in Figure \ref{fig:position} are positive, further illustrating the effectiveness of our innovation in combining both global and local features, no matter what type of structure is used to construct the Gang module.
Second, models with LSTM and GRU outperform those with CNN and MLP, indicating that recurrent structures are more promising in short sequence feature extraction.
Since the recurrent structures can effectively capture position information by its token-by-token manner and help the model understand word-word relations based on their relative positions, we may conclude that position information is vital for improving performance.
Third, the comparison between CNN and MLP shows the power of CNN in extracting features from sub-sequences since CNN can leverage more fine-grained features, such as n-gram.
}
\begin{figure}[t]
\centering
\includegraphics[width=0.49\textwidth, trim=0 10 0 10]{position.pdf}
\caption{\textcolor{black}{The improvement values (\%) compared to Base models (i.e., BERT for general domain datasets and BioBERT for biomedical datasets) in terms of F-1 score from different Gang modules, MLP, CNN, RNN, GRU, and LSTM, respectively.}}
\label{fig:position}
 \vskip -1em
\end{figure}
\paragraph{Ablation studies}
In this subsection, we compare our multi-window model with single-window models, and the improvements compared with Base model are shown in Figure \ref{fig:ablation}.
We have following observations.
First of all, illustrated by the comparisons among Base (i.e., BERT) and others, models with sliding windows achieve better performance, where all the improvement values in Figure \ref{fig:ablation} are positive.
This illustrates that both single window and multi-window recurrent structures can help to enhance token representation and bring different degrees of improvement, which further shows the importance of local features in this task.
Second, we can observe that the optimal single window sizes for different datasets are also different.
For example, the optimal single window size of W17 is 5, while that for BC2GM is 7, which indicates that the best length of the local sequence depends on the characteristics of datasets to some extent.
Third, compared with those models using a single window, the multi-window recurrent module obtains better performance, illustrating that features extracted from multiple sub-sequences are more effective than those captured from a single one.
The reason could be that multi-window can help the model pay attention to different local context sub-sequences and give them appropriate weights through the multi-window attention mechanism, such that it can provide more reasonable local information and alleviate the impact of the characteristics of the datasets themselves.
\begin{figure}[t]
\centering
\includegraphics[width=0.49\textwidth, trim=0 10 0 10]{ablation.pdf}
\caption{\textcolor{black}{The improvement values (\%) of models with single windows or multi-window compared to Base models (BERT or BioBERT w.r.t. datasets), where 3, 5, 7, 9, 11 represents the single window size when models only use a single window to construct the Gang module.}}
\label{fig:ablation}
 \vskip -1em
\end{figure}
\begin{figure*}[t]
\centering
\includegraphics[width=0.99\textwidth, trim=0 10 0 5]{case.pdf}
\caption{\textcolor{black}{Examples of two predicted labels from \textsc{Base} and \textsc{Ours} as well as their corresponding source sentence and real label. Note that the \textsc{Base} for these two cases are \textsc{Bert} and \textsc{BioBERT}, respectively.}}
\label{fig:case}
 \vskip -1em
\end{figure*}

\paragraph{Case Study}
\textcolor{black}{
To further show the validity of our model, we perform qualitative analysis on some cases with their real labels and predicted labels from different models.
Figure \ref{fig:case} shows two cases from ON5e and BC5-C, respectively.
\textcolor{black}{We can observe that our model can predict more complete entities than Base. 
Specifically, in the first case, our model can recognize all the words in the entity \textit{"a period of years"} while Base model only recognizes the word \textit{"years"}.
In the second case, our model is able to identify \textit{"Monosodium glutamate"}, but Base model regards these words as two different entities.}
In addition, in the first example, compared with real labels, our model can label two \textit{"of"} correctly with the help of local features, which are O and I-date, respectively, while Base classifies both \textit{"of"} as O.
The sub-sequence (i.e.,\textit{"a period of years,"}) from the second \textit{"of"} is usually used to describe time such that this information is able to assist the model in marking the \textit{"of"} as I-date.
 %
However, for the first \textit{"of"}, its sub-sequence \textit{"divest themselves of such speculative"} does not contain any meaning related to the entity themes, and thus the model marks the corresponding \textit{"of"} as O.
}

\section{Related Work}
\textcolor{black}{
NER is a fundamental task in NLP \cite{huang2015bidirectional}, which has drawn substantial attention over the past years and there have been many studies to address this task.
Recently, deep learning has played a dominant role in NER due to its effectiveness in capturing contextual information from sequences.
The recurrent neural networks (RNN), including its variants such as LSTM \cite{hochreiter1997long}, and GRU \cite{GRU}, is a promising structure for solving this task since it can effectively learn sequence information with its recurrent mechanism \cite{ma2016end, huang2015bidirectional, chiu2016named, zhu2019can}.
However, it is ineffective for RNN to learn long sequences due to the gradients exploding and vanishing.
Thus, Transformer-based models, such as BERT \cite{BERT}, BioBERT \cite{biobert}, and XLNET \cite{yangxlnet}, are proposed to alleviate these problems with the help of the self-attention mechanism.
Compared to RNN, Transformer is able to capture long-distance information through multiple multi-head attention layers and has achieved impressive performance in this task \cite{nie2020named,luo2020hierarchical,yamada2020luke, gui2019lexicon}.
}

However, multi-head attention usually treats every position identically, which lead to the loss of position information.
To mitigate this problem, several approaches have been proposed to advance the Transformer \cite{dai2019transformer,shaw2018self,yan2019tener}.
\citet{shaw2018self} proposed cross-lingual position representation to help self-attention alleviate word order divergences in different languages and learn position information.
\textcolor{black}{
\citet{yan2019tener} introduced the directional relative positional encoding and an adapted Transformer Encoder to model the character-level and word-level features.
}
Although these position embeddings are able to help the model learn position information, they are still not enough to solve the issue appropriately \cite{Rtransformer, huang2020improve,qu2021explore}.
Besides, Transformer-based approaches cannot effectively extract local features that are also important for sequence learning tasks, and some studies have been proposed to alleviate this problem \cite{xu2017local,li2019enhancing,yang2019convolutional}.
\citet{xu2017local} proposed to use the fixed-size ordinally forgetting encoding to model sentence fragments, which is then used to predict the label for each text fragment.
\citet{li2019enhancing} utilized convolutional self-attention by producing queries and keys with causal convolution to incorporate local contextual information into the attention mechanism.
To address these issues, we offer an alternative solution, namely Hero-Gang Neural model, to enhance local and position information extraction via multiple recurrent structures under the guidance of global information.

\section{Conclusion}
\textcolor{black}{
In this paper, we propose a novel Hero-Gang {N}eural (HGN) structure to effectively combine global and local features for enhancing NER.
In detail, the Hero module aims to capture global understanding by a Transformer-based encoder, which is then used to guide the Gang to extract local features and relative position information through a multi-window recurrent module.
Afterward, we utilize the multi-window attention to elaborately combine the global information and local features for enhancing representations that are then used to predict the entity label for each token.
Empirically, our proposed model achieves new state-of-the-art results on several NER benchmark datasets, including both general and biomedical domains.
Besides, we compare different structures to construct the Gang model and investigate the effect of the number of sliding windows, which further illustrates the effectiveness of our proposed model.
}



\section*{Acknowledgements}
This work is supported by Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101350001), NSFC under the project “The Essential Algorithms and Technologies for Standardized Analytics of Clinical Texts” (12026610) and the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen.


\subsection{Hyper-parameter Settings}
\label{appendix:hyperparameter}
\begin{table*}[t]
\centering
\vspace{1mm}
\begin{tabular}{c|c|c|c|c|c|c|c}
\toprule
\multirow{2}{*}{\textsc{\textbf{Model}}} & \multirow{2}{*}{\textsc{\textbf{Hypter.}}} & \multicolumn{3}{c|}{\textsc{\textbf{NLP Data}}} & \multicolumn{3}{c}{\textsc{\textbf{Biomedicine Data}}} \\ \cmidrule(l){3-8} 
 &  & \textsc{W16} & \textsc{W17} & \textsc{ON5e} & \textsc{BC2GM} & \textsc{BC5-D} & \textsc{BC5-C} \\ \midrule
\multirow{3}{*}{\textsc{HGN (MLP)}} & Window Size &\{1,3,5,7\} &  \{3,5,7\}
&  \{5,7,9\}
&  \{1,3,5\}
&  \{5,7,11\}
&  \{5,7,11\}
\\
 & Learning Rate & 3e-5 & 5e-5 & 1e-5 & 1e-5 & 9e-6 &1e-5  \\
 & Batch Size & 32  & 32 & 32 & 32 & 32 &32  \\ \midrule
\multirow{3}{*}{\textsc{HGM (DOT)}} & Window Size & \{3,5,7\} & \{5,7,9\}
&  \{3,5,7\}
&  \{3,5,7\}
&  \{5,7,9\}
&  \{5,7,11\}
\\
 & Learning Rate &3e-5  &5e-5  & 1e-5 & 1e-5 &9e-6  &9e-6  \\
 & Batch Size & 32  & 32 & 32 & 32 & 32 &32  \\ \bottomrule
\end{tabular}
\caption{The hyper-parameters for best models that we have experimented on the given datasets.}
\label{tab:hypter-parameter}
\end{table*}
We have tested several combinations of hyper-parameters in tuning our models for all NLP and Biomedical benchmark datasets (i.e., W16, W17, \textsc{ON5e}, BC5CDR-disease, BC2GM, and BC5CDR-chem). Table \ref{tab:hypter-parameter} reports the combinations that achieve the highest F-1 score for each dataset.




