\section{Introduction}
Named Entity Recognition (NER) \cite{Sundheim1995NamedET} has attracted increasing attention in natural language processing community. It has been applied to a lot of domains such as news \citep{tjong-kim-sang-2002-introduction,tjong-kim-sang-de-meulder-2003-introduction}, E-commerce \citep{10.1145/3404835.3463102}, social media \cite{strauss-etal-2016-results,derczynski-etal-2017-results} and bio-medicine \citep{dougan2014ncbi,li2016biocreative}. Several recent studies focus on improving the accuracy of NER models through utilizing image information (MNER) in tweets \citep{zhang2018adaptive,moon-etal-2018-multimodal,lu-etal-2018-visual}. Most approaches to MNER use the attention mechanism to model the interaction between image and text representations \citep{yu-etal-2020-improving-multimodal,zhang2021multi,Sun2021RpBERTAT}, in which image representations are from a pretrained feature extractor, i.e. ResNet \citep{he2016deep}, and text representations are extracted from pretrained textual embeddings, i.e. BERT \citep{devlin-etal-2019-bert}. Since these models are separately trained on datasets of different modalities and their feature representations are not aligned, it is difficult for the attention mechanism to model the interaction between the two modalities.



Recently, pretrained vision-language (V+L) models such as LXMERT \citep{Tan2019LXMERTLC}, UNITER \citep{chen2020uniter} and Oscar \citep{li2020oscar} have achieved significant improvement on several cross-modal tasks such as image captioning, VQA \citep{Agrawal2015VQAVQ}, NLVR \citep{Young2014FromID} and image-text retrieval \citep{Suhr2019ACF}. Most pretrained V+L models are trained on image-text pairs and simply concatenate text features and image features as the input of pretraining. There are, however, two problems. First, texts in these datasets mainly contain common nouns instead of named entities\footnote{\url{https://visualgenome.org/data_analysis/statistics}} which leads to an inductive bias over common nouns and images. Second, despite its important role in pretraining V+L models,  the image modality only plays an auxiliary role in MNER for disambiguation, and can sometimes even be discarded. These problems make pretrained V+L models perform weaker than pretrained language models for MNER.

Pretrained textual embeddings such as BERT, XLM-RoBERTa \citep{conneau-etal-2020-unsupervised} and LUKE \citep{yamada-etal-2020-luke} have achieved state-of-the-art performance on various NER datasets through simple fine-tuning of pretrained textual embeddings. Since most of the transformer-based pretrained textual embeddings are trained over long texts, recent work \citep{akbik-etal-2019-pooled,schweter2020flert,yamada-etal-2020-luke,wang-etal-2021-improving} has shown that introducing document-level contexts can significantly improve the accuracy of a NER model. The attention mechanism in transformer-based pretrained textual embeddings can utilize contexts to improve the token representation of a sequence. Moreover, pretrained V+L models such as Oscar and VinVL \citep{zhang2021vinvl} can use object tags detected in images to significantly ease the alignments between text and image features. Therefore, the images in MNER can be converted to texts as well so that the image representations can be aligned to the space of text representations. As a result, the attention module of the pretrained textual embeddings have the capability to easily model the interactions between aligned image and text representations, without introducing a new attention module. In this paper, we propose ITA, a simple but effective framework for {\bf I}mage-{\bf T}ext {\bf A}lignments. ITA converts an image into visual contexts in textual space by multi-level alignments. We concatenate the NER texts with the visual contexts as a new cross-modal input view and then feed it into a pretrained textual embedding model to improve the token representations of NER texts, which are fed into a linear-chain CRF \citep{10.5555/645530.655813} layer for prediction.
In practice, a MNER model should be robust when there is only text information, as images may be unavailable or can introduce noises. Sometimes it is even undesirable to use images as image feature extraction can be inefficient in online serving. Therefore, we further propose to utilize the cross-modal input view to improve the accuracy of textual input view, based on cross-view alignment that minimizes the KL divergence over the probability distributions of the two views. 

ITA can be summarized in four aspects:
\begin{enumerate}[leftmargin=*]
    \item Object Tags as Local Alignment: ITA locally extracts object tags and its corresponding attributes of image regions from an object detector.
    \item Image Captions as Global Alignment: ITA summarizes what the image is describing through predicting image captions from an image captionilng model.
    \item Optical Character Alignment: ITA extracts the texts presented in the image via optical character recognition (OCR).
    \item Cross-View Alignment: we calculate the KL divergence between the output distributions of two input views.
\end{enumerate}
We show in experiments that ITA can significantly improve the model accuracy on MNER datasets and achieve the state-of-the-art. The cross-view alignment module can significantly improve both the cross-modal and textual input views, and bridge the performance gap between the two views.



\begin{figure*}[ht]
	\centering
	\includegraphics[scale=0.61]{model.pdf}
	\caption{The architecture of ITA. ITA aligns an image into object tags, image captions and texts from OCR. ITA takes them as visual contexts and then feeds them together with the input texts into the transformer-based embeddings. In the cross-view alignment module, ITA minimizes the distance between the output distribution of cross-modal inputs and textual inputs.
	}
	\label{fig:architecture}
\end{figure*}


\section{Approaches}
We consider the NER task as a sequence labeling problem. Given a sentence ${\bm{w}} = \{w_1, \cdots, w_n\}$ with $n$ tokens and its corresponding image $I$, an sequence labeling model aims to predict a label sequence ${\bm{y}} = \{y_1, \cdots, y_n\}$ at each position.
In our framework, we focus on incorporating visual information to improve the representations of the input tokens by aligning visual and textual information effectively. We use a visual context generator to convert the image $I$ into texts forming visual contexts ${\bm{w}}^\prime = \{w_1^\prime, \cdots, w_m^\prime\}$ with $m$ tokens. We then concatenate the input text and visual contexts as a cross-modal text+image (\textbf{I+T}) input view instead of the text (\textbf{T}) input view. We feed the \textbf{I+T} input into a pretrained textual embeddings model to get stronger token representations of the input sentence. Then the token representations are fed into a linear-chain CRF layer to get the label sequence ${\bm{y}}$. To further improve the model accuracy of both input views, we use the cross-view alignment module to align the output distributions of \textbf{I+T} and \textbf{T} input views during training. The architecture of our framework is shown in Figure \ref{fig:architecture}.


\subsection{NER Model Architecture}
We use a neural model with a linear-chain CRF layer, a widely used approach for the sequence labeling problem \citep{Huang2015BidirectionalLM,akbik-etal-2018-contextual,devlin-etal-2019-bert}. The input is fed into a transformer-based pretrained textual embeddings model and the output token representations $\{{\bm{r}}_1, \cdots, {\bm{r}}_n\}$ are fed into the CRF layer:
\begin{align}
    p_\theta({\bm{y}}|{\bm{w}}) &= \frac{\prod\limits_{i=1}^{n} \psi(y_{i-1}, y_i, {\bm{r}}_i)}{\sum\limits_{{\bm{y}}' \in \mathcal{Y}({\bm{w}})} \prod\limits_{i=1}^{n} \psi(y'_{i-1}, y'_i, {\bm{r}}_i)}\nonumber
\end{align}
where $\theta$ is the model parameters, $\mathcal{Y}({\bm{w}})$ is the set of all possible label sequences given the input ${\bm{w}}$. Given the gold label sequence $\hat{{\bm{y}}}$ in the training data, the objective function of the model for the \textbf{T} input view is:
\begin{align}
\mathcal{L}_\text{T}(\theta) = - \log p_\theta(\hat{{\bm{y}}}|{\bm{w}}) \label{eq:nll_loss}
\end{align}
The loss can be calculated using Forward algorithm.

\subsection{Image-text Alignments}
The transformer-based pretrained textual embeddings have strong representations over texts. Therefore, ITA converts the image information into textual space through generating texts from the image so that the learning of the self-attention in the transformer-based model can be significantly eased compared with simply using image features from an object detector. We propose a local (LA), a global (GA) and an optical character alignment (OCA) approaches for alignments.
\paragraph{Object Tags as Local Alignment} Given an image, the image information can be decomposed into a set of objects in local regions. The object tags of each region textually describe the local information in the image. To extract the objects, we use an object detector $\textbf{OD}$ to identify and locate the objects in the image: 
\begin{align}
&{\bm{a}}, {\bm{o}} = \textbf{OD}(I); \text{where }  \nonumber\\
&{\bm{a}} = \{{\bm{a}}_1,{\bm{a}}_2,\cdots,{\bm{a}}_l\} \text{ and } {\bm{o}} = \{{o}_1,{o}_2,\cdots,{o}_l\} \nonumber
\end{align}
The attribute predictions from the object detector contain multiple attribute tags ${\bm{a}}_i$ for each object ${o}_i$. We linearize and sort the objects in a descending order based on the confidences of the detection model. For each object, we heuristically keep 0 to 3 attributes with confidence scores above a threshold $m$. We linearize the attributes and put the attributes before the corresponding objects since the attributes are the adjectives describing the object tags. 
As a result, we take the predicted $l$ object tags ${\bm{o}}$ and their attribute tags ${\bm{a}}$ from the object detector as the locally aligned visual contexts ${\bm{w}}^{\text{LA}}$:
\begin{align}
{\bm{w}}^{\text{LA}} = \{{\bm{a}}_1,{o}_1,{\bm{a}}_2,{o}_2,\cdots,{\bm{a}}_l,{o}_l\}\nonumber
\end{align}


\paragraph{Image Captions as Global Alignment} Though the local alignment can localize the image into objects, the objects cannot fully describe the of the whole image. Image captioning is a task that predicts the meaning of an image. Therefore, we align the image into $k$ image captions by an image captioning model $\textbf{IC}$:
\begin{align}
\{{\bm{w}}^1,{\bm{w}}^2,\cdots,{\bm{w}}^k\} = \textbf{IC}(I) \nonumber
\end{align}
where $\{{\bm{w}}^1,{\bm{w}}^2,\cdots,{\bm{w}}^k\}$ are captions generated from beam search with $k$ beams. We concatenate the $k$ captions together with a special separate token [X] to form the aligned global visual contexts ${\bm{w}}^{\text{GA}}$:
\begin{align}
 {\bm{w}}^{\text{GA}}  {=} [{\bm{w}}^1,\text{[X]},{\bm{w}}^2,\text{[X]},\cdots,\text{[X]},{\bm{w}}^k] \nonumber
\end{align}
The exact label (e.g. ``[SEP]'' in BERT) of the special [X] token depends on the selection of embeddings.


\paragraph{Optical Character Alignment} Some image contain text when they are created to enrich the semantic information that the images want to convey. In order to better understand this type of image, we use an \textbf{OCR} model to identify and extract the texts in the image:
\begin{align}
{\bm{w}}^{\text{OCA}} = \textbf{OCR}(I) \nonumber
\end{align}
where ${\bm{w}}^{\text{OCA}}$ are the texts extracted by the \textbf{OCR} model. Note that ${\bm{w}}^{\text{OCA}}$ may be an empty text if there is no text in the image.

We concatenate the input sentence and our aligned visual contexts to form the \textbf{I+T} input view $\hat{{\bm{w}}}=[{\bm{w}};{\bm{w}}^\prime]$, where ${\bm{w}}^\prime$ can be one of ${\bm{w}}^{\text{LA}}$, ${\bm{w}}^{\text{GA}}$, ${\bm{w}}^{\text{OCA}}$ or the concatenation of all (we denote it as \textbf{All}). The transformer-based embeddings are fed with the \textbf{I+T} input view and then output image-text fused token representations for each token $\{{\bm{r}}_1^\prime, \cdots, {\bm{r}}_n^\prime\}$. The token representations are fed into the CRF layer to get the probability distribution $p_\theta({\bm{y}}|\hat{{\bm{w}}})$. Similar to Eq. \ref{eq:nll_loss}, the objective function of the model for the \textbf{I+T} input view is:
\begin{align}
\mathcal{L}_{\text{I+T}}(\theta) = - \log p_\theta(\hat{{\bm{y}}}|\hat{{\bm{w}}}) \label{eq:nll_x}
\end{align}
\paragraph{Cross-View Alignment}
There are several limitations in incorporating images into NER prediction: 1) the images may not available in testing; 2) aligning images to texts requires several pipelines in pre-processing instead of an end-to-end manner, which is so time-consuming that it is not applicable to some time-critical scenes such as online serving; 3) the noises in the image can mislead the MNER model to make wrong predictions. To alleviate these issues, we propose Cross-View Alignment (CVA), which targets at reducing the gap between the \textbf{I+T} and \textbf{T} input views over the output distributions so that the MNER model can better utilize the textual information in the input.
During training, CVA minimizes the KL divergence over the probability distribution of \textbf{I+T} and \textbf{T} input views:
\begin{align}
\mathcal{L}_{\text{CVA}}(\theta)&{=}\text{KL}(p_\theta({\bm{y}}|\hat{{\bm{w}}})||p_\theta({\bm{y}}|{\bm{w}})) \label{eq:mda_loss}
\end{align}
Since the \textbf{I+T} input view has additional visual information in the input and we want the \textbf{T} input view to match the accuracy of \textbf{I+T} input view, we only back-propagate through $p_\theta({\bm{y}}|{\bm{w}})$ in Eq. \ref{eq:mda_loss}. Therefore, Eq. \ref{eq:mda_loss} is equivalent to calculating the cross-entropy loss over the two distributions:
\begin{align}
\mathcal{L}_{\text{CVA}}(\theta)&{=}\sum_{{\bm{y}} \in \mathcal{Y}({\bm{x}})} p_\theta({\bm{y}}|\hat{{\bm{w}}}) \log p_\theta({\bm{y}}|{\bm{w}}) \label{eq:xe_loss}
\end{align}
As the set of all possible label sequences $ \mathcal{Y}({\bm{x}})$ is exponential in size, we calculate the posterior distributions of each position $p_\theta({y}_i|{\bm{w}})$ and $p_\theta({y}_i|\hat{{\bm{w}}})$ through forward-backward algorithm to approximate Eq. \ref{eq:xe_loss}:
\begin{align}
&p_\theta({y}_k|*) {\propto} \sum\limits_{\{y_0,\dots,y_{k-1}\}} {\prod\limits_{i=1}^{k}} \psi(y_{i-1}, y_i, {\bm{r}}^*_i) \nonumber\\
&{\times} \sum\limits_{\{y_{k{+}1},\dots,y_n\}} {\prod\limits_{i{=}k{+}1}^{n}} \psi(y_{i{-}1}, y_i, {\bm{r}}^*_i) \nonumber \\
&\mathcal{L}_{\text{CVA}}(\theta){=}\sum_{i=1}^n p_\theta({y}_i|\hat{{\bm{w}}}) \log p_\theta({y}_i|{\bm{w}})) \label{eq:mda} 
\end{align}
where ${\bm{r}}^*_i$ represents either ${\bm{r}}_i$ or ${\bm{r}}_i^\prime$.
\paragraph{Training} During training, we jointly train \textbf{T} and \textbf{I+T} input views with the training objective in Eq. \ref{eq:nll_loss} and \ref{eq:nll_x} together with the CVA alignment training objective in Eq. \ref{eq:mda}. As a result, the final training objective for ITA is:
\begin{align}
\mathcal{L}_{\text{ITA}} = \mathcal{L}_{\text{CVA}} + \mathcal{L}_{\text{T}} + \mathcal{L}_{\text{I+T}} \nonumber
\end{align}





\section{Experiments}
\label{sec:exp}
We conduct experiments on three MNER datasets. To show the effectiveness of our approaches, we use two embedding settings and compare our approaches with previous multi-modal approaches.
\subsection{Settings}

\paragraph{Datasets}
We show the effectiveness of our approaches on Twitter-15, Twitter-17 and SNAP Twitter datasets\footnote{Twitter-15 and 17 datasets are available at \url{https://github.com/jefferyYu/UMT}.} containing 4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459 sentences in train/development/test split respectively. The Twitter-15 dataset is constructed by \citet{zhang2018adaptive}. The SNAP dataset is constructed by \citet{lu-etal-2018-visual} and the Twitter-17 dataset is a filtered version of SNAP constructed by \citet{yu-etal-2020-improving-multimodal}.

\paragraph{Model Configuration}
For token representations, we use BERT base model to fairly compare with most of the recent work \citep{yu-etal-2020-improving-multimodal,zhang2021multi,Sun2021RpBERTAT}. Recently, XLM-RoBERTa has achieved state-of-the-art accuracy on various NER datasets by feeding the input together with contexts to the model. To further utilize the visual contexts in transformer-based embeddings, we use XLM-RoBERTa large (XLMR) model as another embedding in our experiments. 
To extract object tags and image captions of the image, we use VinVL \citep{zhang2021vinvl}, which is a pretrained V+L model based on a newly pretrained large-scale object detector based on the ResNeXt-152 C4 architecture. We use the object detection module of VinVL to predict object tags and their corresponding attributes. The number of object tags and attributes varies over the images and is no more than $100$. We set the threshold $m$ to be $0.1$ for keeping the attributes of each object. For image captions, we use VinVL large model finetuned on MS-COCO \citep{lin2014microsoft} captions\footnote{\url{github.com/microsoft/Oscar}} with CIDEr optimization \citep{rennie2017self}. In our experiments, we use a beam size of $5$ with at most 20 tokens for prediction and keep all the $5$ captions as the visual contexts. For OCR, we use Tesseract OCR\footnote{\url{github.com/tesseract-ocr/tesseract}} \citep{smith2007overview}, which is an open source OCR engine. We use the default configuration of the engine to extract texts in the image\footnote{Please refer to Appendix \ref{app:oca} for more statistics.}.

\paragraph{Training Configuration}
During training, we finetune the pretrained textual embedding model by AdamW \citep{loshchilov2018decoupled} optimizer. In experiments we use the grid search to find the learning rate for the embeddings within $[1\times 10^{-6}, 5\times 10^{-4}]$. For BERT embeddings, we finetune the embeddings with a learning rate of $5\times 10^{-5}$ with a batch size of $16$. For XLMR embeddings, we use a learning rate of $5\times 10^{-6}$ and a batch size of $4$ instead. For the learning rate of the CRF layer, we use a grid search over $[0.05,0.5]$ and $[0.005,0.05]$ for BERT and XLMR respectively. The MNER models are trained for $10$ epochs and we report the average results from $5$ runs with different random seeds for each setting.



\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{1pt}
\begin{tabular}{c|l||cc|cc|cc}
\hlineB{4}
\multicolumn{2}{c||}{} & \multicolumn{2}{c|}{Twitter-15} & \multicolumn{2}{c|}{Twitter-17} & \multicolumn{2}{c}{SNAP} \\
 \hline
\multirow{3}{*}{\makecell{Train\\Modal}} & \multirow{3}{*}{Approach} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c}{\bf Eval}\\
 & & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c}{\bf Modal}\\
&  & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}}\\
\hline\hline
 \multicolumn{8}{c}{\bf \textsc{BERT-CRF}}\\
\hline
T & {\textbf{BERT-CRF}} & 74.79 & - & 85.18 & - & 85.98 & -\\
\hline
\multirow{7}{*}{I+T} & {\textbf{ITA-LA}} & - & 75.18 & - & 85.67& - & 86.26\\
 & {\textbf{ITA-GA}} & - & 75.17  & - & 85.75  & - & 86.72\\
 & {\textbf{ITA-OCA}} & - & 75.01 & - & 85.64& - & 86.52\\
 & {\textbf{ITA-All}} & - & 75.15 & - & 85.78 & - & 86.79\\
 & {\textbf{ITA-LA$_{\text{+CVA}}$}} & 75.26 & 75.20 & 85.72 & 85.62& 86.51 & 86.41\\
 & {\textbf{ITA-GA$_{\text{+CVA}}$}} & 75.45 & 75.52 & 85.96 & \textbf{85.85}& 86.42 & 86.39\\
 & {\textbf{ITA-OCA$_{\text{+CVA}}$}} & 75.26 & 75.30 & 85.73 & 85.79& 86.64 & 86.59\\
 & {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{75.67} & \textbf{75.60} & \textbf{85.98} & 85.72&  \textbf{86.83} & \textbf{86.75} \\
\hline
\multicolumn{8}{c}{\bf \textsc{XLMR-CRF}}\\
\hline
T & {\textbf{XLMR-CRF}} & 77.37 & - & 88.73 & - & 89.39 & - \\
\hline
\multirow{7}{*}{I+T} & {\textbf{ITA-LA}} & - & 77.64 & - & 89.29& - & 89.68\\
 & {\textbf{ITA-GA}} & - & 77.78 & - & 89.32& - & 89.78 \\
& {\textbf{ITA-OCA}} & - & 77.94 & - & 89.31& - & 89.64\\
& {\textbf{ITA-All}} & - & 77.81 & - & 89.62 & - & 90.10\\
& {\textbf{ITA-LA$_{\text{+CVA}}$}} & 77.87 & 77.93 & 89.45 & \textbf{89.90}& 89.85 & 89.91\\
 & {\textbf{ITA-GA$_{\text{+CVA}}$}} & 78.03 & 78.02 & 89.41 & 89.62& 89.85 & 90.09 \\
& {\textbf{ITA-OCA$_{\text{+CVA}}$}} & 77.57 & 77.59 & 89.32 &89.55& 89.90 & 89.84\\
& {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{78.25} & \textbf{78.03} & \textbf{89.47} & 89.75& \textbf{90.02} & \textbf{90.15}\\
\hlineB{4}
\end{tabular}
\caption{A comparison of ITA and our baseline.}
\label{tab:main}
\end{table}


\begin{table}[t!]
\small
\centering
\begin{tabular}{l||c|c|c}
\hlineB{4}
\multicolumn{1}{c||}{Approach} & \multicolumn{1}{c|}{Twitter-15} & \multicolumn{1}{c|}{Twitter-17} & \multicolumn{1}{c}{SNAP} \\
\hline
\multicolumn{4}{c}{\bf \textsc{Reported F1 of Previous Approaches}}\\
\hline
{\textbf{BERT-CRF}}$^\dagger$ & 71.81 &  83.44 & -\\
{\textbf{OCSGA}}$^\clubsuit$ & 72.92 & - & - \\
{\textbf{UMT}}$^\dagger$ & 73.41 & 85.31 & -\\
{\textbf{RIVA}}$^\ddagger$ & 73.80 & - & 86.80 \\ 
{\textbf{RpBERT$_{\text{base}}$}}$^\spadesuit$ &74.40 & - & 87.40 \\
{\textbf{UMGF}}$^\diamond$ & 74.85 & 85.51 & - \\
\hline
\multicolumn{4}{c}{\bf \textsc{Our Reproductions}}\\
\hline
{\textbf{BERT-CRF}} & 74.79 & 85.18 & 85.98\\
{\textbf{UMT}} & 72.83 & 84.88 & - \\
{\textbf{UMGF}} & 74.42 & 85.27 & - \\ 
{\textbf{RpBERT$_{\text{base}}$}} & 67.21 & - & 62.14 \\ 
{Ours: \textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{76.01} & \textbf{86.45} & \textbf{87.44} \\ 
\hlineB{4}
\end{tabular}
\caption{A comparison of our approaches and state-of-the-art approaches. $\clubsuit$: \citet{10.1145/3394171.3413650}; $\dagger$: results are from \citet{yu-etal-2020-improving-multimodal}; $\ddagger$: \citet{sun-etal-2020-riva}, $\spadesuit$: \citet{Sun2021RpBERTAT}, note that {\textbf{RpBERT$_{\text{base}}$}} uses the test set to select the best model; $\diamond$: results are from \citet{zhang2021multi}.}
\label{tab:previous_sota}
\end{table}





\subsection{Results}
In Table \ref{tab:main}, we compare our approaches with our baselines with different training and evaluation modalities (\textbf{T} for the text-only input view and \textbf{I+T} for the multi-modal input view). Results show that ITA models are significantly stronger than our {\textbf{BERT-CRF}} and {\textbf{XLMR-CRF}} baselines (Student's t-test with $p<0.05$). For the aligned visual contexts, LA, GA and OCA are competitive in most of the cases. To show the effectiveness of CVA, we report the evaluation results of both input views in evaluation. With CVA, the accuracy of both input views can be improved, especially the \textbf{T} input view. CVA can improve the \textbf{T} input view to be competitive with \textbf{I+T} input view. Moreover, the combination of all the alignments \textbf{ITA-All$_{\text{+CVA}}$} can further improve the model accuracy in most of the cases. The accuracy of the MNER models can be significantly improved if we use XLMR embeddings, which shows the importance of the text modality in MNER. With XLMR embeddings, the model accuracy can be further improved with ITA. The relative improvements over the baseline models are sometimes higher with XLMR than with BERT, which shows that the visual contexts can be further utilized with stronger embeddings. 

In Table \ref{tab:previous_sota}, we compare \textbf{ITA} with previous state-of-the-art approaches. For previous approaches, we report the results including
\textbf{OCSGA}, \textbf{UMT}, \textbf{RIVA}, \textbf{RpBERT}, \textbf{UMGF}, which are the proposed approaches of \citet{10.1145/3394171.3413650}, \citet{yu-etal-2020-improving-multimodal}, \citet{sun-etal-2020-riva}, \citet{Sun2021RpBERTAT} and \citet{zhang2021multi} respectively. For fair comparison, we report the results of these models based on the BERT base embeddings. Moreover, since most of these previous approaches report the best model accuracy instead of the averaged model accuracy, we use the best model accuracy of \textbf{ITA-All$_{\text{+CVA}}$} over $5$ runs. We also report our reproduced results of \textbf{UMT}, {\textbf{RpBERT}} and \textbf{UMGF} on the corresponding datasets.
The results show that \textbf{ITA-All$_{\text{+CVA}}$} outperforms all of the previous approaches. On the SNAP dataset, the reported accuracy of {\textbf{RpBERT$_{\text{base}}$}} is competitive with \textbf{ITA-All$_{\text{+CVA}}$}. However, we find that the accuracy of our reproduced {\textbf{RpBERT$_{\text{base}}$}}\footnote{We reproduced the results based on the official code for {\textbf{RpBERT$_{\text{base}}$}}: \url{https://github.com/Multimodal-NER/RpBERT}} is significantly lower than the reported accuracy, even after careful check of the source code and hyper-parameter tuning.
Moreover, the fact that our \textbf{BERT-CRF} baseline achieves competitive accuracy with previous state-of-the-art multi-modal approaches shows that most of the previous work has not fully explored the strength of the text representations for the task. 



\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l||c|c}
\hlineB{4}
Approaches & \multicolumn{1}{c|}{Twitter-15} & \multicolumn{1}{c|}{Twitter-17} \\%& \multicolumn{1}{c}{SNAP} \\
\hline\hline
{\textbf{BERT-CRF$_{\text{UMT}}$}} & 71.81 &  83.44 \\% & - \\
{\textbf{BERT-CRF$_{\text{Ours}}$}} & 74.79 & 85.18 \\% & 85.98\\
\hline
\multicolumn{3}{c}{\bf \textsc{Our Reproductions}}\\
\hline
{\textbf{BERT-CRF$_{\text{UMT}}$}} & 71.74 & 84.20 \\%& -\\
{\textbf{BERT-CRF$_{\text{UMT-Improved}}$}} & 72.53 & 84.48 \\%& -\\
{\textbf{UMT}} & 72.83 & 84.88 \\%& -\\
{\textbf{UMT$_{\text{Improved}}$}}  & 72.96 & 84.50 \\%& -\\
\hlineB{4}
\end{tabular}
\caption{Our reproductions of previous baselines and approaches. ``Improved'' means our improved models based on the UMT code base.}
\label{tab:reproduction}
\end{table}

\paragraph{Discussion about Textual Modules}
As we have shown in Table \ref{tab:main} and \ref{tab:previous_sota}, the textual baselines (i.e. {\textbf{BERT-CRF}}) of previous work are significantly lower than that of ours. In most of the previous MNER architectures, the textual modules are mainly based on the baseline architectures with some modifications. We further show the baselines of previous work are not well-trained and how the multi-modal approaches perform with stronger textual modules. In Table \ref{tab:reproduction}, we rerun the \textbf{BERT-CRF} baseline based on the released codes of \textbf{UMT}\footnote{\url{https://github.com/jefferyYu/UMT}}. Based on the code of \textbf{UMT}, we tried to improve the baseline models in the code by using the same loss function as ours\footnote{The details are discussed in Appendix \ref{app:bug}}. The accuracy of \textbf{BERT-CRF} models in the code are significantly improved but the \textbf{UMT} models based on the improved code are not improved and even get worse in Twitter-17. Therefore, we suspect the \textbf{UMT} model cannot be further improved even with stronger textual modules. \citet{zhang2021multi} also reported the baseline based on the implementation of \citet{yu-etal-2020-improving-multimodal}, so we suspect the \textbf{UMGF} model cannot be improved as well. 
Therefore, the under-trained textual baselines of previous work make the effectiveness of the images unclear and we show that some of the MNER models perform even weaker than our \textbf{BERT-CRF} model.





\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{2pt}
\begin{tabular}{l||cc|cc|cc}
\hlineB{4}
\multicolumn{1}{c||}{} & \multicolumn{2}{c|}{Twitter-15} & \multicolumn{2}{c|}{Twitter-17}  & \multicolumn{2}{c}{SNAP} \\
 \hline
\multirow{3}{*}{Approach} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c}{\bf Eval}\\
 & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c}{\bf Modal}\\
 & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}}\\
\hline\hline
{\textbf{ITA-Random}} & - & 74.67 & - & 84.98 & - & 85.82 \\
 {\textbf{ITA-GA$_{\text{BU}}$}} & - & 75.10 & - & 85.77 & - & 86.51 \\
{\textbf{ITA-LA$_{\text{BU}}$}} & - & 75.18 & - & 85.59  & - &  86.57\\
{\textbf{ITA-OCA$_{\text{Paddle}}$}} & - & 75.12 & - & \textbf{85.87}  & - & 86.66 \\
 {\textbf{BERT-CRF$_{\text{+ImgFeat}}$}} & - & 74.70 & - & 84.99 & - & 85.90\\
 {\textbf{VinVL-CRF}} & - & 60.58 & - & 75.55 & - & 74.53\\
 {\textbf{BERT+VinVL-CRF}} & - & 74.89 & - & 85.19 & - & 86.14 \\
 {\textbf{ITA-Joint}} & 74.88 & 75.22 & 85.31 & 85.60& 86.06 & 86.34  \\
\hline
 \multicolumn{7}{c}{\bf \textsc{References}}\\
\hline
 {\textbf{RpBERT w/o Rp}} & - & 72.60 & - & - & - & 86.20\\
 {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{75.67} & \textbf{75.60} & \textbf{85.98} & 85.72 &  \textbf{86.83} & \textbf{86.75} \\
\hlineB{4}
\end{tabular}
\caption{A comparison of other variants of MNER models. }
\label{tab:comparison}
\end{table}

\subsection{Comparison with Other Variants}
To further show the effectiveness of ITA, we perform several comparisons between ITA and the following variants of the MNER model in Table \ref{tab:comparison}:
\paragraph{\textsc{\bf ITA-Random}:} We generate random image-text pairs for the model. For each sentence, we randomly select the image in the dataset and generate the corresponding visual contexts. The noises of random visual contexts make the model accuracy drop slightly comparing with our \textsc{\bf BERT-CRF} baseline, which shows the improvement of our approach is from the visual contexts rather than extending the input sequence length the embeddings.
\paragraph{\textsc{\bf ITA-Joint}:} It is an ablated model of \textsc{\bf ITA-All$_{\text{+CVA}}$}. We train the \textsc{\bf ITA-All} model for both input views without the CVA loss in Eq. \ref{eq:mda}. The model accuracy is improved moderately with only the \textbf{T} input view while our \textsc{\bf ITA-All$_{\text{+CVA}}$} can improve both input views significantly, which shows the effectiveness of the CVA module of ITA.
\paragraph{\textsc{\bf ITA-LA$_{\text{BU}}$} and \textsc{\bf ITA-GA$_{\text{BU}}$}:} We conduct experiments to see how the accuracy changes when using weaker image features. We use \textbf{B}ottom-\textbf{U}p features proposed by \citet{Anderson2017up-down} for object detection and image captioning. The captioning model is a pretrained image captioning model\footnote{\url{https://github.com/ruotianluo/self-critical.pytorch}} proposed by \citet{Luo2018DiscriminabilityOF} with the Bottom-Up features and self-critical training \citep{rennie2017self}. Results show that there is no significant difference between the visual contexts from Bottom-Up features and VinVL features. Therefore, our approaches can utilize other off-the-shelf vision models to extract visual contexts.
\paragraph{\textsc{\bf ITA-OCA$_{\text{Paddle}}$}:} We conduct experiments to see how the accuracy changes when using stronger OCR models. We use PaddleOCR\footnote{\url{https://github.com/PaddlePaddle/PaddleOCR}} for the experiment, which is one of the newest open resource lightweight OCR system. Results show that the model accuracy can be slightly improved comparing with \textsc{\bf ITA-OCA}, which shows the ITA models can be improved by using better OCR models.
\paragraph{{\textbf{BERT-CRF$_{\text{+ImgFeat}}$}}:} Instead of \textbf{ITA}, we can directly feed the image region features generated from an object detector into the BERT. We use ResNet-152 model to generate region features and then feed the features into a linear layer to project the region features into the same space of text features in the BERT. Moreover, we compare the model with {\textbf{RpBERT w/o Rp}}, which is an ablated model of {\textbf{RpBERT}} and is equivalent to {\textbf{BERT-CRF+$_{\text{+ImgFeat}}$}} over the usage of BERT embeddings. \citet{Sun2021RpBERTAT} showed {\textbf{RpBERT w/o Rp}} can improve the model accuracy compared with their baseline. However, our results show that the model accuracy slightly drops comparing with our {\textbf{BERT-CRF}}, which shows that it is difficult for the attention module of BERT to learn the relations of the unaligned representations of two modalities. 
\paragraph{\textsc{\bf VinVL-CRF}:} To show how the pretrained V+L models perform on the NER task, we use VinVL since it is a very recent state-of-the-art pretrained V+L model on a lot of multi-modal tasks. We feed the VinVL model with texts and images in the MNER datasets and finetune the model over the task. We take the text representations output from VinVL as the input of the CRF layer. The accuracy of the finetuned VinVL model drops significantly compared to the BERT model, which shows that the inductive bias of the pretrained V+L model hurts the model accuracy on MNER.
\paragraph{\textsc{\bf BERT+VinVL-CRF}:} As the VinVL model may lead to an inductive bias over the common nouns and the image, we jointly finetune the BERT and VinVL models and concatenate the output text representations of the two models. The accuracy is improved on a moderate scale, which shows BERT is complementary to VinVL for MNER.




\subsection{Analysis}


\begin{filecontents}{twitter15.dat}
beam	acc
0	74.79
1	75.07	
2	75.17
3	75.14
4	74.97	
5	74.95	
\end{filecontents}

\begin{filecontents}{twitter17.dat}
beam acc
0	85.18
1	85.61
2	85.75
3	85.58
4	85.41
5	85.27
\end{filecontents}

\begin{filecontents}{snap.dat}
beam acc
0   85.98
1	86.3
2	86.41
3	86.35
4	86.39
5	86.45

\end{filecontents}

\begin{figure}[t!]
\begin{minipage}{1.0\linewidth}
\centering
\begin{tikzpicture}
   
   
    \begin{axis}[
        xshift=-10cm,
       
        name=ner,
       
        width=0.42\textwidth,
        height=0.35\textwidth,
       
       
       
        legend columns=2, 
       
        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},
        xtick={0,1,2,3,4,5},
        ylabel style={font=\tiny,yshift=-0.2cm},
        ]
        \addplot[blue!80,mark=*] table[x=beam,y=acc] {twitter15.dat};
        \legend{Twitter-15}
    \end{axis}
    \begin{axis}[
       
        at={(ner.south west)},
        xshift=2.6cm,
       
        width=0.42\textwidth,
        height=0.35\textwidth,
       
       
       
        legend columns=2, 
       
        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},
       
        xtick={0,1,2,3,4,5},
       
        ylabel style={font=\tiny,yshift=-0.5cm},
        ]
        \addplot[red!80!yellow!45,mark=*] table[x=beam,y=acc] {twitter17.dat};
        \legend{Twitter-17}
    \end{axis}
    \begin{axis}[
       
        at={(ner.south west)},
        xshift=5.2cm,
       
        width=0.42\textwidth,
        height=0.35\textwidth,
       
       
       
        legend columns=2, 
       
        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},
       
        xtick={0,1,2,3,4,5},
       
        ylabel style={font=\tiny,yshift=-0.5cm},
        ymax=86.7,
        ymin=85.9,
        ]
        \addplot[yellow!80!blue!65,mark=*] table[x=beam,y=acc] {snap.dat};
        \legend{SNAP}
    \end{axis}
\end{tikzpicture}
\caption{A relation between the number of captions input to the MNER model and model accuracy. The x-axis is the number of captions. The y-axis is the averaged F1 score on the test set.}
\label{fig:test_curve}
\end{minipage}
\end{figure}


\paragraph{Effect of the Number of Captions}
Using more captions output from the captioning model can improve diversities of the visual contexts but can add noises to them as well. To better understand how the number of captions affects the model accuracy, we change the beam size and keep all the sentences output from the captioning model. The trends in Figure \ref{fig:test_curve} show that the model accuracy increases until $5$ captions for all the datasets and gradually drops when the number of captions further increases for Twitter-15 and 17 datasets. The observation shows that using $5$ captions keeps a good balance between the diversities and correctness of the captions.


\begin{filecontents}{mask_embed.dat}
type base ours ours-mda base_std ours_std ours-mda_std
Twitter-15 0.1848 0.3591 0.1581 0.0844 0.0324 0.0286
Twitter-17 0.3312 0.4725 0.1755 0.0879 0.0354 0.0112
SNAP 0.40 0.83 0.32 0.18 0.10 0.009
\end{filecontents}


\begin{figure}[t]
\centering
\begin{tikzpicture}

\begin{axis}[
    ybar=0pt,
   
   
    bar width = {1em},
    width=0.49\textwidth,
    height=0.18\textwidth,
    enlarge x limits={abs=1cm},
    symbolic x coords={Twitter-15,Twitter-17, SNAP},
   
    xticklabel style={
       
       
       
       
         align=center,
         font=\scriptsize,
      },
    yticklabel style={font=\small},
    ylabel = {L2 Distance},
    ylabel style = {yshift=-0.3cm,font=\small},
    legend style={font=\tiny,at={(0.5,-0.83)},anchor=south},
   
    legend columns=3, 
    xtick={Twitter-15,Twitter-17, SNAP},
    legend image code/.code={%
      \draw[#1] (0cm,-0.1cm) rectangle (0.15cm,0.2cm);
    },
    ytick={0,0.5,1.0},
    ymin=0,
    ymax=1.0,
    ]
    \addplot[ybar, fill=blue!20, error bars/.cd,y dir=both,y explicit] table [x=type, y=base,y error=base_std] {mask_embed.dat};
    \addplot[ybar, fill=red!60!yellow!25, error bars/.cd,y dir=both,y explicit] table [x=type, y=ours,y error=ours_std] {mask_embed.dat};
    \addplot[ybar, fill=yellow!25, error bars/.cd,y dir=both,y explicit] table [x=type, y=ours-mda,y error=ours-mda_std] {mask_embed.dat};
    \legend{{\textbf{ BERT-CRF+ImgFeat}}, {\textbf{ITA-All}}, {\textbf{ITA-All$_{\text{+CVA}}$}}};
\end{axis}
\end{tikzpicture}
\caption{Averaged L2 distance between the token representations without image input (${\bm{r}}_i$) and with image input (${\bm{r}}^\prime_i$). The error bars mean the standard deviation over $5$ runs.}
\label{fig:l2dist}
\end{figure}

\paragraph{How ITA Eases the Cross-Modal Alignments}
Previous work such as \citet{moon-etal-2018-multimodal,Sun2021RpBERTAT} visualized modality attention in several cases to show the effectiveness of their approaches. However, visualizing the multi-layer attention in transformer-based embeddings is relatively difficult. Instead of studying special cases, we statistically calculate the averaged L2 distance between token representations ${\bm{r}}_i$ and ${\bm{r}}^{\prime}_i$ from two input modalities to show how the token representations depend on image information. In Figure \ref{fig:l2dist}, the L2 distance \textbf{ITA-All} is significantly larger than that of \textbf{BERT-CRF+ImgFeat}. Besides, the standard deviation of \textbf{BERT-CRF+ImgFeat} is very large. The observations show the image region features make the alignment become difficult and unstable while our visual contexts can significantly ease the cross-modal alignments. Moreover, with CVA, the L2 distance becomes much smaller and stable as CVA aligns the two input views to reduce the dependence on images, which shows the MNER model can better utilize the textual information with CVA.



\begin{table*}[t!]
\small
\centering
\begin{tabular}{l|ccc|ccc|ccc|ccc}
\hlineB{4}
 & \multicolumn{3}{c|}{LOC} & \multicolumn{3}{c|}{ORG} & \multicolumn{3}{c|}{PER} & \multicolumn{3}{c}{OTHER}\\% & \multicolumn{3}{c|}{LOC} \\% & \multicolumn{3}{c|}{ORG} & \multicolumn{3}{c|}{PER} & \multicolumn{3}{c}{OTHER}\\
 & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1\\
\hline\hline
 & \multicolumn{12}{c}{Twitter-15}\\
 \hline
{\textbf{BERT-CRF}} & 80.0 & 83.8 & 81.8 & 65.9 & 61.0 & 63.3 & 84.2 & 86.8 & 85.4 & 44.2 & 44.2 & 44.1\\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 81.1 & 84.2 & 82.6 & 68.8 & 60.6 & 64.4 & 84.0 & 87.2 & 85.6 & 44.9 & 44.6 & 44.8 \\
$\mathbf{\Delta}$ & 1.1  & 0.4  & 0.8  & 2.8  & -0.4 & 1.1  & -0.2 & 0.4  & 0.1  & 0.8  & 0.5  & 0.6 \\
\hline\hline
 & \multicolumn{12}{c}{Twitter-17} \\
\hline
{\textbf{BERT-CRF}} & 85.5 & 84.4 & 84.9 & 83.5 & 83.8 & 83.7 & 90.7 & 90.8 & 90.7 & 68.9 & 65.1 & 66.9 \\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 86.0 & 83.7 & 84.8 & 83.9 & 84.2 & 84.0 & 91.9 & 90.9 & 91.4 & 73.7 & 64.3 & 68.6 \\
$\mathbf{\Delta}$ & 0.5  & -0.7 & -0.1 & 0.3  & 0.4  & 0.4  & 1.2  & 0.1  & 0.7  & 4.8  & -0.8 & 1.7 \\
\hline\hline
 & \multicolumn{12}{c}{SNAP} \\
\hline
{\textbf{BERT-CRF}} & 82.1 &82.8 &82.5 &87.8 &86.9 &87.3 &91.0 &91.5 &91.2 &72.3 &75.1 &73.7 \\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 80.3 &81.7 &81.0 &87.8 &86.5 &87.1 &90.1 &91.2 &90.6 &70.1 &73.2 &71.6\\
$\mathbf{\Delta}$ & 1.9 &1.1 &1.5 &0.6 &0.5 &0.5 &0.9 &0.3 &0.6 &2.2 &1.9 &2.1 \\
\hlineB{4}
\end{tabular}
\caption{A comparison between our ITA ({\textbf{ITA-All$_{\text{+CVA}}$}} with \textbf{I+T} inputs) model and the baseline ({\textbf{BERT-CRF}}) in precision (P), recall (R) and F1. $\mathbf{\Delta}$ represents the relevant improvement of ITA over the Baseline.}
\label{tab:prf}
\end{table*}




\paragraph{How Images Affect the NER Prediction}
To study the effectiveness of the images over each label, we show a comparison between our model and our baselines in Table \ref{tab:prf}. When the relative improvement of the F1 score is larger than $0.5$, the relative improvement of precision is larger than that of recall. The observation shows that the main improvement of MNER is mainly because the images can help the model to reduce false-positive predictions for disambiguation on uncertain entities.\footnote{In Appendix \ref{app:case}, we show several cases to show the effectiveness of ITA to affect NER prediction.}








\section{Related Work}


\paragraph{Multi-modal Named Entity Recognition} 
Most of the previous approaches to MNER focus on the interaction between image and text features through attention mechanisms. \citet{moon-etal-2018-multimodal} proposed a modality attention network to fuse the text and image features before the input to the BiLSTM layer. \citet{lu-etal-2018-visual} additionally used a visual attention gate for the output features of the BiLSTM layer. \citet{zhang2018adaptive} proposed an adaptive co-attention network after the BiLSTM layer to model the interaction between image and text. Recently, \citet{10.1145/3394171.3413650} proposed OCSGA, which use object labels to model the interaction between image and object labels in an additional dense co-attention layer.
Compared with the work, we show a simpler and more effective way to utilize object labels and additionally use other alignment approaches to further improve the model accuracy. \citet{yu-etal-2020-improving-multimodal} proposed UMT, which utilized a multi-modal interaction module and an auxiliary entity span detection module for MNER. \citet{zhang2021multi} proposed UMGF, which utilizes a pretrained parser to create the graph connection between visual object tags and textual words. They used a graph attention network to fuse the textual and visual features. In order to better model whether the image is related to the text, \citet{Sun2021RpBERTAT} proposed RpBERT, which additionally trains on a text-image relation classification dataset proposed by \citet{vempala-preotiuc-pietro-2019-categorizing} to prevent the negative effect of noisy images. Comparing with RpBERT, we use CVA to let the NER model better utilize the input sentences without such kinds of supervision. All of these approaches focus on fusing the image and text features through the attention mechanism but ignore the gap between the image and text features while we propose to fully utilize the attention mechanism in the pretrained textual embeddings through aligning image features into textual space. Besides, some cross-media research also shows the effectiveness of OCR texts \citep{chen2016context,wang-etal-2020-cross-media} and object tags \citep{7780398} have been shown. Most of the approaches introduced a new attention module over cross-modal features while in comparison ITA effectively utilizes the attention module in the pretrained textual embeddings.



\paragraph{Pretrained Vision-Language Models}
Inspired by related work on language model pretraining, visual-language pretraining (VLP) has recently attracted a lot of attention \citep{li2019visualbert,Lu2019ViLBERTPT,chen2020uniter,Tan2019LXMERTLC,Li2020OscarOA,Yu2021ERNIEViLKE,zhang2021vinvl}. The pretrained V+L models are pretrained on large-scale image-text pairs and have achieved state-of-the-art accuracy over various vision-language tasks such as image captioning, VQA, NLVR and image-text retrieval. 
Recently, \citet{Li2020OscarOA} proposed Oscar to add object tags in pretraining so that self-attention can learn the image-text alignments easily. Following Oscar, \citet{zhang2021vinvl} proposed VinVL to train a large-scale object detector to improve the pretrained V+L model's accuracy. Comparing with VLP, MNER is a totally different task.
Firstly, the image-caption pairs are given in VLP and the image and text are equally important in pretraining for general representations. Therefore, using global alignment is meaningless for VLP but makes sense for MNER. In MNER, the input text is not the caption of the image and the image may not adds additional information to the input text.
Secondly, though captions and object tags are often utilized in VLP, how to effectively utilize the captions and object tags of the image in MNER is rarely considered. Finally, besides the local and global alignments, another aspect of ITA is the optical character alignment and cross-view alignment, which is rarely considered in VLP.

\section{Conclusion}
In this paper, we propose Image-Text Alignments for multi-modal named entity recognition, which convert images into object labels, captions and OCR texts to align the image representations into textual space in a multi-level manner and form a cross-modal input view. The model can effectively utilize attention module of the transformer-based embeddings. Considering noises, availability of images and inference speed for practical use, we propose cross-view alignment, which let the MNER models better utilize the text information in the input. In our experiments, we show that ITA significantly outperforms previous state-of-the-art approaches on MNER datasets. We also show that most of the previous work failed to train a good textual baseline while our textual baseline can easily match or even outperform previous multi-modal approaches. In analysis, we further analyze how ITA eases the cross-modal alignments and how the images affect the NER prediction.

\section*{Acknowledgements}
This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.


