\section{Introduction}

\IEEEPARstart{I}{mage} captioning is one of the important tasks in the intersection of vision and language, which aims to automatically describe given images with natural language captions. Some recent studies have achieved superior performance and are even comparable to humans under certain circumstances \cite{anderson2018bottom,zhang2018high,yu2021dual}. However, these traditional image captioning models perform unsatisfactorily when describing scenes containing text. As shown in Case (a) (cf. \cref{fig:intro}), ``a bottle of Coca-Cola'' is obviously specific than ``a bottle''. To solve this defect, Sidorov \textit{et al.} \cite{sidorov2020textcaps} propose text-based image captioning and collect a high-quality dataset TextCaps, in which captions contain scene text from the~images.
\begin{figure}[t]
  \centering
 
   \includegraphics[width=1.0\linewidth]{./Intro5.pdf}
   \caption{Our DEVICE significantly improves the quality of captions. In Case a, with depth information and a depth-enhanced feature updating module, DEVICE correctly models 3D relationships between scene texts. With a salient visual object concepts extractor and a semantic-guided alignment module, DEVICE generates more accurate and comprehensive captions, cf. Case b. Scene text is represented in \textcolor[rgb]{0,0.69,0.23}{Green}, and objects are represented in \textcolor[rgb]{0,0.76,0.98}{Blue}.}
   \label{fig:intro}
\end{figure}

To utilize the scene text in pictures, extracting Optical Character Recognition (OCR) tokens from given images is the precondition. M4C-Captioner\cite{sidorov2020textcaps} utilizes Rosetta\cite{borisyuk2018rosetta} OCR system to obtain OCR tokens, and then employ a multimodal transformer directly to model the relationships between visual objects and OCR tokens. However, it is inappropriate to treat all OCR tokens equally, because only part of OCR tokens are crucial to the understanding of the scene. Wang \textit{et al.} \cite{wang2021confidence} creatively introduce a concept of OCR confidence to choose the noteworthy OCR tokens, which has greatly alleviated this issue. Then Wang \textit{et al.}\cite{wang2020multimodal, wang2021improving} are committed to improving the spatial relationship construct methods for OCR tokens, which have been proven effective, and the proposed model LSTM-R\cite{wang2021improving} achieves the state-of-the-art performance. However, current methods are not accurate and comprehensive enough when describing the scenes, which can be mainly manifested in the following two-fold.

Firstly, the real-world is three-dimensional (3D). However, current studies\cite{sidorov2020textcaps,wang2021confidence,wang2020multimodal,wang2021improving, yang2021tap} only exploit the spatial relationships of OCR tokens at the plane-level, which leads to visual location information being utilized incompletely, and even results in generating inaccurate scene text. As illustrated in the Case (a) of \cref{fig:intro}, CNMT\cite{wang2021confidence} mistakenly regards ``victorian'' and ``coca'' as closely related OCR tokens in the same plane, which are irrelevant and actually one behind the other. Naturally, we consider that introducing depth information based on two-dimensional (2D) information can simulate real-world geometric information for improving the accuracy of scene text in captions.

Secondly, scenes contain abundant but complex visual entities (visual objects and scene text), existing methods suffer from coarse-grained descriptions of visual objects and the absence of crucial visual entities in captions. As shown in the Case (b) on the right of \cref{fig:intro}, CNMT\cite{wang2021confidence} generates the coarse-grained word ``person'' for the visual object ``athlete". Besides, CNMT ignores an important entity ``bike'', causing the OCR token ``spinner'' on it cannot be utilized effectively. One possible reason for the absence of ``bike'' is that the visual features of mutilated objects are not conducive to modeling. Intuitively, we consider that the semantic information of salient visual object concepts can alleviate coarse-grained and partial captioning.

To tackle the aforementioned issues, we propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps. It mainly comprises a reading module, a depth-enhanced feature updating module, a salient visual object concepts extractor, a semantic-guided alignment module, and a multimodal transformer. Concretely, in the reading module, we employ a monocular depth estimator BTS\cite{lee2019big} to generate depth maps, then we extract depth values for visual objects and OCR tokens, respectively. The depth-enhanced feature updating module is designed to optimize the visual features of OCR tokens, which is beneficial for 3D geometry relationship construction. In the salient visual object concepts semantic extractor, we first employ a pre-trained visual-language model CLIP\cite{radford2021learning} as a filter to generate $K$ concepts most related to the given picture. Then we adopt the semantic information of visual object concepts as augmentations to improve the overall expressiveness of our model. To utilize visual concepts more efficiently, we introduce the Semantic-guided Alignment Module to interact aligned visual concepts with corresponding scene text. Subsequent experiments in \cref{sec4} demonstrate the effectiveness of our proposed DEVICE.

Our main contributions can be summarized as follows:
\begin{itemize}
\item We propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps. DEVICE performs better than previous methods as it improves the accuracy and completeness of captions.
\item We devise a Depth-enhanced Feature Updating Module (DeFUM) to enhance the 3D spatial relationship between OCR tokens, which effectively helps the model evaluate the relevance of OCR tokens. To our knowledge, we are the first to apply depth information to the text-based image captioning task.
\item We introduce semantic information of salient visual objects by the Visual Object Concepts Extractor (VOC), and we construct a Semantic-guided Alignment Module (SgAM) to improve the availability of visual concepts. Our DEVICE is capable of generating fine-grained objects in captions and better understanding holistic scenes.
\item Our DEVICE outperforms the state-of-the-art models on the TextCaps test set, inspiringly boosting CIDEr-D from 100.8 to 110.0.
\end{itemize}
\section{Related Work}
\subsection{Text-based Image Captioning}
Image captioning task has made a lot of progress in recent years\cite{vinyals2015show,anderson2018bottom,huang2019attention,huang2021image,cornia2020meshed}, but most of them perform poorly facing images with scene text. Text-based image captioning was naturally born to alleviate this problem, which was proposed by Sidorov \textit{et al.}\cite{sidorov2020textcaps}. The authors introduced the Textcaps dataset and a multimodal Transformer-based\cite{vaswani2017attention} baseline model M4C-Captioner. The M4C-Captioner was modified from a Text-VQA\cite{singh2019towards,biten2022latr,jin2022token,zeng2021beyond} model M4C\cite{hu2020iterative}, and can encode both image and OCR tokens to generate more property captions. However,  this paradigm suffers from a lack of diversity and information integrity. Zhang \textit{et al.}\cite{zhang2022magic} and Xu \textit{et al.}\cite{xu2021towards} were committed to solving this problem by introducing additional captions. Although introducing multiple captions alleviates the problem of missing scene information, it is inherently hard to summarize the complex entities of a scene within one caption well. Apart from this, M4C-Captioner is less than satisfactory performed on Textcaps due to insufficient use of visual and semantic information of images. To distinguish misidentified OCR recognition results and pick out the crucial OCR tokens, Wang \textit{et al.}\cite{wang2021confidence} introduced confidence embedding and alleviated the adverse effects of inaccurate OCR tokens to a certain extent. Yang \textit{et al.}\cite{yang2021tap} designed a pre-training model that can exploit OCR tokens and achieved strong results on multiple tasks, including text-based image captioning. Wang \textit{et al.}\cite{wang2020multimodal,wang2021improving} and Tang \textit{et al.}\cite{tang2022ocr} employ the 2D spatial relationships to measure correlations between OCR tokens. Nevertheless, we find that simply leveraging two-dimensional spatial relationships may cause incorrect captions. Therefore, we ameliorate this problem by utilizing depth information.

\subsection{Depth Information For Multimodal Tasks}
Monocular depth estimation is a challenging problem of computer vision, whose purpose is to use depth values to measure the distance of objects relative to the camera\cite{couprie2013indoor,geiger2012we,lee2019big}. With depth estimation technology, significant improvements have been achieved on many multimodal tasks\cite{banerjee2021weakly, wang2018look, liao2021scene}. In the visual question answer (VQA) task, Banerjee \textit{et al.}\cite{banerjee2021weakly} and Liu \textit{et al.}\cite{9830073} proved that exploiting three-dimensional relationships by depth information could enhance the spatial reasoning ability of VQA models. Apart from this, Qiu \textit{et al.}\cite{qiu20203d} and Liao \textit{et al.}\cite{liao2021scene} point out that ignoring the depth information could tremendously place restrictions on applications of scene change captioning in the real world. At the same time, we notice that text-based image captioning is facing the same problem. Therefore, we improve the 3D relationship modeling method to apply in text-based image captioning effectively.

\begin{figure*}[ht]
  \centering
 
   \includegraphics[width=0.96\linewidth]{./Model3.pdf}
   \caption{Overview of our DEpth and VIsual ConcEpts Aware Transformer for TextCaps (DEVICE). In the reading module, we first extract visual objects and OCR tokens along with their depth values. Then a depth-enhanced feature updating module is constructed to enhance OCR appearance features with the help of depth values. After that, DEVICE utilizes depth information and plane geometry relationships to construct 3D positional relationships. Semantic information of salient visual object concepts promotes fine-grained and holistic captioning, and the proposed semantic-guided alignment module improves the model's efficiency to utilize visual concepts. Words in \textcolor[rgb]{0,0.76,0.98}{Blue}, \textcolor[rgb]{0,0.75,0.23}{Green} and \textcolor[rgb]{0.95,0.65,0}{Orange} represent objects, scene text, and visual concepts, respectively.}
   \label{fig:model}
\end{figure*}

\section{Approach}

\subsection{Overview}

Given an image \textit{I}, text-based image captioning models aim to automatically generate caption $C=[y_0, y_1, ... , y_n]$ based on scene text \textit{S} in the image.  

As depicted in \cref{fig:model}, our DEVICE mainly consists of a reading module, a depth-enhanced feature updating module, a salient visual object concepts extractor, a semantic-guided alignment module, a multimodal transformer, and a sentence decoder. Concretely, the input image is first put into an object detector and two OCR detectors to extract \textit{N} visual objects and \textit{M} OCR tokens along with their 2D regional coordinates. Then we utilize a pre-trained depth estimation model to generate the pixel-level depth map \textit{D} and construct 3D coordinates of visual entities. After that, we update the visual features of OCR tokens with the depth-enhanced feature updating module. Depth information is included in the coordinates of both scene text and objects, hence the multimodal transformer could establish 3D relations between visual entities. Meanwhile, to generate salient visual object concepts for each image, we employ a pre-trained cross-modal tool CLIP\cite{radford2021learning} to match the top-\textit{K} related visual concepts (e.g., ``Van'' and ``Vehicle'') in vocabulary. Then we propose the semantic-guided alignment module to integrate the information of aligned visual concepts into the representation of relevant OCR tokens. Ultimately, the multimodal transformer and sentence decoder generate captions in the auto-regressive paradigm.
\subsection{Multimodal Embedding}
First of all, we exploit pre-trained Faster R-CNN\cite{ren2015faster} model to get the bounding box coordinates $\{b^{obj}_n\}_{n=1:N}$ of objects $\{a^{obj}_n\}_{n=1:N}$. To ensure high quality OCR tokens, following LSTM-R\cite{wang2021improving}, we apply external OCR systems\cite{borisyuk2018rosetta,Googleocr} to obtain both OCR tokens $\{a^{ocr}_m\}_{m=1:M}$ and corresponding bounding box coordinates $\{b^{ocr}_m\}_{m=1:M}$, respectively. After that, we extract the $d$-dim appearance features $\{x^{of}_n\}_{n=1:N}$ of objects and appearance features $\{x^{tf}_m\}_{m=1:M}$ of OCR tokens by Faster R-CNN\cite{ren2015faster}. To generate depth maps of images in the TextCaps dataset, we utilize a monocular depth estimation model BTS\cite{lee2019big}.

\textbf{Depth values.}
To measure the relative distance of objects and OCR tokens to the observer, we employ a pre-trained depth estimation model BTS\cite{lee2019big} to generate depth maps. Subsequently, we unify the values of the depth maps to the range of 0 to 255. For object $a^{obj}_n$ and OCR token $a^{ocr}_m$, we count the grayscale distribution of bounding box region of $a^{obj}_n$ and $a^{ocr}_m$ in the depth map at pixel scale. We take the gray values with the highest frequency as the depth values for object $a^{obj}_n$ and OCR token $a^{ocr}_m$ (cf. \cref{fig:depth}). We denote $\{dv^{obj}_n\}_{n=1:N}$ as the depth values for $\{a^{obj}_n\}_{n=1:N}$ and $\{dv^{ocr}_m\}_{m=1:M}$ for $\{a^{ocr}_m\}_{m=1:M}$, respectively. 

\textbf{Embedding of objects.}
To measure the relationship between visual entities more effectively, we incorporate depth information into object features. For the object $a^{obj}_n$, we denote its 2D geometry coordinate by $b^{obj}_n = [x^{tl}/W; y^{tl}/H; x^{br}/W; y^{br}/H]$, where $x^{tl}$ and $y^{tl}$ represent the coordinates of the upper left corner of object $a^{obj}_n$, $x^{br}$ and $y^{br}$ represent the coordinates of the bottom right corner of object $a^{obj}_n$, respectively. Then we denote the 3D spatial location feature of $a^{obj}_n$ by $x^{os}_n = [b^{obj}_n; dv^{obj}_n/255]$. The embedding of object $a^{obj}_n$ is calculated as
\begin{equation}
    x^{obj}_n = LN(W_{of}x^{of}_n) + LN(W_{s}x^{os}_n),
\end{equation}
where $W_{fr} \in \mathbb{R}^{t\times{d}}$ and $W_{s} \in \mathbb{R}^{t\times{5}}$, $t$ represents the~dimension of common space in multimodal transformer, and $LN$ represents layer normalization.

\begin{figure}[t]
  \centering
 
   \includegraphics[width=0.9\linewidth]{./depth_value.pdf}
   \caption{Illustrations for the process of getting the depth values and 3D geometric relationship modeling. The pie chart represents the gray value distribution of object $a^{obj}_n$. We take the gray values with the highest proportion as the depth value (0 means nearest) of object $a^{obj}_n$ and OCR token $a^{ocr}_m$. $b^{obj}_n$ and $b^{ocr}_m$ denote bounding boxes of $a^{obj}_n$ and $a^{ocr}_m$, respectively. With depth information, relationships between visual entities become more clear. }
   \label{fig:depth}
\end{figure}

\textbf{Embedding of OCR tokens.}
Unlike objects, OCR tokens not only contain rich visual clues but also contain rich semantic information. To get rich representations of OCR tokens, following Hu \textit{et al.}\cite{hu2020iterative} and Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we apply FastText\cite{bojanowski2017enriching} to extract the 300-dim sub-word feature $x^{ft}_m$ for OCR token $a^{ocr}_m$, and employ PHOC (Pyramidal Histogram of Characters)\cite{almazan2014word} to extract 604-dim character-level feature $x^{ph}_m$ for $a^{ocr}_m$. Similarly, the spatial location of $a^{ocr}_m$ is denoted as $x^{ts}_m = [b^{ocr}_m; dv^{ocr}_m/255]$, where $b^{ocr}_m = [x^{tl}/W; y^{tl}/H; x^{br}/W; y^{br}/H]$. To better fuse the three-dimensional relative spatial relationships into OCR features, we propose a Depth-enhanced Feature Updating Module (DeFUM, cf. \cref{subsec c}). In the real world, scene texts are attached to the surface of objects. We consider that visual features of objects can help model the visual relationship of OCR tokens. For example, if several different OCR tokens are on the same object, then these OCR tokens are likely to have a strong correlation. Therefore we fed the OCR token appearance features $x^{tf}_{1:M}$ into DeFUM along with object appearance features $x^{of}_{1:N}$, we can get the depth-enhanced OCR appearance feature $x^{tf'}_{1:M}$: 
\begin{equation}
    x^{tf'}_{1:M} = DeFUM(x^{of}_{1:N},x^{tf}_{1:M},dv^{obj}_{1:N},dv^{ocr}_{1:M}),
\end{equation}
where $dv^{obj}_{1:N} \in \mathbb{R}^{N\times{1}}$ and $dv^{ocr}_{1:M} \in \mathbb{R}^{M\times{1}}$, respectively.
Considering the gap between visual features and semantic features, we do not update FastText and PHOC features (semantic embeddings).
Then we utilize the Semantic-guided Alignment Module (SgAM, cf. \cref{subsec e}) to integrate aligned visual concept information into the semantic representation of relevant OCR tokens
\begin{equation}
    x^{ft'}_{1:M} = SgAM(FT(a^{voc}_{1:K}),x^{ft}_{1:M}),
\end{equation}
where $x^{ft'}_{1:M}$ and $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, $FT(a^{voc}_k)$ represent the FastText embedding of visual concept $a^{voc}_k$ (cf. \cref{subsec d}).

Following Wang \textit{et al.}\cite{wang2021confidence}, we adopt confidence features of detected OCR tokens $\{x^{conf}_m\}_{m=1:M}$ due to some OCR tokens may be amiss. The final embedding of OCR token $a^{ocr}_m$ could be calculated by
\begin{equation}
\begin{aligned}
    x^{ocr}_m = LN(W_{tf}x^{tf'}_m + W_{ft}x^{ft'}_m + W_{ph}x^{ph}_m)\\ + LN(W_{s}x^{ts}_m) + LN(W_{conf}x^{conf}_m),
\end{aligned}
\end{equation}
where $W_{tf} \in \mathbb{R}^{t\times{d}}$, $W_{ft} \in \mathbb{R}^{t\times{300}}$,  $W_{ph} \in \mathbb{R}^{t\times{604}}$, $W_{s} \in \mathbb{R}^{t\times{5}}$ and $W_{conf} \in \mathbb{R}^{t\times{1}}$ are learnable parameters, and $LN$ represents layer normalization.

\begin{figure}[t]
  \centering
 
   \includegraphics[width=1.0\linewidth]{./dem.pdf}
   \caption{Illustration of the Depth-enhanced Feature Updating Module (DeFUM), which consists of a depth-aware self-attention module and $N$ layers of transformer encoder. The DeFUM could effectively enhance the appearance features of OCR tokens with the help of depth information, which is conducive to constructing the three-dimensional geometric relationships and improving the accuracy of captions.}
   \label{fig:dem}
\end{figure}
\subsection{Depth-enhanced Feature Updating Module}
\label{subsec c}
Unlike other multimodal tasks, such as VQA, which require high-quality relationships between objects, text-based image captioning requires models with stronger scene text relationship modeling capabilities. Therefore the quality of the OCR token features is more critical to the generated captions. For this reason, we propose the DeFUM to enhance the OCR appearance features $\{x^{tf}_m\}_{m=1:M}$ with the help of depth information. Intuitively, we consider that object features can play the role of a bridge linking its adjacent scene text. Therefore, we first obtain appearance embedding $x^v$ of visual entities by concatenating the visual features $\{x^{of}_n\}_{n=1:N}$ of objects $\{a^{obj}_n\}_{n=1:N}$ and the visual features $\{x^{tf}_m\}_{m=1:M}$ of OCR tokens $\{a^{ocr}_m\}_{m=1:M}$: 
\begin{equation}
    x^v = Concat(x^{of}_{1:N}, x^{tf}_{1:M}),
\end{equation}
where $x^v \in \mathbb{R}^{(m+n)\times{d}}$. Subsequently, we calculate the query $Q$, key $K$, and value $V$ as follows:
\begin{equation}
    \left\{ 
    \begin{aligned}
    Q &= x^{v}W_Q \\
    K &= x^{v}W_K,\\
    V &= x^{v}W_V \\
    \end{aligned} 
    \right.
\end{equation}
where query $Q \in \mathbb{R}^{(m+n)\times{d}}$, key $K \in \mathbb{R}^{(m+n)\times{d}}$, value $V \in \mathbb{R}^{(m+n)\times{d}}$. Then, we can obtain the attention weights matrix $A$ by: 
\begin{equation}
    A = \frac{QK^T}{\sqrt{d}},
\end{equation}
where $K \in \mathbb{R}^{(m+n)\times{(m+n)}}$. To measure the relative depth between different visual entities, we calculate the relative depth score of $x^v_i$ and $x^v_j$ by
\begin{equation}
    R_{i,j} = log(\frac{dv_j}{dv_i}),
\end{equation}
where $dv = Concat(dv^{obj},dv^{ocr})$ and the relative depth map $R\in \mathbb{R}^{(m+n)\times{(m+n)}}$. Then the output of Depth-aware Self-Attention module (cf. \cref{fig:dem}) is updated by 
\begin{equation}
    x^{vti} = LN(x^v + softmax(A + R)V),
\end{equation}
where the depth-aware visual feature $x^{vti} \in \mathbb{R}^{(m+n)\times{d}}$, $LN$ denotes layer normalization. Eventually, $x^{vti}$ is fed into $N$ layers of transformer\cite{vaswani2017attention} to update appearance features with depth information 
\begin{equation}
    x^{vto} = transformer(x^{vti}),
\end{equation}
where the depth-enhanced appearance feature $x^{vto}\in \mathbb{R}^{(m+n)\times{d}}$ ($x^{vto} = [x^{of'}, x^{tf'}]$), $x^{of'}\in\mathbb{R}^{n\times{d}}$, $x^{tf'}\in \mathbb{R}^{m\times{d}}$, respectively.

\subsection{Salient Visual Object Concepts Extractor}
\label{subsec d}
To improve the modeling ability of the multimodal transformer to the scene objects, avoid the lack of vital objects, and generate more accurate captions, we propose a visual object concepts extractor to introduce semantic information of visual object concepts. Specifically, we employ a pre-trained model CLIP\cite{radford2021learning} to match $15$ object concepts related to the given image from vocabulary. Then we sort them by similarity score and take the top-K as visual object concepts. Visual object concepts and similarity score are represented as $\{a^{voc}_k\}_{k=1:K}$ and $\{a^{score}_k\}_{k=1:K}$. The embedding of visual object concepts are calculated as follows:
\begin{equation}
     x^{voc}_k = LN(W_{voc}FT(a^{voc}_k))\\ + LN(W_{score}a^{score}_k),
\end{equation}
where $FT(a^{voc}_k)$ means the FastText\cite{bojanowski2017enriching} embedding of $a^{voc}_k$, $W_{voc} \in \mathbb{R}^{t\times{300}}$ and $W_{score} \in \mathbb{R}^{t\times{1}}$.

\begin{figure}[t]
  \centering
 
   \includegraphics[width=1.0\linewidth]{./sgam.pdf}
   \caption{Illustration of the Semantic-guided Alignment Module (SgAM), which utilizes semantic attention to transfer the information of visual concepts to semantically aligned OCR Tokens' sub-word embedding. This operation is capable of making model emphasize visual concepts that are highly relevant to the scene text. Besides, SgAM helps multimodal transformer roughly model the spatial location of semantically aligned visual concepts.}
   \label{fig:sgam}
\end{figure}
\subsection{Semantic-guided Alignment Module}
\label{subsec e}
CLIP\cite{radford2021learning} is a large-scale pre-trained model based on contrastive learning, whose powerful representation ability has been fully demonstrated by the application of downstream tasks\cite{wu2022wav2clip, zhang2022pointclip}. However, CLIP embeds the whole image and sentence into textual and visual representations, which makes it difficult for CLIP's image encoder to capture fine-grained features\cite{ma2022x}. Therefore, it is hard to ground the area depicted by these visual concepts, which may prevent the model from effectively understanding the representations of visual concepts. In addition, to avoid the introduction of out-of-domain words, we do not use words that are not included in the vocabulary provided by M4C-Captioner\cite{sidorov2020textcaps} for visual object concept extraction. Accordingly, a small part of visual concepts may be invalid, because there may be less than $K$ words related to the given image in the limited vocabulary.

To alleviate the above potential problems, we propose a Semantic-guided Alignment Module (SgAM, cf. \cref{fig:sgam}). The sub-word embedding extracting model FastText is trained on Wikipedia data. Therefore, related words (e.g., Van and Vehicle) have relatively high semantic similarity. Taking advantage of this factor, we integrate semantic information of visual concepts into related OCR tokens features. The semantic integration operation not only helps the model distinguish visual concepts related to OCR tokens but also roughly models the spatial position of some visual concepts during the interaction with spatial position information of OCR tokens in the multimodal transformer.

Concretely, we first convert FastText embedding of visual concepts $FT(a^{voc}_{1:K})$ to query $Q_S$ and convert FastText embedding of OCR tokens $x^{ft}_{1:M}$ to value $K_S$, respectively:
\begin{equation}
    \left\{ 
    \begin{aligned}
    Q_S &= FT(a^{voc}_{1:K})W_{Q_S} \\
    K_S &= x^{ft}_{1:M}W_{K_S}, \\
    \end{aligned} 
    \right.
\end{equation}
where $FT(a^{voc}_{1:K})\in \mathbb{R}^{K\times{300}}$, $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, while $W_{Q_S}$ and $W_{K_S}\in \mathbb{R}^{300\times{300}}$.

Subsequently, we adopt semantic attention to interact semantic information of visual concepts and OCR tokens
\begin{equation}
     Q'_S = softmax(\frac{Q_SK_S^T}{\sqrt{d_s}})^TFT(a^{voc}_{1:K}),
\end{equation}
where $Q'_S\in \mathbb{R}^{M\times{300}}$. Then we can get the final semantic embedding $x^{ft'}_{1:M}$ of OCR tokens
\begin{equation}
     x^{ft'}_{1:M} = L2Norm(x^{ft}_{1:M} + Q'_S),
\end{equation}
where $x^{ft'}_{1:M}$ and $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, while $L2Norm$ represents L2 Normalization.

\subsection{Reasoning and Generation Module}
Following Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we adopt a multimodal transformer to encode all the obtained embeddings. Considering most OCR tokens (e.g., ``coca'' and ``spinner'' in \cref{fig:intro}) are not common words, therefore it is not appropriate to generate captions based on a fixed vocabulary. Following M4C-Captioner\cite{sidorov2020textcaps}, we utilize two classifiers for common vocabulary and candidate OCR tokens separately to generate words. Overall, multimodal transformer models multimodal embedding and generates common words $y'_t$. Afterward, a dynamic pointer network\cite{vinyals2015pointer} is employed to generate final words $y_t$:
\begin{equation}
\begin{aligned}
     y'_t, x^{tocr}_{1:M} = mmt(x^{obj}_{1:N}, x^{ocr}_{1:M}, x^{voc}_{1:K}, x^{dec}_{t-1}),
\end{aligned}
\end{equation}
where $mmt$ represents multimodal transformer, $x^{tocr}_m$ indicates the feature of $x^{ocr}_m$ updated by $mmt$, and $x^{dec}_{t-1}$ is the embedding of previous output $y'_{t-1}$, respectively. Eventually, the dynamic pointer network makes the final prediction from the common word in the fixed vocabulary and the special words $x^{tocr}_{1:m}$ from OCR tokens:
\begin{equation}
     y_t = argmax([l_{m}^{w}(y'_t),PN(y'_t, x^{tocr}_{1:m})]),
\end{equation}
where $l_{m}^{w}(\cdot)$ denotes a linear classifier for fixed common vocabulary, $PN(\cdot)$ denotes pointer network, and the caption $C$ consists of predicted words ${y_0, ..., y_t}$. We train our model by optimizing the Cross-Entropy loss:
\begin{equation}
     L(\theta) = -\sum_{t=1}^{T}log(\hat{y}_t|y_{1:t-1}, x^{tocr}_{1:m}),
\end{equation}
where $\hat{y}_t$ is the corresponding token in the ground truth.

\section{Experiments}
\label{sec4}

\subsection{Dataset and Settings}
\textbf{Dataset and evaluation metrics.}
The TextCaps dataset\cite{sidorov2020textcaps} is constructed for text-based image captioning, which contains 28408 images with 5 captions per image.  The training, validation, and test set contain 21953, 3166, and 3289 images, respectively. For evaluation metrics, we choose five widely used evaluation metrics for image captioning, i.e., BLEU-4\cite{papineni2002bleu}, ROUGE-L\cite{liao2017textboxes}, METEOR\cite{banerjee2005meteor}, SPICE\cite{anderson2016spice}, and CIDEr-D\cite{vedantam2015cider}. Following Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we focus on the CIDEr-D metric when comparing different methods, because CIDEr-D has a high correlation with human evaluation scores and puts more weight on informative special tokens (e.g., OCR tokens) in the captions.

\begin{table*}
  \centering
    \caption{Comparison of our performance with other baseline models on the TextCaps validation set. TAP$^{\dagger}$ and ConCap$^{\dagger}$ are pre-trained models. Our DEVICE outperforms state-of-the-art models on METOR, ROUGE-L, SPICE, and CIDEr-D.}
    \label{tab1}
  \resizebox{1.5\columnwidth}{!}{
  \begin{tabular}{l|ccccc}
    \hline
    Model & BLEU-4 & METEOR & ROUGE-L & SPICE & CIDEr-D \\
    \hline
    \hline
    Up-Down (CVPR 2018)\cite{anderson2018bottom} & 20.1 & 17.8 & 42.9 & 11.7 & 41.9 \\
    AoA (ICCV 2019)\cite{huang2019attention} & 20.4 & 18.9 & 42.9 & 13.2 & 42.7 \\
    M4C-Captioner (ECCV 2020)\cite{sidorov2020textcaps} & 23.3 & 22.0 & 46.2 & 15.6 & 89.6 \\
    MMA-SR (ACM MM 2020)\cite{wang2020multimodal} & 24.6 & 23.0 & 47.3 & 16.2 & 98.0\\
    SS-Baseline (AAAI 2021)\cite{zhu2021simple} & 24.9 & 22.7 & 47.2 & 15.7 & 98.8\\
    CNMT (AAAI 2021)\cite{wang2021confidence} & 24.8 & 23.0 & 47.1 & 16.3 & 101.7\\
    ACGs-Captioner (CVPR 2021)\cite{xu2021towards} & 24.7 & 22.5 & 47.1 & 15.9 & 95.5\\
    MAGIC (AAAI 2022)\cite{zhang2022magic} & 22.2 & 20.5 & 42.3 & 13.8 & 76.6\\
    OMOT (ICMR 2022)\cite{tang2022ocr} & 26.4 & 22.2 & 47.5 & 14.9 & 95.1\\
    LSTM-R (CVPR 2021)\cite{wang2021improving} & \textbf{27.9} & 23.7 & 49.1 & 16.6 & 109.3\\
    \hline
    DEVICE (Ours) & 27.6 & \textbf{24.6} & \textbf{49.3} & \textbf{17.7} & \textbf{117.1}\\
    \hline
    M4C-Captioner(w/ GT OCRs)\cite{sidorov2020textcaps} & \textcolor[rgb]{0.5,0.5,0.5}{26.0} & \textcolor[rgb]{0.5,0.5,0.5}{23.2} & \textcolor[rgb]{0.5,0.5,0.5}{47.8} & \textcolor[rgb]{0.5,0.5,0.5}{16.2} & \textcolor[rgb]{0.5,0.5,0.5}{104.3}\\
    TAP$^{\dagger}$ (CVPR 2021)\cite{yang2021tap} & 25.8 & 23.8 & 47.9 & 17.1 & 109.2\\
    ConCap$^{\dagger}$ (AAAI 2023)\cite{wang2022controllable} & 31.3 & - & - & - & 116.7\\
    \hline
  \end{tabular}
  }
\end{table*}

\begin{table*}
  \centering
    \caption{Comparison of our performance with other baseline models on the TextCaps test set. Our DEVICE outperforms state-of-the-art models on all metrics, notably boosting CIDEr-D from 100.8 to 110.0.}
    \label{tab2}
  \resizebox{1.5\columnwidth}{!}{
  \begin{tabular}{l|ccccc}
    \hline
    Model & BLEU-4 & METEOR & ROUGE-L & SPICE & CIDEr-D \\
    \hline
    \hline
    Up-Down (CVPR 2018)\cite{anderson2018bottom} & 14.9 & 15.2 & 39.9 & 8.8 & 33.8 \\
    AoA (ICCV 2019)\cite{huang2019attention} & 15.9 & 16.6 & 40.4 & 10.5 & 34.6 \\
    M4C-Captioner (ECCV 2020)\cite{sidorov2020textcaps} & 18.9 & 19.8 & 43.2 & 12.8 & 81.0 \\
    MMA-SR (ACM MM 2020)\cite{wang2020multimodal} & 19.8 & 20.6 & 44.0 & 13.2 & 88.0\\
    SS-Baseline (AAAI 2021)\cite{zhu2021simple} & 20.2 & 20.3 & 44.2 & 12.8 & 89.6\\
    CNMT (AAAI 2021)\cite{wang2021confidence} & 20.0 & 20.8 & 44.4 & 13.4 & 93.0\\
    ACGs-Captioner (CVPR 2021)\cite{xu2021towards} & 20.7 & 20.7 & 44.6 & 13.4 & 87.4\\
    OMOT (ICMR 2022)\cite{tang2022ocr} & 21.2 & 19.6 & 44.4 & 12.1 & 84.9\\
    LSTM-R (CVPR 2021)\cite{wang2021improving} & 22.9 & 21.3 & 46.1 & 13.8 & 100.8\\
    \hline
    DEVICE (Ours) & \textbf{23.1} & \textbf{22.5} & \textbf{46.7} & \textbf{15.0} & \textbf{110.0}\\
    \hline
    M4C-Captioner(w/ GT OCRs)\cite{sidorov2020textcaps} & \textcolor[rgb]{0.5,0.5,0.5}{21.3} & \textcolor[rgb]{0.5,0.5,0.5}{21.1} & \textcolor[rgb]{0.5,0.5,0.5}{45.0} & \textcolor[rgb]{0.5,0.5,0.5}{13.5} & \textcolor[rgb]{0.5,0.5,0.5}{97.2}\\
    TAP$^{\dagger}$ (CVPR 2021)\cite{yang2021tap} & 21.9 & 21.8 & 45.6 & 14.6 & 103.2\\
    ConCap$^{\dagger}$ (AAAI 2023)\cite{wang2022controllable} & 27.4 & - & - & - & 105.6\\
    Human\cite{sidorov2020textcaps} & 24.4 & 26.1 & 47.0 & 18.8 & 125.5\\
    \hline
  \end{tabular}
  }
\end{table*}

\textbf{Settings and implementation details.}
To improve the robustness of the OCR system and increase the accuracy of detected OCR tokens, we employ the Rosetta OCR system\cite{borisyuk2018rosetta} and Google OCR system\cite{Googleocr} following Wang \textit{et al.}\cite{wang2021improving}. The number of OCR tokens for each image is limited to 80 at most. We extract 100 object appearance features for each picture, the dimension $d$ of appearance features is 2048, and the dimension $t$ of common embedding space in the multimodal transformer is 768. The maximum generation length is 30. DeFUM contains two layers of transformer encoder in our experiments, we set the layer number of the multimodal transformer to 4 and the number of self-attention heads to 12. We adopt default settings for other parameters following BERT-BASE\cite{devlin2018bert}. The fixed common vocabulary has 6736 words. To get visual object concepts, we filter top-$k$ ($k=5$ in our implement) visual concepts from the vocabulary by CLIP\cite{radford2021learning}. We retain the similarity score given by CLIP as a reference for measuring the relevance of visual concepts to the image. To get the confidence of OCR tokens, for those OCR tokens detected by Rosetta, we adopted the confidence provided by Wang \textit{et al.}\cite{wang2021confidence}. For those OCR tokens that only appear in one OCR system, we set the confidence to $0.9$.

We train the model for about 5 epochs (9000 iterations) on a single 3090 Ti GPU, and the batch size is 64. We adapt Adam\cite{kingad2015} optimizer, the initial learning rate is $1e$-$4$ and is declined to 0.1 times at 7000 iterations. We monitored the CIDEr-D metric to choose the best model and evaluate it on both the validation set and test set. All the experimental results are computed by Eval AI online platform submissions.

\subsection{Experimental Results}
\textbf{Compared models.}
We make comparisons with other models in \cref{tab1} and \cref{tab2}. \textbf{M4C-Captioner}\cite{sidorov2020textcaps}, \textbf{SS-Baseline}\cite{zhu2021simple}, \textbf{CNMT}\cite{wang2021confidence}, and \textbf{ACGs-Captioner}\cite{xu2021towards} are Transformer-based strong baseline models. \textbf{MMA-SR}\cite{wang2020multimodal}, \textbf{MAGIC}\cite{zhang2022magic}, \textbf{OMOT}\cite{tang2022ocr}, and \textbf{LSTM-R}\cite{wang2021improving} are LSTM-based strong baseline models.
\begin{itemize}
 \item \textbf{Up-Down}\cite{anderson2018bottom} and \textbf{AOA}\cite{huang2019attention} are widely used image captioning models. Both of them are trained and inferred without OCR tokens.
 \item \textbf{M4C-Captioner}\cite{sidorov2020textcaps} is a Transformer-based strong baseline model, which firstly introduces the OCR system into the TextCaps dataset. 
 \item \textbf{SS-Baseline}\cite{zhu2021simple} introduces attention branches for visual and linguistic features of scene text.
 \item \textbf{CNMT}\cite{wang2021confidence} is the state-of-the-art Transformer-based model, which adopts the confidence of scene text, and introduces a non-repetitive generation paradigm.
 \item \textbf{ACGs-Captioner}\cite{xu2021towards} introduces the anchor-centred graph (ACG), and based on different ACGs generate diverse captions.
 \item \textbf{MAGIC}\cite{zhang2022magic} constructs multiple multimodal relational graphs to improve the diversity of captions and employs a cascaded generative adversarial network to generate unpaired captions. 
 \item \textbf{MMA-SR}\cite{wang2020multimodal} is the first to explore sufficient two-dimensional spatial relationships between text tokens, and the model performance is remarkable.  
 \item \textbf{OMOT}\cite{tang2022ocr} explores the master object for each OCR token, then builds scene graphs based on these master objects. 
 \item \textbf{LSTM-R}\cite{wang2021improving} is the state-of-the-art LSTM-based model, which expands the two-dimensional geometric relationship measurement of OCR tokens on the basis of MMA-SR, such as IOU. LSTM-R can select high-quality OCR tokens with the help of a relation-aware pointer network. 
 \item \textbf{TAP$^{\pmb{\dagger}}$}\cite{yang2021tap} is a text-aware pre-trained model, which is trained on TextVQA\cite{singh2019towards}, STVQA\cite{biten2019scene}, TextCaps\cite{sidorov2020textcaps}, and OCR-CC\cite{yang2021tap}. 
 \item \textbf{ConCap$^{\pmb{\dagger}}$}\cite{wang2022controllable} is a pre-trained image captioning framework, which caption style can be controlled by prompts. ConCap$^{\dagger}$ is pre-trained on  Visual Genome\cite{krishna2017visual},  COCO\cite{lin2014microsoft}, Conceptual Caption\cite{sharma2018conceptual}, SBU Captions\cite{ordonez2011im2text}, and a filtered version of LAION\cite{schuhmann2021laion}.
 \item \textbf{M4C-Captioner (w/ GT OCRs)}\cite{sidorov2020textcaps} indicates \textbf{M4C-Cpationer} takes ground truth OCR tokens as input and evaluates on a subset of the TextCaps test set. 
 \item \textbf{Human}\cite{sidorov2020textcaps} represents human-generated captions.
\end{itemize}

\begin{figure*}[t]
  \centering
 
   \includegraphics[width=1.0\linewidth]{./case_study3.pdf}
   \caption{Example of captions generated by strong baseline CNMT\cite{wang2021confidence}, DEVICE, and GT (ground truth). \textcolor{red}{Red} indicates inappropriate sence text generated by CNMT. The depth maps and visual object concepts are displayed in the middle. The score of visual object concepts measures how similar these concepts are to the image. \textcolor[rgb]{0,0.76,0.98}{Blue} indicates scene text and \textcolor[rgb]{0,0.69,0.23}{Green} indicates visual object concepts.}
   \label{fig:case}
\end{figure*}

\textbf{Main results on the TextCaps validation set.}
The comparisons on the validation set between our DEVICE and other models are shown in \cref{tab1}. Traditional methods UP-Down and AOA perform less well than other models with OCR tokens due to they cannot generate scene text. CNMT proves the effectiveness of confidence embedding and repetition mask, which improves CIDEr-D from $89.6$ to $101.7$ compared with M4C-Captioner. LSTM-R constructs sufficient 2D spatial relations and boosts all metrics significantly. By incorporating depth information and semantic information of visual object concepts with transformer, compared with SOTA model LSTM-R\cite{wang2021improving}, our DEVICE improves the performance on METEOR, SPICE, CIDEr-D by $0.9$, $1.1$, and $7.8$, respectively. The performance of DEVICE on BLEU-4 ranks second only to LSTM-R (0.3 below). Note that LSTM-R has cleaned the training captions by removing the undetected OCR symbols, which increases BLEU-4 by 0.6\cite{wang2021improving}. For a fair comparison, we train our model with raw captions provided by TextCaps\cite{sidorov2020textcaps}, following the majority\cite{sidorov2020textcaps, wang2021confidence, zhu2021simple, yang2021tap}. Moreover, CIDEr-D and METEOR have the highest correlation with human scores\cite{sidorov2020textcaps}, which shows that DEVICE significantly outperforms other baseline models from a human perspective. The performance of our model on the TextCaps validation set demonstrates the effectiveness of our DEVICE.
\begin{table}[t]
  \centering
  \caption{Comparison of iterations and time of DEVICE and other baseline models required to converge to the optimal CIDER.}
  \label{tab4}
  \resizebox{1.\columnwidth}{!}{
  \begin{tabular}{l|cccc}
    \hline
    Model & Batch Size & Iterations/epochs & Time\\
    \hline
    \hline
    M4C-Captioner & 128 & 12000/14 & 11h\\
    CNMT & 128 & 12000/14 & 12h\\
    ACGs-Captioner & 128 & 12000/14 & 12h\\
    SS-Baseline & 128 & 15000/17 & 14h\\
    LSTM-R & 50 & -/30 & -\\
    \hline
    DEVICE(w/o SgAM) & 64 & 12000/7 & 11h\\
    DEVICE & 64 & 9000/5 & 8h\\
    \hline
    TAP$^{\dagger}$ & - & 480000/- & -\\
    ConCap$^{\dagger}$ & 2880 & -/37 & -\\
    \hline
  \end{tabular}
  }
\end{table}
\begin{table}[t]
  \centering
  \caption{Ablation of each module in DEVICE on the TextCaps validation set. Google denotes Google OCR system, DI denotes Depth Information, DeFUM represents the Depth-enhanced Feature Updating Module, VOC indicates semantic information of Visual Object Concepts, and SgAm represents Semantic-guided Alignment Module.}
  \label{tab3}
  \resizebox{1\columnwidth}{!}{
  \begin{tabular}{lccccc|cc}
    \hline
    & Google & DI & DeFUM & VOC & SgAM & BLEU-4 & CIDEr-D \\
    \hline
    \hline
    1 & & & & & & 24.5 &101.5 \\
    2 & \checkmark & & & & & 24.9 & 105.3 \\
    3 & \checkmark & \checkmark & & & & 25.6 & 109.8 \\
    4 & \checkmark & \checkmark & \checkmark & & & 26.1 & 113.9\\
    5 & \checkmark & & & \checkmark & & 26.3 & 112.4\\
    6 & \checkmark & & & \checkmark & \checkmark & 26.5 & 114.4\\
    7 & \checkmark & \checkmark & \checkmark & \checkmark & & 27.4 & 115.9\\
    8 & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \textbf{27.6} & \textbf{117.1}\\
    \hline
  \end{tabular}
  }
\end{table}

\textbf{Main results on the TextCaps test set.}
Compared with LSTM-R, our model shows superiority on the TextCaps test set, which gains rises of $0.2$, $1.2$, $0.6$, $1.2$, $9.2$ on BLEU-4, METEOR, ROUGE-L, SPICE, CIDEr-D, respectively. Significantly, the inspiring improvement on the CIDEr-D shows that our DEVICE generates more accurate scene text. The results of m4c-captioner (w/ GT OCRs)\cite{sidorov2020textcaps} are evaluated on a subset of the TextCaps test set, excluding those samples without OCR annotations. DEVICE outperforms m4c-captioner(w/ GT OCRs) verifies that our DEVICE is capable of compensating the negative impact of the error of OCR systems to some extent. Encouragingly, our model outperforms the pre-trained model TAP$^{\dagger}$ on all metrics, demonstrating that DEVICE utilizes limited training data more efficiently. The BLEU-4 score of ConCap$^{\dagger}$ is 27.3, which shows that ConCap$^{\dagger}$ has learned human language ability well from the large-scale corpus. However, DEVICE outperforms ConCap$^{\dagger}$ on CIDEr-D, which shows that the scene text in the captions generated by DEVICE is of higher quality. Finally, the gap between machines and humans narrows on the text-based image captioning task. 

\textbf{Analysis of model convergence speed.}
In our experiments, we find that the SgAM module can significantly improve the model's converge speed, and achieve better results with fewer iterations. In \cref{tab4}, we list different models' iteration numbers and training epochs when achieving the optimal CIDEr-D. With SgAM, the required training time for DEVICE is reduced from 11 hours to 8 hours. This proves that semantic attention can help the multimodal transformer quickly select effective visual entities. M4C-Captioner, CNMT, ACGs-Captioner, and SS-Baseline are trained on 2 RTX 3090 Ti GPUs in \cref{tab4}.

\begin{table}[t]
  \centering
  \caption{Analysis of visual object concepts number $K$ on validation set. VOC indicates Visual Object Concepts.}
  \label{tab5}
  \resizebox{.85\columnwidth}{!}{
  \begin{tabular}{lc|ccc}
    \hline
    Model & Number of VOC & CIDEr-D \\
    \hline
    \hline
    DEVICE & $K$ = 3 & 115.9 & \textcolor{red}{$\downarrow$}1.2\\
    DEVICE & $K$ = 5 & \textbf{117.1} &\\
    DEVICE & $K$ = 8 & 116.5 & \textcolor{red}{$\downarrow$}0.6\\
    DEVICE & $K$ = 10 & 116.1 & \textcolor{red}{$\downarrow$}1.0\\
    \hline
  \end{tabular}
  }
\end{table}

\subsection{Ablation Studies}
To analyze the respective effects of each module on the model performance, we conduct ablation experiments, as shown in \cref{tab3}. Comparing Line 1 and Line 2, CIDEr-D is boosted from $101.5$ to $105.3$. It is evident that more affluent and more accurate OCR tokens positively impact the model performance. While the improvement on BLEU-4 is relatively moderate, we think one reason may be that BLEU-4 treats all matched words equally, but CIDEr-D pays more attention to OCR tokens. Line $2$ and $3$ prove that simply concatenating depth values with two-dimensional space coordinates boosts the performance substantially, which increases BLEU-4 by $0.7$ and CIDEr-D by $4.5$, respectively. Comparing Line $3$ and $4$, we find that updating the appearance features of OCR tokens under the supervision of depth information boosts all metrics. The significant improvements between Line $4$ and $7$ demonstrate the effectiveness of visual object concepts' semantic information, which boosts BLEU-4 by $1.3$ and CIDEr-D by $2.0$, respectively. The introduction of SgAM reduces the iterations for the model to converge to the optimum, and improves the model's performance on CIDEr-D by $2.0$, respectively. More remarkably, the total increase ratio of CIDEr-D is $11.8$ and BLEU-4 is $2.7$. This demonstrates that all modules work together very efficiently. 

We also evaluate the influences of the number $K$ of visual object concepts on the TextCaps validation (cf. \cref{tab5}). The experimental results indicate that some salient visual object concepts may be missed when $K$ is less than~5. Meanwhile, irrelevant visual concepts may be retrieved when $K$ is larger than 7. Therefore, we set $K$ to $5$.

\subsection{Qualitative Analysis}
We show some cases which are generated by CNMT\cite{wang2021confidence} and DEVICE on the TextCaps validation set in \cref{fig:case},  where ``GT'' means ground truth. We pick CNMT\cite{wang2021confidence} for comparison because our geometric relationship construction methods are the same except for depth, and both are Transformer-based structures. Case 1 and Case 2 demonstrate the effectiveness of visual entities' 3D spatial relationship modeling ability. Case 3 and 4 illustrate the influences of effectively utilize visual object concepts by SgAM. In Case 1, CNMT mistakenly takes ``ats'' and ``mobile'' as tokens in the same plane and connects them incorrectly. By introducing 3D geometric relationship, our DEVICE generates accurate scene text. In Case 2, CNMT incompletely represents the text in the sign and confuses the positional relationship between the airplane and the text on the sign. DEVICE enhances the correlation of scene text on the warning sign and clarifies the spatial relationship between visual entities by introducing 3D geometric relationships. For Case 3, CNMT only generates relatively coarse-grained words such as ``many books''. In contrast, with the help of semantic information of visual object concepts and SgAM, DEVICE is capable of generating more accurate words ``comic books'', like human. In the last case, unlike CNMT, which ignores the visual object ``mascots'', DEVICE generates a more comprehensive description, enabling rational use of scene text ``pylon''. The aforementioned cases demonstrate the effectiveness of DEVICE well.

\begin{figure}[t]
  \centering
 
   \includegraphics[width=1.0\linewidth]{./case_study5.pdf}
   \caption{Negative samples of CNMT and DEVICE on the TextCaps validation set. The objects are inside the \textcolor[rgb]{0,0.69,0.23}{Green} bounding boxes, the OCR tokens are inside the \textcolor[rgb]{0,0.76,0.98}{Blue} bounding boxes. GT represents the Ground Truth.}
   \label{fig:case2}
\end{figure}

However, in Case 4, we can see the gap between our DEVICE and human, we consider one possible reason is that humans have common knowledge. Although adopting CLIP\cite{radford2021learning} to extract the visual object concept ``mascot'' in Case 4 can also be regarded as a way of implicitly using external knowledge, this knowledge is not fine-grained and accurate enough compared to ``Mario'' and ``Luigi''.

We also visualize some negative examples generated by our model on TextCaps validation dataset in \cref{fig:case2}. In Case a, strong baseline CNMT simply depicts the characters on the dial. In contrast, our model attempts to generate higher-level information, namely time. However, our model reads the time of the watch incorrectly. We consider the reason is that the ability of DEVICE to model the relationship between objects and scene text is still limited. Besides, it is difficult for models to read the time on the dial without numbers. In Case b, CNMT and DEVICE cannot correctly capture the information of ``measuring length'', which shows that current models are generally less capable of perceiving intent in scenes.

\section{Conclusion}

In this paper, we propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps, which is capable of generating accurate and comprehensive captions for given images. We introduce depth information and design a depth-enhanced feature updating module to improve OCR appearance features, which is capable of facilitating the construction of 3D geometric relations. Meanwhile, we introduce the semantic information of salient visual object concepts and propose a semantic-guided alignment module to interact visual concepts with aligned  scene text, which improves the integrity of captions. More remarkably, our model achieves state-of-the-art performances on the TextCaps test set. By comparing experimental results with human annotations, we consider that extracting more explicit external knowledge can further improve this task.

{
\bibliographystyle{IEEEtran}
