\section{Results}
\label{sec:results}
In this section, we report quantitative results in Section~\ref{sec:results:quantitative} and qualitative results in Section~\ref{sec:results:qualitative} on each of the three datasets. We highlight the best numbers in \textbf{bold}.

\subsection{Quantitative Results}
\label{sec:results:quantitative}
\paragraph{FashionIQ Dataset}
Table~\ref{tab:results:fashionIQ:origSplit} shows the quantitative performance on the evaluation set, as proposed with the benchmark in \cite{fashionIQdataset}. As highlighted by the results in \textbf{bold}, \textit{TRACE}\xspace significantly outperforms all existing baselines on each of the three data subsets - Dress, TopTee, Shirt by a significant margin. For instance, \textit{TRACE}\xspace outperforms VAL by ~6\%, ~4.5\%, and ~7\% on the Dress, TopTee, and the Shirt subsets respectively with an average improvement of ~5.7\% in R@10 metric. Similarly, \textit{TRACE}\xspace improves the results on the R@50 metric by an average of ~8\% across all the subsets. These results further show that our model outperforms other approaches by a substantial margin.

\textit{\underline{Discussion}:} The most recent baseline VAL~\cite{VALpaper} used a separate evaluation split in comparison to the original split proposed with the dataset~\cite{fashionIQdataset}. In the split used by VAL (we denote it by \textit{VAL-Split}), a subset of the testing dataset containing only correct candidate and target images is indexed for the recall computation as part of R@K evaluation. For consistency in comparison with existing approaches and to ensure completeness in our reporting, we also evaluate the performance for all baselines on this \textit{VAL-Split}. We report results separately for the original split in Table~\ref{tab:results:fashionIQ:origSplit}. As highlighted by the results in Table~\ref{tab:results:fashionIQ:valSplit}, \textit{TRACE}\xspace still achieves SOTA performance by consistently outperforming all existing baselines by similar margins as in the original split.
\paragraph{Shoes Dataset}
Table~\ref{tab:results:shoes} presents performance comparison on the shoes dataset. As highlighted in the table, \textit{TRACE}\xspace significantly outperforms the existing techniques by a good margin. E.g., the best results from \textit{TRACE}\xspace beat the VAL (the previous best algorithm) by at least 3\% in the R@K evaluation metric.
\begin{table}[hbt!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|}
        \hline
         Method & R@1 & R@10 & R@50  \\ \hline
         Image Only & 6.07 & 25.6 &  47.87 \\ \hline
         Text Only & 0.60 & 6.20 &  19.42 \\ \hline
         Concat & 5.70 & 20.32 & 39.97 \\ \hline
         TIRG \cite{TIRG} & 7.89 & 26.53 & 51.05  \\ \hline
       
       
       
         VAL~\cite{VALpaper} & 15.44 & 47.08 & 74.74 \\ \hline\hline
         \textit{TRACE}\xspace w/ \textit{Random Emb.} & 18.11 & \textbf{52.41} & 75.42 \\
         \textit{TRACE}\xspace w/ BERT & \textbf{18.5} & 51.73 & \textbf{77.28} \\ \hline
    \end{tabular}
    \caption{Quantitative results on the Shoes Dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and BERT embeddings.}
    \label{tab:results:shoes}
   
\end{table}%
\begin{figure}[th!]
    \centering
    \begin{subfigure}{0.47\textwidth}
        \centering
        \includegraphics[page=1,width=0.9\linewidth]{figures/paper-comparative-results.pdf}
        \caption{}
        \label{fig:results:qualitative-comp:figA}
    \end{subfigure}\\
    \begin{subfigure}{0.47\textwidth}
        \centering
        \includegraphics[page=3,width=0.9\linewidth]{figures/paper-comparative-results.pdf}
        \caption{}
        \label{fig:results:qualitative-comp:figB}
    \end{subfigure}
    \caption{Qualitative results from \textit{TRACE}\xspace compared with VAL~\cite{VALpaper}. First row shows the reference image and the query text. \textcolor{red}{Red} box highlights the target image from the dataset.}
    \label{fig:results:qualitative-comp}
\end{figure}%
\paragraph{Birds-to-Words Dataset}
Table~\ref{tab:results:birds2words} presents the performance comparison with multiple baselines on the Birds-to-Words dataset. \textit{TRACE}\xspace significantly outperforms the existing techniques by a significant margin. For instance, \textit{TRACE}\xspace gives a improvement of more than 5\% in R@10 evaluation metric and almost 7\% in R@50 metric. This major qualitative improvements with \textit{TRACE}\xspace also shows clearly the efficacy of \textit{TRACE}\xspace with longer text descriptions. For VAL baseline, it was not computationally infeasible to run the author provided code because of its huge memory requirements.
\begin{table}[!ht]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|}
        \hline
         Method &  R@10 & R@50  \\
        \hline
         Image Only & 15.45 & 32.14 \\ \hline
         Text Only & 1.69 & 8.34 \\ \hline
         Concat & 12.05 & 34.27 \\ \hline
         TIRG~\cite{TIRG} & 15.8 & 38.65 \\ \hline
         VAL~\cite{VALpaper} & - & - \\ \hline
       
         \textit{TRACE}\xspace w/ \textit{Random Emb.} & \textbf{20.34} & 44.94 \\
         \textit{TRACE}\xspace w/ BERT & 19.56 & \textbf{45.24} \\ \hline
    \end{tabular}
    \caption{Quantitative results on Birds-to-Words dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and \textit{BERT} embeddings.}
    \label{tab:results:birds2words}
   
\end{table}%

We also provide results with pre-trained BERT embeddings and with Random Embeddings in Tables~\ref{tab:results:shoes} to \ref{tab:results:fashionIQ:valSplit}. We observe similar performance with both the embeddings with slightly better performance with BERT embeddings for Shoes and FashionIQ datasets. Overall using either embedding outperforms the existing best approaches.


\subsection{Qualitative Results}
\label{sec:results:qualitative}
To corroborate our quantitative observations, we also present extensive qualitative analysis for \textit{TRACE}\xspace. First, in Figure~\ref{fig:results:qualitative-comp}, we present comparative visual results for \textit{TRACE}\xspace with VAL, the strongest baseline, on the FashionIQ and Shoes datasets. We notice that our model is able to i) retrieve new images while changing certain attributes conditioned on text feedback (from Figure~\ref{fig:results:qualitative-comp:figA}, \textit{TRACE}\xspace retrieves shirt which is ``blue colored'' with ``less buttons'' when VAL incorrectly retrieves ``orange'' colored clothes) ii) concurrently focus on global appearance and multiple local fine-grained designs (from Figure~\ref{fig:results:qualitative-comp:figB}, \textit{TRACE}\xspace retrieves ``\underline{shoes}'' that have ``silver color'' and ``ice blue accents'').

Figure~\ref{fig:results:qualitative-ours} presents additional qualitative results for \textit{TRACE}\xspace on each of the three datasets. We observe that \textit{TRACE}\xspace is able to a) retrieve new images while changing certain attributes conditioned on text feedback eg. color, material (from row 1, \textit{TRACE}\xspace captures the ``autumn-colored'' while preserving the ``longer'' property) b) ingest multiple visual attributes and properties in the natural language text feedback (from row 2, ``solid'', ``black'' and ``orange-and-beige trim'' all focus on different semantics of the image. \textit{TRACE}\xspace captures all of the semantics in the retrieved image) c) can jointly comprehend global appearance and local details for image search (from row 2, \textit{TRACE}\xspace captures the overall ``black'' look across the retrieved results and attempts to find the appropriate local variations in the design) d) aggregate multiple fine-grained semantic concepts within query sentence for image search (from row 3, \textit{TRACE}\xspace captures the fine-grained changes like ``black breast'', ``green flashes'' and ``longer beak'' in a single query and aggregates these concepts effectively).

We omit more qualitative results because of space constraints and instead provide them in Appendix~\ref{appendix}.



\begin{figure*}[ht!]
    \centering
    \includegraphics[page=1,width=0.95\linewidth]{figures/paper-additional-results.pdf}\vspace{-0.09cm}
    \includegraphics[page=5,width=0.95\linewidth]{figures/paper-additional-results.pdf}\vspace{-0.1cm}
    \includegraphics[page=9,width=0.95\linewidth]{figures/paper-additional-results.pdf}
    \caption{Qualitative results (one for each dataset) obtained from our proposed approach \textit{TRACE}\xspace. Image in the first column denote the reference image along with the query text. \textcolor{red}{Red} boxes highlight the target image. Additional qualitative results are provided in Appendix~\ref{appendix}.}
    \label{fig:results:qualitative-ours}
\end{figure*}
\section{Approach}
\label{sec:approach}
\begin{figure*}[ht!]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/main-architecture.pdf}
    \caption{An overview of our proposed \textit{TRACE}\xspace framework. We highlight the 3 main components in the figure: \textbf{(1)} \textbf{Semantic Feature Transformation} module, \textbf{(2)} the \textbf{Hierarchical Feature Aggregation} module and \textbf{(3)} the \textbf{Visio-Linguistic Composition} module.}
   
    \label{fig:main-architecture}
\end{figure*}
Given a \textit{Query Image ($I_{q}$)} and the \textit{Support Text Description ($D_{t}$)}, our training objective is to learn a composite feature embedding that aligns with the \textit{Target Image ($I_{t}$)} embedding. First, we encode the Query Image with an \textit{Image Encoder}, generating the multi-scale visual embeddings and then the Support Text with a \textit{Text Encoder}, generating the textual feature representation. Then we \textit{Transform}, \textit{Aggregate} and \textit{Compose} the features from multiple scales with modules described in detail in Sections~\ref{sec:approach:sft}, \ref{sec:approach:hfa} and \ref{sec:approach:vlc} to obtain a composite multi-modal representation. All components are jointly optimized using a combination of loss functions (explained in Section~\ref{sec:approach:losses}) specifically designed to regularize the visual and textual features when learning composite embeddings. Figure~\ref{fig:main-architecture} provides the framework overview for \textit{TRACE}\xspace. 
\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/tbir_attention_module.pdf}
    \caption{Visio-linguistic Pooling at $\ell^{th}$ level of the feature pyramid}
    \label{fig:tcfs}
   
\end{figure}
\subsection{Representing Image and Text}
\label{sec:approach:encode}
\paragraph{Image Encoder.} To obtain a discriminative representation of the image we use a standard convolutional neural network (CNN). Since CNNs encode visual concepts with increasing abstraction over the multiple layers, we hierarchically extract multiple feature maps from the CNN to encapsulate information across different granularities. Concretely, the resultant visual feature pyramids $F_{q}$ and $F_{t}$ for the query and the target image respectively can be represented as,
\begin{equation}
    \begin{split}
        F_{q} = {\{V_{q}^1, V_{q}^2,\dotsc, V_{q}^L\}} = \phi_\text{CNN}(I_{q}) \\
        F_{t} = {\{V_{t}^1, V_{t}^2,\dotsc, V_{t}^L\}} = \phi_\text{CNN}(I_{t}) \\
    \end{split}
\end{equation}
Here, $I_{q}$ and $I_{t}$ represent the query and target images respectively; $F_{q}$ and $F_{t}$ the feature pyramids and $V_{q}^{\ell}$ represents the $\ell^{th}$ feature map from the $L$ levels of the feature pyramid. Throughout the text, we refer to any level in the feature pyramid as $\ell \in [1, L]$.

\paragraph{Text Encoder.} Here, we use a GRU~\cite{GRU} followed by $L$ parallel fully connected layers to generate $L$ number of text embeddings corresponding to the visual feature map at each scale.

Given the support text $D_{t}$ with a maximum of $N$ words, we perform text tokenization and use the obtained words as input to obtain a sequence of word-level embedding features $F_{word} \in \mathbb{R}^{1 \times 768}$ which are then passed through a GRU to obtain the support text feature $F_{sent} \in \mathbb{R}^{1 \times 1024}$ as
\begin{equation}
    F_{sent} = \text{GRU}([F_{word}^{1},~F_{word}^{2},~\cdots,~ F_{word}^{N}])
\end{equation}

The textual features $T^{\ell}$ at any level $\ell$ are then computed using,
\begin{equation}
    T^{\ell} = %
    \begin{cases}%
        F_{sent},                   & \text{if}~l=1\\
        \Omega_{\ell}(F_{sent}),    & \text{otherwise}
    \end{cases}
    ~~\forall~\ell \in [1, L]
   
\end{equation}
where $\Omega_{\ell}$ denotes a linear projection layer for a level $\ell$.

\subsection{SFT: Semantic Feature Transformation}
\label{sec:approach:sft}
After we obtain the image features $V_{q}^{\ell}$ and the textual features $T^{\ell}$ at any level $\ell$, we transform the features to project them in a more coherent feature space.
\todo{add why projecting in cohere space helps}
\todo{Once we obtain these coherent vectors we then compose them to obtain a multi-modal representation of the image-text pair in the image search space.}

Formally, given a visual feature map of the query image $V_{q}^{\ell} \in \mathbb{R}^{C_{\ell} \times H_{\ell} \times W_{\ell}}$ and the corresponding text embedding $T^{\ell} \in \mathbb{R}^{C_{\ell}}$, this module learns an attended image-text embedding $O^{\ell}$ which encodes information from both the modalities. We now provide the steps of the transformation operations below.

\underline{\textbf{Attentional Visual Transformation:}}
When learning joint visio-linguistic representations, capturing apriori long-range contextual relationships within the visual embedding ($V_{q}^{\ell}$) helps in enhancing representational capabilities. To adaptively aggregate the spatial context, we thus leverage a positional attention mechanism~\cite{sagan,danet} to transform $V_{q}^{\ell}$ into volumetric representation $\overline{V}_{q}^{\ell} \in \mathbb{R}^{C_{\ell} \times H_{\ell} \times W_{\ell}}$. For this, $V_{q}^{\ell}$ is passed through parallel convolutional layers (denoted by $\Theta_q$, $\Theta_k$, $\Theta_v$) and then the obtained volume is reshaped to obtain new query and key feature maps denoted by $(Q^{\ell}, K^{\ell}) \in \mathbb{R}^{C_{\ell} \times N_{\ell}}$, where $N_{\ell} = H_{\ell} \times W_{\ell}$.
\begin{equation}
    \mathcal{Q}^{\ell} = \Theta_{q}(V_{q}^{\ell}),~~~ \mathcal{K}^{\ell} = \Theta_{k}(V_{q}^{\ell}),~~~ \mathcal{V}^{\ell} = \Theta_{v} (V_{q}^{\ell})
\end{equation}
Next, a spatial attention map is calculated by performing matrix multiplication between transpose of $Q^{\ell}$ and $K^{\ell}$ followed by softmax.
\begin{equation}
    \mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}} = \text{softmax}({Q^{\ell}}^{T} {K^{\ell}})
\end{equation}
We now generate an intermediate representation $E^{\ell}$ by performing matrix multiplication of $\mathcal{V}_{q}^{\ell}$ with the spatial attention map $\mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}}$. Finally, the transformed attentive visual feature map is obtained as,
\begin{equation}
    \begin{split}
        E^{\ell} &= \mathcal{V}^{\ell} {\mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}}}^T \\
        \overline{V}_{q}^{\ell} &= \gamma * E^{\ell}  + V_{q}^{\ell}
    \end{split}
\end{equation}
where $\gamma$ is a learnable scalar parameter.

The obtained feature vector $\overline{V}_{q}^{\ell}$ encodes the global visual information along with selectively aggregated spatial context which improves the semantic consistency in the representation. 

\underline{\textbf{Visiolinguistic Pooling:}}
\label{sec:approach:vision-linguistic-pooling}
The learnt attentive visual representation $\overline{V}_{q}^{\ell}$ is now used to obtain the joint image-text features. For this, $\overline{V}_{q}^{\ell}$ is convolved with the corresponding text representation $T^{\ell}$ to obtain a \textit{cross-modal} feature map $\mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}} \in \mathbb{R}^{H_{i} \times W_{i}} $.
\begin{equation}
    \mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}} = \overline V_{q}^{\ell} \circledast T^{\ell}
\end{equation}
$\mathcal{A}^{\ell}_{cross}$ is passed through softmax, with temperature $\mathcal{T}$, to obtain a feature-weightage mask $M^{\ell}$ as,
\begin{equation}
    M^{\ell} = \text{softmax}\Bigg(\frac{\mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}}}{\mathcal{T}}\Bigg)
\end{equation}
We provide a clear representation of the operations performed in Figure~\ref{fig:tcfs}. We then use the obtained feature-weighted mask $M^{\ell}$ as a kernel to pool each channel in the attentional visual feature map $\overline{V}_{q}^{\ell}$ to generate a visio-linguistic feature $S^{\ell} \in \mathbb{R}^{C_{\ell} \times 1}$ given as,
\begin{equation}
    \begin{split}
        S^{\ell}(c) = \sum_{h=1}^{H_{\ell}}\sum_{w=1}^{W_{\ell}} M^{\ell}(h,w)~*~& \overline{V}_{q}^{\ell}(c, h, w)
    \end{split}
\end{equation}
where $1 \leq c \leq \mathcal{C}^{\ell}$ and $\mathcal{C}^{\ell}$ denotes the number of channels in the $\ell$-th representation from the $L$ levels of visual feature pyramid.

Finally, the \textit{granular} text-conditioned visual embedding $O^{\ell}_{q}$ is obtained by a weighted addition of the visio-linguistic feature $S^{\ell}$ with pooled attentive visual feature map (we use the generalized-mean pooling technique \textbf{GeM}~\cite{gem} that generalizes over the max and average pooling and helps in retrieval tasks).
\begin{equation}
    O^{\ell}_{q} = \textit{Pool}(\overline{V}_{q}^{\ell}) + \beta \times S^{\ell}
\end{equation}
here, $\beta$ is a learnable scalar used for weighting. The pooled visual embedding for the target image is also obtained as
\begin{equation}
    O^{\ell}_{t} = \textit{Pool}(V^{\ell}_{t})
\end{equation}
The obtained embeddings for the query image $O^{\ell}_{q}$, across all the $L$ hierarchical visual feature volumes, form the visiolinguistic feature pyramid ${F}_{q}^{\text{trans}}$ as,
\begin{equation}
    {F}_{q}^{\text{trans}} = \{ O^{1}_{q},\dotsc,O^{\ell}_{q},\dotsc,O^{L}_{q} \}
\end{equation}
Similarly, for target image we obtain the transformed feature pyramid as,
\begin{equation}
    {F}_{t}^{\text{trans}} = \{ O^{1}_{t},\dotsc,O^{\ell}_{t},\dotsc,O^{L}_{t} \}
\end{equation}


\subsection{HFA: Hierarchical Feature Aggregation}
\label{sec:approach:hfa}
After obtaining the visiolinguistic feature pyramid ${F}_{q}^{\text{trans}}$ and the feature pyramid for the target image ${F}_{t}^{\text{trans}}$, the next step is to aggregate them into a single feature vector for retrieval. The existing approaches use \textit{concatenation} (used in \citet{VALpaper}), \textit{add} or \textit{hadamard} for aggregating the features at different levels. These operations, by assigning equal importance to the operands, fail to capture the relative  importance of the pyramidal-features and hence seem to under-perform (more details in Section~\ref{sec:ablations:composed_representations:feature_aggregation}). As convolutional networks learn features hierarchically with increasing abstraction, the embeddings in $F^{\text{trans}}_{q}$ also conform to a coarse sequential structure. Hence, to obtain an efficient representation, we need an aggregation function which can learn to encapsulate the varying granularities encoded in the pyramidal features. For this, each of the features $O^{\ell}_{q}$ and $O^{\ell}_{t}$ are first linearly projected to a fixed dimensionality ($\mathbb{R}^{C_{\ell} \times 1} \xrightarrow{} \mathbb{R}^{C_{L} \times 1}$) corresponding to the finest granularity.
\begin{equation}
    G^{\ell}_{q} = \Omega^{\ell}_{\text{hfa}}(O^{\ell}_{q}),~~~~~G^{\ell}_{t} = \Omega^{\ell}_{\text{hfa}}(O^{\ell}_{t})
\end{equation}
We then use an LSTM module $\text{LSTM}_{\text{hfa}}$ unrolled over $L$ timesteps to aggregate these transformed pyramidal embeddings $G^{\ell}_{q}$s into $H_{\text{hfa}}$ followed by passing them through a BatchNormalization layer and a fully-connected layer to obtain $f_{agg}$. This corresponds to a \textit{coarse} visiolinguistic representation.
\begin{equation}
    \begin{split}
        H_{\text{hfa}} = \text{LSTM}_{\text{hfa}}([G^{1}_{q},\dotsc,{G}^{L}_{q}]) \\
        f_{agg} = \Omega_\text{hfa}(\text{BatchNorm}(H_{\text{hfa}})) \\
    \end{split}
    \label{eq:gated-aggregation}
\end{equation}
We discuss the particular impact of this gated aggregation strategy in Section~\ref{sec:ablations}.

Similarly, to obtain the composite embedding $f_{tgt}$ for the target image $I_{t}$, the pooled embeddings in visual feature pyramid $F_{t}^{\text{trans}}$ (defined in Section~\ref{sec:approach:sft}) are aggregated using HFA to generate the composite embedding $f_{tgt}$ as,
\begin{equation}
        f_{tgt} = \text{LSTM}_{\text{hfa}}([G_{t}^{1},\dotsc,G_{t}^{L}])
\end{equation}

\subsection{VLC: Visual-Linguistic Composition}
\label{sec:approach:vlc}
The final image-text embedding is obtained by unifying the coarse visiolinguistic representation $f_{agg}$ with the global text representation $f_{text}$. First we obtain the global text representation $f_{text}$ using a linear projection of $F_{sent}$ into the visiolinguistic feature space.
\begin{equation}
    f_{text} = \Omega_{g}(F_{sent})
\end{equation}
where $\Omega_{g}$ is a learnable linear projection layer and $F_{sent}$ is a sentence representation introduced previously.

Then we obtain the \textit{composed} image-text representation $f_{com}$ (as shown in Figure~\ref{fig:main-architecture}) by residual offsetting of $f_{agg}$ with $f_{text}$ followed by vector normalization as,
\begin{equation}
    f_{com} = \delta \frac{f_{agg} + f_{text}}{{\lVert{f_{agg} + f_{text}}\rVert}_{2}}
    \label{eq:residual-composition}
\end{equation}
where $\delta$ parameter denotes the learnable normalization scale and ${\lVert .\rVert }_{2}$ denotes the $L_{2}$ norm. We discuss the particular impact of this residual composition strategy in Section~\ref{sec:ablations}.

$f_{com}$, $f_{tgt}$ and $f_{text}$ are used as inputs to the loss functions detailed in the next section.

\subsection{Loss Functions}
\label{sec:approach:losses}
The training dataset is characterised by $3$-tuples consisting of ($I_{q}, D_{t}, I_{t}$). Correspondingly, $f_{com}$ represents the composed text-conditioned image embedding for ($I_{q}, D_{t}$), $f_{text}$ represents the latent embedding for $D_{t}$ and $f_{tgt}^{+}$ represents the latent embedding for $I_{t}$. Consider another image $I_{n}$ sampled from the training set ($I_{train}$), s.t. $I_{n} \notin \{ I_{q} \cup I_{t}\}$ where $f^{-}_{tgt}$ represents its latent visual embedding which is generated using the same pipeline as for $f_{tgt}$. With this formalization, we next explain the combination of loss functions used to train \textit{TRACE}\xspace.

\paragraph{Triplet Loss} is the primary training objective which seeks to constrain the \textit{anchor} $f_{com}$ to align with the \textit{target} $f_{tgt}^{+}$ by simultaneously contrasting with the embedding for a \textit{negative} image $f_{tgt}^{-}$. The loss function is defined as
\begin{equation}
   
    \mathcal{L}_{triplet} = \log(1+e^{{\lVert  f_{com} - f_{tgt}^{+} \rVert }_{2}~-~{\lVert  f_{com} - f_{tgt}^{-}\rVert }_{2}})
\end{equation}
where ${\lVert .\rVert }_{2}$ operator denotes the $L_{2}$ norm.

To help learn discriminative representations, we employ a hard negative strategy that interleaves the random selection of $I_{n}$ with an online distance-based sampling technique. This sampling weighs $I_{n} \in I_{train}$ using the \textit{$L_{2}$}-distance of the corresponding embedding, $f_{tgt}^{-}$ with $f_{com}$, with smaller distance weighted higher. Additional details are included in the supplementary material.

\paragraph{Discriminator Loss} helps improve the alignment of $f_{com}$ with $f_{tgt}$ by utilizing a discriminator that penalizes distributional divergence of linear projections of these embeddings. The loss is defined as
\begin{equation}
    \mathcal{L}_{disc} = - \mathbb{E} \big[log(\mathcal{D}(f_{tgt})\big] -  \mathbb{E}\big[log(1-\mathcal{D}(f_{com}))\big]
\end{equation}
where $\mathcal{D}$ is the discriminator network which is trained end-to-end along with the entire model.
We discuss the particular impact of using this discriminator loss in Section~\ref{sec:ablations:loss_functions}.

\paragraph{Consistency Loss} constraints visual and linguistic projections of $f_{com}$, denoted by $f_{gen}^{img}$ and $f_{gen}^{text}$, to align with latent embeddings $f_{tgt}$ and $f_{text}$ respectively. The objective helps regularize visual-semantic representations in the learned composed embedding $f_{com}$ and is defined as:
\begin{equation}
    \mathcal{L}_{cons} = \alpha_{t}{\lVert  f_{gen}^{text} - f_{text}\rVert }_{2} ~+~ \alpha_{i}{\lVert  f_{gen}^{img} - f_{tgt} \rVert }_{2}
\end{equation}
$\alpha_{t}$, $\alpha_{i}$ are scalar hyper-parameters and ${\lVert .\rVert }_{2}$ is the $L_{2}$ norm.

In the above equation, we project the vector $f_{com}$ using learnable transformations to obtain $f_{gen}^{img}$ and $f_{gen}^{text}$ as,
\begin{equation}
    f_{gen}^{img} = \Omega^{c}_{img}(f_{com})~~~~f_{gen}^{text}  = \Omega^{c}_{text}(f_{com}) \\
\end{equation}
where $\Omega^{c}_{img}$ and $\Omega^{c}_{text}$ are learnable transformations of and are trained end-to-end alongside the model.
 
We discuss the particular impact of using this consistency loss function in Section~\ref{sec:ablations:loss_functions}.

\textbf{Total Loss} used for training is denoted by $\mathcal{L}_{total}$ and is defined as
\begin{equation}
    \mathcal{L}_{total} = \lambda_{1}\mathcal{L}_{triplet} + \lambda_{2}\mathcal{L}_{disc} + \lambda_{3}\mathcal{L}_{cons}
\end{equation}
with $\lambda_{1}, \lambda_{2}, \lambda_{2}$ as scalar hyperparameters.
\section{Experiments}
\label{sec:experiments}
In this section, we formalize the experimental configuration including datasets, baselines, implementation details and evaluation metrics.

\subsection{Datasets}
\label{sec:experiments:datasets}
We evaluate the efficacy of the proposed method, we conduct experimentation on multiple benchmark datasets. In particular, the datasets are selected to ensure maximum diversity in length of the natural language description. As indicated by Figure~\ref{tab:dataset-statistics}, the average number of words in the support text varies from 5 to 31.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/dataset-statistics.pdf}
    \caption{Average length of support text across the datasets}
    \label{tab:dataset-statistics}
\end{figure}%
\paragraph{Shoes}
Shoes~\cite{showsdataset} is a dataset originally crawled from like.com. It is further tagged with relative captions in natural language for dialog-based interactive retrieval~\cite{shoes_guo}. Following~\cite{shoes_guo}, we use 10,000 samples for training and 4,658 samples for evaluation. This dataset is characterised by ``short'' natural language queries with an average text description length of 5.32 words.
\paragraph{FashionIQ}
FashionIQ~\cite{fashionIQdataset} is a natural language-based interactive fashion product retrieval dataset. This dataset is characterized by ``medium'' natural language text descriptions with an average length of 10.69 words. It contains 77,684 images crawled from Amazon.com, covering three categories: Dress, Toptee, and Shirt. Among the 46,609 training images, there are 18,000 image pairs, with each pair accompanied with around two natural language sentences that describe one or multiple visual properties to modify in the reference image, such as ``is darker'' and ``has short sleeves and is longer and more flowing''.
\paragraph{Birds-to-Words}
Birds-to-Words~\cite{birds2words} consists of images of birds from iNaturalist\footnote{www.inaturalist.org} combined with human-annotated paragraphs to describe the difference between these pairs of images. This dataset is characterized by ``long'' natural language queries with every one of 3,347 image pairs having on average 4.8 paragraphs, each describing the differences between the pair of birds in an average of 31.38 words. Birds-to-Words provides richer text descriptions in each example than any of the other datasets we study, although the number of examples is small. \citet{birds2words} studied the generation of these fine-grained relative captions, but we adapt their dataset for the task of retrieving described second image given the relative caption.

\subsection{Baselines}
\label{sec:experiments:baselines}
We compare with the following baselines
\begin{itemize}[itemsep=1.2pt,topsep=1.4pt]
    \item \underline{Image Only} uses only image representation as composed embedding i.e $f_{com} = f_{image}$
    \item \underline{Text Only} uses only text representation as composed embedding i.e. $f_{com} = f_{text}$
    \item \underline{Concat Only} concatenates (denoted by $\mdoubleplus$) and linear transforms the image and text representations (following details from ~\cite{TIRG}) to obtain the composed embedding i.e.
    $f_{com}$ = $\text{MLP}(f_{image}\mdoubleplus f_{text})$
    \item \underline{TIRG}~\cite{TIRG} composes visual and textual representations by
    concatenation, followed by learning a gating connection and a residual connection to obtain composed embedding
    \item \underline{VAL}~\cite{VALpaper} composes the textual representation with visual representations at multiple CNN layers using a composite transformer to obtain composed embedding.
\end{itemize}
VAL is the most recent state-of-the-art technique, and hence the strongest baseline for the purpose of our experimental study. For VAL~\cite{VALpaper} and TIRG~\cite{TIRG} baselines, we refer to author-provided implementations with the recommended parameter settings to ensure consistency in comparison.

\subsection{Implementation Details}
\label{sec:experiments:implementation_details}

\begin{table*}[tbp!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average}\\
        \cline{2-9}
        & R@10 & R@50 & R@10 & R@50 & R@10 & R@50 & R@10 & R@50\\ 
        \hline
        Image Only & 2.30 & 7.51 & 3.32 & 9.42 & 3.24 & 8.15 & 2.95 & 8.36 \\ \hline
        Text Only & 6.39 & 20.92 & 4.60 & 15.8 & 5.94 & 21.10 & 5.64 & 19.27 \\ \hline    
        Concat & 7.30 & 22.31 & 5.73 & 17.87 & 7.29 & 21.31 & 6.77 & 20.50 \\ \hline
        TIRG~\cite{TIRG} & 11.11 & 29.15 & 12.95 & 30.80 & 10.20 & 25.46 & 11.42 & 28.47\\ \hline
        VAL~\cite{VALpaper} & 16.81 & 37.88 & 19.73 & 42.73 & 13.98 & 33.46 & 16.82 & 38.02 \\ \hline\hline
        \textit{TRACE}\xspace w/ \textit{Random Emb.} & 22.41 & \textbf{46.26} & 23.76 & 49.36 & 19.14 & 39.45 & 21.77 & 45.02 \\ 
        \textit{TRACE}\xspace w/ BERT & \textbf{22.70} & 44.91 & \textbf{24.22} & \textbf{49.80} & \textbf{20.80} & \textbf{40.80} & \textbf{22.57} & \textbf{46.19} \\ \hline
    \end{tabular}
    \caption{Quantitative results on Original Split of FashionIQ dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and \textit{BERT} embeddings. Best numbers are highlighted in \textbf{bold}.}
    \label{tab:results:fashionIQ:origSplit}
\end{table*}%
\begin{table*}[tbp!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average}\\
        \cline{2-9}
        & R@10 & R@50 & R@10 & R@50 & R@10 & R@50 & R@10 & R@50\\ \hline
        Image Only & 2.92 & 10.10 & 4.53 & 11.63 & 5.34 & 14.62 & 4.26 & 12.12\\ \hline
        Text Only & 8.67 & 25.08 & 9.68 & 28.25 & 8.30 & 25.02 & 8.88 & 26.11 \\ \hline    
        Concat & 9.06 & 27.27 & 10.45 & 29.83 & 9.66 & 28.06 & 9.72 & 28.33 \\ \hline
       
       
        TIRG~\cite{TIRG} & 13.43 & 33.56 & 17.23 & 40.13 & 17.46 & 40.18 & 16.04 & 37.90 \\ \hline
       
        VAL~\cite{VALpaper} & 20.87 & 43.14 &  27.03 & 52.12 & 20.90 & 46.42 & 22.90 & 47.22 \\ \hline
        \hline
        \textit{TRACE}\xspace w/ \textit{Random Emb.} & 26.13 & \textbf{52.10} & 31.16 & 59.05 & 26.20 & 50.93 & 27.83 & 54.02 \\
        \textit{TRACE}\xspace w/ BERT & \textbf{26.52} & 51.01 & \textbf{32.70} & \textbf{61.23} & \textbf{28.02} & \textbf{51.86} & \textbf{29.08} & \textbf{54.70} \\ \hline
    \end{tabular}
    \caption{Quantitative results on \textit{VAL-Split} of the FashionIQ dataset for \textit{TRACE}\xspace compared with existing techniques. Best numbers are highlighted in \textbf{bold}.}
    \label{tab:results:fashionIQ:valSplit}
\end{table*}
We conduct our experiments on a machine with 4$\times$TESLA V-100 GPU's with 16 GB RAM using Pytorch~\cite{pytorch}. We use Resnet-50~\cite{resnet} (output features = 512) pre-trained on ImageNet~\cite{imagenet} as our backbone for image encoder and a GRU (output features = 512) to encode the support text descriptions. For our experiments with pretrained text embeddings, we use BertForQuestionAnswering~\cite{BertForQuestionAnswering}.

During training, we use the Adam~\cite{kingma2014adam}(initial learning rate=$1e^{-3}$, batch size=32) optimizer for image, text encoders and the SGD optimizer (initial learning rate=$2e^{-4}$, batch size=32) for discriminator. The learning rate was divided by 2 for the Adam optimizer and divided by 10 for the SGD optimizer when the loss plateaued on the validation set until it reached $1e^{-6}$. For the image encoder, we allow for the gradual fine-tuning by unfreezing it's weights only after first few epochs(10 in our case) of training. We use a temperature of  $\mathcal{T}=8$.
We choose the values $\lambda_{1}$ = 1, $\lambda_{2}$ = 0.6, $\lambda_{3}$ = 0.1 as the hyper-parameters of our loss functions. We take $\alpha_{t}$ = 1 and $\alpha_{i}$ = 0.1 in the consistency loss. 

\subsection{Evaluation Techniques}
\label{sec:experiments:eval_techniques}
Quantitative evaluation is conducted using the Recall@K metric (R@K) which computes the percentage of evaluation queries where the target image is found within the top-K retrieved images. To ensure consistent and efficient comparison on each dataset, we report performance with K $\in \{1, 10, 50\}$ for \textit{TRACE}\xspace compared with the other baselines.



\section{Introduction}
The global retail e-commerce market was valued at \$4.25 trillion in 2019. Furthermore, recent social-cultural transformations have necessitated growth with renewed estimates that project increase at a compound annual growth rate (CAGR) of 9.4\% from 2020 to 2027. Consequently, there is significant interest in providing smart, intuitive experiences for online commerce including fine-grained tagging \cite{finegrained2,attribute}, virtual try-on \cite{tryon1,Aux_Ton, Mult_Scale,tryon2}, product recommendations \cite{recommend,recommend2} and image search \cite{imagesearch,similar_search,GSN}. In particular, the ability to search for product images over an indexed catalog is a fundamental task that serves as a cornerstone for several allied user experiences.

\label{sec:introduction}
\begin{figure}[ht!]
    \centering
    \includegraphics[width=0.7\linewidth]{figures/intro_image.pdf}
    \caption{Given a reference image and the descriptive feedback provided by the user, we retrieve the results which not only resemble the original image but also incorporate the changes as requested in the user query}
    \label{fig:problem-vision}
   
\end{figure}


The most ubiquitous frameworks in image search either take image or text as input query to search for relevant items \cite{applications_2,imagesearch}. However, a key limitation of these frameworks is the infeasibility to capture intricate user requirements which cannot be precisely encapsulated with a single image or a combination of keywords. Correspondingly, several interactive paradigms are being explored which can incorporate user feedback to help tailor retrieved results to specific user intentions. These interactions involve refining upon a reference query image through feedback in form of spatial layouts \cite{spatial}, scene-graphs \cite{scene_graph, scene2, Scene_4} or relative and certain attributes \cite{relative_attribute,fashion200Kdataset,attribute2,attribute4}. Furthermore, text feedback via natural language queries presents immense promise in providing more flexibility to the user for interactive image search \cite{shoes_guo}. In this work, we focus on incorporating diverse, unconstrained natural language descriptions as feedback for interactive image search with the specific task denoted as text conditioned image retrieval (\textit{TCIR}). Formally, given a  reference image as input and a support text description as feedback, \textit{TCIR} is concerned with retrieving the best matching images that satisfying similarity constraints imposed by both components of the multi-modal input.

Learning joint image-text representations is key to several popular task such as language grounding~\cite{grounding1,grounding2,mausoom_ground}, captioning~\cite{caption1}, Visual Question Answering (VQA) \cite{VQA,patro_question}, visual reasoning \cite{AdvTextMatch,cross_attn_matching,cvpr20_matching}, etc. These tasks use the text input to primarily guide image saliency and are interested in addressing the \textit{where-to-look} problem. Tasks like VQA focus on incorporating the global text semantics to localize objects in an image~\cite{singh2019towards,AdvTextMatch,cross_attn_matching,cvpr20_matching}. Correspondingly, research in the domain has introduced several attention mechanisms~\cite{yu2017multi} to encode textual context in visual representations. Although novel and effective, these techniques are inadequate for our task.


The task of text-conditioned image retrieval is more involved and requires learning multi-modal representations that can simultaneously address \textit{``where-to-look''} and \textit{``how-to-change''}. In particular, our work focuses on natural language queries of variable length as text feedback. Major challenges are concerned with accommodating the multiple, spatially, and semantically disentangled, visual transformations that a user may request. Figure~\ref{fig:problem-vision} shows some of the examples relevant to our task. As in Row 1, a user can parallelly propose edits in both shape and style (color, length, pattern, geometry) of multiple distinct regions (collar, sleeves, neckline). For eg, \textit{with \underline{short sleeves} and \underline{peasant neckline}}.  Also, as in Row 2, text descriptions can request multi-granular, local, and global semantic edits. For eg, \textit{is a \underline{turquoise scoop neck tshirt} with \underline{white decorative motif} and has a \underline{decal design}}.


In this work, we focus on learning image-text representations that can incorporate textual cues with multiple fine-grained semantic edits with visual representations.  Recent works have shown that ConvNets learn image features hierarchically with increasing abstractions across the pyramidal levels \cite{deep,visualizing_dnn}. Consequently, we focus on utilizing this granular feature abstraction for attentional learning of image features with text queries. We introduce a pyramidal attention and fusion module consisting of three key phases: \underline{\textbf{TR}}ansform \underline{\textbf{A}}ggregate \underline{\textbf{C}}ompos\underline{\textbf{E}} (or \textbf{TRACE}). First, granular visual representations are \textit{semantically} transformed through textual co-attention. Next, transformed features across the pyramid are hierarchically \textit{aggregated} to generate a coarse visio-linguistic representation. This aggregated representation is utilized as a residual offset and \textit{composed} with the textual features to generate a multi-modal compositional embedding.
Further, discriminative and consistency objectives are used to reinforce the utilization of semantic information in the composed embedding. 

Our contributions can be summarized as follows,
\begin{itemize}
    \item We introduce \textit{TRACE}\xspace, a framework to systematically \textit{transform}, \textit{aggregate} and \textit{compose} visio-linguistic embeddings for text conditioned image retrieval, and use discriminative and consistency objectives to regularize representation learning.
    \item We perform detailed quantitative and qualitative analysis on 3 benchmark datasets with varying lengths of natural language queries and consistently achieve SOTA results.
    \item We also conduct extensive ablative studies to analyze the impact of each component of the proposed framework.
\end{itemize}







    


\section{Ablation Studies}
\label{sec:ablations}
In this section, we conduct ablation studies to probe the impact of different design choices in \textit{TRACE}\xspace. Specifically, we design independent studies for both the Network Architecture and the Loss Functions. For ease of exposition and analysis, we restrict our scope to the FashionIQ dataset and use the original evaluation, as proposed with the benchmark.
Finally, we conclude the section with a short discussion on the physical significance of the recall evaluation metric in this task setting and open opportunities for future work in the domain.
\vspace{-0.5cm}
\begin{table*}[h!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c|c||c|c|}
        \hline
        \multicolumn{2}{|c|}{Operators}         & \multicolumn{2}{c|}{Dress}                                         & \multicolumn{2}{c|}{Toptee}                                        & \multicolumn{2}{c||}{Shirt}                                         & \multicolumn{2}{c|}{Average}                                       \\
        \hline
        HFA              & VLC             & R@10                            & R@50                            & R@10                            & R@50                            & R@10                            & R@50                            & R@10                            & R@50                            \\
        \hline
        Addition         & Residual Offsetting        & 13.53                           & 30.29                           & 12.34                           & 28.66                           & 11.28                           & 26.25                           & 12.38                           & 28.4                            \\
        Concat           & Residual Offsetting        & 15.46                           & 34.90                           & 15.91                           & 37.27                           & 12.70                           & 30.7                            & 14.69                           & 34.29                           \\
        Hadamard Product & Residual Offsetting       & 16.16                           & 33.91                           & 16.01                           & 35.03                           & 14.76                           & 32.48                           & 15.64                           & 33.80                           \\
        \hline
        LSTM             & Residual Offsetting        & \textbf{22.41} & \textbf{46.26} & \textbf{23.76} & \textbf{49.36} & \textbf{19.14} & \textbf{39.45} & \textbf{21.77} & \textbf{45.02} \\
        \hline
        &  &  &  &  &  &  &  & & 
        \\
        \hline
        LSTM             & Concat          & 19.68                           & 42.89                           & 19.17                           & 43.65                           & 15.89                           & 36.75                           & 18.24                           & 41.09                           \\
        LSTM             & Residual Gating & 19.28                           & 42.19                           & 19.88                           & 45.48                           & 15.94                           & 37.78                           & 18.33                           & 41.81                           \\
        LSTM             & Hadamard        & 19.48                           & 42.89                           & 21.06                           & 44.67                           & 17.02                           & 36.70                           & 19.18                           & 41.42                           \\
        \hline
        LSTM             & Residual Offsetting        & \textbf{22.41} & \textbf{46.26} & \textbf{23.76} & \textbf{49.36} & \textbf{19.14} & \textbf{39.45} & \textbf{21.77} & \textbf{45.02} \\
        \hline
    \end{tabular}
    \caption{Ablation study for the gated feature fusion module}
    \label{tab:ablations:fusion}
\end{table*}
\begin{table*}[h!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Loss Functions} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average} \\
        \cline{2-9}
                                        & R@10        & R@50        & R@10         & R@50        & R@10        & R@50        & R@10         & R@50         \\
        \hline
        Tr + Cons                        & 20.52       & 44.17       & 21.62        & 46.66       & 17.32       & 38.51       & 19.82        & 43.11        \\
        Tr + Disc                       & 19.93       & 45.06       & 22.30        & 46.70       & 17.32       & 38.91       & 19.65        & 43.55        \\
        \hline \hline
        Tr + Disc + Cons                 & \textbf{22.41}       & \textbf{46.26}       & \textbf{23.76}        & \textbf{49.36}       & \textbf{19.14}       & \textbf{39.45}       & \textbf{21.77}        & \textbf{45.02} \\
        \hline
    \end{tabular}
    \caption{Ablation study for the loss functions}
    \label{tab:ablations:losses}
\end{table*}

\paragraph{Effect of Gated Feature Aggregation}
\label{sec:ablations:composed_representations:feature_aggregation}
In \textit{TRACE}\xspace, we aggregate the image-text representations over multiple scales to obtain a single feature vector ($f_{agg}$) for retrieval (refer to Equation~\ref{eq:gated-aggregation}). Since CNNs learn features over varying levels of abstraction, we posit that granular representations over scales exhibit a coarse sequential ordering and therefore utilize an LSTM-based operator for aggregation. To further investigate this, here we contrast with the following design choices:
\begin{itemize}[itemsep=1.2pt]
    \item \textit{Addition}: Vector addition for vectors in $F^{\text{trans}}_{q}$ as $\sum\limits_{l=1}^L O_{l}$
    \item \textit{Concat}: Concatenates all $O_{l} \in F^{\text{trans}}_{q}$
    \item \textit{Hadamard Product}: Performs the Hadamard Product amongst all the $O_{l} \in F^{\text{trans}}_{q}$ vectors
    \item \textit{LSTM}: For each vector $O_{l} \in F^{\text{trans}}_{q}$, first perform the gating and then perform vector addition
\end{itemize}
Our observations are presented in the upper-half of Table~\ref{tab:ablations:fusion}. As highlighted in the results, using the LSTM based aggregation significantly outperforms all other operators. On average, we observe a gain of $6.11\%$ on R@10 and $11.22\%$ on R@50 over \textit{Hadamard Product}.

\paragraph{Effect of Residual Offsetting}
\label{sec:ablations:composed_representations:residual_composition}
In \textit{TRACE}\xspace, $f_{com}$ is generated by residual offsetting of $f_{agg}$ with $f_{text}$ which is a linear projection of the sentence embedding $f_{sent}$. Similar visiolinguistic compositional operations have been routinely utilized in prior work \cite{film,TIRG,showandtell},. Correspondingly, we validate this design choice by contrasting with the following operators:
\begin{itemize}[itemsep=1.2pt]
    \item \textit{Concat}: $f_{com}$ is obtained by linear projection of embedding generated by concatenation of $f_{agg}$ and $f_{text}$.
    \item \textit{Residual Gating}: $f_{com}$ is obtained using residual gating, from the baseline \cite{TIRG}, on $f_{agg}$ and $f_{text}$.
    \item \textit{Hadamard Product}: $f_{com}$ is obtained by multiplication of $f_{agg}$ and $f_{text}$.
    \item \textit{Residual Offsetting}: $f_{com}$ is obtained by residual offset of $f_{agg}$ with $f_{text}$.
 \end{itemize}
The results are presented in the lower-half of Table~\ref{tab:ablations:fusion}. As highlighted by the results, using residual operator significantly outperforms all the alternate designs. On average, residual offsetting improves over residual gating by 2.59\% at R@10 and 3.60\% at R@50.
\begin{figure*}[th!]
    \centering
    \includegraphics[page=1,width=0.7\linewidth]{figures/paper-not-target-results_hor.pdf}
    \caption{On the left, we provide the query image and the corresponding textual query along with the ground truth from the FashionIQ dataset. On the right, we provide the retrieved results ranked by decreasing relevance results from our approach \textit{TRACE}\xspace. Though the retrieved results don't contain the actual annotated sample, the results are extremely relevant to the query}
    \label{fig:ablations:label-not-found}
\end{figure*}


\subsection{Loss Functions}
\label{sec:ablations:loss_functions}
Alongside the primary triplet loss objective, we additionally introduced a discriminator and a reconstruction loss to regularize the visio-linguistic representation learning. Here, we seek to study the individual impact of each of these auxiliary constraints. We present the quantitative observations in Table~\ref{tab:ablations:losses}.
\paragraph{Effect of Discriminator Loss}
\label{sec:ablations:loss_functions:discriminator_loss}
This loss operates between $f_{com}$ and $f_{tgt}$ to condition visual representations in the learned multi-modal embedding, owing to the weak training supervision. As highlighted by the results, using the $\mathcal{L}_{disc}$ results in an average gain of 1.95\% on R@10 and 1.91\% on R@50.
\paragraph{Effect of Consistency Loss}
\label{sec:ablations:loss_functions:reconstruction_loss}
This loss is designed to reinforce the balanced utilization of both text and image representations in the learned multi-modal embedding. As highlighted by the results, using the $\mathcal{L}_{rec}$ results in an average gain of 2.12\% on R@10 and 1.47\% on R@50.
\subsection{Next Steps}
The task of text-conditioned image retrieval is pertinent owing to the immense value in industry and academia. In the benchmark datasets, each input image text pair of $I_{q}$ and $D_{t}$ is mapped to a unique target image $I_{t}$. However, owing to the generality/ambiguity of the support text $D_{t}$, multiple indexed target images may satisfy the constraint. Thus, while the Recall metric is correct, it can be too strict to provide a coherent signal of algorithmic fitness. As highlighted by results in Figure~\ref{fig:ablations:label-not-found}, the predictions are qualitative consistent but this is not captured by the metric since the specific target image is not a part of the retrieved set. Therefore, richer annotations for benchmark datasets or working towards standardized metrics can be extremely useful.

\section{Additional Results}
\label{appendix}








\todo{Add reproducibility checklist}
\todo{Add code appendix and code}
\todo{This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; the amount of memory; operating system; names, and versions of relevant software libraries and frameworks.}
\todo{add code at least for paper submission and reviewers}


\begin{figure*}[htb!]
    \centering
    \includegraphics[page=2,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=3,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=4,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=6,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=7,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=8,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=10,width=0.9\linewidth]{figures/paper-additional-results.pdf}
    \caption{Additional Qualitative results on the FashionIQ-Shoes dataset}
    \label{tab:results:additional_results}
\end{figure*}


\section{Related Work}

\paragraph{Product Search and Image Retrieval} has attracted significant research interest owing to it's diverse practical applicability \cite{applications_image_retrieval,applications_2}. Conventionally, search has focus on using uni-modal (image or text) queries to retrieve visually similar \cite{GSN,similar_search} or compatible \cite{Compatibility_MDSR} images. More recently, we have witnessed a surge in interactive multi-modal techniques that incorporates user feedback to navigate visual search. The user interactions can manifest in form of attributes \cite{attribute,attribute2,attribute4}, spatial layouts \cite{spatial, spatial_TPAMI,spatial_WACV}, sketches \cite{sketch1,sketch2, Sketch_3,sketch_example}, scene-graph \cite{scene_graph, scene2, Scene_4} and text descriptions \cite{TIRG,VALpaper, shoes_guo}. Owing to the ubiquity of text queries in existing search engines and their flexibility of articulation, using textual guidance can facilitate fine-grained specificity in user queries. In this work, we pursue visual search with textual feedback and propose a framework that efficiently handles unconstrained natural language descriptions of varying lengths.
\paragraph{Visiolinguistic Representation Learning} involves jointly processing image and text inputs to learn unified multi-modal embeddings. This is central to several \textit{vision-to-language} tasks such as VQA \cite{VQA, patro_question, mcan}, Image-Text Matching \cite{AdvTextMatch, cross_attn_matching, cvpr20_matching}, language grounding \cite{grounding1,mausoom_ground,grounding2} which use the text input to primarily address the problem of \textit{``where to look"} in the image. Earlier research has proposed various attention mechanisms to generate a coarse visual fixation. Other self-attention and co-attention techniques have been developed to learn intra-modal and inter-modal latent attention \cite{intermodal, DFAF}. Here, we focus on obtaining Visio-linguistic embeddings for the task of text-based image retrieval, which involves the need for fine-grained spatial/semantic co-attention to jointly addressing \textit{``how to change"} along-with \textit{``where to look"}. To get to this goal, we introduce a hierarchical co-attention scheme that learns to aggregate Visio-linguistic attention across multiple visual granularities. Our proposed approach is independent of pre-trained object detectors \cite{ren2015faster, fast_rcnn} and generalizes well to fine-grained visual search with diverse natural language queries.
\paragraph{Text Conditioned Image Retrieval} can be visualized as learning a composite representation of an image and text. Although CNNs inherently learn the composition of visual parts, they can not efficiently compose visual representations and natural-language semantics. TIRG~\cite{TIRG} introduced a residual gating operation to fuse latent image and text embeddings, and VAL~\cite{VALpaper} concatenates the text embedding at multiple layers of the image CNN to extract the composite representations.\todo{it is somewhat similar to our TRANSFORM step but we provide an effective way to utilize the features from different scales} We introduce a pyramidal gated fusion mechanism exploiting the CNNs hierarchical visual representations across varying abstractions~\cite{deep, visualizing_dnn}. Our method generates granular Visio-linguistic embeddings and leverages their coarse hierarchy to learn the final compositional representation.
\section{Introduction}
\todo{$j_{composed}$ should be j_aggregated}

- Online commerce is important, espp with changing social-cultural dynamics:
    - Market size, CAGR, etc. -> AI is central
    - Applications in research: image tagging, virtual try-on and search/recommendations.
    - searching items in a catalog is still the fundamental challenge -> object may exist but needs to be retrieved -> image search is very critical.

- Image Search
    - image-to-image or text-to-image: Similar and compatible: lacks flexibility
    - more flexibility in accepting user specifications: feedback based interactive tech
        - specific attributes or relative attribute
        - very recent, natural language feedback: tailor search with image and text cues -> satisfy both: our focus

\textit{transform, aggregate and compose}
transformation - of granular visual features based on text feedback. -> spatial disentanglement
\textbf{aggregation} - learned pyramidal visio-linguistic \textit{transformations}. -> semantic disentanglement
-> this gives us an aggregate transformation
\textbf{composition} - visiolinguistic transformation with the text representation. -> easy of convergence: learn an additional residual offset to reinforce semantics utilization in the learned embedding.
-> get the final embedding

- TBIR
    - vision-and-language: captioning, VQA, language grounding: limited text use to help guide image saliency: "where-to-look". attention mechanism proposed but inadequate.
    - our task: multi-modal representations -> "where-to-look" and "how-to-edit". 
        - spatial disentanglement vs semantic hierarchy

        - visual features over granular abstractions
    - Challenges:
        - need efficient compositional learning mechanism for
            - fine-grained specificity: Eg: change color/collar etc. -- good attn
            - multiple semantics: red color and blue (under-constrained NL queries).
        - need to jointly focus on content and style transformations
    - ConvNets learn features hierarchically with increasing abstractions. Hence, attentional learning of image with NL text queries -> ConvNets: content/style learning disentangled across pyramidal levels [CITE] -- granularity of representational details is abstracted across the pyramidal levels.
    - need to leverage the granular representation -> so, we propose to hierarchically aggregated visiolinguistic attention to learn composed image-text features.
    
- Contributions:
    - Focus on task of TBIR with natural language queries. Introduce FuseNet which contains hierarchical feature aggregation module to learn composite representations. Consistency training objective to regularize visiolinguistic representation learning
    - Achieve SOTA performance on varying benchmark datasets, specifically chosen to stress test performance across diff num tokens.
    - Extensive ablative studies to analyse impact of each component of FuseNet.

    
    
Another challenge is to
learn a composite representation that can jointly encapsulate visual and linguistic contents from coarse to fine-grain.
Since the text feedback may convey multi-level semantics
(Fig. 1), the composite representation is also expected to
capture the multi-granular visiolinguistic information. To
address these challenges in a unified solution, we propose
a novel Visiolinguistic Attention Learning (VAL) framework, which fuses vision and language features via attention learning at varying representation depths.





\section{Results}
\label{sec:results}
In this section, we report quantitative results in Section~\ref{sec:results:quantitative} and qualitative results in Section~\ref{sec:results:qualitative} on each of the three datasets. We highlight the best numbers in \textbf{bold}.

\subsection{Quantitative Results}
\label{sec:results:quantitative}
\paragraph{FashionIQ Dataset}
Table~\ref{tab:results:fashionIQ:origSplit} shows the quantitative performance on the evaluation set, as proposed with the benchmark in \cite{fashionIQdataset}. As highlighted by the results in \textbf{bold}, \textit{TRACE}\xspace significantly outperforms all existing baselines on each of the three data subsets - Dress, TopTee, Shirt by a significant margin. For instance, \textit{TRACE}\xspace outperforms VAL by ~6\%, ~4.5\%, and ~7\% on the Dress, TopTee, and the Shirt subsets respectively with an average improvement of ~5.7\% in R@10 metric. Similarly, \textit{TRACE}\xspace improves the results on the R@50 metric by an average of ~8\% across all the subsets. These results further show that our model outperforms other approaches by a substantial margin.

\textit{\underline{Discussion}:} The most recent baseline VAL~\cite{VALpaper} used a separate evaluation split in comparison to the original split proposed with the dataset~\cite{fashionIQdataset}. In the split used by VAL (we denote it by \textit{VAL-Split}), a subset of the testing dataset containing only correct candidate and target images is indexed for the recall computation as part of R@K evaluation. For consistency in comparison with existing approaches and to ensure completeness in our reporting, we also evaluate the performance for all baselines on this \textit{VAL-Split}. We report results separately for the original split in Table~\ref{tab:results:fashionIQ:origSplit}. As highlighted by the results in Table~\ref{tab:results:fashionIQ:valSplit}, \textit{TRACE}\xspace still achieves SOTA performance by consistently outperforming all existing baselines by similar margins as in the original split.
\paragraph{Shoes Dataset}
Table~\ref{tab:results:shoes} presents performance comparison on the shoes dataset. As highlighted in the table, \textit{TRACE}\xspace significantly outperforms the existing techniques by a good margin. E.g., the best results from \textit{TRACE}\xspace beat the VAL (the previous best algorithm) by at least 3\% in the R@K evaluation metric.
\begin{table}[hbt!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|}
        \hline
         Method & R@1 & R@10 & R@50  \\ \hline
         Image Only & 6.07 & 25.6 &  47.87 \\ \hline
         Text Only & 0.60 & 6.20 &  19.42 \\ \hline
         Concat & 5.70 & 20.32 & 39.97 \\ \hline
         TIRG \cite{TIRG} & 7.89 & 26.53 & 51.05  \\ \hline
       
       
       
         VAL~\cite{VALpaper} & 15.44 & 47.08 & 74.74 \\ \hline\hline
         \textit{TRACE}\xspace w/ \textit{Random Emb.} & 18.11 & \textbf{52.41} & 75.42 \\
         \textit{TRACE}\xspace w/ BERT & \textbf{18.5} & 51.73 & \textbf{77.28} \\ \hline
    \end{tabular}
    \caption{Quantitative results on the Shoes Dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and BERT embeddings.}
    \label{tab:results:shoes}
   
\end{table}%
\begin{figure}[th!]
    \centering
    \begin{subfigure}{0.47\textwidth}
        \centering
        \includegraphics[page=1,width=0.9\linewidth]{figures/paper-comparative-results.pdf}
        \caption{}
        \label{fig:results:qualitative-comp:figA}
    \end{subfigure}\\
    \begin{subfigure}{0.47\textwidth}
        \centering
        \includegraphics[page=3,width=0.9\linewidth]{figures/paper-comparative-results.pdf}
        \caption{}
        \label{fig:results:qualitative-comp:figB}
    \end{subfigure}
    \caption{Qualitative results from \textit{TRACE}\xspace compared with VAL~\cite{VALpaper}. First row shows the reference image and the query text. \textcolor{red}{Red} box highlights the target image from the dataset.}
    \label{fig:results:qualitative-comp}
\end{figure}%
\paragraph{Birds-to-Words Dataset}
Table~\ref{tab:results:birds2words} presents the performance comparison with multiple baselines on the Birds-to-Words dataset. \textit{TRACE}\xspace significantly outperforms the existing techniques by a significant margin. For instance, \textit{TRACE}\xspace gives a improvement of more than 5\% in R@10 evaluation metric and almost 7\% in R@50 metric. This major qualitative improvements with \textit{TRACE}\xspace also shows clearly the efficacy of \textit{TRACE}\xspace with longer text descriptions. For VAL baseline, it was not computationally infeasible to run the author provided code because of its huge memory requirements.
\begin{table}[!ht]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|}
        \hline
         Method &  R@10 & R@50  \\
        \hline
         Image Only & 15.45 & 32.14 \\ \hline
         Text Only & 1.69 & 8.34 \\ \hline
         Concat & 12.05 & 34.27 \\ \hline
         TIRG~\cite{TIRG} & 15.8 & 38.65 \\ \hline
         VAL~\cite{VALpaper} & - & - \\ \hline
       
         \textit{TRACE}\xspace w/ \textit{Random Emb.} & \textbf{20.34} & 44.94 \\
         \textit{TRACE}\xspace w/ BERT & 19.56 & \textbf{45.24} \\ \hline
    \end{tabular}
    \caption{Quantitative results on Birds-to-Words dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and \textit{BERT} embeddings.}
    \label{tab:results:birds2words}
   
\end{table}%

We also provide results with pre-trained BERT embeddings and with Random Embeddings in Tables~\ref{tab:results:shoes} to \ref{tab:results:fashionIQ:valSplit}. We observe similar performance with both the embeddings with slightly better performance with BERT embeddings for Shoes and FashionIQ datasets. Overall using either embedding outperforms the existing best approaches.


\subsection{Qualitative Results}
\label{sec:results:qualitative}
To corroborate our quantitative observations, we also present extensive qualitative analysis for \textit{TRACE}\xspace. First, in Figure~\ref{fig:results:qualitative-comp}, we present comparative visual results for \textit{TRACE}\xspace with VAL, the strongest baseline, on the FashionIQ and Shoes datasets. We notice that our model is able to i) retrieve new images while changing certain attributes conditioned on text feedback (from Figure~\ref{fig:results:qualitative-comp:figA}, \textit{TRACE}\xspace retrieves shirt which is ``blue colored'' with ``less buttons'' when VAL incorrectly retrieves ``orange'' colored clothes) ii) concurrently focus on global appearance and multiple local fine-grained designs (from Figure~\ref{fig:results:qualitative-comp:figB}, \textit{TRACE}\xspace retrieves ``\underline{shoes}'' that have ``silver color'' and ``ice blue accents'').

Figure~\ref{fig:results:qualitative-ours} presents additional qualitative results for \textit{TRACE}\xspace on each of the three datasets. We observe that \textit{TRACE}\xspace is able to a) retrieve new images while changing certain attributes conditioned on text feedback eg. color, material (from row 1, \textit{TRACE}\xspace captures the ``autumn-colored'' while preserving the ``longer'' property) b) ingest multiple visual attributes and properties in the natural language text feedback (from row 2, ``solid'', ``black'' and ``orange-and-beige trim'' all focus on different semantics of the image. \textit{TRACE}\xspace captures all of the semantics in the retrieved image) c) can jointly comprehend global appearance and local details for image search (from row 2, \textit{TRACE}\xspace captures the overall ``black'' look across the retrieved results and attempts to find the appropriate local variations in the design) d) aggregate multiple fine-grained semantic concepts within query sentence for image search (from row 3, \textit{TRACE}\xspace captures the fine-grained changes like ``black breast'', ``green flashes'' and ``longer beak'' in a single query and aggregates these concepts effectively).

We omit more qualitative results because of space constraints and instead provide them in Appendix~\ref{appendix}.



\begin{figure*}[ht!]
    \centering
    \includegraphics[page=1,width=0.95\linewidth]{figures/paper-additional-results.pdf}\vspace{-0.09cm}
    \includegraphics[page=5,width=0.95\linewidth]{figures/paper-additional-results.pdf}\vspace{-0.1cm}
    \includegraphics[page=9,width=0.95\linewidth]{figures/paper-additional-results.pdf}
    \caption{Qualitative results (one for each dataset) obtained from our proposed approach \textit{TRACE}\xspace. Image in the first column denote the reference image along with the query text. \textcolor{red}{Red} boxes highlight the target image. Additional qualitative results are provided in Appendix~\ref{appendix}.}
    \label{fig:results:qualitative-ours}
\end{figure*}
\section{Approach}
\label{sec:approach}
\begin{figure*}[ht!]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/main-architecture.pdf}
    \caption{An overview of our proposed \textit{TRACE}\xspace framework. We highlight the 3 main components in the figure: \textbf{(1)} \textbf{Semantic Feature Transformation} module, \textbf{(2)} the \textbf{Hierarchical Feature Aggregation} module and \textbf{(3)} the \textbf{Visio-Linguistic Composition} module.}
   
    \label{fig:main-architecture}
\end{figure*}
Given a \textit{Query Image ($I_{q}$)} and the \textit{Support Text Description ($D_{t}$)}, our training objective is to learn a composite feature embedding that aligns with the \textit{Target Image ($I_{t}$)} embedding. First, we encode the Query Image with an \textit{Image Encoder}, generating the multi-scale visual embeddings and then the Support Text with a \textit{Text Encoder}, generating the textual feature representation. Then we \textit{Transform}, \textit{Aggregate} and \textit{Compose} the features from multiple scales with modules described in detail in Sections~\ref{sec:approach:sft}, \ref{sec:approach:hfa} and \ref{sec:approach:vlc} to obtain a composite multi-modal representation. All components are jointly optimized using a combination of loss functions (explained in Section~\ref{sec:approach:losses}) specifically designed to regularize the visual and textual features when learning composite embeddings. Figure~\ref{fig:main-architecture} provides the framework overview for \textit{TRACE}\xspace. 
\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/tbir_attention_module.pdf}
    \caption{Visio-linguistic Pooling at $\ell^{th}$ level of the feature pyramid}
    \label{fig:tcfs}
   
\end{figure}
\subsection{Representing Image and Text}
\label{sec:approach:encode}
\paragraph{Image Encoder.} To obtain a discriminative representation of the image we use a standard convolutional neural network (CNN). Since CNNs encode visual concepts with increasing abstraction over the multiple layers, we hierarchically extract multiple feature maps from the CNN to encapsulate information across different granularities. Concretely, the resultant visual feature pyramids $F_{q}$ and $F_{t}$ for the query and the target image respectively can be represented as,
\begin{equation}
    \begin{split}
        F_{q} = {\{V_{q}^1, V_{q}^2,\dotsc, V_{q}^L\}} = \phi_\text{CNN}(I_{q}) \\
        F_{t} = {\{V_{t}^1, V_{t}^2,\dotsc, V_{t}^L\}} = \phi_\text{CNN}(I_{t}) \\
    \end{split}
\end{equation}
Here, $I_{q}$ and $I_{t}$ represent the query and target images respectively; $F_{q}$ and $F_{t}$ the feature pyramids and $V_{q}^{\ell}$ represents the $\ell^{th}$ feature map from the $L$ levels of the feature pyramid. Throughout the text, we refer to any level in the feature pyramid as $\ell \in [1, L]$.

\paragraph{Text Encoder.} Here, we use a GRU~\cite{GRU} followed by $L$ parallel fully connected layers to generate $L$ number of text embeddings corresponding to the visual feature map at each scale.

Given the support text $D_{t}$ with a maximum of $N$ words, we perform text tokenization and use the obtained words as input to obtain a sequence of word-level embedding features $F_{word} \in \mathbb{R}^{1 \times 768}$ which are then passed through a GRU to obtain the support text feature $F_{sent} \in \mathbb{R}^{1 \times 1024}$ as
\begin{equation}
    F_{sent} = \text{GRU}([F_{word}^{1},~F_{word}^{2},~\cdots,~ F_{word}^{N}])
\end{equation}

The textual features $T^{\ell}$ at any level $\ell$ are then computed using,
\begin{equation}
    T^{\ell} = %
    \begin{cases}%
        F_{sent},                   & \text{if}~l=1\\
        \Omega_{\ell}(F_{sent}),    & \text{otherwise}
    \end{cases}
    ~~\forall~\ell \in [1, L]
   
\end{equation}
where $\Omega_{\ell}$ denotes a linear projection layer for a level $\ell$.

\subsection{SFT: Semantic Feature Transformation}
\label{sec:approach:sft}
After we obtain the image features $V_{q}^{\ell}$ and the textual features $T^{\ell}$ at any level $\ell$, we transform the features to project them in a more coherent feature space.
\todo{add why projecting in cohere space helps}
\todo{Once we obtain these coherent vectors we then compose them to obtain a multi-modal representation of the image-text pair in the image search space.}

Formally, given a visual feature map of the query image $V_{q}^{\ell} \in \mathbb{R}^{C_{\ell} \times H_{\ell} \times W_{\ell}}$ and the corresponding text embedding $T^{\ell} \in \mathbb{R}^{C_{\ell}}$, this module learns an attended image-text embedding $O^{\ell}$ which encodes information from both the modalities. We now provide the steps of the transformation operations below.

\underline{\textbf{Attentional Visual Transformation:}}
When learning joint visio-linguistic representations, capturing apriori long-range contextual relationships within the visual embedding ($V_{q}^{\ell}$) helps in enhancing representational capabilities. To adaptively aggregate the spatial context, we thus leverage a positional attention mechanism~\cite{sagan,danet} to transform $V_{q}^{\ell}$ into volumetric representation $\overline{V}_{q}^{\ell} \in \mathbb{R}^{C_{\ell} \times H_{\ell} \times W_{\ell}}$. For this, $V_{q}^{\ell}$ is passed through parallel convolutional layers (denoted by $\Theta_q$, $\Theta_k$, $\Theta_v$) and then the obtained volume is reshaped to obtain new query and key feature maps denoted by $(Q^{\ell}, K^{\ell}) \in \mathbb{R}^{C_{\ell} \times N_{\ell}}$, where $N_{\ell} = H_{\ell} \times W_{\ell}$.
\begin{equation}
    \mathcal{Q}^{\ell} = \Theta_{q}(V_{q}^{\ell}),~~~ \mathcal{K}^{\ell} = \Theta_{k}(V_{q}^{\ell}),~~~ \mathcal{V}^{\ell} = \Theta_{v} (V_{q}^{\ell})
\end{equation}
Next, a spatial attention map is calculated by performing matrix multiplication between transpose of $Q^{\ell}$ and $K^{\ell}$ followed by softmax.
\begin{equation}
    \mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}} = \text{softmax}({Q^{\ell}}^{T} {K^{\ell}})
\end{equation}
We now generate an intermediate representation $E^{\ell}$ by performing matrix multiplication of $\mathcal{V}_{q}^{\ell}$ with the spatial attention map $\mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}}$. Finally, the transformed attentive visual feature map is obtained as,
\begin{equation}
    \begin{split}
        E^{\ell} &= \mathcal{V}^{\ell} {\mathbf{\mathcal{A}}^{\ell}_{\mathbf{self}}}^T \\
        \overline{V}_{q}^{\ell} &= \gamma * E^{\ell}  + V_{q}^{\ell}
    \end{split}
\end{equation}
where $\gamma$ is a learnable scalar parameter.

The obtained feature vector $\overline{V}_{q}^{\ell}$ encodes the global visual information along with selectively aggregated spatial context which improves the semantic consistency in the representation. 

\underline{\textbf{Visiolinguistic Pooling:}}
\label{sec:approach:vision-linguistic-pooling}
The learnt attentive visual representation $\overline{V}_{q}^{\ell}$ is now used to obtain the joint image-text features. For this, $\overline{V}_{q}^{\ell}$ is convolved with the corresponding text representation $T^{\ell}$ to obtain a \textit{cross-modal} feature map $\mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}} \in \mathbb{R}^{H_{i} \times W_{i}} $.
\begin{equation}
    \mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}} = \overline V_{q}^{\ell} \circledast T^{\ell}
\end{equation}
$\mathcal{A}^{\ell}_{cross}$ is passed through softmax, with temperature $\mathcal{T}$, to obtain a feature-weightage mask $M^{\ell}$ as,
\begin{equation}
    M^{\ell} = \text{softmax}\Bigg(\frac{\mathbf{\mathcal{A}}^{\ell}_{\mathbf{cross}}}{\mathcal{T}}\Bigg)
\end{equation}
We provide a clear representation of the operations performed in Figure~\ref{fig:tcfs}. We then use the obtained feature-weighted mask $M^{\ell}$ as a kernel to pool each channel in the attentional visual feature map $\overline{V}_{q}^{\ell}$ to generate a visio-linguistic feature $S^{\ell} \in \mathbb{R}^{C_{\ell} \times 1}$ given as,
\begin{equation}
    \begin{split}
        S^{\ell}(c) = \sum_{h=1}^{H_{\ell}}\sum_{w=1}^{W_{\ell}} M^{\ell}(h,w)~*~& \overline{V}_{q}^{\ell}(c, h, w)
    \end{split}
\end{equation}
where $1 \leq c \leq \mathcal{C}^{\ell}$ and $\mathcal{C}^{\ell}$ denotes the number of channels in the $\ell$-th representation from the $L$ levels of visual feature pyramid.

Finally, the \textit{granular} text-conditioned visual embedding $O^{\ell}_{q}$ is obtained by a weighted addition of the visio-linguistic feature $S^{\ell}$ with pooled attentive visual feature map (we use the generalized-mean pooling technique \textbf{GeM}~\cite{gem} that generalizes over the max and average pooling and helps in retrieval tasks).
\begin{equation}
    O^{\ell}_{q} = \textit{Pool}(\overline{V}_{q}^{\ell}) + \beta \times S^{\ell}
\end{equation}
here, $\beta$ is a learnable scalar used for weighting. The pooled visual embedding for the target image is also obtained as
\begin{equation}
    O^{\ell}_{t} = \textit{Pool}(V^{\ell}_{t})
\end{equation}
The obtained embeddings for the query image $O^{\ell}_{q}$, across all the $L$ hierarchical visual feature volumes, form the visiolinguistic feature pyramid ${F}_{q}^{\text{trans}}$ as,
\begin{equation}
    {F}_{q}^{\text{trans}} = \{ O^{1}_{q},\dotsc,O^{\ell}_{q},\dotsc,O^{L}_{q} \}
\end{equation}
Similarly, for target image we obtain the transformed feature pyramid as,
\begin{equation}
    {F}_{t}^{\text{trans}} = \{ O^{1}_{t},\dotsc,O^{\ell}_{t},\dotsc,O^{L}_{t} \}
\end{equation}


\subsection{HFA: Hierarchical Feature Aggregation}
\label{sec:approach:hfa}
After obtaining the visiolinguistic feature pyramid ${F}_{q}^{\text{trans}}$ and the feature pyramid for the target image ${F}_{t}^{\text{trans}}$, the next step is to aggregate them into a single feature vector for retrieval. The existing approaches use \textit{concatenation} (used in \citet{VALpaper}), \textit{add} or \textit{hadamard} for aggregating the features at different levels. These operations, by assigning equal importance to the operands, fail to capture the relative  importance of the pyramidal-features and hence seem to under-perform (more details in Section~\ref{sec:ablations:composed_representations:feature_aggregation}). As convolutional networks learn features hierarchically with increasing abstraction, the embeddings in $F^{\text{trans}}_{q}$ also conform to a coarse sequential structure. Hence, to obtain an efficient representation, we need an aggregation function which can learn to encapsulate the varying granularities encoded in the pyramidal features. For this, each of the features $O^{\ell}_{q}$ and $O^{\ell}_{t}$ are first linearly projected to a fixed dimensionality ($\mathbb{R}^{C_{\ell} \times 1} \xrightarrow{} \mathbb{R}^{C_{L} \times 1}$) corresponding to the finest granularity.
\begin{equation}
    G^{\ell}_{q} = \Omega^{\ell}_{\text{hfa}}(O^{\ell}_{q}),~~~~~G^{\ell}_{t} = \Omega^{\ell}_{\text{hfa}}(O^{\ell}_{t})
\end{equation}
We then use an LSTM module $\text{LSTM}_{\text{hfa}}$ unrolled over $L$ timesteps to aggregate these transformed pyramidal embeddings $G^{\ell}_{q}$s into $H_{\text{hfa}}$ followed by passing them through a BatchNormalization layer and a fully-connected layer to obtain $f_{agg}$. This corresponds to a \textit{coarse} visiolinguistic representation.
\begin{equation}
    \begin{split}
        H_{\text{hfa}} = \text{LSTM}_{\text{hfa}}([G^{1}_{q},\dotsc,{G}^{L}_{q}]) \\
        f_{agg} = \Omega_\text{hfa}(\text{BatchNorm}(H_{\text{hfa}})) \\
    \end{split}
    \label{eq:gated-aggregation}
\end{equation}
We discuss the particular impact of this gated aggregation strategy in Section~\ref{sec:ablations}.

Similarly, to obtain the composite embedding $f_{tgt}$ for the target image $I_{t}$, the pooled embeddings in visual feature pyramid $F_{t}^{\text{trans}}$ (defined in Section~\ref{sec:approach:sft}) are aggregated using HFA to generate the composite embedding $f_{tgt}$ as,
\begin{equation}
        f_{tgt} = \text{LSTM}_{\text{hfa}}([G_{t}^{1},\dotsc,G_{t}^{L}])
\end{equation}

\subsection{VLC: Visual-Linguistic Composition}
\label{sec:approach:vlc}
The final image-text embedding is obtained by unifying the coarse visiolinguistic representation $f_{agg}$ with the global text representation $f_{text}$. First we obtain the global text representation $f_{text}$ using a linear projection of $F_{sent}$ into the visiolinguistic feature space.
\begin{equation}
    f_{text} = \Omega_{g}(F_{sent})
\end{equation}
where $\Omega_{g}$ is a learnable linear projection layer and $F_{sent}$ is a sentence representation introduced previously.

Then we obtain the \textit{composed} image-text representation $f_{com}$ (as shown in Figure~\ref{fig:main-architecture}) by residual offsetting of $f_{agg}$ with $f_{text}$ followed by vector normalization as,
\begin{equation}
    f_{com} = \delta \frac{f_{agg} + f_{text}}{{\lVert{f_{agg} + f_{text}}\rVert}_{2}}
    \label{eq:residual-composition}
\end{equation}
where $\delta$ parameter denotes the learnable normalization scale and ${\lVert .\rVert }_{2}$ denotes the $L_{2}$ norm. We discuss the particular impact of this residual composition strategy in Section~\ref{sec:ablations}.

$f_{com}$, $f_{tgt}$ and $f_{text}$ are used as inputs to the loss functions detailed in the next section.

\subsection{Loss Functions}
\label{sec:approach:losses}
The training dataset is characterised by $3$-tuples consisting of ($I_{q}, D_{t}, I_{t}$). Correspondingly, $f_{com}$ represents the composed text-conditioned image embedding for ($I_{q}, D_{t}$), $f_{text}$ represents the latent embedding for $D_{t}$ and $f_{tgt}^{+}$ represents the latent embedding for $I_{t}$. Consider another image $I_{n}$ sampled from the training set ($I_{train}$), s.t. $I_{n} \notin \{ I_{q} \cup I_{t}\}$ where $f^{-}_{tgt}$ represents its latent visual embedding which is generated using the same pipeline as for $f_{tgt}$. With this formalization, we next explain the combination of loss functions used to train \textit{TRACE}\xspace.

\paragraph{Triplet Loss} is the primary training objective which seeks to constrain the \textit{anchor} $f_{com}$ to align with the \textit{target} $f_{tgt}^{+}$ by simultaneously contrasting with the embedding for a \textit{negative} image $f_{tgt}^{-}$. The loss function is defined as
\begin{equation}
   
    \mathcal{L}_{triplet} = \log(1+e^{{\lVert  f_{com} - f_{tgt}^{+} \rVert }_{2}~-~{\lVert  f_{com} - f_{tgt}^{-}\rVert }_{2}})
\end{equation}
where ${\lVert .\rVert }_{2}$ operator denotes the $L_{2}$ norm.

To help learn discriminative representations, we employ a hard negative strategy that interleaves the random selection of $I_{n}$ with an online distance-based sampling technique. This sampling weighs $I_{n} \in I_{train}$ using the \textit{$L_{2}$}-distance of the corresponding embedding, $f_{tgt}^{-}$ with $f_{com}$, with smaller distance weighted higher. Additional details are included in the supplementary material.

\paragraph{Discriminator Loss} helps improve the alignment of $f_{com}$ with $f_{tgt}$ by utilizing a discriminator that penalizes distributional divergence of linear projections of these embeddings. The loss is defined as
\begin{equation}
    \mathcal{L}_{disc} = - \mathbb{E} \big[log(\mathcal{D}(f_{tgt})\big] -  \mathbb{E}\big[log(1-\mathcal{D}(f_{com}))\big]
\end{equation}
where $\mathcal{D}$ is the discriminator network which is trained end-to-end along with the entire model.
We discuss the particular impact of using this discriminator loss in Section~\ref{sec:ablations:loss_functions}.

\paragraph{Consistency Loss} constraints visual and linguistic projections of $f_{com}$, denoted by $f_{gen}^{img}$ and $f_{gen}^{text}$, to align with latent embeddings $f_{tgt}$ and $f_{text}$ respectively. The objective helps regularize visual-semantic representations in the learned composed embedding $f_{com}$ and is defined as:
\begin{equation}
    \mathcal{L}_{cons} = \alpha_{t}{\lVert  f_{gen}^{text} - f_{text}\rVert }_{2} ~+~ \alpha_{i}{\lVert  f_{gen}^{img} - f_{tgt} \rVert }_{2}
\end{equation}
$\alpha_{t}$, $\alpha_{i}$ are scalar hyper-parameters and ${\lVert .\rVert }_{2}$ is the $L_{2}$ norm.

In the above equation, we project the vector $f_{com}$ using learnable transformations to obtain $f_{gen}^{img}$ and $f_{gen}^{text}$ as,
\begin{equation}
    f_{gen}^{img} = \Omega^{c}_{img}(f_{com})~~~~f_{gen}^{text}  = \Omega^{c}_{text}(f_{com}) \\
\end{equation}
where $\Omega^{c}_{img}$ and $\Omega^{c}_{text}$ are learnable transformations of and are trained end-to-end alongside the model.
 
We discuss the particular impact of using this consistency loss function in Section~\ref{sec:ablations:loss_functions}.

\textbf{Total Loss} used for training is denoted by $\mathcal{L}_{total}$ and is defined as
\begin{equation}
    \mathcal{L}_{total} = \lambda_{1}\mathcal{L}_{triplet} + \lambda_{2}\mathcal{L}_{disc} + \lambda_{3}\mathcal{L}_{cons}
\end{equation}
with $\lambda_{1}, \lambda_{2}, \lambda_{2}$ as scalar hyperparameters.
\section{Experiments}
\label{sec:experiments}
In this section, we formalize the experimental configuration including datasets, baselines, implementation details and evaluation metrics.

\subsection{Datasets}
\label{sec:experiments:datasets}
We evaluate the efficacy of the proposed method, we conduct experimentation on multiple benchmark datasets. In particular, the datasets are selected to ensure maximum diversity in length of the natural language description. As indicated by Figure~\ref{tab:dataset-statistics}, the average number of words in the support text varies from 5 to 31.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/dataset-statistics.pdf}
    \caption{Average length of support text across the datasets}
    \label{tab:dataset-statistics}
\end{figure}%
\paragraph{Shoes}
Shoes~\cite{showsdataset} is a dataset originally crawled from like.com. It is further tagged with relative captions in natural language for dialog-based interactive retrieval~\cite{shoes_guo}. Following~\cite{shoes_guo}, we use 10,000 samples for training and 4,658 samples for evaluation. This dataset is characterised by ``short'' natural language queries with an average text description length of 5.32 words.
\paragraph{FashionIQ}
FashionIQ~\cite{fashionIQdataset} is a natural language-based interactive fashion product retrieval dataset. This dataset is characterized by ``medium'' natural language text descriptions with an average length of 10.69 words. It contains 77,684 images crawled from Amazon.com, covering three categories: Dress, Toptee, and Shirt. Among the 46,609 training images, there are 18,000 image pairs, with each pair accompanied with around two natural language sentences that describe one or multiple visual properties to modify in the reference image, such as ``is darker'' and ``has short sleeves and is longer and more flowing''.
\paragraph{Birds-to-Words}
Birds-to-Words~\cite{birds2words} consists of images of birds from iNaturalist\footnote{www.inaturalist.org} combined with human-annotated paragraphs to describe the difference between these pairs of images. This dataset is characterized by ``long'' natural language queries with every one of 3,347 image pairs having on average 4.8 paragraphs, each describing the differences between the pair of birds in an average of 31.38 words. Birds-to-Words provides richer text descriptions in each example than any of the other datasets we study, although the number of examples is small. \citet{birds2words} studied the generation of these fine-grained relative captions, but we adapt their dataset for the task of retrieving described second image given the relative caption.

\subsection{Baselines}
\label{sec:experiments:baselines}
We compare with the following baselines
\begin{itemize}[itemsep=1.2pt,topsep=1.4pt]
    \item \underline{Image Only} uses only image representation as composed embedding i.e $f_{com} = f_{image}$
    \item \underline{Text Only} uses only text representation as composed embedding i.e. $f_{com} = f_{text}$
    \item \underline{Concat Only} concatenates (denoted by $\mdoubleplus$) and linear transforms the image and text representations (following details from ~\cite{TIRG}) to obtain the composed embedding i.e.
    $f_{com}$ = $\text{MLP}(f_{image}\mdoubleplus f_{text})$
    \item \underline{TIRG}~\cite{TIRG} composes visual and textual representations by
    concatenation, followed by learning a gating connection and a residual connection to obtain composed embedding
    \item \underline{VAL}~\cite{VALpaper} composes the textual representation with visual representations at multiple CNN layers using a composite transformer to obtain composed embedding.
\end{itemize}
VAL is the most recent state-of-the-art technique, and hence the strongest baseline for the purpose of our experimental study. For VAL~\cite{VALpaper} and TIRG~\cite{TIRG} baselines, we refer to author-provided implementations with the recommended parameter settings to ensure consistency in comparison.

\subsection{Implementation Details}
\label{sec:experiments:implementation_details}

\begin{table*}[tbp!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average}\\
        \cline{2-9}
        & R@10 & R@50 & R@10 & R@50 & R@10 & R@50 & R@10 & R@50\\ 
        \hline
        Image Only & 2.30 & 7.51 & 3.32 & 9.42 & 3.24 & 8.15 & 2.95 & 8.36 \\ \hline
        Text Only & 6.39 & 20.92 & 4.60 & 15.8 & 5.94 & 21.10 & 5.64 & 19.27 \\ \hline    
        Concat & 7.30 & 22.31 & 5.73 & 17.87 & 7.29 & 21.31 & 6.77 & 20.50 \\ \hline
        TIRG~\cite{TIRG} & 11.11 & 29.15 & 12.95 & 30.80 & 10.20 & 25.46 & 11.42 & 28.47\\ \hline
        VAL~\cite{VALpaper} & 16.81 & 37.88 & 19.73 & 42.73 & 13.98 & 33.46 & 16.82 & 38.02 \\ \hline\hline
        \textit{TRACE}\xspace w/ \textit{Random Emb.} & 22.41 & \textbf{46.26} & 23.76 & 49.36 & 19.14 & 39.45 & 21.77 & 45.02 \\ 
        \textit{TRACE}\xspace w/ BERT & \textbf{22.70} & 44.91 & \textbf{24.22} & \textbf{49.80} & \textbf{20.80} & \textbf{40.80} & \textbf{22.57} & \textbf{46.19} \\ \hline
    \end{tabular}
    \caption{Quantitative results on Original Split of FashionIQ dataset for \textit{TRACE}\xspace w/ \textit{Random Emb} and \textit{BERT} embeddings. Best numbers are highlighted in \textbf{bold}.}
    \label{tab:results:fashionIQ:origSplit}
\end{table*}%
\begin{table*}[tbp!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average}\\
        \cline{2-9}
        & R@10 & R@50 & R@10 & R@50 & R@10 & R@50 & R@10 & R@50\\ \hline
        Image Only & 2.92 & 10.10 & 4.53 & 11.63 & 5.34 & 14.62 & 4.26 & 12.12\\ \hline
        Text Only & 8.67 & 25.08 & 9.68 & 28.25 & 8.30 & 25.02 & 8.88 & 26.11 \\ \hline    
        Concat & 9.06 & 27.27 & 10.45 & 29.83 & 9.66 & 28.06 & 9.72 & 28.33 \\ \hline
       
       
        TIRG~\cite{TIRG} & 13.43 & 33.56 & 17.23 & 40.13 & 17.46 & 40.18 & 16.04 & 37.90 \\ \hline
       
        VAL~\cite{VALpaper} & 20.87 & 43.14 &  27.03 & 52.12 & 20.90 & 46.42 & 22.90 & 47.22 \\ \hline
        \hline
        \textit{TRACE}\xspace w/ \textit{Random Emb.} & 26.13 & \textbf{52.10} & 31.16 & 59.05 & 26.20 & 50.93 & 27.83 & 54.02 \\
        \textit{TRACE}\xspace w/ BERT & \textbf{26.52} & 51.01 & \textbf{32.70} & \textbf{61.23} & \textbf{28.02} & \textbf{51.86} & \textbf{29.08} & \textbf{54.70} \\ \hline
    \end{tabular}
    \caption{Quantitative results on \textit{VAL-Split} of the FashionIQ dataset for \textit{TRACE}\xspace compared with existing techniques. Best numbers are highlighted in \textbf{bold}.}
    \label{tab:results:fashionIQ:valSplit}
\end{table*}
We conduct our experiments on a machine with 4$\times$TESLA V-100 GPU's with 16 GB RAM using Pytorch~\cite{pytorch}. We use Resnet-50~\cite{resnet} (output features = 512) pre-trained on ImageNet~\cite{imagenet} as our backbone for image encoder and a GRU (output features = 512) to encode the support text descriptions. For our experiments with pretrained text embeddings, we use BertForQuestionAnswering~\cite{BertForQuestionAnswering}.

During training, we use the Adam~\cite{kingma2014adam}(initial learning rate=$1e^{-3}$, batch size=32) optimizer for image, text encoders and the SGD optimizer (initial learning rate=$2e^{-4}$, batch size=32) for discriminator. The learning rate was divided by 2 for the Adam optimizer and divided by 10 for the SGD optimizer when the loss plateaued on the validation set until it reached $1e^{-6}$. For the image encoder, we allow for the gradual fine-tuning by unfreezing it's weights only after first few epochs(10 in our case) of training. We use a temperature of  $\mathcal{T}=8$.
We choose the values $\lambda_{1}$ = 1, $\lambda_{2}$ = 0.6, $\lambda_{3}$ = 0.1 as the hyper-parameters of our loss functions. We take $\alpha_{t}$ = 1 and $\alpha_{i}$ = 0.1 in the consistency loss. 

\subsection{Evaluation Techniques}
\label{sec:experiments:eval_techniques}
Quantitative evaluation is conducted using the Recall@K metric (R@K) which computes the percentage of evaluation queries where the target image is found within the top-K retrieved images. To ensure consistent and efficient comparison on each dataset, we report performance with K $\in \{1, 10, 50\}$ for \textit{TRACE}\xspace compared with the other baselines.



\section{Introduction}
The global retail e-commerce market was valued at \$4.25 trillion in 2019. Furthermore, recent social-cultural transformations have necessitated growth with renewed estimates that project increase at a compound annual growth rate (CAGR) of 9.4\% from 2020 to 2027. Consequently, there is significant interest in providing smart, intuitive experiences for online commerce including fine-grained tagging \cite{finegrained2,attribute}, virtual try-on \cite{tryon1,Aux_Ton, Mult_Scale,tryon2}, product recommendations \cite{recommend,recommend2} and image search \cite{imagesearch,similar_search,GSN}. In particular, the ability to search for product images over an indexed catalog is a fundamental task that serves as a cornerstone for several allied user experiences.

\label{sec:introduction}
\begin{figure}[ht!]
    \centering
    \includegraphics[width=0.7\linewidth]{figures/intro_image.pdf}
    \caption{Given a reference image and the descriptive feedback provided by the user, we retrieve the results which not only resemble the original image but also incorporate the changes as requested in the user query}
    \label{fig:problem-vision}
   
\end{figure}


The most ubiquitous frameworks in image search either take image or text as input query to search for relevant items \cite{applications_2,imagesearch}. However, a key limitation of these frameworks is the infeasibility to capture intricate user requirements which cannot be precisely encapsulated with a single image or a combination of keywords. Correspondingly, several interactive paradigms are being explored which can incorporate user feedback to help tailor retrieved results to specific user intentions. These interactions involve refining upon a reference query image through feedback in form of spatial layouts \cite{spatial}, scene-graphs \cite{scene_graph, scene2, Scene_4} or relative and certain attributes \cite{relative_attribute,fashion200Kdataset,attribute2,attribute4}. Furthermore, text feedback via natural language queries presents immense promise in providing more flexibility to the user for interactive image search \cite{shoes_guo}. In this work, we focus on incorporating diverse, unconstrained natural language descriptions as feedback for interactive image search with the specific task denoted as text conditioned image retrieval (\textit{TCIR}). Formally, given a  reference image as input and a support text description as feedback, \textit{TCIR} is concerned with retrieving the best matching images that satisfying similarity constraints imposed by both components of the multi-modal input.

Learning joint image-text representations is key to several popular task such as language grounding~\cite{grounding1,grounding2,mausoom_ground}, captioning~\cite{caption1}, Visual Question Answering (VQA) \cite{VQA,patro_question}, visual reasoning \cite{AdvTextMatch,cross_attn_matching,cvpr20_matching}, etc. These tasks use the text input to primarily guide image saliency and are interested in addressing the \textit{where-to-look} problem. Tasks like VQA focus on incorporating the global text semantics to localize objects in an image~\cite{singh2019towards,AdvTextMatch,cross_attn_matching,cvpr20_matching}. Correspondingly, research in the domain has introduced several attention mechanisms~\cite{yu2017multi} to encode textual context in visual representations. Although novel and effective, these techniques are inadequate for our task.


The task of text-conditioned image retrieval is more involved and requires learning multi-modal representations that can simultaneously address \textit{``where-to-look''} and \textit{``how-to-change''}. In particular, our work focuses on natural language queries of variable length as text feedback. Major challenges are concerned with accommodating the multiple, spatially, and semantically disentangled, visual transformations that a user may request. Figure~\ref{fig:problem-vision} shows some of the examples relevant to our task. As in Row 1, a user can parallelly propose edits in both shape and style (color, length, pattern, geometry) of multiple distinct regions (collar, sleeves, neckline). For eg, \textit{with \underline{short sleeves} and \underline{peasant neckline}}.  Also, as in Row 2, text descriptions can request multi-granular, local, and global semantic edits. For eg, \textit{is a \underline{turquoise scoop neck tshirt} with \underline{white decorative motif} and has a \underline{decal design}}.


In this work, we focus on learning image-text representations that can incorporate textual cues with multiple fine-grained semantic edits with visual representations.  Recent works have shown that ConvNets learn image features hierarchically with increasing abstractions across the pyramidal levels \cite{deep,visualizing_dnn}. Consequently, we focus on utilizing this granular feature abstraction for attentional learning of image features with text queries. We introduce a pyramidal attention and fusion module consisting of three key phases: \underline{\textbf{TR}}ansform \underline{\textbf{A}}ggregate \underline{\textbf{C}}ompos\underline{\textbf{E}} (or \textbf{TRACE}). First, granular visual representations are \textit{semantically} transformed through textual co-attention. Next, transformed features across the pyramid are hierarchically \textit{aggregated} to generate a coarse visio-linguistic representation. This aggregated representation is utilized as a residual offset and \textit{composed} with the textual features to generate a multi-modal compositional embedding.
Further, discriminative and consistency objectives are used to reinforce the utilization of semantic information in the composed embedding. 

Our contributions can be summarized as follows,
\begin{itemize}
    \item We introduce \textit{TRACE}\xspace, a framework to systematically \textit{transform}, \textit{aggregate} and \textit{compose} visio-linguistic embeddings for text conditioned image retrieval, and use discriminative and consistency objectives to regularize representation learning.
    \item We perform detailed quantitative and qualitative analysis on 3 benchmark datasets with varying lengths of natural language queries and consistently achieve SOTA results.
    \item We also conduct extensive ablative studies to analyze the impact of each component of the proposed framework.
\end{itemize}







    


\section{Ablation Studies}
\label{sec:ablations}
In this section, we conduct ablation studies to probe the impact of different design choices in \textit{TRACE}\xspace. Specifically, we design independent studies for both the Network Architecture and the Loss Functions. For ease of exposition and analysis, we restrict our scope to the FashionIQ dataset and use the original evaluation, as proposed with the benchmark.
Finally, we conclude the section with a short discussion on the physical significance of the recall evaluation metric in this task setting and open opportunities for future work in the domain.
\vspace{-0.5cm}
\begin{table*}[h!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c|c||c|c|}
        \hline
        \multicolumn{2}{|c|}{Operators}         & \multicolumn{2}{c|}{Dress}                                         & \multicolumn{2}{c|}{Toptee}                                        & \multicolumn{2}{c||}{Shirt}                                         & \multicolumn{2}{c|}{Average}                                       \\
        \hline
        HFA              & VLC             & R@10                            & R@50                            & R@10                            & R@50                            & R@10                            & R@50                            & R@10                            & R@50                            \\
        \hline
        Addition         & Residual Offsetting        & 13.53                           & 30.29                           & 12.34                           & 28.66                           & 11.28                           & 26.25                           & 12.38                           & 28.4                            \\
        Concat           & Residual Offsetting        & 15.46                           & 34.90                           & 15.91                           & 37.27                           & 12.70                           & 30.7                            & 14.69                           & 34.29                           \\
        Hadamard Product & Residual Offsetting       & 16.16                           & 33.91                           & 16.01                           & 35.03                           & 14.76                           & 32.48                           & 15.64                           & 33.80                           \\
        \hline
        LSTM             & Residual Offsetting        & \textbf{22.41} & \textbf{46.26} & \textbf{23.76} & \textbf{49.36} & \textbf{19.14} & \textbf{39.45} & \textbf{21.77} & \textbf{45.02} \\
        \hline
        &  &  &  &  &  &  &  & & 
        \\
        \hline
        LSTM             & Concat          & 19.68                           & 42.89                           & 19.17                           & 43.65                           & 15.89                           & 36.75                           & 18.24                           & 41.09                           \\
        LSTM             & Residual Gating & 19.28                           & 42.19                           & 19.88                           & 45.48                           & 15.94                           & 37.78                           & 18.33                           & 41.81                           \\
        LSTM             & Hadamard        & 19.48                           & 42.89                           & 21.06                           & 44.67                           & 17.02                           & 36.70                           & 19.18                           & 41.42                           \\
        \hline
        LSTM             & Residual Offsetting        & \textbf{22.41} & \textbf{46.26} & \textbf{23.76} & \textbf{49.36} & \textbf{19.14} & \textbf{39.45} & \textbf{21.77} & \textbf{45.02} \\
        \hline
    \end{tabular}
    \caption{Ablation study for the gated feature fusion module}
    \label{tab:ablations:fusion}
\end{table*}
\begin{table*}[h!]
    \centering
    \small
    \begin{tabular}{|c|c|c|c|c|c|c||c|c|}
        \hline
        \multirow{2}{*}{Loss Functions} & \multicolumn{2}{c|}{Dress} & \multicolumn{2}{c|}{Toptee} & \multicolumn{2}{c||}{Shirt} & \multicolumn{2}{c|}{Average} \\
        \cline{2-9}
                                        & R@10        & R@50        & R@10         & R@50        & R@10        & R@50        & R@10         & R@50         \\
        \hline
        Tr + Cons                        & 20.52       & 44.17       & 21.62        & 46.66       & 17.32       & 38.51       & 19.82        & 43.11        \\
        Tr + Disc                       & 19.93       & 45.06       & 22.30        & 46.70       & 17.32       & 38.91       & 19.65        & 43.55        \\
        \hline \hline
        Tr + Disc + Cons                 & \textbf{22.41}       & \textbf{46.26}       & \textbf{23.76}        & \textbf{49.36}       & \textbf{19.14}       & \textbf{39.45}       & \textbf{21.77}        & \textbf{45.02} \\
        \hline
    \end{tabular}
    \caption{Ablation study for the loss functions}
    \label{tab:ablations:losses}
\end{table*}

\paragraph{Effect of Gated Feature Aggregation}
\label{sec:ablations:composed_representations:feature_aggregation}
In \textit{TRACE}\xspace, we aggregate the image-text representations over multiple scales to obtain a single feature vector ($f_{agg}$) for retrieval (refer to Equation~\ref{eq:gated-aggregation}). Since CNNs learn features over varying levels of abstraction, we posit that granular representations over scales exhibit a coarse sequential ordering and therefore utilize an LSTM-based operator for aggregation. To further investigate this, here we contrast with the following design choices:
\begin{itemize}[itemsep=1.2pt]
    \item \textit{Addition}: Vector addition for vectors in $F^{\text{trans}}_{q}$ as $\sum\limits_{l=1}^L O_{l}$
    \item \textit{Concat}: Concatenates all $O_{l} \in F^{\text{trans}}_{q}$
    \item \textit{Hadamard Product}: Performs the Hadamard Product amongst all the $O_{l} \in F^{\text{trans}}_{q}$ vectors
    \item \textit{LSTM}: For each vector $O_{l} \in F^{\text{trans}}_{q}$, first perform the gating and then perform vector addition
\end{itemize}
Our observations are presented in the upper-half of Table~\ref{tab:ablations:fusion}. As highlighted in the results, using the LSTM based aggregation significantly outperforms all other operators. On average, we observe a gain of $6.11\%$ on R@10 and $11.22\%$ on R@50 over \textit{Hadamard Product}.

\paragraph{Effect of Residual Offsetting}
\label{sec:ablations:composed_representations:residual_composition}
In \textit{TRACE}\xspace, $f_{com}$ is generated by residual offsetting of $f_{agg}$ with $f_{text}$ which is a linear projection of the sentence embedding $f_{sent}$. Similar visiolinguistic compositional operations have been routinely utilized in prior work \cite{film,TIRG,showandtell},. Correspondingly, we validate this design choice by contrasting with the following operators:
\begin{itemize}[itemsep=1.2pt]
    \item \textit{Concat}: $f_{com}$ is obtained by linear projection of embedding generated by concatenation of $f_{agg}$ and $f_{text}$.
    \item \textit{Residual Gating}: $f_{com}$ is obtained using residual gating, from the baseline \cite{TIRG}, on $f_{agg}$ and $f_{text}$.
    \item \textit{Hadamard Product}: $f_{com}$ is obtained by multiplication of $f_{agg}$ and $f_{text}$.
    \item \textit{Residual Offsetting}: $f_{com}$ is obtained by residual offset of $f_{agg}$ with $f_{text}$.
 \end{itemize}
The results are presented in the lower-half of Table~\ref{tab:ablations:fusion}. As highlighted by the results, using residual operator significantly outperforms all the alternate designs. On average, residual offsetting improves over residual gating by 2.59\% at R@10 and 3.60\% at R@50.
\begin{figure*}[th!]
    \centering
    \includegraphics[page=1,width=0.7\linewidth]{figures/paper-not-target-results_hor.pdf}
    \caption{On the left, we provide the query image and the corresponding textual query along with the ground truth from the FashionIQ dataset. On the right, we provide the retrieved results ranked by decreasing relevance results from our approach \textit{TRACE}\xspace. Though the retrieved results don't contain the actual annotated sample, the results are extremely relevant to the query}
    \label{fig:ablations:label-not-found}
\end{figure*}


\subsection{Loss Functions}
\label{sec:ablations:loss_functions}
Alongside the primary triplet loss objective, we additionally introduced a discriminator and a reconstruction loss to regularize the visio-linguistic representation learning. Here, we seek to study the individual impact of each of these auxiliary constraints. We present the quantitative observations in Table~\ref{tab:ablations:losses}.
\paragraph{Effect of Discriminator Loss}
\label{sec:ablations:loss_functions:discriminator_loss}
This loss operates between $f_{com}$ and $f_{tgt}$ to condition visual representations in the learned multi-modal embedding, owing to the weak training supervision. As highlighted by the results, using the $\mathcal{L}_{disc}$ results in an average gain of 1.95\% on R@10 and 1.91\% on R@50.
\paragraph{Effect of Consistency Loss}
\label{sec:ablations:loss_functions:reconstruction_loss}
This loss is designed to reinforce the balanced utilization of both text and image representations in the learned multi-modal embedding. As highlighted by the results, using the $\mathcal{L}_{rec}$ results in an average gain of 2.12\% on R@10 and 1.47\% on R@50.
\subsection{Next Steps}
The task of text-conditioned image retrieval is pertinent owing to the immense value in industry and academia. In the benchmark datasets, each input image text pair of $I_{q}$ and $D_{t}$ is mapped to a unique target image $I_{t}$. However, owing to the generality/ambiguity of the support text $D_{t}$, multiple indexed target images may satisfy the constraint. Thus, while the Recall metric is correct, it can be too strict to provide a coherent signal of algorithmic fitness. As highlighted by results in Figure~\ref{fig:ablations:label-not-found}, the predictions are qualitative consistent but this is not captured by the metric since the specific target image is not a part of the retrieved set. Therefore, richer annotations for benchmark datasets or working towards standardized metrics can be extremely useful.

\section{Additional Results}
\label{appendix}








\todo{Add reproducibility checklist}
\todo{Add code appendix and code}
\todo{This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; the amount of memory; operating system; names, and versions of relevant software libraries and frameworks.}
\todo{add code at least for paper submission and reviewers}


\begin{figure*}[htb!]
    \centering
    \includegraphics[page=2,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=3,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=4,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=6,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=7,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=8,width=0.9\linewidth]{figures/paper-additional-results.pdf}\\
    \includegraphics[page=10,width=0.9\linewidth]{figures/paper-additional-results.pdf}
    \caption{Additional Qualitative results on the FashionIQ-Shoes dataset}
    \label{tab:results:additional_results}
\end{figure*}


\section{Related Work}

\paragraph{Product Search and Image Retrieval} has attracted significant research interest owing to it's diverse practical applicability \cite{applications_image_retrieval,applications_2}. Conventionally, search has focus on using uni-modal (image or text) queries to retrieve visually similar \cite{GSN,similar_search} or compatible \cite{Compatibility_MDSR} images. More recently, we have witnessed a surge in interactive multi-modal techniques that incorporates user feedback to navigate visual search. The user interactions can manifest in form of attributes \cite{attribute,attribute2,attribute4}, spatial layouts \cite{spatial, spatial_TPAMI,spatial_WACV}, sketches \cite{sketch1,sketch2, Sketch_3,sketch_example}, scene-graph \cite{scene_graph, scene2, Scene_4} and text descriptions \cite{TIRG,VALpaper, shoes_guo}. Owing to the ubiquity of text queries in existing search engines and their flexibility of articulation, using textual guidance can facilitate fine-grained specificity in user queries. In this work, we pursue visual search with textual feedback and propose a framework that efficiently handles unconstrained natural language descriptions of varying lengths.
\paragraph{Visiolinguistic Representation Learning} involves jointly processing image and text inputs to learn unified multi-modal embeddings. This is central to several \textit{vision-to-language} tasks such as VQA \cite{VQA, patro_question, mcan}, Image-Text Matching \cite{AdvTextMatch, cross_attn_matching, cvpr20_matching}, language grounding \cite{grounding1,mausoom_ground,grounding2} which use the text input to primarily address the problem of \textit{``where to look"} in the image. Earlier research has proposed various attention mechanisms to generate a coarse visual fixation. Other self-attention and co-attention techniques have been developed to learn intra-modal and inter-modal latent attention \cite{intermodal, DFAF}. Here, we focus on obtaining Visio-linguistic embeddings for the task of text-based image retrieval, which involves the need for fine-grained spatial/semantic co-attention to jointly addressing \textit{``how to change"} along-with \textit{``where to look"}. To get to this goal, we introduce a hierarchical co-attention scheme that learns to aggregate Visio-linguistic attention across multiple visual granularities. Our proposed approach is independent of pre-trained object detectors \cite{ren2015faster, fast_rcnn} and generalizes well to fine-grained visual search with diverse natural language queries.
\paragraph{Text Conditioned Image Retrieval} can be visualized as learning a composite representation of an image and text. Although CNNs inherently learn the composition of visual parts, they can not efficiently compose visual representations and natural-language semantics. TIRG~\cite{TIRG} introduced a residual gating operation to fuse latent image and text embeddings, and VAL~\cite{VALpaper} concatenates the text embedding at multiple layers of the image CNN to extract the composite representations.\todo{it is somewhat similar to our TRANSFORM step but we provide an effective way to utilize the features from different scales} We introduce a pyramidal gated fusion mechanism exploiting the CNNs hierarchical visual representations across varying abstractions~\cite{deep, visualizing_dnn}. Our method generates granular Visio-linguistic embeddings and leverages their coarse hierarchy to learn the final compositional representation.
\section{Introduction}
\todo{$j_{composed}$ should be j_aggregated}

- Online commerce is important, espp with changing social-cultural dynamics:
    - Market size, CAGR, etc. -> AI is central
    - Applications in research: image tagging, virtual try-on and search/recommendations.
    - searching items in a catalog is still the fundamental challenge -> object may exist but needs to be retrieved -> image search is very critical.

- Image Search
    - image-to-image or text-to-image: Similar and compatible: lacks flexibility
    - more flexibility in accepting user specifications: feedback based interactive tech
        - specific attributes or relative attribute
        - very recent, natural language feedback: tailor search with image and text cues -> satisfy both: our focus

\textit{transform, aggregate and compose}
transformation - of granular visual features based on text feedback. -> spatial disentanglement
\textbf{aggregation} - learned pyramidal visio-linguistic \textit{transformations}. -> semantic disentanglement
-> this gives us an aggregate transformation
\textbf{composition} - visiolinguistic transformation with the text representation. -> easy of convergence: learn an additional residual offset to reinforce semantics utilization in the learned embedding.
-> get the final embedding

- TBIR
    - vision-and-language: captioning, VQA, language grounding: limited text use to help guide image saliency: "where-to-look". attention mechanism proposed but inadequate.
    - our task: multi-modal representations -> "where-to-look" and "how-to-edit". 
        - spatial disentanglement vs semantic hierarchy

        - visual features over granular abstractions
    - Challenges:
        - need efficient compositional learning mechanism for
            - fine-grained specificity: Eg: change color/collar etc. -- good attn
            - multiple semantics: red color and blue (under-constrained NL queries).
        - need to jointly focus on content and style transformations
    - ConvNets learn features hierarchically with increasing abstractions. Hence, attentional learning of image with NL text queries -> ConvNets: content/style learning disentangled across pyramidal levels [CITE] -- granularity of representational details is abstracted across the pyramidal levels.
    - need to leverage the granular representation -> so, we propose to hierarchically aggregated visiolinguistic attention to learn composed image-text features.
    
- Contributions:
    - Focus on task of TBIR with natural language queries. Introduce FuseNet which contains hierarchical feature aggregation module to learn composite representations. Consistency training objective to regularize visiolinguistic representation learning
    - Achieve SOTA performance on varying benchmark datasets, specifically chosen to stress test performance across diff num tokens.
    - Extensive ablative studies to analyse impact of each component of FuseNet.

    
    
Another challenge is to
learn a composite representation that can jointly encapsulate visual and linguistic contents from coarse to fine-grain.
Since the text feedback may convey multi-level semantics
(Fig. 1), the composite representation is also expected to
capture the multi-granular visiolinguistic information. To
address these challenges in a unified solution, we propose
a novel Visiolinguistic Attention Learning (VAL) framework, which fuses vision and language features via attention learning at varying representation depths.


