
\section*{Appendices}

\section{Conclusion}
We presented PAN, a method of incorporating additional attribute annotations in image datasets to learn a better similarity predictor. We saw that PAN's method of decomposing similarity prediction into multiple conditions is general, functions with a range of different image encoders and is flexible in using attribute annotation, possibly sparse, when available. PAN outperformed state of the art on a diverse set of three tasks---by 4-9\% on fashion item compatibility prediction on Polyvore Outfits, 5\% on few shot classification on CUB and over 1\% Recall@1 on In-Shop Clothing Retrieval---contrary to prior approaches of using attribute supervision, which were unable to outperform methods that automatically learned concepts in different similarity conditions. In showing these benefits of PAN we factored out contributions from training parameters like batch-sizes, hopefully informing future work with our analysis.
\smallskip


\noindent\textbf{Acknowledgements:} This work was funded in part by DARPA and by the Hariri Data Science Faculty Fellowship.
\section{Datasets and Tasks}




\noindent\textbf{Polyvore Outfits~ \cite{vasileva2018learning}} contains 53K outfits (sets of fashion items) for training, 5K for validation and 10K for testing. It also provides fine-grained category information and text descriptions of items.  We use the 205 sparsely annotated attributes from \cite{plummerSimilarity} as labels for supervising similarity conditions. %
Evaluation involves two tasks.  First, in the fill-in-the-blank (FITB) outfit completion task a model is given a partial outfit and has to select from four possible answers what item would best complete it.  Performance is measured by how often the answer was correct. Second, in outfit compatibility a model is asked to discriminate between good and bad outfits. Performance is measured using area under a receiver operating characteristic curve (AUC). Following~\cite{vasileva2018learning,cucurull2019context}, outfit compatibility scores are computed by averaging the similarity prediction over all pairs of items in the outfit. There are 10K FITB questions and 10K each positive and negative samples for outfit compatibility (20K total) in the test split.  

Since current methods get almost perfect performance on the original outfit compatibility task, we created a more challenging testing set of the same size by modifying the procedure outlined in \cite{vasileva2018learning}, that we refer to as the \emph{resampled} set. For outfit compatibility we collected new negative outfits by replacing only part of a ground truth outfit, unlike the original split which replaced all items.  We randomly selected the number of items to replace, and each item is replaced with another of the same type in the same split (\ie, a top could only be replaced with another top).   Similarly, we made a more challenging FITB task, where a model must select between 10 candidate answers (the original test had 4).  As with the original sampling, we ensured any replaced items and candidate answers were of the same type.
\smallskip


\noindent\textbf{CUB-200-2011 \cite{wah2011caltech}} consists of 200 classes and a total of 11,768 images of birds. We use the split provided by \cite{chen2019closer} for our experiments, which contains 100 base classes, 50 validation and 50 novel classes. The CUB dataset also has 312 fine-grained binary attributes labeled for each image, with an accompanying score on a 4 point scale indicating the confidence of the assigned label. We drop all attribute labels that have a confidence score less than or equal to 2.

We use the 5-way 5-shot classification task for evaluation. Reported accuracies are averaged over 3 training runs from different random initializations accompanied by 95\% confidence intervals. A test \emph{episode} consists of a random sample of 5 classes and 5 support images from the 50 classes in the novel split of the dataset. 16 query images, distinct from support images, are also sampled for each of these 5 classes. The accuracy for an episode is the 5-way accuracy of a classifier over the 16 x 5 = 80 query images. A few shot learning model is evaluated using its average classification accuracy over 600 randomly generated test episodes. 
\smallskip

\noindent\textbf{In-Shop Retrieval \cite{liu2016deepfashion}} contains 52,712 images of clothes from 11,967 classes. There are 14,218 query images and 12,612 gallery images for testing. Given a query image, the task is to retrieve an image of the same item from the gallery set. Note that the query and gallery sets do not overlap with the training set. There are 463 attributes of clothes in total, we use these attribute labels for our PAN-Supervised model. Methods are ranked based on Recall@1.













\section{Results} \label{subsec:results}

\subsection{Comparison with prior work}
Table \ref{tab:compare_prior_work_po}, Table \ref{tab:compare_prior_work_cub}, and Table \ref{tab: inshop_results} compare the best settings (encoder, number of unsupervised similarity conditions, \etc) used by our model %
to representative state-of-the-art results reported in prior work on Polyvore Outfits, CUB, and In-Shop Retrieval, respectively. As shown in Table~\ref{tab:compare_prior_work_po} we obtain a 4\% better FITB accuracy and 9\% AUC boost over the state-of-the-art on the fashion compatibility task using our more challenging resampled test set for both tasks, while also increasing FITB accuracy by 8\% on the original split.  Similarly, in Table~\ref{tab:compare_prior_work_cub} and Table \ref{tab: inshop_results} we observe a 5\% and 1\% performance improvement over the state-of-the-art on fine-grained few shot classification and In-Shop Retrieval, respectively. Improvement over the diverse set of tasks demonstrates PAN's ability to generalize.  Our PAN model can also be useful when no supervision is provided, as our PAN-Unsupervised model obtains a 3-4\% gain over prior work on Polyvore Outfits and CUB, while also boosting performance on In-Shop Retrieval.  Note that fashion compatibility benefited from using a graph image encoder (GE), while few-shot classification reported best performance with a CNN encoder, which we shall discuss further in Section~\ref{subsec:context}.


In addition to comparison with prior work, Table \ref{tab:compare_prior_work_po}, Table \ref{tab:compare_prior_work_cub}, and Table \ref{tab: inshop_results} also provide two alternative methods of using attributes in an image similarity model.  In ``X + Attr.\ Multitask'' we use a hard parameter sharing multitask approach~\cite{caruanaMULTI-task}, where we share an image encoder, but have separate output heads for each of the two tasks (one of them being attribute classification, the other similarity link prediction). In ``Attr.\ Similarity'' we use a standard framework where we predict attributes for each image and then learn a classifier implemented as a fully connected layer that takes both attributes as input and predicts similarity (the general framework used by~\cite{farhadi2009describing}).  Notably, both baseline methods that use attributes only improve performance on few-shot classification, but either make no difference, or are even harmful to performance on the other two datasets (\eg, Attr. Similarity \emph{reduces} FITB performance by 5-8\% compared with the CGAE baseline).  In contrast, our PAN-supervised model outperforms all other methods, including on the fashion compatibility task where we report a staggering 6-17\% boost over the attribute baselines on the resampled test set.
\smallskip

\input{tables/po_results}

\input{tables/cub_results}

\input{tables/inshop_results}









 

\begin{figure}[t]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/plot_lam_polyvore_testval.pdf}
    \caption{FITB accuracy of PAN-Supervised-GE on Polyvore Outfits.}
    \end{subfigure}
    \begin{subfigure}[t]{0.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/plot_lam_testval.pdf}
    \caption{5-way 5-shot classification accuracy of PAN-Supervised on CUB}
    \end{subfigure}
    \caption{Sensitivity to $\lambda$ from Eq.~(\ref{eq:total_loss}). }
    \label{fig:plot-lam}
    \vspace{-5mm}
\end{figure}


\input{tables/analysis}

\input{tables/att_combn_type}





\subsection{Choice of image encoder and batch size}
\label{subsec:context}

Prior work has been inconsistent in its training methods and controlling for hyperparameters like batch size, which can significantly affect performance.  Table~\ref{tab:analysis} remedies this by comparing training with the whole training split vs. using minibatches. It also compares the effect of using a graph encoder (GE) instead of a simple CNN. We see comparing the numbers of row 3 of Table \ref{tab:analysis} to those of prior methods in Tables \ref{tab:compare_prior_work_po} and \ref{tab:compare_prior_work_cub} that training with the whole training set can significantly improve performance, making even a simple Siamese network trained using a triplet loss outperform most recent methods on both tasks. We note here that in our experiments with single batch training, we use a pre-trained CNN to extract image features, and do not finetune it.


It is also notable that using GE performs worse than the simpler Siamese Network baseline on CUB, which we believe is due to differences between tasks. Specifically, in fashion compatibility links exist between items that may be very different from each other. Thus, the additional context provided through the GE may be more important than in CUB, which has less variation between linked items since they all contain the same bird. 






\subsection{Choice of attribute combination $f_a$}
\label{subsec:logical_function}

As discussed in Section \ref{subsec:def-similarity-conds}, choosing $f_a$ as one of common logical functions can lead to different interpretations of the attribute supervision provided.
Table~\ref{tab:att_comb_type} compares these functions that convert image attribute labels to pairwise labels for use in our PAN model. 



At first glance, predicting $1$ when either both images have an attribute or neither of them does (\ie using $XNOR$) seems like an intuitive choice. This would indicate how many attributes match between two images. %
However, some functions like fashion compatibility, where images may match \emph{because} they contain different attributes, $XNOR$ would not be appropriate since any non-shared attributes would be ignored (\ie have a supervision label $0$). $AND$, which encourages models to predict $1$ only when both images have the attribute has same issue. However $OR$ would not face the issue since it can be $1$ when either of the images has an attribute, allowing the model to use its relevance weights to decide whether a combination of attributes is relevant for similarity. In Table \ref{tab:att_comb_type}, we see $OR$ perform the best on fashion compatibility on Polyvore Outfits.

Notably, we also see that $OR$ is still competitive with $XNOR$ on the CUB dataset, where the goal is to determine similarity in the more conventional sense, \ie, similar images should have matching attributes. This task seems like a good fit for $XNOR$. 
However, on CUB, many of the attributes are mutually exclusive (\eg, a bird either has a ``red beak'' or a ``black beak'', but not both). 
If a model can reason jointly about dissimilar attributes inferring that they should not coexist, it can correctly function on this task. Relevance weights in our model allow for this joint reasoning over different attributes.
Thus, training the model to predict $1$ when either image has an attribute ($OR$ supervision), can also perform well on this task, and from the empirical results in Table \ref{tab:att_comb_type}, we see that it does. 

Providing both the $AND$ and $XOR$ outputs (as a concatenation) seems lucrative since it seems more informative than $OR$, but we found that the model uses its additional capacity to overfit to training data. This is also challenging to use since attribute predictions are noisy (see Appendix \ref{sec:attr_recog} for attribute recognition performance).

The In-Shop retrieval task involves fetching matches from a gallery of different views of an object. $XNOR$ is ideal for matching different views in this case, since matching attributes can be directly translated to the inference that two views belong to the same object, therefore, are similar.







\subsection{PAN sensitivity to $\lambda$}
\label{subsec:lambda}
Figure \ref{fig:plot-lam} shows PAN's sensitivity  to different weights ($\lambda$) of the attribute supervision loss. Performance is plotted for both the testing and validation sets. On Polyvore Outfits, we see that the PAN-Supervised-GE model performs well when attribute supervision weight is relatively high (best accuracy at $\lambda=10$). Model performance decreases on either side of this, with a heavier decrease when $\lambda$ is increased significantly (to $100$). On the CUB dataset, we see a somewhat different behavior where the best model performance is achieved at $\lambda=10^{-5}$, which is much lower, indicating relatively lower attribute supervision is optimal for the task.  Note that on the In-Shop task we set $\lambda=1$ and did not tune it on that task, demonstrating that PAN can be readily adapted to improve performance on other tasks/models.





\section{Introduction}

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.95\linewidth,trim=0cm 0cm 0cm 0cm,clip]{figures/intro_fig_v2.pdf}
    \caption{In prior work (\eg~\cite{koh2020concept,plummerPLCLC2017,vasileva2018learning,yang2019interpretable}), shown in (a), attributes used for image similarity are predicted for each image and then are used as input to the image similarity model.  However, this can result in loss of important information about how attributes are expressed (\eg, different shades of the attribute \emph{yellow breast}).  Thus, in our work, shown in (b), we avoid this loss of information by using a joint representation of the two images to compute multiple disentangled similarity scores, each corresponding to an attribute, and relevances of each similarity score in the final similarity prediction. This allows for more fine-grained reasoning about different attribute manifestations, boosting performance.}
    \label{fig:comparison}
\end{figure}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/pipeline_v2.pdf}
    \caption{\textbf{PAN overview.}  Given a pair of images, the goal of PAN is produce its similarity score.  We begin by using the image encoder to generate feature vectors for input images. %
    The image features are then fed into the Concept-conditioned Similarity Module (CSM) that uses these features to generate a set of similarity scores with corresponding relevance weights. This enables PAN to identify that two images do contain the same attribute, but that they are not relevant to the similarity score since they are different manifestations of the attribute (see Fig~\ref{fig:comparison} for an example).  The final similarity score $p \in [0, 1]$ is produced using a weighted combination of the similarity conditions and their relevance. Note that the different colored lines (blue, pink) represent information flow pertaining to individual images.  %
    }
    \label{fig:pipeline}
\end{figure*}

Learning similarity metrics between images is a central problem in computer vision with wide-ranging applications such as face recognition \cite{liu2017sphereface,schroff2015facenet}, image retrieval \cite{gordo2016deep,noh2017large,yang2019efficient}, prototype based few shot image classification \cite{koch2015siamese,snell2017prototypical,sung2018learning,vinyals2016matching}, continual learning of image classification \cite{castro2018end,rebuffi2017icarl,stojanov2019incremental}, and fashion compatibility or recommendation  \cite{cucurull2019context,tan2019learning,vasileva2018learning,veit2017conditional,veit2015learning,zhang2018become}. There has been a recent trend of learning these metrics by decomposing the problem into multiple axes of similarity or \emph{similarity conditions}, which has improved performance on a variety of tasks~\cite{HeICDM2016,LIN_2020_CVPR,plummerHint2019,plummer2018conditional,plummerPLCLC2017,tan2019learning,vasileva2018learning,veit2017conditional}. Generally speaking, methods that automatically learn what these conditions represent~\cite{plummer2018conditional,tan2019learning} have reported better performance than those that predefine this knowledge using information like labeled image attributes and item categories~\cite{LIN_2020_CVPR,veit2017conditional,vasileva2018learning,plummerPLCLC2017}. 
We argue this is primarily due to prior work using attributes to predict their presence on single images (\eg~\cite{koh2020concept,yang2019interpretable,vasileva2018learning,plummerPLCLC2017}), and subsequently using these predictions for predicting similarity (Fig \ref{fig:comparison} (a)). This incurs a loss in information about the different manifestations of an attribute, differences that could affect similarity prediction, but may not be distinguishable in attribute annotations. While this could be addressed by collecting a complete set of annotations of every possible attribute and their different manifestations that could affect similarity, such a collection would be expensive.  In addition, it is often impossible to articulate every fine-grained attribute that may affect similarity.



In this paper, we introduce a Pairwise Attribute-informed similarity Network (PAN) that effectively learns to use supervisory information in the form of attribute labels, avoiding information loss, to create a powerful image similarity model that performs well on a range of diverse tasks.   
To illustrate how we do this we refer to the example in Fig \ref{fig:comparison}(b). The figure shows two birds (of different categories) from the Caltech-UCSD Birds (CUB) dataset \cite{wah2011caltech}, where they are both labeled positively for the binary attribute \texttt{has\_breast\_color::yellow}, indicating that they have yellow breasts. Prior work (\eg~\cite{koh2020concept,plummerPLCLC2017,vasileva2018learning,yang2019interpretable}) directly predicts attributes for each image,  which tends to lose information about subtle differences in the manifestations of the attributes, like the shades of the color yellow. 
Our PAN model avoids this issue by first comparing images in a feature space rather than attribute space, as illustrated in Fig~\ref{fig:comparison}(b). Using the joint image features it then predicts both a similarity score and a relevance for different similarity conditions defined by the attributes. Even when the similarity score may coarsely indicate that the two images are similar since they have the same attribute, the model can pick up on finer attribute differences and decide that the mere presence of the same attribute is of low relevance to a positive similarity prediction.
As our experiments will show, this difference can make a dramatic impact on the performance of the learned image similarity model.  






One major challenge we face is the considerable difference in how attributes relate to the similarity functions that arise in different tasks.  For example, in few-shot classification, where we use the labeled support images in a nearest-neighbors classifier, the goal of a visual similarity classifier would be to simply measure similarity by matching attributes between the test and support images.  In contrast, for tasks like fashion compatibility, where two images are deemed similar if they complement each other when worn together in an outfit, image pairs with different attributes (\eg, black and orange) can indicate they are highly compatible.  Furthermore, simply modeling which attribute pairs indicate compatibility is insufficient, since two attributes which often result in compatible pairs could be deemed incompatible depending on the other attributes that are present.  For example, \emph{black} and \emph{orange} items are often compatible, except when some other attributes like \emph{red} are also present.  Thus, visual similarity models must learn a far more complex set of relationships between attributes when learning fashion compatibility.  These differences mean that methods that do well on few-shot classification often perform poorly on fashion compatibly and vice-versa. PAN, however, can take this into account via the method we use to convert the incomplete attribute labels for single images into supervisory signals for image pairs and improve performance across diverse tasks.

As we will discuss in Sec \ref{sec:model} (and illustrated in Fig \ref{fig:pipeline}), PAN naturally allows for training and automatically learning similarity conditions in the absence of any additional data like attributes. We also find that PAN can improve performance even in cases where only sparse attribute labels are available.






Summarizing our contributions:
\begin{itemize}
    \item We propose a Pairwise Attribute-informed similarity Network (PAN), which incorporates fine-grained attribute information during training based on a joint representation of two images enabling us to avoid the loss of information suffered by prior work.
    \item While prior methods of incorporating attribute information under-perform prior art, PAN outperforms them on three diverse tasks---by 4-9\% on fashion item compatibility prediction on Polyvore Outfits~\cite{vasileva2018learning}, 5\% on few shot classification on CUB \cite{wah2011caltech} and over 1\% Recall@1 on In-Shop Clothing Retrieval~\cite{liu2016deepfashion}, demonstrating PAN's generality. In comparison to prior approaches of incorporating attribute supervision, PAN is better by a wider margin, \eg, it outperforms them by a sizeable 6-17 \% on Polyvore Outfits.
    \item We propose different methods of using attributes for supervising predictions along similarity conditions, delving into the interpretations of each, providing insights for their applicability in different tasks.  
    \item Our analysis also outlines the contributions of the training procedures, specifically training batch-size. This has commonly been overlooked in prior work, but could have significant impact on final model performance. In doing so, we factor out contribution of the training procedure in demonstrating PAN's benefits. 
\end{itemize}
























\section{Pairwise Attribute-informed similarity Network (PAN)} \label{sec:model}




 Given two images $\boldsymbol x_1$ and $\boldsymbol x_2$, the goal of our PAN model is to output a score in $[0, 1]$ representing the probability that the two images are similar.  The primary contribution of our approach lies in its Concept-conditioned Similarity Module (Sec~\ref{subsec:dec-desc}), which takes features  $\boldsymbol h_1, \boldsymbol h_2 \in \mathbb{R}^d$ representing the input images (computed using an image encoder from Sec~\ref{subsec:enc-desc}), and predicts their similarity $p$ using a weighted combination of predictions along multiple axes of similarity $\boldsymbol\rho$ and their relevance $\boldsymbol\omega$.  As discussed in the introduction, learning relevance weights over the attribute defined similarity conditions can help us selectively ignore them when said attributes are present, but are not quite relevant in determining similarity between the images. 
 
As our experiments demonstrate, the similarity conditions can be unsupervised as in prior work~\cite{plummer2018conditional,tan2019learning}, but the aim of PAN is to learn to supervise these conditions so they represent a particular concept (\ie, a particular attribute).  These attributes can represent any concept that has been annotated in a dataset.  For example, for CUB~\cite{wah2011caltech} these represent parts of birds (\ie, the type of beak or tail features).  In Polyvore Outfits~\cite{vasileva2018learning} and InShop Retrieval~\cite{liu2016deepfashion} these attributes can contain low-level concepts like colors and textures as well as high-level concepts like ``formal'' and ``fashionable''.  We discuss how we convert the attributes/concepts, which are annotated per image, into labels for image pairs in Sec~\ref{subsec:def-similarity-conds}. Fig \ref{fig:pipeline} provides an overview of our approach.
 
 








\subsection{Concept-conditioned Similarity Module (CSM)} \label{subsec:dec-desc}



Given features $\boldsymbol{h}_i, \boldsymbol{h}_j$ $\in \mathbb{R}^{d}$ of two images, our Concept-conditioned Similarity Module (CSM) generates a set of $M$ similarity scores  $\boldsymbol{\rho} = [\rho_1, \ldots, \rho_M] \in \mathbb{R}^M$ and corresponding relevance weights $\boldsymbol{\omega} = [\omega_1, \ldots, \omega_M] \in \mathbb{R}^M$ which represent the importance of each similarity condition to the final similarity score:

\begin{align}
    \boldsymbol{\rho} &= \sigma \left( \boldsymbol{W}_1^\top |\boldsymbol{h}_i - \boldsymbol{h}_j| + \boldsymbol{b}_1 \right) \\ 
    \boldsymbol{\omega} &= \text{softmax} \left( \boldsymbol{W}_2^\top |\boldsymbol{h}_i - \boldsymbol{h}_j| + \boldsymbol{b}_2 \right)
\end{align}
where $M$ is the number of distinct similarity conditions, $|\cdot|$ represents an element-wise absolute value, and $\sigma(\cdot)$ an element-wise sigmoid function. $\boldsymbol{W}_1, \boldsymbol{W}_2 \in \mathbb{R}^{d \times M}$, and $\boldsymbol{b}_1, \boldsymbol{b}_2 \in \mathbb{R}^{M}$ are learnable parameters.  Note that $\boldsymbol\rho$ is supervised using attribute labels, but the relevance scores $\boldsymbol\omega$ are treated as a latent variable and automatically learned.  
The final similarity score $p \in [0, 1]$ is calculated as the sum of similarity conditions weighed by their relevance, \ie,
\begin{align}
    p = \sum_{m=1}^{M} \rho_m \omega_m = \boldsymbol{\rho}^\top \boldsymbol{\omega}.
\end{align}

Note that prior work that predicted multiple similarity conditions in both the supervised and unsupervised setting did so based off the features of a single image (\eg,~\cite{LIN_2020_CVPR,plummerHint2019,vasileva2018learning,veit2017conditional}). In contrast, CSM predicts these conditions off a joint representation of both images.  As we show in Appendix \ref{sec:more_questions}, this results in a significant boost to performance when combined with relevance scores.  We believe this is due, in part, to the fact that this joint representation makes it easier to identify differences in attribute manifestations (due to taking the difference of features for the two images). Thus, our approach can more accurately identify when to ignore attribute predictions.


\subsubsection{Defining Similarity Conditions} \label{subsec:def-similarity-conds}

Depending on the availability of labelled attributes for the images, we can choose to supervise similarity conditions to give them semantic meaning. This choice results in two kinds of similarity conditions as described below: 
\smallskip

\noindent\textbf{Unsupervised similarity conditions.}  Similarity conditions are treated as latent variables as done in \cite{tan2019learning}.  The benefit of this approach is that it requires no additional annotations.  Note that all the conditions we predict are based on a joint representation of the two images, unlike in \cite{tan2019learning} where they were computed per-image.
\smallskip

\noindent\textbf{Supervised similarity conditions.} Unsupervised similarity conditions need no attribute annotations. However, we would expect that with some expert knowledge of what might be important in the image, as provided with attributes, we can improve performance using these attributes effectively. Hence, rather than treating each similarity condition as a latent variable, supervised similarity conditions are trained to reflect a specific concept. Since attribute annotations are defined per image, and we predict attributes based off a joint representation, we convert these labels to represent both images, as described next.

Suppose the images have $M$ labeled binary attributes. Each image $i$ is then accompanied with an $M$ dimensional vector $\boldsymbol{a}_i \in \{0, 1\}^{M}$. For a pair of images $i$ and $j$, we can use a function $f_{a} : \{0, 1\} \times \{0, 1\} \rightarrow [0, 1]$ to get an $M$ dimensional vector $\boldsymbol{a}_{i, j} = f_{a}\left( \boldsymbol{a}_i, \boldsymbol{a}_j \right)$. Elements of $\boldsymbol{a}_{i, j}$ can then be used as labels for supervising the similarity conditions in the model output scores $\boldsymbol{\rho}$. Note that if there are missing entries in $\boldsymbol{a}_{i, j}$ because of missing attribute labels, these can be handled by zeroing out the loss resulting from them using a binary mask over the indices of $\boldsymbol{a}_{i, j}$. %

In Section~\ref{subsec:logical_function}, we experiment with common logical functions as $f_a$, which map to clear semantic meanings. For instance, using $f_a = logical~AND$, with the similarity score $\boldsymbol \rho$, the model is asked to predict whether a given attribute appears in both images. Similarly with $OR$, model predicts whether the attribute is in either image, with $XOR$ it predicts whether it is exclusively in one image, and with $XNOR$ it predicts whether the attribute is in both images or in neither. A more detailed discussion of opting for these 4 choices for $f_a$ is in Appendix \ref{sec:more_questions}.  In practice for a given dataset, a logical function can be selected using some prior knowledge about how the attributes relate to the similarity score, or can be selected empirically using held-out data.







\subsection{Types of Image Encoder} \label{subsec:enc-desc}

As mentioned previously, the image encoder generates a lower dimensional feature representation $\boldsymbol h$ for an image $\boldsymbol x$. We experiment with three different image encoders.

\smallskip

\noindent\textbf{Convolutional Network.}  Unless specified otherwise, we use a simple convolutional neural network (CNN), specifically a ResNet~\cite{he2016deep} to obtain our image feature representation (details in the Appendix \ref{sec:implementation_details}).
\smallskip

\noindent\textbf{Graph Encoder (GE)~\cite{cucurull2019context}.} For some image similarity tasks like fashion compatibility, context can be an important cue in determining how similar two items are. Thus, the second encoder we explore is a graph convolutional network (GCN) that operates on features extracted by a CNN. The GE (composed of the CNN and GCN) takes in as input both images from a dataset and an adjacency matrix over them and simultaneously generates features for all images. This encoder was also used by Cucurull~\etal~\cite{cucurull2019context} and we refer readers to Section 3.1 of their paper for complete details. 
\smallskip

\noindent\textbf{ProxyNCA++~\cite{wern2020proxynca++}.} Many tasks may also find context unhelpful or that GCNs may be too computationally expensive to use~\cite{LIN_2020_CVPR}.  For example, in retrieval tasks a particular emphasis is placed on speed as methods may have to search through millions of images in order to locate the desired item.  As such, for our last encoder we use a state-of-the-art retrieval method when evaluating on the In-Shop Retrieval task~\cite{liu2016deepfashion}. ProxyNCA++ at its core learns a distance metric between images based on learning proxy feature representations for each class. Consequently, it relies on annotated categories for images, and cannot directly be applied for similarity metric learning where no such annotation is available. We refer readers to \cite{wern2020proxynca++} for complete details of this encoder.

\subsection{Model Objective and Training} \label{subsec:model-objective}

The final objective function on a pair of images $\boldsymbol x_i$ and $\boldsymbol x_j$ is then defined as:
\begin{align}
    \mathcal{L}(\boldsymbol{x}_i, \boldsymbol{x}_j, e_{i, j}, \boldsymbol{a}_{i, j}) = \mathcal{L}^{BCE} (e_{i, j}, p) + \lambda \mathcal{L}_{el}^{BCE} (\boldsymbol{a}_{i, j}, \boldsymbol{\rho}),
    \label{eq:total_loss}
\end{align}
where $\lambda$ is a tunable hyperparameter, $e_{i, j} \in \{ 0, 1 \}$ is the ground truth similarity label between images $\boldsymbol x_i$ and $\boldsymbol x_j$, $\mathcal{L}^{BCE}$ is the binary cross-entropy loss and $\mathcal{L}_{el}^{BCE}$ is the mean element-wise binary cross-entropy. Note that when there are no supervision attributes, the second term in Equation \ref{eq:total_loss} is $0$. For training, an equal number of positive and negative pairs are sampled randomly from the training split and the model is trained to predict similarity between them. Details regarding the exact process for each encoder are in Appendic \ref{sec:implementation_details}. 














\section{Related Work}


\noindent\textbf{Visual Similarity Learning.} Learning visual similarity can be used for a wide range of visual tasks. A couple widely used evaluation tasks for similarity learning are face recognition \cite{chopra2005learning,liu2017sphereface,nguyen2010cosine,schroff2015facenet} and image retrieval \cite{noh2017large,gordo2016deep,yang2019efficient}, the latter itself subcategorized multiple ways depending on type of images involved, \eg fashion images, natural images. Image retrieval has a fairly direct application in e-commerce. 
A major portion of this industry consists of websites selling clothing and accessories, giving rise to the challenging task of predicting fashion compatibility \cite{veit2015learning, kang2017visually, vasileva2018learning}, which, as discussed in the introduction, is a form of visual similarity different from the conventional notion. However, we use similarity and fashion compatibility interchangeably as they are addressed in the same manner in our paper.

Some prior visual similarity learning approaches focused on learning a single similarity space~\cite{veit2015learning,kang2017visually,han2017learning}. More recent work \cite{veit2017conditional,plummerHint2019,vasileva2018learning,tan2019learning,LIN_2020_CVPR} found that learning multiple \emph{similarity conditions}, each capturing a different concept, performs better. Concurrent to some of these, the role of using contextual information in image encodings was discussed by Cucurull~\etal~\cite{cucurull2019context}. They used a graph convolutional network (GCN) on images, with similarity links defining the graph structure, to achieve state of the art performance in fashion compatibility prediction. 

Up until now, methods using predefined similarity conditions~\cite{plummerHint2019,veit2017conditional,vasileva2018learning} have underperformed methods that automatically learned these conditions~\cite{tan2019learning,LIN_2020_CVPR}.  Via PAN, we show a method of incorporating additional attribute annotations to supervised similarity conditions while improving final task performance, breaking this trend.
\smallskip





 


\noindent\textbf{Few Shot Learning.} 
Given the cost associated with acquiring human-annotated labels, learning with few labelled examples is well sought after in computed vision, with a range of prior work exploring possible solutions~\cite{miller2000learning,li2006one,lake2011one,koch2015siamese}. Given a few examples with labeled classes, \cite{vinyals2016matching} classified novel examples using attention weights to compute a probability distribution over known classes. They introduced an episodic training paradigm, 
later also adopted by \cite{ravi2016optimization,snell2017prototypical}. Each episode in an N-way K-shot classification task is a sample of N classes, with K images from each class available to the learning algorithm as ``labeled'' or ``support'' examples for learning few shot classification. Each episode is also accompanied by some query examples which the learner is supposed to predict class labels for. For training, the learning algorithm is provided episodes sampled from a base training dataset, and it is typically evaluated on test episodes sampled from a novel split of the dataset, containing classes different from those appearing in the base split. Both \cite{vinyals2016matching} and \cite{snell2017prototypical} adopted a strategy of minimizing a distance metric in feature space between query images and a prototypical support vector for training. Subsequent work \cite{sung2018learning} improved few shot classification performance by learning a parametric distance measure rather than using a closed-form metric without learnable parameters. 

Another body of work uses meta-learning \cite{finn2017model,Qiao_2019_ICCV,Sun_2019_CVPR} to initialize a classification model's parameters allowing for fast adaptation to a new few-shot task using just a few parameter updates. \cite{chen2019closer} performed a thorough study of several recent few-shot learning methods and proposed a strong baseline classifier using cosine similarity and data augmentation. %
Some recent methods have also employed graph neural networks for few shot classification~\cite{garcia2018fewshot,Kim_2019_CVPR,Schonfeld_2019_CVPR}.  

Like \cite{veit2017conditional, vasileva2018learning, tan2019learning}, PAN lies in the space of decomposing similarity prediction into multiple similarity conditions, but unlike them, PAN can use attribute supervision when available to supervise these conditions and improve performance. PAN is a general similarity learning approach and its benefits with multiple similarity condition spaces and attribute supervision can be seen with different image encoders, be it a simple CNN or a contextual GCN encoder similar to that used by \cite{cucurull2019context}, and on different tasks, like fashion compatibility and few shot classification.







