\subsection{Retrieval of style files}

Although deep-learning vision models can sometimes predict aspects of human vision
\citep[e.g.,][]{Dubey2015,Elsayed2018,Rajalingham2018,Feather2019,Golan2020}, their behavior often contrasts sharply
with human expectations. For instance, small perturbations that 
are imperceptible to humans can dramatically affect model classification  decisions \citep{Goodfellow2015};
and texture and local image features drive classifiers \citep{Baker2018,Geirhos2019}, whereas
humans are more strongly influenced by Gestalt shape.
Given that differences exist in how humans and machines represent the world,
our goal is to develop techniques that bring their representations into better correspondence.
This goal is important for two reasons. First, human vision is robust and visual
representations contain a wealth of information about objects and their properties.
Bringing representations into alignment might expand the range of tasks for which 
deep nets are useful \citep[e.g.,][]{Hsieh2017,Roads2020,Wilber2015}. Second, if 
the correspondence is strong, deep nets can serve as a human surrogate for prediction and
optimization, allowing us to efficiently determine, say, the best training procedures for people
\cite{Davidson2020,RoadsMozer2020,Peterson2018,Sanders2020}.

Let's be more specific about the representations that need to be aligned.
In a deep net trained to classify images, the representation in the penultimate layer 
(prior to the softmax layer) serves as a \emph{deep embedding} of the image. This 
representation necessarily contains the features essential for discriminating object categories.  
One might hope to align this representation with the activity pattern in higher cortical
areas of the human brain, i.e., a \emph{neural embedding}, but it is not feasible 
to read out large-scale brain activation at a sufficiently fine spatial and temporal resolution.
Instead, one might hope to align \emph{psychological embeddings}---a representation of the
features essential for human classification, judgment, decision making, and information processing.

A common method to obtain psychological embeddings requires collecting similarity ratings between
pairs of items in a domain and then inferring an embedding in which more
similar pairs are closer in the embedding space than less similar pairs.
Multidimensional scaling (MDS) \cite{Torgerson1958} has been used for over half a century
to obtain psychological embeddings for a fixed set of items whose pairwise similarity matrix is 
provided. Even at a large scale, it can obtain low-dimensional interpretable embeddings that 
generalize to behavioral tasks \cite{Hebart2020}.
Although MDS can be used given partial or noisy similarity matrices, it is not
productive in the sense that it allows one to predict representations
and similarities only for items contained in the original similarity matrix. Ideally, one
desires an open-set method, not one that works only for the fixed, previously rated set.

\vspace{-.07in}
\section{Background and Related Research}
\vspace{-.05in}

Toward the goal of being able to obtain a psychological embedding for \emph{any} image, methods have been proposed that leverage 
human similarity judgments in conjunction with deep nets. These nets have the advantage  over MDS that they can in principle embed 
novel images.

Sanders and Nosofsky \cite{Sanders2020} trained a fully-connected net to re-map from a deep embedding
of a pretrained classifier to an MDS representation. They found that the resulting re-mapping
generalized well to images held out from the re-mapping training set. While this approach demonstrates
that psychological embeddings can be extracted from deep embeddings, the approach is limited in that 
it produces a representation that is no richer than the MDS embedding. The dimensionality of MDS embeddings 
is limited by the quantity of human judgment data available; deep embeddings are not.

Peterson, Abbott, and Griffiths \cite{Peterson2018} bypassed the MDS embedding and used a deep embedding
to directly predict human similarity judgments. These judgments, made on image pairs
using a 0--10 scale with larger values indicating greater similarity, were placed into a
symmetric similarity matrix $\bm{S}$, where element $s_{ij}$ is the judgment for 
image pair $i$ and $j$. The matrix is modeled with $\smash{\hat{\bm{S}}}$, where
\begin{equation}
\hat{s}_{ij} = {\bm{z}}_i^{\mathrm{T}} ~ {\bm{W}} ~{\bm{z}}_j ,
\label{eq:peterson}
\end{equation}
and ${\bm{z}}_i$ and ${\bm{z}}_j$ are the deep embeddings for images $i$ and $j$, and ${\bm{W}}$
is a diagonal matrix. The parameters of ${\bm{W}}$ are obtained
by optimization of a squared loss, $\smash{|| \S - {\hat{\bm{S}}}||^2}$, with an $L_2$ regularizer
to prevent overfitting. Peterson et al.\ found that the best fits to human data are obtained
by a VGG architecture \cite{Simonyan2015}, whose embedding layer is 4096 dimensional.

Peterson et al.\ placed no constraint on the sign of the elements of ${\bm{W}}$. However, with non-negative elements, ${\bm{W}}$ can be decomposed as ${\bm{V}}^{\mathrm{T}} {\bm{V}}$, and Peterson et al.'s method can be 
viewed as a form of deep metric learning \cite{Hoffer2015,Kaya2019}. That is, the similarity function can
be interpreted as computing a dot product of linearly transformed embeddings, i.e.,
$\hat{s}_{ij} = ({\bm{V}} {\bm{z}}_i)^{\mathrm{T}} ({\bm{V}} {\bm{z}}_j)$. In this case, the linear transform rescales individual
features (vector elements) of the embedding. Such a transform makes the most sense if these features can be ascribed
psychological meaning: when comparing vectors, the rescaling permits some features to matter more, some less. 
However, the basis used for representing the embedding is completely arbitrary: any rotation of the basis is a functionally
equivalent solution to the classifier (because a linear transform of the embedding
is performed in the softmax output layer). Consequently, we question whether it is well motivated to
restrict transforms to dilating a representation that lies in an arbitrary basis. Peterson et al.\ likely made
this choice because a full ${\bm{W}}$ would have 16M parameters and would be underconstrained by the
relatively small number of human judgments. We describe a potential solution to this dilemma that both gives
the embedding a non-arbitrary basis---hopefully one with psychological validity---and allows us to vary
the number of free parameters in the model.

We have two further concerns with Peterson et al.'s method which our solution addresses. First, Peterson et al.\ 
$z$-score normalize the embeddings from the pretrained model before using the embeddings to compute similarity.
Variance normalization does not matter because any such normalization can be inverted via ${\bm{W}}$. However, the zero centering 
of $z$-scoring alters the angles between embedding vectors, and those angles are not arbitrary: the original classifier uses
these angles to compute softmax probabilities. Second, Peterson et al.\ use absolute similarity ratings for model training. Such 
ratings are subject to sequential dependencies \cite{Mozer2007}, reducing the signal they convey. Relative 
judgments---of the form `is $X$ more similar to $Y$ than to $Z$?'---tend to be more reliable, despite outwardly conveying
less information \cite{Demiralp2014,Li2016,Wilber2014}.


\vspace{-.06in}
\section{Methodology}
\vspace{-.09in}
\subsection{Data set}
\vspace{-.04in}
We used a previously collected data set of similarity judgments for bird images \cite{RoadsMozer2020}. The image set
contains four bird families (Orioles, Warblers, Sparrows, and Cardinals), four distinct species within each family, and thirteen
distinct images of each species. Examples are presented in Figure 2 of \cite{RoadsMozer2020}.  Mechanical Turk participants were shown 
a  \emph{query} image along with two \emph{reference} images and chose which reference was most similar
to the query.  See the left edge of Figure~\ref{fig:model} for a sample triple. The resulting judgment providess a 
\emph{triplet inequality constraint} (\emph{TIC}). Some participants were shown the query with \emph{eight} reference images and 
were asked to choose the \emph{two} most similar. Data from these trials provide 12 TICs: each of the two chosen references is more 
similar to the query than each of the six non-chosen references. The complete data set consisted of 112,784 TICs.



\subsection{Models to be evaluated}

\begin{SCfigure}[10][b!]
\centering
\includegraphics[width=.75\linewidth]{figures/fig1.pdf}
\caption{We model human triplet judgments of the form `Is the query more similar to reference 1 or 2?' After embedding each image,
pairwise similarities are computed and the relative similarity determines the probability of choosing one reference or the other.}
\label{fig:model}
\end{SCfigure}
Figure~\ref{fig:model} sketches the structure of our approach. To obtain deep embeddings, we use a headless VGG16 classifier pretrained
on ImageNet from the Keras library.  The penultimate layer of VGG16 has 4,096 units. To model the TIC, we pass the query and two 
references through VGG16 and then compute pairwise similarities between the query and each of the references.
For a query $q$ and reference $r$, we generalize the learned similarity function of Equation~\ref{eq:peterson} as follows:
\begin{equation}
\hat{s}_{qr} = f({\bm{z}}_q)^{\mathrm{T}} ~ {\bm{W}} ~f({\bm{z}}_r),
\label{eq:ours}
\end{equation}
where $f: \mathbb{R}^{4096}  \rightarrow \mathbb{R}^k$ performs dimensionality reduction on the original 4096-dimensional deep embedding.
Variants of this model are specified via choice of $f(.)$, $k$, and the constraints placed on ${\bm{W}}$.  We explore these specific constraints on ${\bm{W}}$:
\begin{itemize}[leftmargin=*]
    \item \emph{Identity}: Use original deep embedding space via ${\bm{W}} = \bm{I}$. This case serves as a baseline.
    \item \emph{Diagonal}: Rescale the original embedding with a diagonal matrix.  In contrast to Peterson 
    et al., We require the diagonal elements to be non-negative by optimizing for an unconstrained diagonal vector $\v \in \mathbb{R}^k$ and using ${\bm{W}}=\mathrm{diag}(|\v|)$. We used this constraint because
    it seems antithetical to posit that greater alignment of representations should reduce similarity.
    \item \emph{Symmetric}: Require symmetry of the $k\times k$ matrix, which allows us to interpret the similarity function as 
    applying an arbitrary linear transform to each embedding and then computing their dot-product similarity (see Introduction). 
    We optimize over an unconstrained matrix, ${\bm{V}} \in \mathbb{R}^{k\times k}$,  where ${\bm{W}} = {\bm{V}}^{\mathrm{T}} {\bm{V}}$.
    This approach is related to that of Ryali et al.\ \cite{Ryali2020}, who learn the parameters of a covariance matrix to compute the Mahalanobis distance between representations. We question whether a Mahalanobis distance
    is the appropriate metric when deep embeddings are trained and used to classify via dot-product softmax functions.
    \item \emph{Unconstrained}: We optimize directly over an unconstrained ${\bm{W}} \in \mathbb{R}^{k \times k}$.
\end{itemize}

In picking $k$ and $f(.)$, we have two goals. First, we wish to reduce the number of free parameters in our models to avoid overfitting the
training data. Second, we wish to impose a basis that is less arbitrary than that of the original deep embedding. As we argued earlier, 
the basis obtained from training VGG is arbitrary; any rank-preserving linear transform is an equivalent solution given that this
transform could  be inverted in the softmax layer to achieve the same output. If ${\bm{W}}$ is constrained to be diagonal, it seems desirable
for the dimensions rescaled by ${\bm{W}}$  to have some psychological reality. To achieve these two goals in the simplest manner possible,
we treat $f$ as the projection of the deep embedding onto the top $k$ principal components of the embedding space. PCA is performed
on the embeddings of 23,400 images of the 18 bird classes in the ImageNet training set. Note that these images are distinct from 
those used for similarity assessment.

To obtain a prediction for the human judgment, we follow a long tradition in choice modeling and assume a logistic function of the 
relative similarities for query $q$ and references $r_1$ and $r_2$:
\begin{equation}
    \Pr(\mathrm{choose~ref}_c | ~q, r_1, r_2) = \mathrm{logistic}(\hat{s}_{q,r_c} - \hat{s}_{q,r_{3-c}})  .
\end{equation}
Note that because of the linearity of $f$ and $\hat{s}$,
\[
\textstyle \Pr(\mathrm{choose~ref}_c | q, r_1, r_2) = \mathrm{logistic}\left[f({\bm{z}}_q)^{\mathrm{T}} {\bm{W}} f({\bm{z}}_{r_c} - {\bm{z}}_{r_{3-c}}) \right] . 
\]

\vspace{-.09in}
\subsection{Simulation details}

Models are trained to maximize log likelihood of the training triples,
\[
\textstyle  \ell = \sum_{(q, r_1, r_2, c) \in \mathcal{T}} \log \Pr(\mathrm{choose~ref}_c | ~q, r_1, r_2)
\]
where $c$ is the index of the reference chosen by the human rater and $\mathcal{T}$ is the training set.
Five-fold cross-validation was performed, yielding 90k and 22.5k triplets in each fold for the training and validation sets, respectively.
We trained models using TensorFlow with an SGD optimizer, with Nesterov momentum of 0.9, an learning rate of  $10^{-9}$ for the Unconstrained model and $10^{-5}$ for all others.
Although trained with likelihood maximization, we evaluate models on \emph{accuracy}, defined as the proportion of 
examples in which the human response matches the most likely model response.  As weak prevention against overfitting, we stopped 
training when the training accuracy did not improve in the last 10 epochs over the 10 epochs previous.  

\vspace{-.04in}
\section{Results}
\vspace{-.05in}

\begin{figure}[b!]
\centering
\includegraphics[width=.48\linewidth]{figures/train_results.pdf}
\hspace{.02in}
\includegraphics[width=.48\linewidth]{figures/eval_results.pdf}
\caption{(a) Training and (b) validation performance as a function of the number of principal components included 
in the embedding ($k$).  Each curve represents a different constraint on ${\bm{W}}$. Error bars reflect $\pm1$ SEM on the five-fold
cross-validation procedure.}
\label{fig:results}
\end{figure}
Figure~\ref{fig:results} shows the outcome of five-fold cross-validation on our data set of human similarity judgments of bird images.
The left and right panels show training and validation set accuracy, respectively. A model prediction is scored as correct
if the model's probability of selecting the reference chosen by the human is greater than 0.5.  Accuracy is plotted as a function of
the number of principal-component loadings included in the deep embedding ($k$), and a separate curve is drawn for each different
constraint on ${\bm{W}}$ that we tested.  The dashed blue line is an implementation of Peterson et al.'s
method: diagonal weights, allowing negative values, the full 4096-element vector, and imposing an $L_2$
penalty on the weights. To give this model its best shot, we searched for the $L_2$ regularization coefficient over many orders of magnitude that maximized performance on the validation set (i.e., we cheated to benefit this method); we also did not $z$-score the embeddings for reasons explained earlier.

Validation performance roughly tracks training performance. However, for the most complex models (Symmetric and Unconstrained
${\bm{W}}$ and large $k$) there appears to be some overfitting: the training curve rises faster than the validation curve.
We expected to observe more severe overfitting, manifested by a \emph{drop} in validation performance with $k$, because
the largest models have a nearly 185:1 (Unconstrained, $k=4096$) and 93:1 (Symmetric, $k=4096$) ratio of free parameters
to training examples (TICs). Contrary to our predictions, the inverted U function is not observed,
possibly due to the linear form of the model, which offers a strong constraint on data patterns that can 
be fit, or to inconsistency in the human judgments that results in violations of transitivity, not fittable 
by our models.
The validation curves in Figure~\ref{fig:results}b allow us to draw some strong and intriguing conclusions:
\vspace{-.08in}
\begin{itemize}[leftmargin=*]
\item \emph{Dilation of the transformed deep-embedding features (blue curves) achieves a significantly better fit to the human data than
by simply using the deep embedding straight from the VGG16 classifier (grey curve).} This result replicates the key finding of
Peterson et al.\ on a new data set and a different response measure. Our work extends Peterson et al.\ by using the transform $f(.)$ and
showing that increasing the dimensionality of the embedding strictly improves the fit to human data, and
that adding the non-negativity constraint slightly improves results (for $k=4096$).
\item \emph{Applying a general linear transform to the deep embedding (purple curve) obtains a better fit to the human data than
a dilation (blue curves).} Peterson et al.\ did not investigate using the broader class of transform because it seemed likely that
overfitting would occur. However we see a strict improvement in performance with the more complex model, regardless of $k$.

\item \emph{Relaxing the symmetry constraint on similarity (i.e., the similarity of A to B does not have to equal the
similarity of B to A) improves the fit to human data (red versus purple curve).} This finding was most surprising to us,
but in retrospect might have been anticipated by the prominent, longstanding finding that human judgments of similarity 
cannot be accounted for by the use of an internal psychological distance metric \cite{Tversky1977}.  For example, individuals
might judge North Korea to be more similar to China (focusing on the leadership) than China is to North Korea (focusing on the
size of the country). To cast this claim
in terms of the judgments we are modeling, consider two different triplets: (query $I_1$, references $I_2$ and $I_3$)
and (query $I_2$, references $I_1$ and $I_3$). The deep embedding of $I_1$ and $I_2$ are interpreted differently depending
on whether they are in the role of query or reference.
\end{itemize}

\begin{wrapfigure}{r}{0.45\textwidth}
    \vspace{-.18in}
    \centering
    \vspace{-.4cm}
    \begin{subfigure}{0.45\textwidth}
    \centering
      \includegraphics[width=\linewidth]{figures/eval_kfold_strategies.pdf}
    \end{subfigure}
  \vspace{-.2cm}
  \caption{Prediction accuracy for $k=2048$ with held-out triplets vs.\ held-out images}
  \label{fig:holdout}
\vspace{-.5cm}
\end{wrapfigure}
In the above results, human-prediction accuracy is assessed on held-out triplets (TICs). To evaluate accuracy for held-out \emph{images},
we ran validation folds in which we randomly selected images to hold out such that the training set was roughly the same
size as in our earlier experiment. To best match our earlier experiment, we perform five fold validation (sampling with replacement each fold)
for both held-out triplets and held-out images with $k=2048$. As Figure~\ref{fig:holdout} indicates, models do generalize to new images.
The ranking of models is the same, but performance does suffer on new images.

\vspace{-.09in}
\section{Discussion}
\vspace{-.06in}
Our models of similarity judgment are able to predict 90.3\% of human binary choices, suggesting that a deep embedding
from a pretrained classifier can be adapted to capture the structure of psychological embeddings of visual images. 
By applying a linear transform to a deep embedding, we are able to boost the accuracy of prediction from a 67.8\% baseline
using the original embedding. We significantly improve on an existing approach in the literature \cite{Peterson2018}, 
which achieves a prediction accuracy on our data set of only 78.0\%.

Our simulations reveal several surprising and intriguing results. First, we observe that overfitting is not
a serious issue for highly overparameterized linear models---models with forty times as many free parameters as training data 
points. We are presently investigating whether this finding is due to model linearity (which restricts the transformations
that can be applied to the deep embedding).
Second, and most notably, we observe a benefit for encoding similarity in form that cannot be expressed in terms of
distance metric in the embedding space. Rather, one item of a pair is treated as an anchor with respect to which the
other item is compared. To the best of our knowledge, researchers in machine learning who model human similar have done
so based on distance metrics that do not permit the sort of asymmetry supported by our data \cite[e.g.,][]{Hsieh2017,Roads2019,Wilber2014,Wilber2015}.
Considering the role of anchoring and context in choice is a productive avenue for future research.


\begin{ack}
We thank Gamaledin Elsayed for helpful feedback on the manuscript.
\end{ack}

