\section{Introduction}
\input{sections/1_introduction}

\section{Method}
\input{sections/2_method}

\section{Experiments}
\input{sections/3_experiment}

\input{sections/4_ablation}

\input{sections/5_relatedwork}

\section{Conclusion}
\input{sections/6_conclusion}









\subsection{Object-Centric Scene Hierarchy}
From simple object co-occurrence statistics~\cite{galleguillos2008object,mensink2014costa} to finer object relationships~\cite{johnson2015image,krishna2017visual}, using hierarchical relationships between objects and scenes to understand images is not new. Previous studies primarily work on an image-level hierarchy by dividing an image into its lower-level elements recursively: a scene contains multiple objects, an object has different parts, and each part may consist of even lower-level features~\cite{parikh2007hierarchical,choi2010exploiting,hinton2021represent}. While this is intuitive, it describes a hierarchical structure contained in the individual images. 
Instead, we study the structure presented among different images. Our goal is to learn a representation space for images of both objects and scenes across the entire dataset. To this end, we argue that it is more natural to consider an \textit{object-centric hierarchy}.


It is known that when training an image classifier, the objects from visually similar classes often lie close to each other in the representation space~\cite{wu2018unsupervised}, which has become the cornerstone of contrastive learning.
Motivated by this observation, we believe that the representation of each scene image should also be close to the object clusters it consists of. However, modeling scenes requires a much larger volume due to the exponential number of possible compositions of objects. Another way to think about the object-centric hierarchy is through the generality and specificity as often discussed in the language literature~\cite{miller1990introduction,nickel2017poincare}. An object concept is general when standing alone in the visual world, and it will become specific when a certain context is given. For example, ``a desk'' is thought to be a more general concept than ``a desk in a classroom with a boy sitting on it''. 

Therefore, we propose to study an object-centric hierarchy across the entire dataset. Formally, given a set of images $\mathcal{S} = \{s_1, s_2, \cdots, s_n\}$, $\mathcal{O}_i = \{o_i^1, o_i^2, \cdots, o_i^{n_i}\}$ are the object bounding boxes contained in the image $s_i$. We define the regions of scene $\mathcal{R}_i = \{r_i^1, r_i^2, \cdots, r_i^{m_i}\}$ to be partial areas of the image $s_i$ that contain multiple objects such that $r_i^j = \displaystyle\cup_k  o_i^k$, where $o_i^k\in \mathcal{O}_i \text{ and object $k$ is in the region } j$. We define the object-centric hierarchy $T = (V, E)$ to be that $V = \mathcal{S} \displaystyle\cup \mathcal{O} \displaystyle\cup \mathcal{R}$, where $ \mathcal{R} =\mathcal{R}_1\cup \cdots \cup \mathcal{R}_n$ and $\mathcal{O} =\mathcal{O}_1\cup \cdots \cup \mathcal{O}_n$. For $u, v\in V$, $e=(u, v)$ is an edge of $T$ if $u \subseteq v$ or $v \subseteq u$. Note that the natural scene images $\mathcal{S}$ are always put as the leaf nodes.


\subsection{Representation Learning beyond Objects}
To describe our proposed model based on this hierarchy, we begin with a brief review of hyperbolic space and its properties used in our model. For comprehensive introductions to Riemannian geometry and hyperbolic space, we refer the readers to \cite{do1992riemannian,lee2018introduction}. 
\subsubsection{Hyperbolic Space}

A hyperbolic space $(\mathbb{H}^m, g)$ is a complete, connected Riemannian manifold with constant negative sectional curvature. These special manifolds are all isometric to each other with the isometries defined as $O^+(m, 1)$. Among these isometries, there are five common models that previous studies often work on~\cite{cannon1997hyperbolic}. In this paper, we choose the Poincaré ball $\mathbb{D}^{n}:=\left\{p \in \mathbb{R}^{n} \mid \|p\|^{2}<r^2\right\}$ as our basic model~\cite{nickel2017poincare,tifrea2018poincare,ganea2018hyperbolicn}, where $r>0$ is the radius of the ball. The Poincaré ball is coupled with a Riemannian metric $g_\mathbb{D}(p)=\frac{4}{\left(1-\|p\|^{2}/r^2\right)^{2}} g_\mathbb{E}$, where $p \in \mathbb{D}^n$ and $g_\mathbb{E}$ is the canonical metric of the Euclidean space. For $p, q \in \mathbb{D}$, the Riemannian distance on the Poincaré ball induced by its metric $g_\mathbb{D}$ is defined as follows:
\begin{equation}
\label{eq:hyp_dis}
    d_\mathbb{D}(p, q)=2r \tanh ^{-1}\left(\frac{\left\|- p \oplus q\right\|}{r}\right),
\end{equation}
where $\oplus$ is the Möbius addition and it is clearly differentiable. In addition, the Poincaré ball can be viewed as a natural counterpart of the hypersphere as it allows all directions, unlike the other models such as the halfspace or hemisphere models that have constraints on the directions. The hyperbolic space is globally differomorphic to the Euclidean space, which is stated in the theorem below:

\begin{theorem}
\textbf{(Cartan–Hadamard).}  For every point $p \in \mathbb{H}^n$ the exponential map $\exp _{p}: T_{p} \mathbb{H}^n \approx \mathbb{R}^n\rightarrow \mathbb{H}^n$ is a smooth covering map. Since $\mathbb{H}^n$ is simply connected, it is diffeomorphic to $\mathbb{R}^n$.
\end{theorem}

Specifically, for $p \in \mathbb{D}^n$ and $v \in T_{p} \mathbb{D}^n \approx \mathbb{R}^n$, the exponential map of the Poincaré ball  $\exp_{p}: T_{p} \mathbb{D}^n \rightarrow \mathbb{D}^n$ is defined as 
\begin{equation}
\label{eq:exp_map}
    \exp _{p}(v):=p \oplus\left(\tanh \left(\frac{r\|v\|}{r^2-\|p\|^{2}}\right) \frac{rv}{\|v\|}\right),
\end{equation} 
The exponential map gives us a way to map the output of a network, which is in the Euclidean space, to the Poincaré ball. In practice, to avoid numerical issues, we clip the maximal norm of $v$ with $r-\varepsilon$ before the projection, where $\varepsilon>0$. During the backpropagation, we perform RSGD~\cite{bonnabel2013stochastic} by scaling the gradients by $g_\mathbb{D}(p)^{-1}$. Intuitively, this forces the optimizer to take a smaller step when $p$ is closer to the boundary. The scaling factor is lower bounded by $\mathcal{O}(\varepsilon^2)$.

The immediate consequence of the negative curvature is that for any point $\boldsymbol{p}\in\mathbb{H}^m$, there are no conjugate points along any geodesic starting from $\boldsymbol{p}$. Therefore, the volume grows exponentially faster in hyperbolic space than in Euclidean space. Such a property makes it suitable to embed the hierarchical structure that has constant branching factors and exponential number of nodes. This is formally stated in the theorem below:

\begin{theorem}\cite{gromov1987hyperbolic}
Given a Poincaré ball $\mathbb{D}^n$ with an arbitrary dimension $n \ge 2$ and any set of points $p_1,\cdots, p_m \in \mathbb{D}^n$, there exists a
finite weighted tree $(T, d_{T})$ and an embedding $f: T \rightarrow \mathbb{D}^n$
such that for all i, j,
$$\left|d_{T}\left(f^{-1}\left(x_{i}\right), f^{-1}\left(x_{j}\right)\right)-d_\mathbb{D}\left(x_{i}, x_{j}\right)\right|=\mathcal{O}(\log (1+\sqrt{2}) \log (m))$$
\end{theorem}

Intuitively, the theorem states that any tree can be embedded into a Poincaré disk ($n=2$) with low distortion. On the contrary, it is known that the Euclidean space with unbounded number of dimensions is not able to achieve such a low distortion~\cite{linial1995geometry}. One useful intuition~\cite{sala2018representation} to help understand the advantage of the hyperbolic space is given two points $p, q \in \mathbb{D}^n$ s.t. $\|p\|=\|q\|$,  
\begin{equation}
    \label{eq:intuition}
    d_\mathbb{D}(p, q) \rightarrow d_\mathbb{D}(p, 0) + d_\mathbb{D}(0, q), \text{ as } \|p\|=\|q\| \rightarrow r
\end{equation}
This property basically reflects the fact that the shortest path in a tree is the path through the earliest common ancestor, and it is reproduced in the Poincaré when points are both close to the boundary.




\subsubsection{Hyperbolic Contrastive Learning}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/model.pdf}
    \caption{Our \textbf{H}yperbolic \textbf{C}ontrastive \textbf{L}earning (HCL) framework has two branches: given a scene image, two object regions are cropped to learn the object representations with a loss defined in the Euclidean space focusing on the representation directions. A scene region as well as a contained object region are used to learn the scene representations with a loss defined in the hyperbolic space that affects the representation norms.}
    \label{fig:model}
\end{figure*}

Given the theoretical benefits of the hyperbolic space stated above, we propose a contrastive learning framework as shown in Figure~\ref{fig:model}. We adopt two losses to learn the object and scene representations. First, to learn object representations, we use the standard normalized temperature-scaled cross-entropy loss, which operates on the hypersphere in Euclidean space. As shown in the top branch of Figure~\ref{fig:model}, we crop two views of a jittered and slightly expanded object region as the positive pairs and feed into the base and momentum encoders to calculate the object representations. We denote the output after the normalization to be $\mathbf{z}_{\text{euc}}^1$ and $\mathbf{z}_{\text{euc}}^2$. 
We follow MoCo~\cite{he2020momentum} and leverage a memory bank to store the negative representations $\boldsymbol{z}^{n}_{\text{euc}}$, which are the features $\mathbf{z}_{\text{euc}}^2$ from the previous batches. Note that our framework can be readily extended to other contrastive learning models. The Euclidean loss for each image is then calculated as:
\begin{equation*}
    \mathcal{L}_{\text{euc}} = -\log \frac{\exp \left(\mathbf{z}_{\text{euc}}^1\cdot \mathbf{z}_{\text{euc}}^2 / \tau\right)}{\exp \left(\mathbf{z}_{\text{euc}}^1\cdot \mathbf{z}_{\text{euc}}^2 / \tau\right)+\sum_{n} \exp \left(\mathbf{z}_{\text{euc}}^1\cdot \mathbf{z}_{\text{euc}}^{n} / \tau\right)},
\end{equation*}
where $\tau$ is a temperature parameter. 

While the loss above aims to learn object representations, we propose a hyperbolic contrastive objective to learn the representations for scene images. We sample positive region pairs $u$ and $v$ from object-centric scene hierarchy $T$ such that $(u, v) \in E$. In other words, as shown in the bottom branch of Figure~\ref{fig:model}, the objects contained in one region are required to be a subset of the objects in the other. We sample the negative samples of $u$ to be $\mathcal{N}_u = \{v | (u, v) \not\in E \}$. However, building and sampling exhaustively from the entire hierarchy explicitly is tricky. 
In practice, given an image $s$, we always sample $u\in \mathcal{R} \cup \{s\}$ to be a scene region, $v\in \mathcal{O}$ to be an object that occurs in $u$, and $\mathcal{N}_u$ to be the other objects that are not in $u$. 


The pair of scene and object images are fed into the base and momentum encoders that share the weights with the Euclidean branch. However, instead of normalizing the output of the encoders, we use the exponential map defined in the~\eqref{eq:exp_map} to project these features in the Euclidean space to the Poincaré ball, which are denoted as $\mathbf{z}_{\text{hyp}}^1$ and $\mathbf{z}_{\text{hyp}}^2$. Further, we replace the inner product in the cross-entropy loss with the negative hyperbolic distance as defined in~\eqref{eq:hyp_dis}. We calculate the hyperbolic contrastive loss as follows:
\begin{equation*}
    \mathcal{L}_{\text{hyp}} = -\log \frac{\exp \left(-\frac{d_\mathbb{D}(\mathbf{z}_{\text{hyp}}^1, \mathbf{z}_{\text{hyp}}^2)}{\tau}\right)}{\exp \left(- \frac{d_\mathbb{D}(\mathbf{z}_{\text{hyp}}^1, \mathbf{z}_{\text{hyp}}^2)}{\tau}\right)+\sum_{n} \exp \left(- \frac{d_\mathbb{D}(\mathbf{z}_{\text{hyp}}^1, \mathbf{z}_{\text{hyp}}^{n})}{\tau}\right)},
\end{equation*}
When minimizing the distances of all the positive pairs, with the intuition from~\eqref{eq:intuition}, it would be beneficial to put the nodes near the root, i.e. objects, close to the center to achieve an overall lower loss.
The overall loss function of our model is as follows:
\begin{equation*}
    \mathcal{L} = \mathcal{L}_{\text{euc}} + \lambda\mathcal{L}_{\text{hyp}},
\end{equation*}
where $\lambda$ is a scaling parameter to control the trade-off between hyperbolic and Euclidean losses. 

\subsection{Implementation Details}



\topic{Pre-training phase.} We pre-train on three datasets: COCO~\cite{Lin2014MicrosoftCC}, the full OpenImages labelled dataset \cite{kuznetsova2020open}($\sim 1.7$ million samples) and a subset of OpenImages ($\sim 212k$) \cite{Mishra2021ObjectAwareCF}. All these datasets are multi-object datasets; OpenImages 
contains 12 objects on average per image and COCO 
contains 6 objects on average. We experiment with both the ground truth bounding box (GT) and using selective search (SS)~\cite{uijlings2013selective} to produce object bounding boxes in an unsupervised fashion, following previous work~\cite{xie2021unsupervised}. As the goal of this paper is not to present another state-of-the-art self-supervised learning method, we implement our sampling procedure and hyperbolic loss on top of three popular contrastive learning methods: MoCo-v2~\cite{chen2020improved}, Dense-CL~\cite{wang2021dense}, and ORL~\cite{xie2021unsupervised}. Dense-CL is a contrastive learning framework which extracts dense features from scene images and generally achieves better object detection results than MoCo-v2. ORL is a pipeline that learns improved object representations from scene images. We also consider HCL without the hyperbolic loss $\mathcal{L}_{\text{hyp}}$. This approach, which we denote as ``HCL w/o $\mathcal{L}_{\text{hyp}}$'', adopts the same cropping strategy as HCL but applies only a standard contrastive loss. We show that adding the hyperbolic loss improves results under various settings. More details on the datasets as well as training setups can be found in Appendix A. 

\label{sec:dataset}
\topic{Downstream tasks.} We evaluate our pre-trained models on image classification, object-detection and semantic segmentation. For classification, we show linear evaluation (lineval) accuracy with MoCo-v2, i.e. we freeze the backbone and only train the final linear layer. We test on VOC~\cite{everingham2010pascal}, ImageNet-100~\cite{tian2019contrastive} and ImageNet-1k~\cite{imagenet_cvpr09} datasets. 
For object detection and semantic segmentation, we show results with all 3 baselines on the COCO 
datasets using Mask R-CNN, following \cite{chen2020improved}. We closely follow the common protocols listed in Detectron2 \cite{wu2019detectron2}.



\begin{table}[t]
\centering
\setlength{\tabcolsep}{4pt}
    \begin{tabular}{lcccccc}
        \toprule
       
        & AP$^{\rm b}$ & AP$^{\rm b}_{50}$ & AP$^{\rm b}_{75}$ & AP$^{\rm m}$ & AP$^{\rm m}_{50}$ & AP$^{\rm m}_{75}$\\
        \midrule
       
       
       
       
       
        \multicolumn{7}{l}{\textit{MoCo-v2 pre-trained on COCO:}}\\
        Baseline & 38.5 & 58.1 & 42.1 & 34.8 & 55.3 & 37.3\\
        Baseline + bbox &  39.7 & 60.1 & 43.4  & 36.0 & 57.3 & 38.8 \\
        \rowcolor[HTML]{DFDFDF} 
        HCL  & \textbf{40.6} &  \textbf{61.1} & \textbf{44.5}  & \textbf{37.0} & \textbf{58.3}
        &\textbf{39.7}\\ \hline
        \multicolumn{7}{l}{\textit{Dense-CL pre-trained on COCO:}}\\
        Baseline & 39.6 & 59.3 & 43.3 & 35.7 & 56.5 & 38.4\\
        Baseline + bbox & 41.3 & 	61.5 & 44.7 & 	37.5 & 59.5 & 40.4 \\
        \rowcolor[HTML]{DFDFDF} 
        HCL & \textbf{42.5} &  \textbf{62.5} & \textbf{45.8}  & \textbf{38.5} & \textbf{60.6}
        &\textbf{41.4}\\ \hline
        \multicolumn{7}{l}{\textit{ORL pre-trained on COCO:}}\\
        Baseline & 40.3 & 60.2 & 44.4 & 36.3 & 57.3	 & 38.9 \\
        \rowcolor[HTML]{DFDFDF} 
        HCL & \textbf{41.4} &  \textbf{61.4} & \textbf{45.5}  & \textbf{37.3} & \textbf{58.5}
        &\textbf{40.0}\\ \bottomrule
        \multicolumn{7}{l}{\textit{Dense-CL pre-trained on OpenImages:}}\\
        Baseline  & 38.2 & 58.9 & 42.6 & 34.8 & 55.3 & 37.8\\
        Baseline + bbox &  41.1 & 	61.5 & 44.4 & 	37.2 & 58.3 & 39.7 \\
        \rowcolor[HTML]{DFDFDF} 
        HCL & \textbf{42.1} &  \textbf{62.6} & \textbf{45.5}  & \textbf{38.3} & \textbf{59.4}
        &\textbf{40.6}\\ 
    \bottomrule
    \end{tabular}
    \captionof{table}{\textbf{Comparison with state-of-the-art methods.} This table shows object detection (columns 1-3) and semantic segmentation (columns 4-6) results on COCO using MoCo-v2, Dense-CL and ORL by pre-training on COCO and OpenImages using unsupervised object bounding boxes generated by the selective search. The first row in each sub-table shows the results using random crops on pre-training datasets. The second and third rows set HCL/$\mathcal{L}_{\text{hyp}}$ to 0, which  means we are pre-training baseline methods on just proposal boxes. Our model consistently improves both object detection and semantic segmentation tasks across multiple contrastive learning baselines by pre-training on both COCO (800 epochs) and the full OpenImages dataset (75 epochs, last 3 rows). 
   
    }
    \label{tab:voc_moco_200}
\end{table}


\begin{table}
\begin{center}{
\setlength{\tabcolsep}{2.6pt}
    \begin{tabular}{lccccc}
        \toprule
        & Pre-train & Bbox & VOC & IN-100 & IN-1k \\
        \midrule
        MoCo-v2    & COCO & - & 64.79 & 64.84 &  51.17 \\
        MoCo-v2 + bbox   & COCO & SS & 73.13 & 73.84 & 54.21\\
        MoCo-v2 + bbox  & COCO & GT & 75.55 & 76.22 & 54.52\\
        \rowcolor[HTML]{DFDFDF} 
        HCL & COCO & SS & 74.19  & 75.16 &  55.03\\ 
        \rowcolor[HTML]{DFDFDF} 
        HCL & COCO & GT & \textbf{76.51} & \textbf{76.74} &  \textbf{55.63}\\
        \midrule
        MoCo-v2    & OpenImages & - & 69.95 & 72.80 & 54.12\\
        MoCo-v2 + bbox  & OpenImages & SS & 71.82 & 75.33 & 56.58\\
        MoCo-v2 + bbox  & OpenImages & GT & 73.79 & 77.36 & 57.57\\
        \rowcolor[HTML]{DFDFDF} 
        HCL & OpenImages & SS & 74.31 & 78.14 &  58.12\\
        \rowcolor[HTML]{DFDFDF} 
        HCL & OpenImages & GT & \textbf{75.40} & \textbf{79.08} &  \textbf{58.51}\\
        \bottomrule
        \end{tabular}
        \captionof{table}{\textbf{Classification results with linear evaluation.} The first row shows the results using random crops on pre-training datasets.
       
        In the last two rows we use our hyperbolic loss and we see improved performance by using both Ground Truth (GT) boxes and Selective Search (SS) boxes. HCL improves scene-level classification on the VOC dataset, and object-level classification on ImageNet-100 and ImageNet-1k datasets.}
    \label{tab:main_classification}}
    \end{center}
\end{table}








\subsection{Main Results}
\topic{Object detection and semantic segmentation.} Table \ref{tab:voc_moco_200} reports the object detection and semantic segmentation results by pre-training on COCO and full OpenImages dataset (last 3 rows) by using selective search boxes. HCL shows consistent improvements over the baselines on COCO object detection and COCO semantic segmentation. Although Dense-CL and ORL improve the object-level downstream performance over MoCo-v2 through improved object representations or dense pixel representations, they still lack the direct modeling of scene images. We show that learning representations for scene images in hyperbolic space is beneficial to object-level downstream performance. Note that pre-training Dense-CL on ImageNet for 200 epochs gives 40.3 mAP~\cite{wang2021dense}, while pre-trainng on OpenImages for only 75 epochs with our method gives 42.1 mAP. 
This shows the importance of efficient pre-training on datasets like OpenImages.


\topic{Image classification.}
As shown in Table \ref{tab:main_classification}, HCL improves image classification on both scene-level (VOC) and object-level (ImageNet) datasets. When pretraining on OpenImages, HCL improves ImageNet lineval accuracy by 0.94\% points and VOC lineval classification accuracy by 1.61 mAP. We observe similar improvements when pretraining on COCO. HCL improves accuracy whether we use ground truth object bounding boxes or boxes generated by selective search. In general, we observe a larger improvement of using HCL on OpenImages than COCO, which supports our hypothesis that HCL provides larger improvements on datasets with more objects per image.



\subsection{Properties of Models Trained with HCL}
The visual representations learned by HCL have several useful properties. In this section, we evaluate the representation norm as an measure of the label uncertainty for image classification datasets, and evaluate the object-scene similarity in terms of out-of-context detection. 


\subsubsection{Label Uncertainty Quantification}


\begin{table}
\centering
    \includegraphics[trim=0 35 0 30, clip, width=0.95\textwidth]{figures/qualitative/imagenet100_norms.png}
   
    \captionof{figure}{Average representation norms of images with different number of labels in ImageNet-ReaL.}
    \label{fig:in_real}
\end{table}
    


ImageNet~\cite{imagenet_cvpr09} is an image classification dataset consisting of object-centered images, each of which has a single label. As the performance on this dataset has gradually saturated, the original labels have been scrutinized more carefully~\cite{recht2019imagenet,tsipras2020imagenet,shankar2020evaluating,beyer2020imagenet,vasudevan2022does}. Prevailing labeling issues in the validation set have been recently identified, including labeling errors, multi-label images with only a single label provided, and so on. Although \cite{beyer2020imagenet} provides reassessed labels for the entire validation set, relabeling the entire training set may be infeasible. 

\begin{table}[t]
\begin{center}
    \begin{tabular}{ll|ll}
    \toprule
    \multirow{2}{*}{Method} &  \multirow{2}{*}{Indicator} & \multicolumn{2}{c}{Datasets} \\
        &  & IN-Real & COCO \\
    \midrule
    MoCo     & Entropy         & 0.633 & 0.791 \\
    Supervised & Entropy         & 0.671 & 0.793 \\
    HCL      & Norm         & 0.655 & \textbf{0.839} \\
    Ensemble & Entropy+Norm       & \textbf{0.717} & 0.823 \\
    \bottomrule
    \end{tabular}
  \captionof{table}{NDCG scores of the image rankings based on the different indicators and models, and evaluated by the the number of labels per image.}
  \label{tab:ndcg}
\end{center}
\end{table}
\begin{figure*}[t]
    \centering
   
    \includegraphics[width=\linewidth]{figures/qualitative/imagenet-norm.pdf}
    \caption{Images from ImageNet training set. The 5 images on the left have the smallest representation norms among all the images from the same class, and the 5 on the right have the largest norms.}
    \vspace{-0.2cm}
    \label{fig:imagenet_norm}
\end{figure*}


Our learned representations provide a potential automatic way to identify images with multiple labels from datasets like ImageNet. Specifically, we first show in Figure~\ref{fig:in_real} that there is a strong correlation between the representation norms and the number of labels per image according to the reassessed labels. For each class of the ImageNet training set, we use a pre-trained OpenImages model and rank the images according to their norms. The extreme images of some classes are shown in Figure~\ref{fig:imagenet_norm} and also in the Appendix. Images with smaller norms tend to capture a single object, while those with larger norms are likely to depict a scene. 

To quantitatively evaluate this property, we report the NDCG metric on the ranked images as shown in Table~\ref{tab:ndcg}. NDCG assesses how often the scene images are ranked at the top. As a baseline, we rank the images based on the entropy of the class probability predicted by a classifier, which is a widely adopted indicator of label uncertainty~\cite{chen2019understanding,northcutt2021confident}. We use both MoCo-v2 and supervised ResNet-50 as the classifier. As shown in Table~\ref{tab:ndcg}, using norms with HCL achieves similar rank quality as using entropy with the supervised ResNet-50 on the ImageNet-ReaL dataset. In addition, when combining two ranks using simple ensemble methods such as Borda count, the score is further improved to $0.717$. This shows that the entropy and the norm provide complimentary signals regarding the existence of multiple labels. For example, the entropy indicator can be affected by the bias of the model and the norm indicator can be wrong on the images with multiple objects from the same class.

Compared to supervised indicators of label uncertainty, HCL has the additional advantage that it is dataset-agnostic and can be applied to new data without further training. To demonstrate this benefit, we report the same metric on the COCO validation, where we also have the number of labels for each image. Our method achieves much better NDCG scores than the supervised ResNet-50 as shown in Table~\ref{tab:ndcg}. This finding can be potentially useful to guide label reassessment, or provide an extra signal for model training.  


\subsubsection{Out-of-Context Detection}

Our hyperbolic loss $\mathcal{L}_{\text{hyp}}$ encourages the model to capture the similarity between the object and scene. We apply the resulting representations to detect out-of-context objects, which can be useful in designing data augmentation for object detection~\cite{dvornik2019importance}. We are especially interested in out-of-context images with conflicting backgrounds. To this end, we use the out-of-context images proposed in the SUN09 dataset~\cite{choi2010exploiting}. We first compute the representations of each object and entire scene image with that object masked out. We then calculate the hyperbolic distance between the representations mapped to the Poincaré ball. Some example images from this dataset as well as the distance of each contained object are shown in Figure~\ref{fig:oot_detection}. We find that the out-of-context objects generally have a large distance, i.e. smaller similarity, to the overall scene image. To quantify this finding, we compute the mAP of the object ranking on each image and obtain $0.61$ for HCL. As a comparison, the MoCo similarity gives mAP $=0.52$ and the random ranking gives mAP $=0.44$.

\section{Main Ablation Studies}
In this section, we report the results of several important ablation studies with respect to HCL. All the models are trained on the subset of the OpenImages dataset and linearly evaluated on the ImageNet-100 dataset. The top-1 accuracy is reported.




\begin{table*}[t]
\RawFloats
\small
\centering
\begin{minipage}[b]{0.4\textwidth}
\centering
    \begin{tabular}{llc}
    \toprule
    Distance  & Center & IN-100 Accuracy \\
    \midrule
    -          & -      & 77.36  \\
    \rowcolor[HTML]{DFDFDF} 
    Hyperbolic & Scene  & 79.08      \\
    Hyperbolic & Object & 76.96      \\
    Euclidean  & Scene  & 76.68     \\
    \bottomrule
    \end{tabular}
  \captionof{table}{Similarity measure and hierarchy center.}
    \label{tab:setup}
\end{minipage}%
\hfill
\begin{minipage}[b]{0.23\textwidth}
\centering
    \begin{tabular}{lc}
    \toprule
    $\lambda$ & IN-100 Accuracy \\
    \midrule
    0.01      & 77.70       \\
    \rowcolor[HTML]{DFDFDF} 
    0.1       & 79.08      \\
    0.2       & 78.64      \\
    0.5       & 0          \\
    \bottomrule
    \end{tabular}
  \captionof{table}{Losses trade-off.}
    \label{tab:lambda}
\end{minipage}
\hfill
\begin{minipage}[b]{0.36\textwidth}
\centering
    \begin{tabular}{llc}
    \toprule
    Optimizer & $\lambda$ & IN-100 Accuracy \\
    \midrule
    \rowcolor[HTML]{DFDFDF} 
    RSGD & 0.1 & 79.08  \\
    RSGD & 0.5 & 0      \\
    SGD  & 0.1 & 70.16  \\
    SGD  & 0.5 & 74.18  \\
    \bottomrule
    \end{tabular}
  \captionof{table}{RSGD versus SGD optimizers.}
    \label{tab:optim}
\end{minipage}
    \vspace{-0.5cm}
\end{table*}

\topic{Similarity measure and the center of the scene-object hierarchy.} We propose to use the negative hyperbolic distance as the similarity measure of the scene-object pairs. As an alternative, one can use cosine similarity on the hypersphere as the measure as in the original contrastive objective. However, this would attempt to maximize the similarity between a single object and multiple objects. It is likely that these objects belong to different classes, and hence this strategy impairs the quality of the representation. As shown in Table~\ref{tab:setup}, replacing the negative hyperbolic distance with the Euclidean similarity impairs downstream performance. The resulting model performs even worse than the baseline without loss function on the scene-object pairs, demonstrating the necessity of using hyperbolic distance. We also validate our choice of an object-centric hierarchy by comparing its performance with that of a scene-centric hierarchy~\cite{parikh2007hierarchical,parikh2009unsupervised} generated by sampling the negative pairs as objects and unpaired scenes. This scene-centric hierarchy leads to substantially lower accuracy (Table~\ref{tab:setup}). 

\topic{Trade-off between the Euclidean and hyperbolic losses.} We adopt the Euclidean loss to learn object-object similarity and the hyperbolic loss to learn object-scene similarity. A hyperparameter $\lambda$ controls the trade-off between them. As shown in Table~\ref{tab:lambda}, we find that a smaller $\lambda=0.01$ leads to marginal improvement. However, we also observe that larger $\lambda$s can lead to unstable and even stalled training. With careful inspection, we find that in the early stage of the training, the gradient provided by the hyperbolic loss can be inaccurate but strong, which pushes the representations to be close to the boundary. As a result, since Riemannian SGD divides gradients by the distance to the boundary, updates become small and training ceases to make progress.

\topic{Optimizer.} Given the observation above, we ask whether RSGD is necessary for practical usage. We replace the RSGD optimizer with SGD. To avoid numerical issues when the representations are too close to the boundary, we increase $\varepsilon$ from $1e^{-5}$ to $1e^{-1}$. This allows a larger $\lambda$ to be used as opposed to the RSGD. However, SGD always yields inferior performance compared to RSGD.

\section{Related Work}
\topic{Representation Learning with Hyperbolic Space.}
Representations are typically learned in Euclidean space.
Hyperbolic space has been adopted for its expressiveness in modeling tree-like structures existing in various domains such as language~\cite{sala2018representation, nickel2017poincare, nickel2018learning}, graphs~\cite{balazevic2019multi,chami2020low,park2021unsupervised}, and vision~\cite{chen2020learning,suris2021learning}. The corresponding deep neural network modules have been designed to boost the progress of such applications~\cite{chami2019hyperbolic,ganea2018hyperbolicn,liu2019hyperbolic,shimizu2021hyperbolic}. The hierarchical structure presented in the datasets can arise from three factors that together motivate the use of hyperbolic space. The first factor is generality: the hypernym-hyponym property is a natural feature of words (e.g. WordNet~\cite{miller1990introduction}) and the hyperbolic space is extensively exploited to learn word and image embeddings that preserve that property~\cite{tifrea2018poincare, ganea2018hyperbolic, sala2018representation,liu2020hyperbolic,yan2021unsupervised,Long_2020_CVPR}. 
The second factor is uncertainty: Several studies have found that applying hyperbolic neural network modules to different tasks leads to a natural modeling of the uncertainty~\cite{ghadimiatigh2022hyperbolic,khrulkov2020hyperbolic,suris2021learning}. The third factor is compositionality of different basic elements to form a natural hierarchy. Motivated by these factors, previous work in computer vision has applied hierarchical representations learned in the hyperbolic space to various tasks such as image classification~\cite{khrulkov2020hyperbolic} or segmentation~\cite{weng2021unsupervised},  zero-/few-shot learning~\cite{liu2020hyperbolic}, action recognition~\cite{Long_2020_CVPR}, and video prediction~\cite{suris2021learning}. In this paper, we focus on learning the representations that capture the hierarchy between the objects and scenes with the goal of learning general-purpose image representations that can transfer to various downstream tasks.


\topic{Self-Supervised Learning on Scenes.} 
Self-Supervised Learning (SSL) has made great strides in closing the performance with supervised methods \cite{chen2020simple,chen2020improved,Ge2021RobustCL} when pretrained on the object-centric datasets like ImageNet. However, recent work has shown that SSL is limited on multi-object datasets like COCO \cite{selvaraju2017grad, wang2021dense, Mishra2020LearningVR} and OpenImages \cite{kuznetsova2020open}. Several papers have tried to address this issue by proposing different techniques. Dense-CL \cite{wang2021dense} operates on pre-average pool features and uses dense features on pixel level to show improved performance on dense tasks such as semantic segmentation. DetCon \cite{henaff2021efficient} uses unsupervised semantic segmentation masks to generate features for the corresponding objects in the two views. 
PixContrast \cite{xie2021propagate} uses pixel-to-propagation consistency pretext task to build features for both dense downstream tasks and discriminative downstream tasks. Pixel-to-Pixel Contrast \cite{Wang_2021_ICCV} uses pixel-level contrastive learning to learn better features for semantic segmentation. Self-EMD \cite{liu2021selfemd} uses earth mover distance with BYOL \cite{grill2020bootstrap} for pretraining on the COCO dataset. ORL \cite{xie2021unsupervised} uses selective search to generate object proposals, then applies object-level contrastive loss to enforce object-level consistency. 
Below-par performance of SSL methods can be attributed to treating scenes and objects using similar techniques, which often results in similar representations. In our work, instead of treating scenes and objects similarly, we use a hyperbolic loss, which builds representation that disambiguates scenes and objects based on the norm of the embeddings. Our method not only separates scenes and objects, but also improves downstream tasks such as image classification.


\section{Acknowledgements}
Songwei Ge, Shlok Mishra, David Jacobs were supported in part by the National Science Foundation under grant no. IIS-1910132 and IIS-2213335. 



\section{Experiment setups}

In this section, we provide additional details of our experiments.

\paragraph{Unsupervised object proposals.} When pretraining on uncurated datasets, acquiring ground truth object bounding boxes using human annotations can be expensive. However, automatically generating unsupervised region proposals is well-studied. We use Selective Search as the unsupervised proposal generation method. Following ORL \cite{xie2021unsupervised} we first generate the proposals using selective search. Then we filter the proposals with 96 pixels as the minimal scale, maximum IOU of 0.5 and  aspect ratio between 1/3 to 3. For every image we generate maximum of 100 proposals and randomly select any image as the object image.

\paragraph{OpenImages dataset.} We use the full OpenImages dataset which have bounding box annotations ($\sim$ 1.9 million images). We also use a subset proposed in \cite{Mishra2021ObjectAwareCF}. This is a subset created from the OpenImages dataset where each image has at least 2 classes present and each class has at least 900 instances. This subset is a balanced subset of OpenImages with an average of 12 object present in an image, making it a good proxy for real-world multi-object images. 


\paragraph{Object and Scene image augmentations.} We find that small objects are always detrimental to performance. Therefore, when sampling object bounding boxes, we drop bounding boxes with size \texttt{width} $\times$ \texttt{height} $\le 56 \times 56$. Further, when sampling objects for the Euclidean branch, if the size of a bounding box \texttt{width} $\times$ \texttt{height} $\le 256 \times 256$, we slightly expand it to either $256 \times 256$ or the maximal size allowed by the original image size. We also apply a small jittering to the  \texttt{width} and \texttt{height} to include different contexts around the objects. Next, we apply random cropping and resizing with the same scale $(0.2, 1.)$ as in MoCo~\cite{he2020momentum}. When sampling objects for the hyperbolic branch, we do not apply jittering and random cropping, but only filter the small boxes and resize to $\le 224 \times 224$. To crop the scene images, we sample another $1$ to $5$ bounding boxes and merge with the selected object bounding box.  

\paragraph{Model details of pre-training.} For the optimizer setups and augmentation recipes, we follow the standard protocol described in MoCo-v2~\cite{chen2020improved}. We find that a base learning rate of 0.3 works better when pre-training on COCO and OpenImage datasets as compared to 0.03. We adopt the linear learning rate scaling receipt that $lr=0.3 \times \text{BatchSize}/256$~\cite{goyal2017accurate} and batch size of $128$ by default on $4$ NVIDIA p6000 gpus. To ensure fair comparison, we also pre-train the baselines with a learning rate of 0.3. We train our models on COCO and the subset of OpenImage datasets for $200$ epochs and full OpenImage dataset for $75$ epochs. We also note that calculating hyperbolic loss itself takes nearly the same time as a normal contrastive loss. The only overhead in pre-training is one additional forward pass to get scene representations. In our setting, MoCo takes 0.616 sec/iter while HCL takes 0.757 sec/iter. For the hyperparameters of our hyperbolic objective, we use $r=4.5$, $\lambda=0.1$, and $\varepsilon=1e^{-5}$ as our default setting.

\section{Additional experimental results}

\subsection{Robustness under Corruption.}

We calculate the mCE error as in Hendrycks et al. \cite{hendrycks2019robustness}. We compare our HCL model trained on OpenImages and lineval on ImageNet dataset with the baseline model without using HCL loss. We see an improvement of 1.9 mCE over the baseline model, demonstrating that our HCL model learns more robust representations as compared to the vanilla MoCo. 

\subsection{Fine-grained class classification}
\begin{table}[h]
    \centering
    \begin{tabular}{lccc}
        \toprule
        Method  & Cars~\cite{KrauseStarkDengFei-Fei_3DRR2013} & DTD~\cite{cimpoi14describing} &  Food~\cite{bossard14}\\
        \midrule
        HCL/$\mathcal{L}_{\text{hyp}}$    &   31.92 &	68.46 &	58.66  \\
        \rowcolor[HTML]{DFDFDF} 
        HCL   & 32.02 &	68.19 &	58.79\\
    \bottomrule
    \end{tabular}
    \captionof{table}{Fine grained classification results.}
    \label{tab:fine_grained}
\end{table}

In Table \ref{tab:fine_grained} we show results on fine-grained classification datasets. We can see that on fine-grained classification our model provides little performance improvement. This could be due to the fact that all classes in these datasets have very similar scene contexts, and hence the hyperbolic objective does not help very much.

\subsection{More ImageNet Examples}

\begin{figure*}[h]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/appendix/imagenet1.pdf}
    \caption{More images from ImageNet training set sorted by their representation norms.}
\end{figure*}
\clearpage
\begin{figure*}
    \centering
    \includegraphics[width=0.95\linewidth]{figures/appendix/imagenet2.pdf}
    \caption{More images from ImageNet training set sorted by their representation norms.}
\end{figure*}
\clearpage

\section{Additional ablation studies}
In this section, we provide more ablation experiments on hyperbolic linear evaluation, model architecture, and the radius of Poincare ball. All the models are trained on the OpenImages dataset and evaluated on the ImageNet-100 (IN-100) or ImageNet-1k (IN-1k) the top-1 accuracy reported.

\paragraph{Radius of the Poincare ball.}
In Table \ref{tab:poincare_radius} we show results by varying the radius of Poincaré ball. The hyperbolic objective improves the performance over all the tested radius. We find that a too small radius may lead to a smaller improvement due to the stronger regularization. 

\paragraph{Configuration of the encoder head.}
In our experiments, the Euclidean and hyperbolic branches share the weights in both the backbone and the head of the encoders. We also try using a separate head for the hyperbolic branch. As shown in Table~\ref{tab:predictor}, this leads to a more stable training when larger learning rate is applied. However, we did not see any improvements brought by this modification.

\paragraph{Hyperbolic linear evaluation.}
Apart from the common linear evaluation in the Euclidean space, we show the hyperbolic linear evaluation results with different optimizers and learning rates in Table \ref{tab:hyperbolic_lineval}. The idea is to test if the representations are more linearly separable in the hyperbolic space. We follow the same setting of hyperbolic softmax regression~\cite{ganea2018hyperbolicn} and train a single hyperbolic linear layer. However, we find the optimization with SGD can easily cause overflow. By contrast, Adam is much more stable with appropriate learning rates.


\begin{minipage}[b]{0.25\textwidth}
\centering
    \begin{tabular}{lccc}
        \toprule
        $c$ & IN-1k Acc. \\
        \midrule
        1 &	58.08 \\
        0.5	& 58.31 \\
        0.1 &	58.29 \\
        \rowcolor[HTML]{DFDFDF} 
        0.05 &	58.51 \\
        0.01 &	58.49 \\
    \bottomrule
    \end{tabular}
    \captionof{table}{Results by varying the radius $r$ of Poincaré ball. $c=\frac{1}{r^2}.$}
    \label{tab:poincare_radius}
\end{minipage}
\hfill
\begin{minipage}[b]{0.33\textwidth}
\centering
    \begin{tabular}{lccc}
        \toprule
        \multicolumn{1}{l}{Head} & $\lambda$ & IN-100 Acc.\\
        \midrule
        \multicolumn{1}{l}{N/A}        & 0         & 77.36  \\
        \rowcolor[HTML]{DFDFDF} 
        \multirow{2}{*}{shared}        & 0.1       & 79.08  \\
                                       & 0.5       & 0      \\
        \multirow{2}{*}{splitted}      & 0.1       & 77.88  \\
                                       & 0.5       & 77.58  \\
    \bottomrule
    \end{tabular}
    \captionof{table}{Different configurations of head in the the Euclidean and hyperbolic branches.}
    \label{tab:predictor}
\end{minipage}
\hfill
\begin{minipage}[b]{0.35\textwidth}
\centering
    \begin{tabular}{llll}
    \toprule
    \multicolumn{2}{c}{SGD} & \multicolumn{2}{c}{Adam} \\
    lr & IN-100 & lr & IN-100 \\
    \midrule
    0.1        & 63.82       & 0.001        & 67.64      \\
    0.2        & 64.22       & 0.0005       & 70.32      \\
    0.3        & 1           & 0.0001       & 72.58      \\
    0.4        & 1           & 0.00005      & 70.5       \\
    \bottomrule
    \end{tabular}
    \captionof{table}{Results of hyperbolic linear evaluation with different optimizers and learning rates. }
    \label{tab:hyperbolic_lineval}
\end{minipage}
