
\subsection{Selection frequency remeasuring experiment}
\label{app:remeasure_setup}
In Section~\ref{sec:remeasure}, we replicate the ImageNet experiment to
remeasure selection frequencies for the ImageNet v1 and v2 datasets. We present
annotators with a grids of 48 images along with an ImageNet class. The
annotators are also provided with the WordNet synsets for the ImageNet class
being queried, along with a Wikipedia link and asked to selected all images
containing instances of that object (ignoring clutter as per the original
dataset creation process). The 48 images in each grid consist of: (a) 10
ImageNet v1 validation set images from that class, (b) 10 ImageNet v2 validation
set images from that class, (c) 22 related Flickr images scraped from Flickr
(using the exact script and queries described in \citet{recht2018imagenet}) and
(c) 6 negative control images (three corresponding to randomly chosen labels,
and 3 corresponding to the ``nearest'' label to the true label in terms of
WordNet path similarity).

We implemented our setup by modifying the code made publicly available by
\citet{recht2018imagenet}~\footnote{\url{https://github.com/modestyachts/ImageNetV2}}.
A screenshot of our interface appears in Figure~\ref{fig:contains_screen}. Each
such grid of images is shown to 40 annotators. For each image-class pair, we can then compute the ``selection frequency''
based on how often it was selected by the annotators.


\paragraph{Deployment Details.}
There are a number of deployment details that could cause variations in results.
We compensated MTurk workers with \$0.23 per assignment (i.e., each completed
grid), which we calibrated to pay a rate of at least \$9/hr for most workers. To
collect 40 separate MTurk annotations for each submitted grid of images, we
obtained 10 annotations on 4 different dates and times, all within the span of
a single week. We placed qualification requirements on the workers allowed to
complete assignments. Specifically, we filtered for workers that (a) agree to
view adult content (as some ImageNet images have content like nudity or gore)
and (b) have a larger than 95\% assignment approval rate (as to ensure the
quality of the results).


\paragraph{Controls.} All of the results presented in this work were run on a
``clean'' and ``raw'' version of our data, i.e., without and with data cleaning
respectively. We find that the inclusion of data cleaning makes the observed gap
between \texttt{v1}\xspace and \texttt{v2}\xspace slightly larger but otherwise does not have a significant
effect on results.

Our data cleaning process is as follows: a given batch is
``flagged'' if: (a) there are less than 6 selected images out of the total 48,
or (b) more than one of the negative controls was selected. We only omit
data, however, from workers whose batches were rejected at a rate of 30\% or
higher (e.g., if an annotator completed 30 batches, but more than 10 of them are
flagged to be low-quality, then all of the annotator's data is omitted).
Finally, to make computing of the statistics easier, we evened out the number
of annotators per image to equal the minimum number of remaining assignments
per image, which was 36 (compared to 40 originally) by randomly discarding
annotations. In total, the entire process corresponds to discarding 10\% of the
annotations.

\subsection{Comparison to the original setup}
\label{app:exp_differences}
\citet{recht2018imagenet} measure the average selection frequency of \texttt{v1}\xspace to be
0.71, whereas our experiment measures the average \texttt{v1}\xspace selection
frequency to be 0.85. While our experiments were modeled closely after that
of~\citet{recht2018imagenet} (and in fact use the same core codebase to minimize
discrepancies in task presentation/inferace), we made a few changes to the setup
to ensure high data quality. We hypothesize that these changes, discussed below,
are what result in the discrepancy between the measured average selection frequencies.
However, since these changes are applied at the task level and annotators are
not told which dataset each image is sourced from, we find it unlikely that these changes
would affect annotations for one dataset more than the other. 
Furthermore, in Appendix~\ref{app:orig_data}, we demonstrate that the bias
identified in this paper can be found even using the original data collected
by~\citet{recht2018imagenet}.

\paragraph{Worker pay and qualifications.} In our experiment, we
paid annotators 20 cents per set of 48 images completed---this was informed by
the average time taken to complete a batch, and was calibrated so that the task
paid approximately 12 dollars per hour. Conversely, the original experiment
of~\citet{recht2018imagenet} pay 10 cents per batch. Although worker pay usually has
only a mild impact on worker reliability on
MTurk~\citep{mason2009financial,buhrmester2011amazon}, higher worker pay has
been recognized as a tool to boost participation rates~\citep{buhrmester2011amazon} and requester
reputation for future experiments~\citep{paolacci2010running}.

Perhaps the most important modification made was our inclusion of worker
qualifications, which only allow annotators who have had 95\% or more of
previous tasks accepted to participate in our task. Prior work has
shown that without these worker qualifications,
crowdsourced data tends to be of significantly worse quality. For
example,~\citet{peer2013reputation} report that 2.6\% of workers with the ``95\%
accepted'' qualification failed an ``attention-check
test,'' compared to 33.9\% of workers without
qualifications\footnote{Attention-check tests are a series of three
attention-check questions (ACQs). ACQs are
questions with right/wrong answers unrelated to the task meant to gauge an
annotator's attentiveness, e.g. ``Have you ever had a fatal heart attack?''. In
the~\citet{peer2013reputation} study, 16.4\% of unqualified workers reported
that they had suffered a {\em fatal} heart attack, compared to 0.4\% of
qualified workers.}. We should therefore expect a significant increase in
annotation reliability (and so in turn some discrepancy) from using
worker qualifications.

\paragraph{Makeup of each batch.} Another difference between the two experiments
is that in our experiment, each batch of images contains 10 images from
ImageNet-v1, and 10 images from ImageNet-v2, in order to ensure that we could
obtain 40 annotations for each \texttt{v1}\xspace and \texttt{v2}\xspace image while keeping to a reasonable
budget constraint. The experiment of~\citet{recht2018imagenet} uses only
three images from ImageNet-v1 per batch (and a variable number of ImageNet-v2
images, since the dataset was not yet realized). Thus, the grids presented in our
experiment contain images that are on average more likely ({\em a priori}) to be
selected. This could in part contribute to the higher average selection
frequency that we observe (though again, we would expect this effect to apply
to both datasets and thus preserve the observed selection frequency gap).

\paragraph{Randomization.} Response-order bias is a well-documented
phenomenon in literature on crowdsourcing (e.g.~\citep{schuman1981question}),
although its effects in the domain of image selection are not
well-understood yet. In our experiment, we randomize the order of the images per
batch per worker (i.e., we used JavaScript to randomize the image order on
page load) to mitigate the potential effects of this bias. In the prior
experiment, however, the image order is deterministic, and thus the study may
have response-order effects.

\paragraph{Worker duplicates.} 
Due to the mechanism by which images were distributed to assignments in the study
of~\citet{recht2018imagenet}, 5.6\% of the annotations are duplicated (i.e.,
5.6\% of the [worker, image] pairs collected are non-unique), with approximately
3\% of the annotations being redundant (unlike the preceding number, this fraction does
not count the ``original copy'' of each non-unique pair). A histogram of the
number of times a single worker labeled a single image is shown in
Figure~\ref{fig:worker_image_hist}. Since duplicate workers violate
sample indpendence and can skew measured selection frequencies for some images,
in our study we ensure that no worker labels the same image more than once.

\paragraph{Data cleaning and controls.} Our study also differs in having
built-in mechanisms for data cleaning (as discussed in the last section),
allowing us to run all of our experiment on the ``cleaned'' and ``raw'' versions
of our data. These results tend to not be substantially different (for the
cleaned data, the selection frequency gap we measure between \texttt{v1}\xspace and \texttt{v2}\xspace 
slightly increases from $4.5\%$ to $4.6\%$). Possible reasons for this
similarity between cleaned and raw results include any of the quality control
protocols outlined in this section.

\begin{figure}[hb]
    \centering
    \input{plots/orig_data/worker_im_pairs}
    \caption{A histogram showing the existence of duplicate (worker, batch)
    pairs present in the original collected data. Each point in the histogram
    is a unique (worker, batch) pair, and the $x$ axis corresponds to the
    number of times that pair is observed in the dataset.}
    \label{fig:worker_image_hist}
\end{figure}

\begin{figure}[!ht]
	\centering
	\includegraphics[width=0.85\textwidth]{figures/contains.jpg}
	\caption{Screenshot of a sample grid to measure selection frequencies
		(Section~\ref{sec:remeasure}).
		Annotators are given a grid of 48 images for a specific label and asked to
		select all images that contain that label. The interface and instructions
		are based on
		\citet{recht2018imagenet}.}
	\label{fig:contains_screen}
\end{figure}

\subsection{Justifying the use of the statistical jackknife}
\label{app:jk_justification}
Recall that in Section~\ref{sec:quantifying}, the quantity of interest is the
following {\em adjusted accuracy}:
\begin{equation}
    \acc{\dvbc}\xspace = \int_s \mathbb{E}_{\ensuremath{\mathcal{D}_2}\xspace}[f(x)|s(x)=s] \cdot p_1(s)\ ds.
\end{equation}
In Section~\ref{sec:naive}, we introduced the following na\"ive estimator
\begin{equation}
    \ensuremath{\estacc{\dvbc}^{n}}\xspace = \sum_{k=0}^{n} \evb{f(x_2)\vert \hs{x_2} = \frac{k}{n}} \cdot p_1\left(\hs{x_1} =\frac{k}{n}\right),
\end{equation}
which evaluates to $\acc{\dvbc}\xspace$ if $\hs{x} = \s{x}$, and assuming the expectations
above can be computed exactly.

\paragraph{Consistency of the na\"ive estimator.} In order for our application of
the statistical jackknife to be valid, we must show that the na\"ive estimator
is a {\em consistent} estimator of $\acc{\dvbc}\xspace$---that is, as $n \rightarrow
\infty$, $\ensuremath{\estacc{\dvbc}^{n}}\xspace \rightarrow \acc{\dvbc}\xspace$. Note that since we operate in the
regime where the number of distinct images $m$ greatly exceeds the number of
annotators per image $n$, we will assume that the expectations above can be
computed exactly. Note that the estimator remains consistent even if this is not
the case, with the additional constraints that $m \rightarrow \infty$ and $m / n
\rightarrow \infty$, but this greatly complicates the proof and we will show
empirically that the estimator is robust to changes in $m$ in the relevant
regime. 


Note that in the ``infinite $m$'' regime, the variance of the na\"ive 
estimator is $0$. Thus, all of the error is due to bias in the estimator. In the
following, we assume that $p_1(s)$, $p_2(s)$, and $p_2(s|f=1)$ are continuous
differentiable densities bounded away from zero and with bounded derivatives
($|d^r/dx^r\ p_i(x)| < C$).
\begin{align}
    p_i\left(\hs{x}=\frac{k}{n}\right) &= \int_0^1 p_i(s)\cdot \binomial{n,k,s}\ ds \\
    &= \int_0^1 p_i(s)\cdot {n \choose k} \cdot s^k \cdot (1-s)^{n-k}\ ds \\
    &= \int_0^1 p_i(s)\cdot \frac{s^k (1-s)^{n-k}}{(n+1)\cdot B(k+1, n-k+1)}\ ds &B(\cdot,\cdot)\text{ is the Euler beta function}\\
    &= \frac{1}{n+1}\int_0^1 p_i(s)\cdot \text{Beta}(s;k+1,n-k+1)\ ds \\
    &= \frac{1}{n+1}\mathbb{E}_{s\sim \text{Beta}(\cdot;k+1,n-k+1)}[p_i(s)]
\end{align}
Using a Taylor expansion of $p_i(s)$ around
$\mathbb{E}_{s\sim\text{Beta}(\cdot;k+1,n-k+1)}[s]$, we can bound the above
expression:
\begin{align*}
    &= \frac{1}{n+1}\mathbb{E}\left[
        p_i(\mathbb{E}[s]) 
        + (s-\mathbb{E}[s])p_i'(E[s]) + \frac{(s-\mathbb{E}[s])^2}{2}p_i''(\mathbb{E}[s])
        + \sum_{r=3}^\infty \frac{(s - \mathbb{E}[s])^r}{r!} p_i^{(r)}(\mathbb{E}[s])
    \right]  \\
    &= \frac{1}{n+1}\left(p_i(\mathbb{E}[s]) + \frac{1}{2}\text{Var}[s]\cdot p_i''(\mathbb{E}[s]) + O\left(\frac{1}{n^{7/2}}\right) \right) \\
    &= \frac{1}{n+1}p_i\left(\frac{k+1}{n+2}\right) + \frac{(k+1)(n-k+1)}{(n+1)(n+2)^2(n+3)}\cdot p_i''(\mathbb{E}[s]) + O\left(\frac{1}{n^{7/2}}\right) 
\end{align*}
Now, using the presumed boundedness of derivatives we can write:
\begin{align}
    \label{eq:term_bound}
    \left| p_i\left(\hs{x}=\frac{k}{n}\right) - 
    \frac{1}{n+1}\cdot p_i\left(\frac{k+1}{n+2}\right) \right| 
    &\leq \frac{(k+1)(n-k+1)}{(n+1)(n+2)^2(n+3)} \cdot p_i''(\mathbb{E}[s]) 
    + O\left(\frac{1}{n^{7/2}}\right) 
\end{align}

\begin{align*}
    |\acc{\dvbc}\xspace - \ensuremath{\estacc{\dvbc}^{n}}\xspace| &= p_2(f(x) = 1)\left|
        \int_0^1 \frac{p_2(s|f(x) = 1)}{p_2(s)} p_1(s)\ ds
        - \sum_{k = 0}^{n} \frac{
            p_2(\hs{x} = \frac{k}{n} | f(x) = 1)
        }{
            p_2(\hs{x} = \frac{k}{n})
        } p_1(\hs{x} = \frac{k}{n})
    \right| \\
    &\leq \left|\int_0^1 \frac{p_2(s|f(x)=1)}{p_2(s)}p_1(s)\ ds 
    -  \frac{1}{n+1}\sum_{k=0}^{n} \frac{
        p_2\left(\frac{k+1}{n+2}\big| f(x) = 1\right)
        }{ p_2\left(\frac{k+1}{n+2}\right)} 
        p_1\left(\frac{k+1}{n+2}\right)  \right| \\
    &+  \left| \frac{1}{n+1}\sum_{k=0}^{n} \frac{
        p_2\left(\frac{k+1}{n+2}\big| f(x) = 1\right)
        }{ p_2\left(\frac{k+1}{n+2}\right)} 
        p_1\left(\frac{k+1}{n+2}\right)  
    - \frac{
        p_2\left(\hs{x} = \frac{k}{n}\big| f(x) = 1\right)
        }{ p_2\left(\hs{x} = \frac{k}{n}\right)} 
        p_1\left(\hs{x} = \frac{k}{n}\right)  \right| 
\end{align*}
Now, note that the first term in the above is simply the error in the Riemmann
sum approximation of the integral, which vanishes as $n \rightarrow \infty$. The
second term is bounded by $n$ times the error in each individual term of the
sum, which we bounded as $O(n^{-2})$ in Equation~\eqref{eq:term_bound}.

\paragraph{Near-linearity of bias and likely underestimation.}
Recall that for the statistical jackknife to yield a reliable estimate of the adjusted
accuracy, the bias in the na\"ive estimator must be analytic in
$\frac{1}{n}$, and in particular should be dominated by a $O(\frac{1}{n})$
term (as this corresponds to precisely the term accounted for by the
jackknife). In Figure~\ref{fig:analytic} we show that the bias estimated by
our jackknife procedure is indeed roughly linear in $1/n$, but grows slightly
faster than $\frac{1}{n}$, likely leading the jackknife to provide an
underestimate. 

\begin{figure}[!h]
	\centering
   \includegraphics[width=0.8\textwidth]{figures/jackknife_justification}
 	\caption{We plot the jackknife adjusted accuracy estimators of 10 models. On
 the y axis is shown the value of the $n$-sample jackknife estimate, with $1/n$
 on the x axis. The fact that the plot is nearly linear suggests that our bias is
     indeed dominated by a $O(1/n)$ term, thus further justifying our use
 of the statistical jackknife in Section~\ref{sec:jackknife}. Furthermore, the
 slightly accelerating slope as one moves left on the plot indicates that any
 error in the jackknife estimate is likely to be an underestimation of bias,
 rather than an overestimation.}
 	\label{fig:analytic}
\end{figure}


\subsection{Confidence Intervals}
To construct 95\% confidence intervals we perform 450 bootstrapped estimates (over
the included images) of adjusted accuracy for each classifier. We then plot the
2.5\% and 97.5\% percentiles from the bootstrap estimates as the confidence
intervals for each classifier.

\subsection{Varying Annotators}
We plot the results of varying the number of annotators in
Figure~\ref{fig:vary_an}.

\begin{figure}[!ht]
	\centering
  \input{plots/varying_annotators.tex}
	\caption{Replicating the \texttt{v1}\xspace vs \texttt{v2}\xspace accuracy plot using different numbers of
    annotators. We obtain similar results even as we decrease the number of
    annotators by less than half.}
	\label{fig:vary_an}
\end{figure}


\subsection{Varying Model Expressiveness}
We plot the results of varying the number of parameters (here, by changing the
number of beta distributions in our mixture) in Figure~\ref{fig:vary_mix}.

\begin{figure}[!ht]
	\centering
    \input{plots/varying_expressiveness.tex}
	\caption{Replicating the \texttt{v1}\xspace vs \texttt{v2}\xspace accuracy plot using different numbers of
    parameters. We obtain similar results even as we increase the number of
    parameters by more than four-fold.}
	\label{fig:vary_mix}
\end{figure}



\subsection{EM Algorithm for Mixture Fitting}
To fit the parameters of the beta-binomial mixture model we apply the
Expectation-Maximization algorithm, optimizing over mixture coefficients
$\{\pi_i\}$, as well as parameters of each mixture element
$\{(\alpha_i, \beta_i)\}$. Our application of the EM algorithm is rather
canonical---first, we compute membership probabilities $p_{j}^{i}$ for each
example $j$ with respect to each mixture element $i$, then minimize the
weighted log-likelihood with respect to the mixture probabilities.
Pseudocode is given in Algorithm~\ref{alg:em}.

\begin{algorithm}
    \caption{Our instantiation of the EM algorithm}
    \label{alg:em}
    \input{algs/em}
\end{algorithm}

\subsection{Sampled dataset accuracy increases with more workers}
\label{app:origdata_jackknife}
We begin by showing that the accuracy we observe on ImageNet-v2 depends on the
number of workers used to sample the dataset. 
We gradually decrease the number of workers $n$ used in computing observed
selection frequencies to study the effect of noise on statistic matching. We
find that model accuracy on the resulting replicated dataset degrades as $n$
decreases. For example, the accuracy gap from \texttt{v1}\xspace to the replication
increases from 12\% when $n = 10$, to 14\% when $n = 5$. This is consistent
with our model of statistic matching bias: fewer annotators means noisier
observed selection frequencies $\hs{x}$, which in turn amplifies the effect
of the bias, driving down model accuracies.

We use the
frequency-adjusted accuracy introduced in Section~\ref{sec:naive},
to estimate model accuracy on a version of the candidate pool reweighted to have
the same selection frequency distribution as ImageNet-v1:
\begin{align}
    \ensuremath{\estacc{\dflickrc}^{n}}\xspace = \sum_{k=0}^{n} 
        \evfl{f(x_{flickr}) \bigg\vert \hs{x_{flickr}} = \frac{k}{n}} 
        \cdot p_1\left(\hs{x_1} =\frac{k}{n}\right).
\end{align}
This estimator is analogous to the ImageNet-v2 selection process
of~\citet{recht2018imagenet}, but operates by reweighting the candidate pool
rather than filtering it. 

We plot this estimator in Figure~\ref{fig:orig_data_naive_est}, varying $n$ from $5$ to $10$. We
find that the gap between the adjusted accuracy and ImageNet accuracy shrinks as
$n$ grows, until shrinking to (and not plateauing at) $12.3\%$ at $n = 10$. This
behavior is predicted by statistic matching bias, and suggests that in the
infinite-annotator limit the ImageNet-v2 accuracy is higher. (Ideally, we
could estimate the infinite-annotator limit using the data
of~\citet{recht2018imagenet}, but there are not enough [10] annotators in
the original data to get a reliable estimate.)

\begin{figure}[hb]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/orig_data_naive_est}
    \caption{The frequency-adjusted accuracy gap between ImageNet-v1 and
    ImageNet-v2, using a varying number of annotators to estimate selection
    frequencies. The gap continually decreases, and does not
    plateau at 10 annotators. Bootstrapped 95\% confidence intervals are
    shown (shaded).} 
    \label{fig:orig_data_naive_est}
\end{figure}

\subsection{Measuring the selection frequency gap using held-out workers}
\label{app:sfgap}
Recall that most images collected by \citet{recht2018imagenet} have 10 annotations
per image, and that to get an unbiased estimate of the selection frequency we
must ``hold out'' some annotations per image (e.g. we should not use the same
annotations for both matching and measuring selection frequencies).
To that end, we split the annotations for each candidate image into a ``train''
set and a ``test'' set. We then mimic the original \texttt{v2}\xspace creation process using
only the train set of annotations, and use the remaining test annotations for
each image to obtain a held-out measurement of selection frequency. 

Since~\citet{recht2018imagenet} collect 10 annotations for most candidate
images, and since the original buckets used in the matching process are split
by boundaries $\{0, 0.2, 0.4, 0.6, 0.8, 1\}$, we use 5
annotations\footnote{We choose 5 annotations specifically since, as in the
10-annotation case, images fall into the same relative locations in each
bin---other choices of annotations per image are severely affected by binning
effects.} for each image in the train set and reserve the rest for
independently measuring the held-out frequency.

The results of this experiment are given in Table~\ref{tab:heldout_sf}. We
repeat the experiment using both 5 and 10 annotators to estimate ImageNet-v1
selection frequencies. In both cases, the
average selection frequency of the in-sample images overestimates the heldout
(true) selection frequency by 2-3\%, and the resulting replicated dataset has
lower selection frequency than ImageNet-v1.

\begin{table}[h]
    \centering
    \caption{Average in-sample and heldout selection frequencies for the
    experiment described in~\ref{app:sfgap}---the top (bottom) table presents the result
    of using 10 (5) annotators per image to estimate ImageNet-v1 selection
    frequencies. We use five annotators per image to 
    estimate selection frqeuencies, then use the filtering algorithm
    of~\citet{recht2018imagenet} to get a replicated dataset
    meant to match the selection frequency distribution of ImageNet-v1. 
    The results show that (a) bias results in the average selection frequency of the
    new sample being {\em lower} than that of ImageNet-v1, and that (b) the bias
    is undetectable without heldout samples.} 
    \vspace{1em}
    \label{tab:heldout_sf}
    \begin{tabular}{@{}lcc@{}} 
        \toprule
        & {\bf ImageNet-v1} & {\bf Sampled replication} \\ \midrule
        Average selection frequency & 0.71 & 0.71 \\
        Heldout selection frequency & 0.71 & 0.69 \\ \bottomrule
    \end{tabular}
    \begin{tabular}{@{}lcc@{}} 
        \toprule
        & {\bf ImageNet-v1} & {\bf Sampled replication} \\ \midrule
        Average selection frequency & 0.71 & 0.73 \\
        Heldout selection frequency & 0.71 & 0.70 \\ \bottomrule
    \end{tabular}
\end{table}

\paragraph{Effect size.}
The difference in held-out mean selection frequencies between \texttt{v1}\xspace and \texttt{v2}\xspace
in the table above is smaller than the one we observe in the newly collected data. 
However, as discussed in Section~\ref{sec:remeasure}, the size
of the average gap is not necessarily predictive of the size of the accuracy
correction. 
The latter depends on the distributional difference between true \texttt{v1}\xspace
and \texttt{v2}\xspace selection frequencies, rather than on just their first moments. 
In particular, observing different mean selection frequencies for \texttt{v1}\xspace and \texttt{v2}\xspace is a
sufficient but not necessary condition for there to be an accuracy gap\footnote{As
a concrete example, suppose that 50\% of the annotators used were low-quality
and did not complete the task (i.e., selected no images)---this
would artificially shrink the mean selection frequency gap by 50\%, but would
not affect the accuracy adjustment.}.

Unfortunately, getting an accurate estimate of the gap on the original data
seems impossible: first, there are insufficient workers to reliably apply any of our
techniques from Section~\ref{sec:quantifying}. Furthermore, with $k$ held-out
workers, we can only estimate the first $k$ moments of the true selection
frequency distributions $p_i(s)$, even if we had infinitely many images.
Overall, the problem is thus largely underdetermined. 

We can, however, show that original data is plausibly consistent with the 4\%
accuracy gap estimated using the new data (i.e., that such a gap is not ruled
out). Specifically, in \ref{app:origdata_jackknife},
the original accuracy difference between Flickr and ImageNet-v1 was 20.8\%. When
using five (ten) annotators per image, this gap shrunk to 14.0\% (12.2\%). 
In the newly collected data, the gaps for
the original distributions, 5-annotator adjustment, and 10-annotator adjustment are
11.7\%, 8.5\%, and 7.3\% respectively (again, by using the same reweighting
scheme as Appendix \ref{app:origdata_jackknife}). 
Therefore, accuracy adjustments incurred by using more
workers on the original data are significantly (about two times) larger than the
corresponding accuracy adjustments on the newly collected data, and so we expect
to see a larger total correction than our estimated 8.1\% correction.

\subsection{Source selection frequencies determine sampled dataset accuracy}
We next explore how the choice of source distribution impacts the resulting
sampled dataset. We use a setup similar to that of the last experiment, in which
we use five workers for the selection process. Then, using four hold-out samples
from each image, we create a new candidate data pool called Flickr-E
(Flickr-Easy) by including only the images which at least two out of the
four heldout workers selected. 


We then perform 5-worker statistic matching, both from Flickr and from Flickr-E
to ImageNet-v1. In the absence of bias, the source distribution should not
affect the accuracy of the resulting classifier. In contrast, we find that the
dataset replication obtained from Flickr-v2 has comparable (within 0.2\%)
average selection frequency, but significantly higher accuracy (by
\textasciitilde 3\%) than the
replication obtained from the unfiltered candidate pool (62\%).
This discrepancy further corroborates the hypothesis that ImageNet-v2 accuracies
are impacted by the statistical bias that we identify in this work.




\section{Introduction}
\label{sec:introduction}
\input{sections/introduction}

\section{Identifying Sources of Reproduction Bias}
\label{sec:id_bias}
\input{sections/id_bias}

\section{Remeasuring Selection Frequencies}
\label{sec:remeasure}
\input{sections/remeasure}

\section{Understanding the Accuracy Gap}
\label{sec:exps}
\input{sections/closer_look}

\section{Quantifying the Bias}
\label{sec:quantifying}
\input{sections/quantifying}

\section{Related Work}
\label{sec:rel_work}
\input{sections/rel_work}

\section{Discussion and Conclusions}
\label{sec:implications}
\input{sections/implications}

\section*{Acknowledgements}
\input{sections/acks}

\pagebreak
\printbibliography

\pagebreak

























\subsection{Notation and terminology}
Here we overview the notation and terminology useful in discussing the bias in
ImageNet-v2 accuracy. 

\paragraph{Selection frequencies.} In Section~\ref{sec:id_bias} we defined the
true selection frequency $s(x)$ for an image $x$ to be the (population) rate at
which crowd annotators select the image as ``correctly labeled.'' The true
selection frequency of an image is unobservable, and often approximated by the
observed selection frequency, $\hs{x} \sim \frac{1}{n} \binomial{\s{x}}$, which
can be estimated from an $n$-annotator MTurk experiment.
When $n$ is clear from context we will often omit it and write $\hat{s}(x)$.

\paragraph{Distributions.} We will use $\ensuremath{\mathcal{D}_1}\xspace$ and $\ensuremath{\mathcal{D}_2}\xspace$ to denote the
distributions of \texttt{v1}\xspace and \texttt{v2}\xspace images respectively,
and $\ensuremath{\mathcal{S}_1}\xspace$ and $\ensuremath{\mathcal{S}_2}\xspace$ to denote the corresponding finite test sets.
As in Section~\ref{sec:id_bias}, we
denote by $p_i(s)$ the probability density of true selection frequencies for
images drawn from $\ensuremath{\mathcal{D}_i}\xspace$ (or Flickr, if $i=flickr$). Similarly, we use
$p_i(\hs{x})$ to denote the probability mass function of the
observed selection frequency for dataset $i$.

We let ${\ensuremath{\mathcal{D}_2\vert s_1}\xspace}$ be the distribution of ImageNet-v2 images reweighted to have the
same selection frequency distribution as ImageNet-v1. Formally, ${\ensuremath{\mathcal{D}_2\vert s_1}\xspace}$ is
the compound distribution $\left( x_2 \sim \ensuremath{\mathcal{D}_2}\xspace \vert s(x_2) 
\sim p_1(s) \right)$. Sampling from ${\ensuremath{\mathcal{D}_2\vert s_1}\xspace}$ corresponds to first sampling a
\texttt{v1}\xspace{} image $x_1$, then sampling an image $x_2$ from the \texttt{v2}\xspace{} distribution
conditioned on $s(x_2) = s(x_1)$.  

\paragraph{Accuracies.} For a classifier $c$, let $f_c(x)$ be an indicator
variable of whether $c$ correctly classifies $x$. Since our analysis applies to
any fixed classifier $c$, we omit it and use $f(x)$.
We then define $\acc{X}$ to represent classifier accuracy on distribution or
test set $X$---for example, classifier accuracy on \texttt{v1}\xspace{} is given by
$$\acc{\ensuremath{\mathcal{D}_1}\xspace} = \pva{f(x_1) = 1} = \eva{f(x_1)}.$$ 

\subsection{Breaking down the accuracy gap} 
The accuracy gap between the \texttt{v1}\xspace{} and \texttt{v2}\xspace{} test sets is given by $\acc{\ensuremath{\mathcal{S}_1}\xspace} -
\acc{\ensuremath{\mathcal{S}_2}\xspace}$. What fraction of this gap can be attributed to bias in selection
frequency? To answer this, we decompose this accuracy gap into three elements
whose contribution can be studied separately:
\begin{align}
  \label{eq:decomp}
    \acc{\ensuremath{\mathcal{S}_1}\xspace} - \acc{\ensuremath{\mathcal{S}_2}\xspace}  
    &= \underbrace{\left(\acc{\ensuremath{\mathcal{S}_1}\xspace} - \acc{\dvbc}\xspace\right)}_{\text{bias-corrected accuracy gap}}
            + \underbrace{\left(\acc{\dvbc}\xspace - \acc{\ensuremath{\mathcal{D}_2}\xspace}\right)}_{\text{selection gap}} +
            \underbrace{\Big(\acc{\ensuremath{\mathcal{D}_2}\xspace} - \acc{\ensuremath{\mathcal{S}_2}\xspace}\Big)}_{\text{finite
            sample gap}\ \approx\ 0}.
\end{align}

\paragraph{Bias-corrected accuracy gap.} The first term of~\eqref{eq:decomp},
called the bias-corrected accuracy gap, captures the portion of the \texttt{v1}\xspace-\texttt{v2}\xspace
accuracy drop that {\em cannot} be explained by a difference in selection
frequency, and instead might be explained by benign distribution shift or
adaptive overfitting\footnote{In fact, this term can be further decomposed into
a sum of an adaptivity gap $(\acc{\ensuremath{\mathcal{S}_1}\xspace} - \acc{\ensuremath{\mathcal{D}_1}\xspace})$, and a distribution shift
gap $(\acc{\ensuremath{\mathcal{D}_1}\xspace} - \acc{\dvbc}\xspace).$ However, since we don't have access to
$\acc{\ensuremath{\mathcal{D}_1}\xspace}$ it is difficult to disentangle these.}.


\paragraph{Selection gap.} The second term of~\eqref{eq:decomp} is accuracy gap
that can {\em only} be attributed to selection frequency, since it
compares accuracy on $\ensuremath{\mathcal{D}_2}\xspace$ to accuracy on a reweighted version of $\ensuremath{\mathcal{D}_2}\xspace$.
If there was no bias, and the distribution of selection frequencies for \texttt{v1}\xspace{}
and \texttt{v2}\xspace{} matched exactly, then this term would equal zero (${\ensuremath{\mathcal{D}_2\vert s_1}\xspace}$ would equal
$\ensuremath{\mathcal{D}_2}\xspace$).  
Thus, the selection gap translates the effect of discrepancy in true selection
frequency between \texttt{v1}\xspace and \texttt{v2}\xspace into a discrepancy in accuracy.
Since we measured \texttt{v1}\xspace as having higher true selection frequency,  we expect the
selection gap to be positive and thus explain a portion of the accuracy gap that
was previously attributed to distribution shift.

\paragraph{Finite-sample error.} The final term refers to the
finite-sample error from using $10,000$ images as a proxy for
distributional accuracy. We believe that this term is negligible, since (a) 95\%
bootstrapped confidence intervals for the classifiers we evaluate are all at
most $0.1\%$, and (b) there can be no adaptive overfitting on $\ensuremath{\mathcal{S}_2}\xspace$ with
respect to $\ensuremath{\mathcal{D}_2}\xspace$. Thus, we drop this term from consideration and instead use
$\acc{\ensuremath{\mathcal{D}_2}\xspace}$ and $\acc{\ensuremath{\mathcal{S}_2}\xspace}$ interchangeably.\\

\paragraph{Computing selection-adjusted accuracy.} We have shown how to
decompose the \texttt{v1}\xspace-\texttt{v2}\xspace accuracy gap into a component explained by selection
frequency (selection gap), and a component unexplained by selection frequency
(bias-corrected accuracy gap). 
The challenge in computing this decomposition is estimating $\acc{\dvbc}\xspace$, the
selection-adjusted \texttt{v2}\xspace{} accuracy. While the closed form of $\acc{\dvbc}\xspace$ is
\begin{align*}
    \int_s \mathbb{E}_{\ensuremath{\mathcal{D}_2}\xspace}[f(x)|s(x)=s] \cdot p_1(s)\ ds,
\end{align*}
we have no access to $p_i(s)$ for any value of $i$ (we do not even have direct
access to $s(x)$ for any image $x$). In the next section, we explore methods for
estimating $\acc{\dvbc}\xspace$ using only the observed selection frequencies that we
collected.


\subsection{Remaining accuracy gap and unmodeled bias}
\paragraph{Worker heterogeniety.} Our study focuses on bias stemming from the
fact that for a given image $x$ one never observes $\s{x}$ but rather $\hs{x} =
\binomial{\s{x}}$. There is another source of bias due to noise that we
do not model here, namely variance in the MTurk annotator population.
Specifically, some annotators are more likely in general to select or reject
independently of what image-label pair they are being shown. This unmodeled
variance likely translates to unmodeled bias, suggesting that more of the gap
might be explained by taking worker heterogeniety into account.

\paragraph{Task shift bias.} At the time of the original ImageNet experiment,
workers judged image-label pairs by some abstract set of criteria $C_1$. Suppose
that at the time of the ImageNet-v2 experiment several years later, annotators
judged image-label pairs based on an overlapping but non-identical set of
criteria $C_2$. Ideally, we should not care about differences between $C_1$ and
$C_2$---indeed, one of the goals of dataset replication is to test robustness to
such benign distribution shifts. The source of the bias lies in the iterated
nature of the filtering experiment. In particular, after both the original
experiment and the replication, images in ImageNet-v1 now meet both $C_1$ and
$C_2$. On the other hand, images in ImageNet-v2 only meet criteria $C_2$, and
may be judged to have low selection frequency under $C_1$---we would thus expect
models to perform better on ImageNet-v1 images due to their increased
qualifications. Although this may contribute towards the remaining accuracy gap,
this type of bias is difficult to study or correct for without more knowledge of
both experiments.

\paragraph{Other sources of error.} The remaining error unexplained by 
bias in data collection could come from one of the gap sources listed in
Section~\ref{sec:exps}, i.e., finite sample error, or distribution shift
and adaptive overfitting. 
Quantifying the potential contribution of the individual terms in the remaining gap
will require more experimentation and future work.

\subsection{Adaptive overfitting and distribution shift}
\paragraph{Identifying sources of distribution shift.} A longstanding goal in
computer vision is to develop models that are less prone to failure under small
distributional shifts. A step in the journey towards this goal is precisely
characterizing the kinds of distribution shifts under which models
fail---examples include rotations and translations of natural
images~\citet{engstrom2019rotation}, or corrupted natural
images~\citep{hendrycks2019benchmarking}. Our findings imply that the drop may
be attributable to differences in selection frequency distribution,
corroborating observations by~\citet{recht2018imagenet} that models are
sensitive to selection frequency. Differences in selection frequency
distribution present another distribution shift to study in depth.

\paragraph{Implications for adaptive overfitting.} A conclusion drawn
from the ImageNet-v2 dataset replication is that 
the slope of best fit line for \texttt{v1}\xspace{} versus \texttt{v2}\xspace{}
accuracy is significantly larger than one. This means that for every point of
progress made on ImageNet, $1.1$ points are made on ImageNet-v2, providing
another point of evidence towards the absence of adaptive overfitting. However,
after correcting for the bias in the \texttt{v2}\xspace{} sampling process, \texttt{v1}\xspace{} versus \texttt{v2}\xspace{}
model accuracies still exhibit a linear fit, but with a slope that is not
conclusively bounded away from $1$ (i.e., slopes are within 95\% confidence
intervals of $1$, c.f. Section~\ref{sec:parametric}). 

\paragraph{Detecting and avoiding bias in dataset replication.} 
More broadly, our analysis identifies statistical modeling of the data
collection pipeline as a useful tool for dataset replication. Indeed,
characterizing the ImageNet and ImageNet-v2 generative processes and isolating
them in a a simple theoretical model allowed for the discovery and correction of
a source of bias in the dataset replication process.





\subsection{Na\"{i}ve approach}
\label{sec:naive}
\input{sections/naive}

\subsection{Estimating bias with the statistical jackknife}
\label{sec:jackknife}
\input{sections/jackknife}

\subsection{Estimating bias with a parametric model}
\label{sec:parametric}
\input{sections/parametric}



