\section{Introduction}

In response to the Covid-19 pandemic, governments implemented a large number of policies in order to try to slow the spread of Covid-19.  Most immediate policy responses were non-pharmaceutical interventions such as lockdowns, school closures, mandates to wear masks, and contact tracing, among others.  In the current paper, we focus on trying to identify and estimate the effect of these sorts of policies.  We are interested in both the direct effect of these Covid-19 related policies on the number of Covid-19 cases as well the effect of these policies on other outcomes (particularly, economic outcomes).  Understanding both of these types of effects is important for evaluating the tradeoffs arising from various policies.


A main challenge with evaluating the effects of policies on the number of Covid-19 cases is that, especially during the early part of the pandemic when testing was not widely available, the confirmed number of cases in a particular location may have been quite different from the actual number of cases.\footnote{To be clear on the terminology, we use the phrase \textit{total cases} or \textit{actual cases} to refer to the total number of Covid-19 cases -- this is in general not observed.  We use the phrase \textit{confirmed cases} to refer to the number of positive tests for Covid-19.}  \Cref{fig:imhe_estimates} plots the number of confirmed cases and estimated number of total cases (these estimates are from the Institute for Health Metrics and Evaluation) from February to December 2020.  Notice the large gap between confirmed and estimated cases and how this gap is largest in the early part of the pandemic.

Besides limited testing early in the pandemic, another complication is that testing was non-random.   In particular (and especially early in the pandemic), individuals who were more likely to have Covid-19 appear to be much more likely to take the test due to both self-selection as well as eligibility requirements for taking the test such as being in a particular risk group and/or exhibiting certain symptoms.  Moreover, the availability of tests varied across locations.  This creates the issue that, holding the number of cases fixed, ``more testing'' can mechanically lead to confirming more cases.  If confirmed cases are treated as true cases, then these issues can lead to faulty inferences regarding the effects of policies on the actual spread of Covid-19.

\FloatBarrier

\begin{figure}[t]
    \centering
    \caption{Estimated and Confirmed Covid-19 Cases}
    \label{fig:imhe_estimates}
    \includegraphics[width=\textwidth]{figures/imhe_estimated_cases.pdf}
    \begin{flushleft}
    { \footnotesize \textit{Notes:} Plots of daily confirmed cases per 100,000 persons (the blue line) and IMHE estimates of new actual cases (the red line) in the United States from February 4, 2020 to December 31, 2020. }
    \end{flushleft}
\end{figure}


In order to evaluate the effects of a Covid-19 related policy early in the pandemic, our strategy is to take treated locations and compare them to untreated locations with similar characteristics before the policy was implemented.  In particular, we compare locations that are similar along dimensions that are related to the spread of Covid-19 (e.g., population size and per capita number of actual cases) as well as the per capita number of confirmed cases and number of tests that have been taken in that location.  This is an unconfoundedness-type strategy, and, unlike difference in differences, is compatible with leading pandemic models in the epidemiology literature (\textcite{callaway-li-2021b}).  


This would be a relatively straightforward exercise if testing were administered randomly; but, as discussed above, that was not the case.    In practice, this creates challenges along two dimensions.  First, ideally we would like to compare locations that had experienced a similar number of actual cases up to the time period when the policy was implemented.  In other words, for evaluating Covid-19 related policies, the actual number of cases in pre-treatment periods plays an important role as a covariate.  But conditioning on the number of pre-treatment cases is not directly possible because actual cases are not observed.  Second, understanding the effect of the policy on actual cases is of primary interest itself, which, of course, is more challenging when actual cases are not directly observed.




In the current paper, we propose a new approach for dealing with nonrandomly missing testing data.  We address the first challenge by introducing a missing at random-type assumption: that the distribution of pre-treatment actual Covid-19 cases is the same for treated and untreated locations conditional on having the same values of observed pre-treatment characteristics (the most important of which are confirmed cases, number of tests, and population size).  Under this sort of condition, the challenge of pre-treatment Covid-19 cases being an unobserved, but important, covariate is effectively solved; this further implies that, under this condition, the effect of a Covid-19 related policy on observed outcomes (such as economic outcomes) is point identified.

Addressing the second issue is more challenging.  Our intuition is that observing a policy that simultaneously decreases the number of confirmed cases while increasing (or at least not decreasing) the number of tests provides a strong indication that the policy is decreasing the actual number of Covid-19 cases in that location (even though these are not fully observed).  We formalize what extra conditions are required for this sort of intuition to hold.  In particular, the key extra condition is that the number of actual cases among the untested is not too large under the policy relative to what the number of actual cases among the untested would have been if the policy had not been implemented (note that the number of actual cases among the untested is not observed in either case).  Under these conditions, differences in (a scaled version of) the number of confirmed cases in a particular location relative to the number of confirmed cases among untreated locations with similar pre-treatment characteristics can be interpreted as an upper bound on the effect of the policy on the actual number of Covid-19 cases.  If this upper bound is negative, it suggests that the policy decreased the number of actual Covid-19 cases.

We also point out some more primitive conditions where the key extra condition that the number of actual cases among the untested under the policy is not larger than the number of actual cases among the untested had the policy not been implemented.  In particular, this extra condition will hold when: (i) the probability of having had Covid-19 is lower among untested individuals than among tested individuals, (ii) among a fixed group of untested individuals (those untested under the policy), the policy does not directly increase the probability of having had Covid-19, (iii) ruling out some forms of negative selection into taking the test under the policy (i.e., ruling out that those who are unlikely to have Covid-19 are systematically more likely to be tested under the policy), and (iv) testing does not become less available under the policy.  These are all relatively weak assumptions, but we show that they provide substantial identifying power and potentially informative bounds on policy effects even in the presence of limited, nonrandom testing.  Thus, the leading case where our approach would indicate that a policy early in the pandemic decreased the number of Covid-19 cases is when (i) the number of confirmed cases decreases in treated locations relative to untreated locations with similar pre-treatment characteristics, (ii) the number of tests does not decrease, on average, in treated locations relative to untreated locations with similar pre-treatment characteristics, and (iii) a researcher feels confident making the extra (typically weak) additional sorts of assumptions mentioned above.

We apply our approach to studying the effects of Tennessee's open-testing policy early in the pandemic.  This policy removed eligibility requirements (such as exhibiting certain symptoms or being in a high risk group) for taking Covid-19 tests in Tennessee.  For observed outcomes such as confirmed cases and trips to work (which are relatively simpler due to only suffering from the first issue mentioned above), our results indicate that the open-testing policy (i) decreased the number of confirmed cases, and (ii) decreased the amount of travel to work in counties that experienced relatively large increases in their number of confirmed cases over time.  For the per capita number of total Covid-19 cases (which is the relatively harder case due to suffering from both issues mentioned above), we obtain non-trivial bounds on the effect of the policy suggesting that the policy has reduced the number of total cases in Tennessee relative to what they would have been in the absence of the policy.  The most important driver of these results is that it simultaneously appears that the open-testing policy has increased the number of tests while decreasing the number of confirmed cases -- together, as discussed above, these form a strong piece of evidence that the policy has reduced total cases even though total cases are unobserved and it is hard to come up with reasonable assumptions that lead to point identification of the effect of the policy on total cases.  Taken together, these results suggest that Tennessee's open-testing policy has had a positive effect along several dimensions.  



\bigskip

\noindent \textbf{Related Work:} 

Our paper is related to a large and rapidly growing literature evaluating the effects of Covid-19 related policies.  Some examples include \textcite{courtemanche-garuccio-le-pinkston-yelowitz-2020}, \textcite{dave-friedson-matsuzawa-mcnichols-sabia-2020}, \textcite{dave-friedson-matsuzawa-sabia-2020}, \textcite{dave-friedson-matsuzawa-sabia-safford-2020}, \textcite{gapen-millar-blerina-sriram-2020}, \textcite{glaeser-jin-leyden-luca-2020}, \textcite{goolsbee-syverson-2021}, \textcite{gupta-montenovo-nguyen-rojas-schmutte-simon-weinburg-2020}, \textcite{juranek-zoutman-2020}, \textcite{kong-prinz-2020}, \textcite{mitze-kosfeld-rode-walde-2020}, and \textcite{ziedan-simon-wing-2020}, among others. Other methodological work on evaluating Covid-19 related policies includes \textcite{allcott-boxell-conway-ferguson-gentzkow-goldman-2020}, \textcite{callaway-li-2021b}, \textcite{chernozhukov-kasaha-schrimpf-2020}, \textcite{goodman-marcus-2020}, and \textcite{rowthorn-maciejowski-2020}.  All of these papers are quite distinct from ours as they propose approaches that result in point identification and are not generally as concerned as we are about nonrandomly missing testing data.  More closely related to our paper is \textcite{manski-molinari-2020} which proposes an approach to bound the actual number of Covid-19 cases in the presence of nonrandomly missing testing data.  Our approach extends these types of arguments to the case where a researcher is interested in policy evaluation. 

On the econometrics side, our approach is related to work on partially identifying treatment effect parameters in the presence of sample selection such as \textcite{lee-2009,lechner-melly-2010}.  Like those papers, the key issues here are that the outcome (in our case, whether or not an individual has had Covid-19) is only observed for some self-selected individuals (in our case, tested individuals) and that the policy can affect both the outcome and the set of individuals for whom the outcome is observed.  Unlike those papers, our main interest is in a different parameter (discussed below), and we invoke different assumptions to obtain bounds.  More generally, the results in this section are related to a large literature on partial identification, particularly partial identification of treatment effect parameters.  See \textcite{tamer-2010,molinari-2020} for recent reviews of this literature.






\section{Methodology} \label{sec:methodology}

This section discusses our methodological approach.  We start by briefly discussing bounds on actual Covid-19 cases in a particular location.  This discussion largely follows \textcite{manski-molinari-2020} and discusses (mild) assumptions that have been made in this context to bound per capita total Covid-19 cases early in the pandemic.  This part also illustrates several of the key issues regarding non-randomly missing data.  Then, we discuss our approach to (point-) identifying Covid-19 related policy effects on observed outcomes in the case where we would like to condition on pre-treatment actual Covid-19 cases for identification.  Finally, we discuss our contribution to developing bounds on the effect of the policy on per capita actual Covid-19 cases.  In both cases, our approach is able to deal with limited/nonrandom Covid-19 testing.

\subsection{Bounding Rates of Total Covid-19 Cases} \label{sec:descriptive-bounds}

\paragraph{Notation:}
\begin{itemize}
\item $C_{ilt}$ -- a binary variable for whether or not individual $i$ in location $l$ has had Covid-19 by time period $t$.
\item $R_{ilt}$ -- a binary variable for whether or not individual $i$ in location $l$ has tested positive for Covid-19 by time period $t$
\item $T_{ilt}$ -- a binary variable for whether or not individual $i$ in location $l$ has taken a test for Covid-19 by time period $t$
\end{itemize}


Our first goal is descriptive:  to see what fraction of the population has had Covid-19 by time period $t$ in a particular location $l$.  That is, our interest centers on $P(C_{ilt}=1)$.  To be clear about the notation here, this is the fraction of the population in location $l$ at time period $t$ that has had Covid-19.  That is, we are averaging over all individuals in a particular location $l$ at time period $t$.  Identifying the fraction of individuals that have ever had Covid-19 is challenging because (i) not all individuals have been tested and (ii) for individuals that have been tested for Covid-19, testing has not been randomly assigned.  The goal of this section is to develop non-trivial bounds on the fraction of total Covid-19 cases in a particular location at a particular time under plausible identifying assumptions.  In particular, following \textcite{manski-molinari-2020},\footnote{Other recent work on estimating the number of cases in the presence of nonrandomly missing data includes \textcite{hortacsu-liu-schwieg-2020} though their approach is substantially different from the approach taken in this section.} notice that
\begin{align} \label{eqn:covid-bound-1}
  P(C_{ilt}=1) = P(C_{ilt}=1|T_{ilt}=1) P(T_{ilt}=1) + P(C_{ilt}=1|T_{ilt}=0) P(T_{ilt}=0)
\end{align}
which follows immediately by the law of total probability.  Next, consider each of these terms individually:
\begin{itemize}
\item $P(C_{ilt}=1|T_{ilt}=1)$.  This is the fraction of the population in location $l$  at time period $t$ that has had Covid-19 conditional on being tested.  We discuss this term in more detail below.
\item $P(T_{ilt}=1)$ is the (observed) fraction of the population in location $l$ at time period $t$ who have been tested for Covid-19.
\item $P(T_{ilt}=0)$ is the (observed) fraction of the population in location $l$ at time period $t$ who have not been tested for Covid-19.
\item $P(C_{ilt}=1|T_{ilt}=0)$ is the (unobserved) fraction of the population that have had Covid-19 but have not been tested in location $l$ by time period $t$.  This term is the hardest to identify, and we discuss plausible assumptions that lead to bounds on this term below.
\end{itemize}
Next, consider $P(C_{ilt}=1|T_{ilt}=1)$.  It can be written as
\begin{align*}
  P(C_{ilt}=1|T_{ilt}=1) &= P(C_{ilt}=1 | T_{ilt}=1, R_{ilt}=1)P(R_{ilt}=1|T_{ilt}=1) \\
                         & \hspace{10pt} + P(C_{ilt}=1, | T_{ilt}=1, R_{ilt}=0) P(R_{ilt}=0|T_{ilt}=1) \\
                         %
                         &= P(R_{ilt}=1|T_{ilt}=1) + P(R_{ilt}=0|T_{ilt}=1,C_{ilt}=1) P(C_{ilt}=1|T_{ilt}=1)
\end{align*}
where the first equality holds by the law of total probability and the second equality holds because (i) $R_{ilt} = 1 \implies T_{ilt}=1$  (i.e., in order to test positive, an individual has to be tested), (ii) we suppose that the false positive rate of the test is equal to 0 which implies that $P(C_{ilt}=1|R_{it}=1)=1$,\footnote{The false positive rate is given by $P(C_{ilt}=0|R_{ilt}=1)$, and there is evidence that the false positive rates for PCR tests are extremely low; see, for example \textcite{sethuraman-jeremiah-ryo-2020}.} and (iii) repeated application of the definition of conditional probability for the second term.  Then, rearranging implies that
\begin{align} \label{eqn:CcondT}
  P(C_{ilt}=1|T_{ilt}=1) = \frac{P(R_{ilt}=1|T_{ilt}=1)}{1-FNR}
\end{align}
where
\begin{itemize}
\item $P(R_{ilt}=1|T_{ilt}=1)$ is the (observed) fraction of tests that have come back positive in location $l$ at time period $t$.
\item $FNR := P(R_{ilt}=0|T_{ilt}=1,C_{ilt}=1)$ is the false negative rate of the test.  This is a property of the test, and we set the false negative rate to be equal to 0.25.\footnote{\textcite{manski-molinari-2020} put bounds on a closely related term called the Negative Predictive Value of the test; we could similarly put bounds on the false negative rate of the test.  We do not do this in the current paper in order to mainly focus on the bounds arising from non-random testing.  In the results presented below, in general, the bounds are not very sensitive to different reasonable values of the false negative rate of the test.  More specifically, a higher false negative rate increases both the lower bound and the upper bound, but it increases the upper bound relatively more under \Cref{ass:covid-bound} (see \Cref{eqn:covid-bound-2} below); thus bounds tend to be somewhat wider (although the actual difference is small) under larger values of the false negative rate of the test.}
\end{itemize}
\Cref{eqn:CcondT} says that the probability of having Covid-19 conditional on being tested is increasing in the fraction of positive tests and the false negative rate of the test.  It also implies that every term in \Cref{eqn:covid-bound-1} is identified except $P(C_{ilt}=1|T_{ilt}=0)$.  Without employing some additional assumption on this term, the bounds on the rate of total cases are given by
\begin{align} \label{eqn:covid-bound-1b}
  \frac{P(R_{ilt}=1)}{1-FNR} \leq P(C_{ilt}=1) \leq \frac{P(R_{ilt}=1)}{1-FNR} + P(T_{ilt}=0)
\end{align}

In most cases, these sorts of bounds would be extremely wide.  For example, in our application about Tennessee's open-testing policy, for the whole state of Tennessee, $P(R_{ilt}=1)$ is about 0.2\% and $P(T_{ilt}=0)$ is about 97\% (i.e., about 3\% of Tennessee's population had been tested and about 0.2\% had a positive test).  If the only restriction on $P(C_{ilt}=1|T_{ilt}=0)$ is that it is bounded between 0 and 1, then this will lead to extremely wide bounds on Covid-19 cases (essentially uninformative).  Instead (and continuing to follow \textcite{manski-molinari-2020}), we make the following assumption.

\begin{assumption}[Covid-19 Bound for Untested Individuals] \label{ass:covid-bound}
  \begin{align*}
    P(C_{ilt}=1|T_{ilt}=0) \leq P(C_{ilt}=1|T_{ilt}=1)
  \end{align*}
\end{assumption}

\Cref{ass:covid-bound} says that the fraction of individuals who have had Covid-19 (in a particular location) is lower among the group of individuals who have not been tested than among those who have been tested.  This is a mild assumption.  This assumption is likely to hold for two reasons.  First, early in the pandemic, tests were predominantly given to individuals expressing Covid-19 symptoms.  Second, even in states (or time periods) where testing has been available to anyone who wants to take a test, (i) individuals expressing symptoms are still among those most likely to take the test and (ii) it seems likely that there is some self-selection into taking the test among individuals who thought they may have Covid-19 even if they did not have the right combination of symptoms to otherwise warrant a test.  It is also helpful to think about the limiting cases of the assumption.  $P(C_{ilt}=1|T_{ilt}=0)=0$ in the case when no untested individuals have had Covid-19.  $P(C_{ilt}|T_{ilt}=0) = P(C_{ilt}=1|T_{ilt}=1)$ if the probability of having had Covid-19 is the same for individuals who have not been tested as for individuals who have been tested.  This condition would hold if testing were randomly assigned.  In practice, either of these limiting conditions would be strong enough to point identify $P(C_{ilt}=1)$; however, based on the above discussion, neither of these limiting conditions seems likely to hold.  Instead, \Cref{ass:covid-bound} imposes the much weaker condition that the probability of having had Covid-19 for the group of individuals who have not been tested falls in between these two limiting cases.

\Cref{ass:covid-bound} does not affect the lower bound on the total number of cases, but it is potentially very useful in lowering the upper bound on the  number of Covid-19 cases in a particular location.  In particular, notice that under \Cref{ass:covid-bound},
\begin{align} \label{eqn:covid-bound-2}
  P(C_{ilt}=1) \leq P(C_{ilt}=1|T_{ilt}=1) 
\end{align}
This can lead to a much tighter bound especially when $P(C_{ilt}=1|T_{ilt}=1)$ is substantially less than one.  For example, in our application, for the whole state of Tennessee, $P(C_{ilt}=1|T_{ilt}=1)$ is roughly equal to 8\%.  This immediately leads to a much tighter bound on the total number of cases relative to not putting any restrictions on $P(C_{ilt}=1|T_{ilt}=0)$.  

\subsection{Policy Evaluation with Nonrandomly Missing Data}

The previous section discussed how to bound the total number of Covid-19 cases in a particular location.  The main goal of the paper is to go beyond these descriptive bounds and evaluate how a policy affects the (unobserved) total number of Covid-19 cases as well as other outcomes such as confirmed cases and trips to work.  We discuss our approach in this section.

For this section, our arguments are about policy effects in certain locations and, therefore, we slightly modify the notation from the previous section.  In particular, define $C_{lt} := P(C_{ilt}=1)$, $R_{lt} := P(R_{ilt}=1)$, $T_{lt} := P(T_{ilt}=1)$.  These are defined for a particular location (rather than for a particular individual) and correspond to the fraction of the population in location $l$ that has had Covid-19, that have tested positive for Covid-19, and that have been tested for Covid-19, respectively.\footnote{Also, notice that we do not need to estimate $R_{lt}$ and $T_{lt}$; rather each of them is exactly observed.}  We also suppose that we have access to location-level covariates $X_l$ that do not vary over time; in practice, the most important covariate is the total population in a particular location.  Some of the results below consider policy effects on other outcomes; in that case we denote the location-level outcome in time period $t$ by $Y_{lt}$.

In order to think about policy effects, we define potential outcomes for location $l$ in time period $t$.  In particular, let $C_{lt}(1)$, $T_{lt}(1)$, $R_{lt}(1)$, and $Y_{lt}(1)$ denote the per capita number of total Covid-19 cases, the per capita number of tests, the per capita number of positive tests, as well as other outcomes that would occur in location $l$ in time period $t$ if the policy were in place.  Similarly, if the policy is not in place for location $l$ in time period $t$, we denote the untreated potential outcomes that would occur in this case by: $C_{lt}(0)$, $T_{lt}(0)$, $R_{lt}(0)$, and $Y_{lt}(0)$.  To conserve on notation, define $Z_{lt}(d) = (Y_{lt}(d), R_{lt}(d), T_{lt}(d), X_l')'$ for $d \in \{0,1\}$.  This collects the covariates and all potential outcomes except for $C_{lt}(d)$. Also, define $Z^*_{lt}(d) = (Z_{lt}(d)',C_{lt}(d))'$ which additionally includes per capita actual cases.

Next, let $D_l$ be a binary variable indicating treatment participation.  For locations that participate in the policy, $D_l=1$; otherwise, $D_l=0$.  Also suppose that  there are two time periods: $t^*$ and $t^*-1$,\footnote{Our results extend immediately to the case where there are more available time periods.} and that the policy is implemented between time periods $t^*$ and $t^*-1$.  In this setup, we observe
\begin{align*}
  Z_{lt^*} = D_l Z_{lt^*}(1) + (1-D_l)Z_{lt^*}(0) \quad \textrm{and} \quad Z_{lt^*-1} = Z_{lt^*-1}(0)
\end{align*}
In other words, in post-treatment time periods we observe treated potential outcomes for locations that participate in the treatment and observe untreated potential outcomes for locations that do not participate in the treatment.  In pre-treatment time periods, we observe untreated potential outcomes for all locations. %



\subsubsection{Policy Effects on Economic Outcomes}
To start with, consider identifying the effect of a Covid-19 related policy on some observed outcome (e.g., in our application on Tennessee's open-testing policy, we consider the effect of the policy on the number of confirmed Covid-19 cases and the number of trips to work) in location $l$ at time period $t^*$.  We start with this case because it is simpler as $Y_{lt}$, the outcome, is fully observed while $C_{lt}$, the per capita number of total cases in location $l$, is not.

Our interest in this section is in identifying
\begin{align*}
  ATT_Y(Z_{lt^*-1}) = E[ Y_{lt^*}(1) - Y_{lt^*}(0) | Z_{lt^*-1}, D_{l}=1] \quad \textrm{and} \quad ATT_Y = E[Y_{lt^*}(1) - Y_{lt^*}(0)|D_l=1]
\end{align*}
$ATT_Y(Z_{lt^*-1})$ is the average effect of the  policy on the outcome among treated locations with pre-treatment characteristics $Z_{lt^*-1}$.  $ATT_Y$ is the overall average effect of the policy among treated locations. 

We make the following assumption
\begin{assumption}[Unconfoundedness] \label{ass:unc}
  \begin{align*}
    E[Y_{lt^*}(0) | Z^*_{lt^*-1}(0), D_l=1] = E[Y_{lt^*}(0) | Z^*_{lt^*-1}(0), D_l=0]
  \end{align*}
\end{assumption}

\Cref{ass:unc} is a standard and widely used assumption to identify the affect of some economic policy (see, for example, \textcite{imbens-wooldridge-2009}).  It says that, if the policy had not been enacted, on average, outcomes in treated locations would have been the same as outcomes in untreated locations that had the same pre-treatment characteristics; i.e., the same outcomes in the previous period, the same per capita number of confirmed cases, the same number of per capita tests, the same population, \textit{as well as the same per capita number of total cases}.\footnote{It is also worth pointing out that \Cref{ass:unc} is not a main requirement of our approach.  Depending on the particular outcome of interest, one could employ an alternative baseline identification strategy.  To give one particular leading example, one could replace the unconfoundedness assumption in \Cref{ass:unc} with a conditional parallel trends assumptions (i.e., that $E[\Delta Y_{lt^*}(0) | Z^*_{lt^*-1}(0), D_l=1] = E[\Delta Y_{lt^*}(0) | Z^*_{lt^*-1}(0), D_l=0]$) and then show an analogous result to \Cref{prop:1} but with $\Delta Y_{lt^*}$ replacing $Y_{lt^*}$ everywhere in that proposition.  Thus, the relatively more important assumption in this section is the one in \Cref{ass:mar}.}

One cannot immediately use \Cref{ass:unc} because $Z^*_{lt^*-1}(0)$ includes $C_{lt^*-1}(0)$ -- the per capita number of total Covid-19 cases in a particular location -- which is unobserved.  But, in practice, most outcomes in period $t^*$ are likely to depend heavily on how widespread Covid-19 has been -- even if it has gone largely undetected.  Therefore, it seems quite important to control for the (unobserved) number of cases.  To address this issue, we make the following assumption

\begin{assumption}[Conditional Independence of Pre-Policy Total Covid-19 Cases] \label{ass:mar}
  \begin{align*}
    C_{lt^*-1}(0) \independent D_l | Z_{lt^*-1}(0)
  \end{align*}
\end{assumption}
\Cref{ass:mar} says that, in the pre-treatment period, the distribution of the per capita number of total Covid-19 cases was the same among treated and untreated locations that had the same pre-policy characteristics.  To be clear here, \Cref{ass:mar} does not imply that the unobserved number of total cases is exactly the same across treated and untreated locations.  Rather, it rules out things like systematic differences in unobserved total cases in the pre-treatment period among treated and untreated locations with similar populations and that had run a similar number of tests and had confirmed a similar number of cases.\footnote{\Cref{ass:mar} is likely to be much more plausible when the observed conditioning variables include location-level number of tests.  In our application on open-testing in Tennessee, we use county-level data.  County-level testing data is not widely available, and its availability for Alabama is a main reason that we focus on comparisons between counties in Tennessee and Alabama that have similar characteristics (including having run a similar number of tests which is not available at the county-level for any of the other states surrounding Tennessee).}  Under these assumptions, we have the following result

\begin{proposition} \label{prop:1}  Under \Cref{ass:unc,ass:mar}, $ATT_Y(Z_{lt^*-1})$ and $ATT_Y$ are identified and given by
  \begin{align*}
    ATT_Y(Z_{lt^*-1}) = E[Y_{lt^*} | Z_{lt^*-1}, D_l=1] - E[Y_{lt^*} | Z_{lt^*-1}, D_l=0]
  \end{align*}
  and
  \begin{align*}
    ATT_Y = E[ATT_Y(Z_{lt^*-1})|D_l=1]
  \end{align*}
\end{proposition}

The proof of \Cref{prop:1} is provided in \Cref{sec:proofs}.  The result in \Cref{prop:1} says that the average effect of the policy across treated locations for a Covid-19 related policy is point identified even in the case where the outcomes themselves could depend on the number of actual cases and the number of actual cases is not observed.  The main conditions for this identification result are (i) some baseline identification strategy (we used unconfoundedness) and (ii) the additional condition that, after conditioning on pre-treatment characteristics (which include pre-treatment outcomes, pre-treatment number of confirmed cases, pre-treatment number of tests, and population), the unobserved number of actual cases (in the absence of the policy) would follow the same distribution as for untreated locations.  Moreover, the $ATT_Y$ can be recovered by comparing outcomes in treated locations to outcomes in untreated locations that had the same observed pre-treatment characteristics.

\subsubsection{Policy Effects on Total Covid-19 Cases}

Next, we consider trying to identify the effect of a policy on the per capita number of total Covid-19 cases.
This is distinctly more challenging than the previous case because the total number of cases is not observed.  Relative to the descriptive bounds presented in \Cref{sec:descriptive-bounds}, developing bounds on policy effects introduces new challenges.  For example, suppose that there are two locations, one treated and one untreated, and we are (i) comfortable with the idea that number of cases experienced in the untreated location is equal to the number of cases that the treated location would have experienced if it had not implemented the policy but (ii) are only able to bound the actual number of cases in each location.  In this case, without further assumptions, the bounds on the effect of the policy are equal to the difference between the upper (or lower) bound for the treated location minus the lower (or upper) bound for the untreated location.  These sorts of bounds are generally very wide and likely to cover 0.  We introduce some additional assumptions that are able to substantially narrow these sorts of bounds; the assumptions that we introduce formalize the idea that if both confirmed cases decrease in the treated location relative to the untreated location while testing does not decrease in the treated location relative to the untreated location, then this would be strong evidence that the policy led to a decrease in total Covid-19 cases relative to what they would have been had the policy not been implemented.  We also formalize what other conditions need to hold in order for this intuition to be correct.


To start with, we continue to make \Cref{ass:mar}, and we modify \Cref{ass:unc} to hold jointly for all untreated potential outcomes and covariates:
\begin{assumption}  [Covid Unconfoundedness] \label{ass:covid-unc}
  \begin{align*}
    Z^*_{lt^*}(0) \independent D | Z^*_{lt^*-1}(0)
  \end{align*}
  %
   %
  %
\end{assumption}
\Cref{ass:covid-unc} is similar to \Cref{ass:unc} in that it is an unconfoundedness type of assumption, but it applies to all untreated potential outcomes and covariates.  It says that, in the absence of the policy intervention,  the distribution of pandemic related variables (e.g., per capita total cases, confirmed cases, and tests) would have been the same for treated locations and untreated locations conditional on having the same pre-treatment characteristics.  The relative merits of unconfoundedness-type identifying assumptions relative to difference in differences-type identifying assumptions are discussed at length in \textcite{callaway-li-2021b} with that paper generally arguing in favor of unconfoundedness rather than difference in differences due to the high degree of nonlinearity (and lack of additively separable location-specific unobserved heterogeneity) in the leading pandemic models in the epidemiology literature.  In practice, in the discussion below, this condition allows us to replace the (unobserved) distributions of outcomes that treated locations would have experienced if they had not been treated with the (observed) distribution of outcomes that untreated locations with similar pre-treatment characteristics actually experienced.

Similarly to the previous section, we focus on identifying
\begin{align}
  ATT_C(Z_{lt^*-1}) = E[C_{lt^*}(1) - C_{lt^*}(0) | Z_{lt^*-1}, D_l=1] \quad \textrm{and} \quad ATT_C = E[C_{lt^*}(1) - C_{lt^*}(0) | D_l=1]
\end{align}
$ATT_C(Z_{lt^*-1})$ is the average effect of the policy on the per capita number of total Covid-19 cases across treated locations with pre-treatment characteristics $Z_{lt^*-1}$.  $ATT_C$ is the overall average effect of the policy on the per capita number of total Covid-19 cases across treated locations.  In addition, the same sorts of arguments as in the previous section continue to go through suggesting that
\begin{align} \label{eqn:attc}
  ATT_C(Z_{lt^*-1}) &= E[C_{lt^*}|Z_{lt^*-1},D_l=1] - E[C_{lt^*}|Z_{lt^*-1},D_l=0] \nonumber \\
                    &= P(C_{ilt^*} = 1| Z_{lt^*-1}, D_l=1) - P(C_{ilt^*} = 1 | Z_{lt^*-1}, D_l=0) 
\end{align}
and
\begin{align*}
  ATT_C &= E[C_{lt^*}|D_l=1] - E\Big[ E[C_{lt^*}|Z_{lt^*-1},D_l=0] | D_l=1 \Big] 
\end{align*}
The problem here is that $C_{lt^*}$ is not directly observed, and, therefore, as in \Cref{sec:descriptive-bounds}, the terms in \Cref{eqn:attc} are only partially identified.  Thus, our approach is to construct bounds on $ATT_C$.  

Before stating these results, we define three more terms to conserve on notation below.  First, for $d \in \{0,1\}$, define
\begin{align*}
      \gamma_d(Z_{lt^*-1}) := \frac{P(R_{ilt}=1|Z_{lt^*-1},D=d)}{1-FNR}
\end{align*}
$\gamma_d(Z_{lt^*-1})$ is a scaled version of the number of confirmed cases in location $l$ in time period $t$.  This term is point identified as we observe the number of confirmed cases and know the false negative rate of the test.  Notice that this term corresponds to the first term in \Cref{eqn:covid-bound-1} (now conditional on $Z_{lt^*-1}$ and $D_l=d$).\footnote{For some of the expressions below and in the proofs, it is also helpful to notice that $\gamma_d(Z_{lt^*-1}) = P(C_{ilt^*}=1, T_{ilt^*}=1 | Z_{lt^*-1}, D_l=d)$ (which follows from the same sorts of arguments as in \Cref{sec:descriptive-bounds}).}  Second, for $d \in \{0,1\}$, define
\begin{align*}
  \tau_{d}(Z_{lt^*-1}) := P(T_{ilt^*}=1|Z_{lt^*-1}, D_l=d)
\end{align*}
which is the probability of being tested conditional on a location's pre-treatment characteristics and treatment status.  This is identified by the sampling process since we observe the number of tests in a particular location.  Finally, define
\begin{align*}
    \phi_d(Z_{lt^*-1}) := P(C_{ilt^*}=1|T_{ilt^*}=0,Z_{lt^*-1}, D_l=d)
\end{align*}
which is the probability of having had Covid-19 conditional on having not been tested, a location's pre-treatment characteristics, and treatment status.  As above, this term is not identified because we do not observe the rate of Covid-19 cases among untested individuals.  Next, we provide an intermediate result that decomposes $ATT_C(Z_{lt^*-1})$ that is useful for developing our main bounds later in this section.

\begin{proposition} \label{prop:attc-decomp} Under \Cref{ass:mar,ass:covid-unc},
  \begin{align*}
      ATT_C(Z_{lt^*-1}) = \big(\gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1}) \big) + (\phi_1(Z_{lt^*-1})(1-\tau_1(Z_{lt^*-1})) - \phi_0(Z_{lt^*-1})(1-\tau_0(Z_{lt^*-1})) \big)
  \end{align*}
\end{proposition}
The proof of \Cref{prop:attc-decomp} is provided in \Cref{sec:proofs}, but it is worth pointing out that it holds almost immediately by (i) the unconfoundedness assumption and the assumption that pre-treatment total cases are missing at random combined with (ii) the arguments in \Cref{sec:descriptive-bounds} for bounds on total cases in a particular location.  It is also worth noting that all the terms in \Cref{prop:attc-decomp} are identified except $\phi_1$ and $\phi_0$ which are the probability of having had Covid-19 conditional on having not been tested (and pre-treatment location characteristics) for the treated group and untreated group, respectively.




Next, we discuss bounds on $ATT_C$ under the additional condition in \Cref{ass:covid-bound} (recall that this assumption says that the probability of having had Covid-19 among those who have not been tested is less than or equal to the probability of having had Covid-19 conditional on being tested).  

\begin{proposition} \label{prop:2} Under \Cref{ass:covid-bound,ass:mar,ass:covid-unc},
  \begin{align*}
    C^{B,L}_{lt^*}(Z_{lt^*-1}) \leq ATT_C(Z_{lt^*-1}) \leq C^{B,U}_{lt^*}(Z_{lt^*-1})
  \end{align*}
  where
  \begin{align*}
    C_{lt^*}^{B,L}(Z_{lt^*-1}) &:= \gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1}) \frac{1-\tau_{0}(Z_{lt^*-1})}{\tau_{0}(Z_{lt^*-1})} \\ %
    C_{lt^*}^{B,U}(Z_{lt^*-1}) &:= \gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1}) + \gamma_1(Z_{lt^*-1}) \frac{1-\tau_{1}(Z_{lt^*-1})}{\tau_{1}(Z_{lt^*-1})}%
  \end{align*}
  and
  \begin{align*}
    E\Big[ C^{B,L}_{lt^*}(Z_{lt^*-1}) | D_l=1 \Big] \leq ATT_C \leq E\Big[ C^{B,U}_{lt^*}(Z_{lt^*-1}) | D_l=1\Big]
  \end{align*}
\end{proposition}

The proof of \Cref{prop:2} is provided in \Cref{sec:proofs}.  These sort of bounds arise under the combination of (i) standard identifying assumptions for policy effects and (ii) \Cref{ass:covid-bound} -- that the probability of having had Covid-19 is lower among untested individuals than among tested individuals.  The term in common for each of the bounds, $\gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1})$, is driven by differences in confirmed cases among treated and untreated locations with similar pre-treatment characteristics.  The extra term for the lower bound comes from setting the fraction of untested individuals in treated locations who have had Covid-19 to be equal to zero while setting the fraction of untested individuals in untreated locations who have had Covid-19 to be equal to the fraction who have had Covid-19 conditional on being testing (this comes from the bound in \Cref{ass:covid-bound}).  The upper bound comes from doing the opposite:  setting the fraction of untested individuals who have had Covid-19 to be equal to zero for untreated locations and setting the fraction of untested individuals who have had Covid-19 to be as large as possible (under \Cref{ass:covid-bound}) for treated locations.\footnote{In practice, the extreme cases that lead to the lower bound and upper bound seem unlikely to hold.  This suggests that these bounds are likely to be quite conservative.}  %
The weights on these terms (the terms involving $\tau_1$ and $\tau_0$) also tend to be very large because the fraction of untested individuals is much larger than the fraction of tested individuals.  This implies that $\tau_d(Z_{lt^*-1}) \ll 1 - \tau_d(Z_{lt^*-1})$ for $d \in \{0,1\}$.

The drawback of these bounds is that they are unlikely to be informative about the sign of the policy effect.  To see this, notice that, especially early in the pandemic, the terms involving $\gamma_d(Z_{lt^*-1})$ are often quite small due to the number of confirmed cases being small.  On the other hand, the extra terms can be orders of magnitude larger.  %
In our application, using these bounds, the bounds cover 0 for all counties and are not very informative.

In order to proceed, it is helpful to re-write the expression for $ATT_C(Z_{lt^*-1})$ in \Cref{prop:attc-decomp} as
\begin{align} 
    ATT_C(Z_{lt^*-1}) &= \big(\gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1}) \big) \label{eqn:attc_decomp2-1}\\ 
    & + \big(\phi_1(Z_{lt^*-1}) - \phi_0(Z_{lt^*-1}) \big)(1-\tau_1(Z_{lt^*-1})) \label{eqn:attc_decomp2-2}\\
    & + \phi_0(Z_{lt^*-1})\big(\tau_0(Z_{lt^*-1}) -\tau_1(Z_{lt^*-1}) \big) \label{eqn:attc_decomp2-3}
\end{align}
Recall that the only terms in this expression that are not identified are $\phi_1$ and $\phi_0$.  In the case where a researcher is interested in trying to determine whether or not the policy decreased total Covid-19 cases, one would be interested in determining if $ATT_C$ is less than or equal to 0.  The sign of the term in \Cref{eqn:attc_decomp2-1} is fully determined by the difference between the number of confirmed cases in treated locations relative to untreated locations with similar pre-treatment characteristics (which is point identified).  But, as we discussed earlier, the effect of the policy is not fully determined by the number of confirmed cases.  Neither expression in either Equation (\ref{eqn:attc_decomp2-2}) or (\ref{eqn:attc_decomp2-3}) is point identified.  However, the sign of the term in \Cref{eqn:attc_decomp2-3} is fully determined by the difference between the number of tests under the policy relative to the number tests among similar untreated locations.  If the treatment does not decrease the number of tests, then this term will be non-positive.  Finally, the sign of the term in \Cref{eqn:attc_decomp2-2} depends on the difference between the number of cases among untested individuals in treated locations relative to the number of cases among untested individuals in similar untreated locations.  This difference is not identified and is the most challenging part to think through (we address this point in substantially more detail below).  For now though, it is important to note that the above expression clarifies our intuition from the introduction: a policy that decreases the number of confirmed cases while not decreasing the number of tests does actually decrease the total number of cases as long as the number of cases among untested individuals under the policy does not increase relative to what the number of cases among untested individuals would have been if the policy had not been implemented.


To formalize the above discussion, we introduce the following assumption.

\begin{assumption} [Bound on Total Cases and Untested Individuals] \label{ass:pol-bound}
  \begin{align*}
    P(C_{ilt^*}(1) = 1, T_{ilt^*}(1) = 0, | Z_{lt^*-1}(0), D_l=1) \leq P(C_{ilt^*}(0) = 1, T_{ilt^*}(0) = 0 |  Z_{lt^*-1}(0), D_l=1)
  \end{align*}
\end{assumption}


\Cref{ass:pol-bound} says that, for individuals in locations that experience the policy, the joint probability of having Covid-19 and not being tested under the policy is less than or equal to the joint probability of having Covid-19 and not being tested in the absence of the policy.  Under \Cref{ass:mar,ass:covid-unc}, this is equivalent to saying that the sum of the terms in \Cref{eqn:attc_decomp2-2,eqn:attc_decomp2-3} are less than or equal to 0.  We provide one set of more primitive conditions and more detailed discussion of this assumption in \Cref{sec:pol-bound-discussion}.  At a high level, though, this assumption is plausible under the conditions that (i) the policy does not make tests less available than they otherwise would have been without the policy, (ii) the policy does not increase the number of Covid-19 cases among the untested (for the fixed group of individuals who would not be tested under the policy) relative to what they would have been without the policy, (iii) there is not negative selection into taking the test among those who would be tested under the policy but would not be tested in the absence of the policy.


Finally for this section, we provide our main result on tighter bounds for evaluating policies early in a pandemic.

\begin{proposition} \label{prop:3} Under \Cref{ass:covid-bound,ass:mar,ass:covid-unc,ass:pol-bound},
  \begin{align*}
    C^{C,L}_{lt^*}(Z_{lt^*-1}) \leq ATT_C(Z_{lt^*-1}) \leq C^{C,U}_{lt^*}(Z_{lt^*-1})
  \end{align*}
  where
  \begin{align*}
    C_{lt^*}^{C,L}(Z_{lt^*-1}) &:= C_{lt^*}^{B,L}(Z_{lt^*-1}) \\
    C_{lt^*}^{C,U}(Z_{lt^*-1}) &:= \gamma_1(Z_{lt^*-1}) - \gamma_0(Z_{lt^*-1})
  \end{align*}  
  and
  \begin{align*}
    E\Big[ C^{C,L}_{lt^*}(Z_{lt^*-1}) | D_l=1 \Big] \leq ATT_C \leq E\Big[ C^{C,U}_{lt^*}(Z_{lt^*-1}) | D_l=1\Big]
  \end{align*}
\end{proposition}

The proof of \Cref{prop:3} is provided in \Cref{sec:proofs}.  Notice that the lower bound is the same as it was in the previous case, but that the upper bound can be substantially tighter.  In particular, the upper bound does not contain the same extra term as in \Cref{prop:2}; as discussed earlier, this term is the ``dominant'' term in the upper bound, and it is removed under the additional condition in \Cref{ass:pol-bound}.  This result provides conditions under which a policy that leads to a decrease in confirmed cases also indicates that the policy decreased actual cases.  







\begin{remark}
The identification results above are constructive and suggest plug-in estimators of each parameter of interest.  There are a number of possibilities here (e.g., regressions or weighting estimators).  In our application, we found it natural to use a matching estimator, and we discuss the particular estimator that we used for the application in more detail in the next section.
\end{remark}




\section{Application: Tennessee's Open-Testing Policy}

 \label{sec:policy}

On Wednesday, April 15, 2020, Tennessee's Republican governor Bill Lee announced free testing in the state for anyone who wanted a test.\footnote{``Gov. Lee expands COVID-19 testing to residents with or without symptoms.'' \textit{Nashville Tennessean}, April 15, 2020. \\ \url{https://www.tennessean.com/videos/news/2020/04/15/gov-lee-expands-covid-19-testing-residents-without-symptoms/5141345002/}}$^,$\footnote{Humbles, Andy. ``More than 6,500 tests conducted at Tennessee's asymptomatic coronavirus testing sites Saturday.'' \textit{Nashville Tennessean}, April 18, 2020. \\ \url{https://www.tennessean.com/story/news/health/2020/04/18/tennessees-asymptomatic-coronavirus-test-sites-draw-huge-crowds/5159081002/}.}  That Saturday, April 18, more than 6,500 Tennessee residents were tested at 20 different testing locations across the state.  Unlike almost all other states at that time, obtaining a test did not require an individual to be showing symptoms or to be in a high risk group.  These tests were also available on the weekends of April 25 and May 2.  Over the course of those three weekends, over 23,000 individuals were tested at a total of 67 different testing sites.\footnote{``Unified-Command Group’s Total Expanded Testing Efforts Exceed 23,000.'' \textit{TN Office of the Governor}, May 4, 2020. \\ \url{https://www.tn.gov/governor/news/2020/5/4/unified-command-group-s-total-expanded-testing-efforts-exceed-23-000.html}.}  Following those three weekends, the open-testing policy was modified.  Tennessee increased their emphasis on testing high-risk populations.  That being said, the requirements to be tested for Covid-19 in Tennessee continued to be milder than in other states even after the policy was modified.\footnote{Hartnett, Kara. ``State officials alter COVID-19 testing strategy.'' \textit{Williamson Home Page}, May 8, 2020. \\ \url{https://www.williamsonhomepage.com/brentwood/state-officials-alter-covid-19-testing-strategy/article_f6650f2e-9143-11ea-a65c-0748e5c7875b.html}.}


Compared to its six neighboring states, Tennessee was already a high testing state when the new policy was implemented (see \Cref{fig:1}).\footnote{One main reason for Tennessee's large number of tests was that, unlike most other states, Tennessee directly paid private labs. (Farmer, Blake.  ``Tennessee’s secret to plentiful Coronavirus testing — Picking up the tab.'' \textit{WPLN News}, April 30, 2020. \\ \url{https://wpln.org/post/tennessees-secret-to-plentiful-coronavirus-testing-picking-up-the-tab/}).}  Out of these seven states, Tennessee was roughly tied with Mississippi for the most tests per capita when the open-testing policy was implemented.  Between April 17 and May 8, the number of tests per capita increased in Tennessee by over 2 percentage points -- more than any of its neighboring states.  And by May 8, Tennessee had conducted more tests per capita than any of its neighboring states (46\% more than Alabama, 68\% more than Arkansas, 66\% more than Georgia, 96\% more than Kentucky, 31\% more than Mississippi, and 109\% more than North Carolina).


\begin{figure}
  \begin{center}
  \caption{Per Capita Tests by State}
  \label{fig:1}
  { \centering \includegraphics[scale=0.6]{figures/percapita-tests2} }
  \end{center}
  { \footnotesize
    
  \textit{Notes:} Per capita tests for Tennessee and its six neighboring states over time.  The vertical black line is set at April 17 which is approximately the start of Tennessee's open-testing policy.

  \textit{Sources:} Covid Tracking Project (\url{https://covidtracking.com/}) and Census Bureau}
\end{figure}

On the other hand, Tennessee was closer to the middle in terms of number of confirmed cases.  On April 17, Tennessee was essentially tied with Alabama for third out of seven in terms of per capita confirmed cases (see \Cref{fig:2}) behind Georgia and Mississippi.  By May 8, Tennessee had moved somewhat ahead of Alabama in terms of per capita number of confirmed cases but was still behind both Georgia and Mississippi.  It is also important to remember that the number of confirmed cases depends on the number of tests especially in cases like Covid-19 where the number of tests is relatively low and there may be a large number of asymptomatic cases or cases with relatively mild symptoms.  Thus, for example, the increase in confirmed cases in Tennessee relative to Alabama could be explained by an increase in actual cases in Tennessee relative to Alabama or just due to a mechanical increase in the number of confirmed cases arising from more extensive testing.

\begin{figure}
  \begin{center}
  \caption{Per Capita Confirmed Cases by State}
  \label{fig:2}
  { \centering \includegraphics[scale=0.6]{figures/percapita-cases2} }
  \end{center}
  { \footnotesize
    
  \textit{Notes:} Per capita confirmed Covid-19 cases for Tennessee and its six neighboring states over time.  The vertical black line is set at April 17 which is approximately the start of Tennessee's open-testing policy.

  \textit{Sources:} Covid Tracking Project (\url{https://covidtracking.com/}) and Census Bureau}
\end{figure}


    %
    %

\subsection{Data} \label{sec:data}

Our analysis uses two main datasets.  The first dataset is state-level and includes data from Tennessee and each of its six bordering states: Alabama, Arkansas, Georgia, Kentucky, Mississippi, and North Carolina.  The state level dataset contains information on the total number of Covid-19 cases by state over time and total number of tests by state over time.  This data comes from the Covid Tracking Project (\url{https://covidtracking.com/}).  This state level data is also merged with state population data from the Census Bureau.  

The second dataset consists of county-level data from Tennessee and Alabama.  The Covid Tracking Project provides confirmed case counts by county over time.  But, many of our main results require county-level testing data.  Tennessee provides historical county-level data at its Department of Health website (\url{https://www.tn.gov/content/tn/health/cedep/ncov/data.html}).  Of its bordering states, the only state that currently provides county-level testing data is Alabama, but it only provides current (not historical) county-level testing data (\url{https://dph1.adph.state.al.us/covid-19/}).  In order to recover historical county-level testing data, we were able to use the Internet Archive's Wayback Machine (\url{web.archive.org}).  The earliest available county-level data for Alabama using this approach is from April 17.  This is after Tennessee's open-testing policy was announced but before individuals in Tennessee could actually get a test under the policy; there is also typically around a 72 hour delay in the test results being available.  Together these suggest that we can treat April 17 as being ``pre-treatment'' for both Alabama and Tennessee.  We use data from May 8 for the post-treatment period.  This is the Friday following the last weekend when open-testing was available in Tennessee.  We use this day as it should allow for enough time to pass that almost all tests from the previous weekend show up in the reported statistics.  \Cref{fig:tn-county} provides county-level data on per capita tests and per capita confirmed cases for Tennessee.

\begin{figure}[t]
  \caption{Tests and Confirmed Cases by TN County}
  \label{fig:tn-county}
  \begin{subfigure}{\textwidth}
    \centering \includegraphics[scale=0.45]{figures/county-percapita-tests-alt2}
    \caption{Per Capita Tests by County}
  \end{subfigure}
  \begin{subfigure}{\textwidth}
    \centering \includegraphics[scale=0.45]{figures/county-percapita-cases-alt2}
    \caption{Per Capita Confirmed Cases by County}
  \end{subfigure}
  %
  %
  %
  %
  { \footnotesize \textit{Notes:} Per capita tests and confirmed cases for all counties in Tennessee except Trousdale and Bledsoe.  The highlighted counties are the most populated counties in Tennessee.  Davidson County includes Nashville.  Williamson County is the largest suburban county of Nashville.  Shelby County includes Memphis.  Knox County includes Knoxville.  The vertical black line is set at April 17 which is approximately the start of Tennessee's open-testing policy.

    \textit{Sources: } Tennessee Department of Health Data (\url{https://www.tn.gov/health/cedep/ncov/data/downloadable-datasets.html}) and Census Bureau
  }
\end{figure}

We merge the county-level testing and confirmed cases data with (i) data from the Census Bureau on county-level population, and (ii) data from Google's Covid-19 Community Mobility Reports (\url{https://www.google.com/covid19/mobility/}).  These are aggregated cell phone data that Google has published to help researchers studying Covid-19.  We focus primarily on county-level trips to work and how this variable evolves over time.  It is reported as a percentage change relative to pre-Covid trips to work.

Finally, it is worth mentioning that Trousdale County and Bledsoe County experienced large Covid-19 outbreaks in prisons.\footnote{Allison, Natalie and Mariah Timms. ``Tennessee to test all inmates, prison staff after massive COVID-19 outbreak at Trousdale Turner.'' \textit{Nashville Tennessean}, May 1, 2020. \\ \url{https://www.tennessean.com/story/news/politics/2020/05/01/tennessee-testing-all-inmates-prison-staff-after-multiple-outbreaks/3067388001/}}$^,$\footnote{Timms, Mariah.  ``Number of coronavirus cases at Bledsoe County Correctional Complex more than triple.'' \textit{Nashville Tennessean}, April 23, 2020. \\ \url{https://www.tennessean.com/story/news/local/2020/04/23/coronavirus-bledsoe-county-prison-inmates/3003595001/}.}  In some of our descriptive analysis, we keep these counties, but in our main results, we drop these counties.

\subsection{Estimation and Inference}
As mentioned above, a number of estimation procedures would be suitable for our proposed approach.  We found it natural to use a matching estimator where, for each county in Tennessee, we found a ``match'' in Alabama based on pre-policy county characteristics, $Z_{lt^*-1}$.  Overall, average effects can be calculated by taking the average outcome experienced by counties in Tennessee and subtracting the overall average outcome experienced by ``matched'' counties in Alabama.

In practice to construct the matched dataset, we match on per capita tests and per capita confirmed cases on April 17 (the pre-treatment date when we observe tests by county in Alabama).  We also match on county population.  And, finally, we match on pre-treatment declines in county-level trips to work from pre-Covid baseline.

To actually construct the matches, for county $l$ in Tennessee, we construct its match by choosing the county in Alabama that minimizes the squared standardized Euclidean distance
\begin{align*}
  d(Z_{lt^*-1}, Z_{jt^*-1}) = (Z_{lt^*-1} - Z_{jt^*-1})'S^{-1}(Z_{lt^*-1} - Z_{jt^*-1})
\end{align*}
where $j$ indexes counties in Alabama and $S$ is a diagonal matrix containing the diagonal elements of the variance matrix of $Z_{jt-1}$.  Finally, we drop all counties that do not have a close match.  This results in a matched dataset that includes 77 (out of 95 total) counties in Tennessee.\footnote{Our matched dataset does not include Davidson County (which contains Nashville) or Shelby County (which contains Memphis).  These are notable exclusions from our dataset, but they occur because it is relatively harder to find a reasonable match for large urban counties relative to less populated and rural counties.}

Another issue is that, in practice, it is not immediately clear how to conduct inference in our case.  We essentially have no sampling uncertainty because we observe all the exact number of confirmed cases and tests in each county for all counties in Tennessee and Alabama.  In light of this,  to conduct inference, we focus on design-based uncertainty (see, for example, \textcite{imbens-rubin-2015,ding-feller-miratrix-2016, abadie-athey-imbens-wooldridge-2020}).  For some parameter of interest, we approximate its distribution under the sharp null hypothesis of no county-level effect of the policy by repeatedly randomly assigning a variable indicating participation in the policy among counties in Tennessee and their match in Alabama.  Then, we construct a p-value by calculating the fraction of times that the original estimated parameter of interest is larger in absolute value than the estimated parameters that arise from repeatedly randomly assigning participating in the policy.  This is similar to the sorts of randomization inference procedures used in the difference-in-differences, synthetic control, and matching literatures (e.g., \textcite{bertrand-duflo-mullainathan-2004},\footnote{Randomization inference is discussed in more detail in the working paper version (\textcite{bertrand-duflo-mullainathan-2002}).} \textcite{abadie-diamond-hainmueller-2010,ferman-2019}).



\subsection{Results} \label{sec:results}

\subsubsection*{Challenges to Identification}

Before presenting our main results, we briefly discuss the timing of other policy decisions made by Tennessee and Alabama.  We list the timing of implementing major policies by Tennessee and Alabama in \Cref{tab:compare}.  The timing of other policies is important in this context because (i) states implemented a number of policies in response to the Covid-19 pandemic and (ii) if the policies themselves or the timing of implementing these policies differed substantially across Tennessee and Alabama, then our results would mix together the effects of the open-testing policy as well other policy differences between Tennessee and Alabama.\footnote{It is also worth pointing out that, although our identification arguments used terminology from the treatment effects literature such as ``treated and untreated potential outcomes,'' the effects that we estimate in this section are not relative to ``no policy'' but rather they are relative to Tennessee not implementing the open-testing policy and continuing to follow the same policy path as Alabama did.}

\ctable[caption=Timing of Other Policies for Tennessee and Alabama, label=tab:compare, pos=t]{rcccc}{\tnote{\textit{Sources:}  Institute for Health Metrics and Evaluation (IMHE).  \url{https://covid19.healthdata.org/united-states-of-america/tennessee} and \url{https://covid19.healthdata.org/united-states-of-america/alabama}}}{
  \FL
  Policy                              & TN Start & TN End   & AL Start & AL End   \ML
  Schools Closed                      & March 20 & -        & March 19 & -        \NN
  Gathering Restrictions              & March 23 & -        & March 19 & -        \NN
  Stay-at-Home Order                  & April 2  & April 29 & April 4  & April 30 \NN
  Non-essential Businesses Closed     & April 1  & April 27 & March 28 & April 30 \LL 
}

\Cref{tab:compare} shows that the timing of major other policies was very similar for Tennessee and Alabama.  To give some examples, Alabama closed public schools on March 19 and Tennessee closed them on March 20.  Tennessee had a stay-at-home order from April 2 to April 29; Alabama had a stay-at-home order from April 4 to April 30.  These close similarities in terms of other policies across states provides one piece of evidence in favor of interpreting our results below as being due to the open-testing policy.

\subsubsection*{Descriptive Bounds on the Per Capita Number of Total Cases across Counties}

Next, we compute bounds on the per capita number of total Covid-19 cases across counties and separately for Tennessee and Alabama.  These results are available in \Cref{fig:covid-bounds} for May 8 and in \Cref{fig:covid-bounds-april-17} in \Cref{app:additional-figures} for April 17.  For both states, the bounds are somewhat narrower by May 8 than they were on April 17 -- this should not be surprising as the number of tests had increased substantially in both states over time.  Focusing on the bounds on May 8, it is immediately clear that the bounds on the per capita number of total cases tend to be noticeably tighter in Tennessee counties than in Alabama counties.  The mean width of the bounds is 0.06 in Tennessee and 0.13 in Alabama.


\begin{figure}[t] 
  \begin{center}
  \caption{Bounds on Per Capita Number of Total Covid-19 Cases by County}
  \label{fig:covid-bounds}
  \begin{subfigure}{.32\textwidth}
    \centering \includegraphics[width=0.95\textwidth]{figures/tn-county-dumbbell3}
    \caption{Tennessee}
  \end{subfigure}
  \begin{subfigure}{.32\textwidth}
    \centering \includegraphics[width=0.95\textwidth]{figures/al-county-dumbbell3}
    \caption{Alabama}
  \end{subfigure}
\end{center}
{ \footnotesize \textit{Notes:} Bounds on the total (unobserved) per capita Covid-19 cases by county in Tennessee and Alabama using the bounds described in the text. These bounds are for May 8, 2020.}
\end{figure}

It is also worth discussing the bounds in some particular cases.  The two counties with the most informative bounds are Trousdale County and Bledsoe County, but these are both rural counties that have experienced large Covid-19 outbreaks in prisons and that have had extensive testing in those prisons.  Besides those two counties, the county with the next highest upper bound on total Covid-19 cases is Davidson at 18\%.  Davidson County is home of Nashville, the capital and largest city in Tennessee.  It is also interesting to consider why the upper bound for Davidson County is high.  Recall that the upper bound mainly depends on %
the number of confirmed cases per test (will tend to be high when this is high).  %
Outside of Trousdale and Bledsoe Counties, Davidson County has the highest number of confirmed cases per test (13\%) which leads to the large upper bound.  Out of the top 10 counties in terms of the upper bound on total Covid-19 cases (excluding Trousdale and Bledsoe), 7 out of 10 are in Davidson County and surrounding suburban counties or Shelby County (Memphis) and surrounding suburban counties.  The other three are Perry County, Bedford County, and Grundy County which are all rural.

The lower bound on the rate of total Covid-19 cases is uniformly quite small as it is primarily driven by the number of confirmed cases.  To give an example, Davidson County has the 4th highest lower bound on total Covid-19 rates, and it is only 0.6\%.



\subsubsection*{Main Results: Policy Effects of Tennessee's Open-Testing Policy}
  
Next, we consider the effect of Tennessee's open-testing policy on several different outcomes.  First, we consider the effect of the policy on the number of tests and on the number of confirmed cases.  These results are available in Panels (a) and (b) of \Cref{fig:tests-cases}.  The results in that figure come from comparing the actual number of tests or confirmed cases for particular counties in Tennessee compared to their ``match'' county in Alabama.  On average, the number of tests per capita increased by 13\% in counties in Tennessee relative to what they would have experienced if they had not enacted the policy.\footnote{This is computed by taking the average number of tests per capita across counties in Tennessee (23.4 per 1000 individuals), subtracting the average number of tests per capita in matched counties (20.7 per 1000 individuals), and dividing by this same term.}  As discussed above, since we observe the full population, it does not seem appropriate to use conventional confidence intervals based on large samples from an infinite population.  Instead, we consider the sharp null hypothesis that the number of tests in a particular county is exactly the same as it would be if the policy had not been enacted.  Then, we can test if the policy is having an effect on the number of tests by comparing the estimated average effect of the policy on the number of tests to the distribution of the number of tests when we randomly assign participation in the policy between each county in Tennessee and its match in Alabama.    Under this null (as well as the maintained assumptions), we compute a p-value as the fraction of times that the estimated effect of the policy is larger in absolute value than its corresponding estimate when  participating in the policy is randomly assigned.  The p-value in this case is 0.359.  This is not statistically significant at conventional significance levels, but it is at least suggestive that the policy tended to increase the number of tests.


\begin{figure}[t]
  \begin{center}
    \caption{Effect of Tennessee's Open-Testing Policy on Tests, Confirmed Cases, and Work Trips}
    \label{fig:tests-cases}
    \begin{subfigure}{.32\textwidth}
      \centering \includegraphics[width=.95\textwidth]{figures/Dtests3}
      \subcaption{Tests}
    \end{subfigure}
    \begin{subfigure}{.32\textwidth}
      \centering \includegraphics[width=.95\textwidth]{figures/Dcases3}
      \subcaption{Confirmed Cases}
    \end{subfigure}
    \begin{subfigure}{.34\textwidth}
      \centering \includegraphics[width=.95\textwidth]{figures/Dcases-Dcommutes3}
      \subcaption{Work Trips}
    \end{subfigure}
  \end{center}

  { \footnotesize \textit{Notes:}  The effect of Tennessee's open-testing policy on the per capita number of tests, confirmed cases, and percentage change in trips to work by county.  In Panel (a), the orange dots are the difference in per capita tests for counties in Tennessee relative to their matched county in Alabama.  In Panel (b), the orange dots are the difference in per capita confirmed cases for counties in Tennessee relative to their matched county in Alabama.  In Panel (c), the blue dots are the percentage change in trips to work for particular counties in Tennessee relative to their matched county in Alabama.  The orange dots are the per capita change in confirmed cases over time for each county in Tennessee.}
\end{figure}


Next, we consider a similar exercise in terms of number of confirmed Covid-19 cases.  These results are available in Panel (b) of \Cref{fig:tests-cases}.  Here, we estimate that, on average, the open-testing policy decreased the number of confirmed cases by 50\% relative to what they would have been in the absence of the policy.\footnote{This estimate comes from averaging the per capita number of confirmed cases across counties in Tennessee (0.83 per 1000 individuals), subtracting the average number of confirmed cases in their matched counties from Alabama (1.68 per 1000 individuals), and dividing by the latter term.}  Using the same randomization inference procedure as above, we get a p-value of 0.006.  These results suggest that Tennessee's open-testing policy substantially decreased the number of confirmed cases.

Next, we consider how the policy is affecting number of trips to work.  Our strategy is to see how the number of trips to work changes as a function of the change in the number of confirmed cases over time.  Panel (c) of \Cref{fig:tests-cases} plots the percentage change in trips to work for counties in Tennessee relative to their matched county in Alabama along with the observed change in confirmed cases between April 17 and May 8 among counties in Tennessee.  We regress the difference in trips to work among matched counties on the observed change in confirmed cases for counties in Tennessee and estimate that the number of trips to work declines by 1.28 percent when the number of confirmed cases per 1000 individuals increased by 1 between April 17 and May 8; this parameter has a p-value of 0.022.\footnote{Calculating the p-value here involves a slight modification to our approach for previous parameters of interest.  As before, we repeatedly randomly assign participating in the policy across matches in Tennessee and Alabama, but now, at each iteration, we run a regression of the difference in number of trips to work between each county and its match (here, the ``treated'' county can be the opposite of what it is in the actual data) on the increase in cases over time.  We calculate a p-value as the fraction of times that the original estimate is larger (in absolute value) than the estimated parameters coming from repeatedly randomly assigning treatment status.}  This result suggests that, at least to some extent, individuals in counties where the number of confirmed cases was increasing the most over time reduced their number of trips to work relative to the number of trips that they would have taken without the policy. 




\begin{figure}[t]
  \begin{center}
    \caption{Bounds on the Effects of Tennessee's Open-Testing Policy on Total Covid-19 Cases}
    \label{fig:covid-policy-bounds}
    \begin{subfigure}{.32\textwidth}
      \centering \includegraphics[width=.95\textwidth]{figures/policy-bounds4}
      \subcaption{Bounds}
    \end{subfigure}
    \begin{subfigure}{.32\textwidth}
      \centering \includegraphics[width=.95\textwidth]{figures/policy-upper-bound4}
      \subcaption{Upper Bound}
    \end{subfigure}
  \end{center}

  { \footnotesize \textit{Notes:}  Panel (a) provides estimates of the overall bounds on effect of Tennessee's open-testing policy on the per capita number of total cases separately by each county in Tennessee.  Panel (b) reproduces the estimates of the upper bound alone with a modified scale.}
\end{figure}


Finally, we consider the effect of Tennessee's open-testing policy on the total number of per-capita Covid-19 cases.  These results are provided in \Cref{fig:covid-policy-bounds}.  The lower bound on the policy effect is quite low -- across counties, the average is a decrease of total cases by 99 per 1000 individuals relative to what they would have been without the policy.  This is probably a fairly unrealistically large effect as it occurs when no untested individuals in Tennessee have had Covid-19, but untested individuals in Alabama have the same probability of having Covid-19 as tested individuals.  It is more interesting to consider the upper bound (this is the smallest possible effect of the policy given the data and identifying assumptions); see Panel (b) of \Cref{fig:covid-policy-bounds}.\footnote{\Cref{prop:3}, in combination with the expression in \Cref{eqn:CcondT}, implies that the upper bound is a scaled version of the difference in confirmed cases across counties in Tennessee and counties in Alabama with similar characteristics.  Therefore Panel (b) of \Cref{fig:covid-policy-bounds} is very similar to Panel (b) \Cref{fig:tests-cases}.  For the same reason, the p-values reported in this section are exactly the same as well.}  %
The average of the upper bound is a reduction of total cases by 1.12 per 1000 individuals.  This seems quite small, but it %
is also useful to compare this number to the number of confirmed cases per 1000 individuals which is only 0.8.  This suggests that even the upper bound on the effect of the policy should be interpreted as a non-trivial reduction in Covid-19 cases.  The randomization p-value is 0.006.  These results suggest that Tennessee's policy led to fewer total cases.  At a higher level, that the upper bound is negative is driven by the fact that confirmed cases appear to have decreased in Tennessee due to the open-testing policy.  This decrease in confirmed cases in the presence of an increase in total tests is a strong piece of evidence that Tennessee's open-testing policy is decreasing the total number of cases -- even if we are not able to provide plausible assumptions that lead to point identification.




\ctable[caption={Sensitivity Analysis},label=tab:sa,pos=t,]{lrrrrr}{\tnote{\textit{Notes:}  The table provides estimates of the percentage reduction of total Covid-19 cases due to Tennessee's open-testing policy for particular values of the sensitivity analysis parameters $\alpha$ and $\beta$ which are defined in \Cref{eqn:sa}.  The last row of the table provides the number of cases per 1000 individuals for particular values of $\alpha$ (and does not depend on $\beta$) in the absence of the open-testing policy.}}{\FL
\multicolumn{1}{l}{\bfseries }&\multicolumn{5}{c}{\bfseries $\alpha$}\NN
\cline{2-6}
\multicolumn{1}{l}{}&\multicolumn{1}{c}{0}&\multicolumn{1}{c}{0.01}&\multicolumn{1}{c}{0.1}&\multicolumn{1}{c}{0.5}&\multicolumn{1}{c}{1}\ML
{\bfseries $\beta$}&&&&&\NN
~~0&$-0.50$&$-0.65$&$-0.91$&$-0.98$&$-0.99$\NN
~~0.5&$-0.50$&$-0.50$&$-0.50$&$-0.50$&$-0.50$\NN
~~0.9&$-0.50$&$-0.38$&$-0.18$&$-0.12$&$-0.11$\NN
~~0.99&$-0.50$&$-0.35$&$-0.10$&$-0.03$&$-0.02$\NN
~~1&$-0.50$&$-0.35$&$-0.09$&$-0.02$&$-0.01$\ML
~~\makecell{Cases Per 1000 \\ w/o policy}&$ 2.24$&$ 3.21$&$12.00$&$51.07$&$99.90$\LL
}


As one final step to help interpret our bounds, we consider a sensitivity analysis.  Recall that our bounds on the effect of the open-testing policy on the per capita number of total cases are due to the number of cases among untested individuals not being known (see \Cref{prop:attc-decomp} and related discussion above).  To perform a sensitivity analysis we parameterize %
\begin{align}
  \phi_0 &=\alpha P(C_{ilt^*}(0)=1|T_{ilt^*}(0)=1, D_l=1) \label{eqn:sa} \\
  \phi_1 &= \beta \phi_0 \nonumber
\end{align}
for $\alpha$ and $\beta$ between 0 and 1.  In this case, $\alpha$ denotes the per capita total number of Covid-19 cases among untested individuals relative to tested individuals (this is also for the case where the policy was not implemented and for the treated group; but note that, by \Cref{ass:mar,ass:covid-unc}, this is equal to the corresponding term for untreated locations and is therefore identified).  And $\beta$ is the per capita fraction of total cases among untested individuals under the policy relative to what it would have been if the policy had not been implemented.  The assumptions in the paper are consistent with any of these possible values for $\alpha$ and $\beta$, but researchers may have some priors on what values these parameters are more likely to take.  Given values for $\alpha$ and $\beta$, this is enough to pin down the per capita total number of Covid-19 cases (both under the policy and in the absence of the policy), and \Cref{tab:sa} reports the average number of Covid-19 cases per 1000 individuals in the absence of the policy and the percentage reduction in total Covid-19 cases due to the policy for those particular values of $\alpha$ and $\beta$.\footnote{In particular, given values of $\alpha$ and $\beta$, (and, for simplicity, ignoring conditioning on covariates), the average effect of the policy, $P(C_{ilt^*}(1) = 1| D_l=1) - P(C_{ilt^*}(0) = 1| D_l=1)$, is given by
  \begin{align*}
    \gamma_1 - \gamma_0 + P(C_{ilt^*}=1|T_{ilt^*}=1, D_l=0) \Big(\alpha \beta (1-\tau_1) - \alpha (1-\tau_0)\Big)
  \end{align*}
  and all terms are identified (this result holds immediately from \Cref{prop:attc-decomp} under \Cref{ass:mar,ass:covid-unc}).}



Notice that the number of total cases strongly depends on $\alpha$; that is, how likely untested individuals are to have had Covid-19 relative to tested individuals.  The assumptions we have made in the paper allow for $\alpha$ to take any value between 0 and 1; it is hard to know precisely what value this should take, but one would suspect that it is at least fairly close to zero.\footnote{Our overall lower bound occurs when $\alpha=1$ and $\beta=0$ -- as discussed above, this is the case where, under the policy, no untested individuals have had Covid-19, but, in the absence of the policy, the probability of having had Covid-19 among the untested is the same as the probability of having had Covid-19 among tested individuals.  This results in a decrease in cases of 99\%, but, in light of the previous discussion, seems like an unlikely case.}  It seems harder to get a good idea about what value $\beta$ is most likely to take though perhaps it is reasonable (or at least conservative) to consider cases where it is relatively close to one.  The open-testing policy has larger effects for small values of $\beta$.  But even for large values of $\beta$, our estimates indicate substantial effects of the open-testing policy in reducing the total number of cases especially when $\alpha$ is not large.  In our view, the most likely estimates for the reduction in per capita Covid-19 cases are between 2\% and 35\%.  These are still wide bounds and reflect a large degree of uncertainty; nevertheless, they indicate that Tennessee's open-testing policy does appear to have decreased the number of actual Covid-19 cases.


\FloatBarrier

\section{Conclusion} \label{sec:conclusion}

In this paper, we have proposed a new approach to evaluating the effects of Covid-19 related policies.  Our approach is particularly suited for evaluating policies early in the pandemic when testing was not widely available.  Our approach is to combine standard policy evaluation identifying assumptions with relatively mild assumptions to deal with total Covid-19 cases in a particular location not being observed.  Our approach leads to bounds on the effect of the policy on total Covid-19 cases and point identification of the effect of the policy on the outcome.


We used this approach to study the effect of Tennessee's open-testing policy that expanded testing and made testing available to anyone who wanted a test.  Overall, we found suggestive evidence that Tennessee's policy  decreased the number of total and confirmed cases in Tennessee and was allowing workers to decrease their trips to work in counties that had experienced greater increases in their confirmed case count over time.  In this sense, it seems that Tennessee's open-testing policy had the intended effects.


\pagebreak

\printbibliography

\pagebreak

