\section{Details of principal stratification model}
\label{sec:ps_details}

\subsection{Priors} 

We adopt weakly-informative priors: 
\begin{equation}
\begin{aligned}
&(\pi^A, \pi^I, \pi^N) \sim \text{Dir}(2,2,2)\\
&\mu^A_0, \mu^A_1, \mu^I_1 \sim  \mathcal{N}(0, 20)\\
&\sigma^A_1 = \sigma^A_0 = \sigma^I_1 \sim \mathcal{N}(0, 1)
\end{aligned}
\end{equation}
\noindent assuming the positive responses are standardized. When covariates are included the prior on ($\pi^A, \pi^I, \pi^N$) is replaced with a prior on $\beta$:
\begin{equation}
\beta \sim \mathcal{N}(0,1)\\
\end{equation}

\subsection{Stan code for principal stratification model}

\linespread{1.1}
\small
\begin{verbatim}
data {
  int<lower=0> N[4]; // responses in each of 4 obs groups Y(1)>0, Y(1)=0, Y(0)>0, Y(0)=0
  vector[N[1] + N[3]] Y; // obs responses > 0 
  real <lower=0, upper=1> Z[N[1] + N[3]]; // treatment indicator for obs responses > 0
}
parameters{
  simplex[3] pi; // strata probabilities indexed by A I N
  vector[2] muA; // indexed by A0, A1
  real muI;
  real<lower=0> sigma; // variances for Y>0 groups 
}
model {
  // priors
  pi ~ dirichlet(rep_vector(2, 3));
  muA ~ normal(0, 20);
  muI ~ normal(0, 20);
  sigma ~ normal(0, 1);
  // observational groups
  N[1] ~ binomial(N[1] + N[2], pi[1] + pi[2]);
  N[3] ~ binomial(N[3] + N[4], pi[1]);
  // outcome (conditional on Y > 0)
  for (n in 1:(N[1] + N[3])) {
    if (Z[n] == 1) {
      target += log_mix(pi[1]/(pi[1] + pi[2]),
                        normal_lpdf(Y[n] | muA[2], sigma),  //A1
                        normal_lpdf(Y[n] | muI, sigma)); //I1
    } else {
      target += normal_lpdf(Y[n] | muA[1], sigma);  //A0
    }
  }
}
generated quantities{
  vector[3] tau; // treatment effects by strata
  real ATE;      // overall average treatment effect
  tau[1] = muA[2] - muA[1];
  tau[2] = muI - 0;
  tau[3] = 0;
  ATE = dot_product(tau, pi);
}
\end{verbatim}
\linespread{1.5}
\normalsize

\subsection{Synthetic data parameter recovery study}

To confirm that the principal stratification model is identified, we simulated data from a model using parameters similar to those estimated in Experiment 2 (see Table \ref{tab:ps_wo_covs}) with 70,000 customers in each group. Table \ref{tab:ps_param_recov} shows that for this synthetic data set, the true value falls within the posterior 95\% credible interval for all parameters. The posterior of $\mu^I_1$ is more diffuse relative to the other parameters, because it is only identified as a small component of the mixture distribution in equation \eqref{eq:treat_purch} and this uncertainty in $\mu^I_1$ is propagated into the estimates of the ATE.  
\linespread{1.1}
\begin{table}[ht]
\caption{Synthetic data parameter recovery for principal stratification model} 
\label{tab:ps_param_recov}
\centering
\begin{tabular}{lccccc}
\hline
& & \multicolumn{4}{c}{Posterior}\\
\cline{3-6}
\multicolumn{1}{l}{Param} & \multicolumn{1}{c}{True} & mean & sd & 2.5\% & 97.5\% \\ 
\hline
$\pi^A$ & 0.200 & 0.199 & 0.001 & 0.197 & 0.202 \\ 
$\pi^I$ & 0.010 & 0.010 & 0.002 & 0.007 & 0.013 \\ 
$\pi^N$ & 0.790 & 0.791 & 0.001 & 0.788 & 0.793 \\
\hline
$\mu^A_0$  & 4.600 & 4.606 & 0.009 & 4.588 & 4.624 \\ 
$\mu^A_1$ & 4.700 & 4.705 & 0.014 & 4.680 & 4.732 \\ 
$\mu^I_1$ & 3.100 & 3.105 & 0.124 & 2.851 & 3.335 \\ 
$\sigma$ & 1.100 & 1.098 & 0.005 & 1.087 & 1.108 \\ 
\hline
ATE ($\tau_{ps}$) & 0.051 & 0.051 & 0.008 & 0.035 & 0.068 \\ 
ATE (\$) & 4.239 & 4.214 & 0.692 & 2.876 & 5.584 \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}


Table \ref{tab:sim_ates} confirms the decrease in posterior variance for the principal stratification model versus estimates from the standard difference-in-means model and the zero-inflated model. 
\linespread{1.1}
\begin{table}[ht]
\caption{Synthetic data estimates of the ATE} 
\label{tab:sim_ates}
\centering
\begin{tabular}{lcccc}
\hline
& \multicolumn{4}{c}{Posterior}\\
\cline{2-5}
Model & mean & sd & 2.5\% & 97.5\% \\ 
\hline
True & 0.0510 &  &  &  \\ 
Diff-in-Means ($\tau_d$) & 0.0418 & 0.0103 & 0.0220 & 0.0619 \\ 
Zero-inflated ($\tau_{zi}$) & 0.0421 & 0.0102 & 0.0218 & 0.0618 \\ 
\hspace{0.1in} with Pos. Lift & 0.0447 & 0.0097 & 0.0259 & 0.0645 \\ Principal Strat ($\tau_{ps}$) & 0.0508 & 0.0084 & 0.0346 & 0.0678 \\
\hline
Variance Reduction (\%) &  & 33.1 &  &  \\ 
StDev Reduction (\%) &  & 18.2 &  &  \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}


A second simulation with 140,000 customers generated according to the model with covariates predicting strata membership is summarized in Table \ref{tab:ps_param_recov_covs}. As discussed in Section \ref{sec:covs}, including covariates improves the identification of the group means $\mu^A_1$ and especially $\mu^I_1$. This is confirmed by narrower posteriors for $\mu^A_1$ and $\mu^I_1$ and, consequently, the ATE, for a similar amount of data. 
\linespread{1.1}
\begin{table}[htbp]
\caption{Synthetic data parameter recovery for principal stratification with covariates} 
\label{tab:ps_param_recov_covs}
\centering
\begin{tabular}{l *{5}{S[table-format=2.3,
         input-decimal-markers={.},
         output-decimal-marker={.}]}}
\hline
& & \multicolumn{4}{c}{Posterior}\\
\cline{3-6}
\multicolumn{1}{l}{Param} & \multicolumn{1}{c}{True} & \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd} & \multicolumn{1}{c}{2.5\%} & \multicolumn{1}{c}{97.5\%}\\
\hline
\multicolumn{5}{l}{Baseline strata probabilities (logit scale)}\\
$\beta^A_0$ & 0 & 0 &  &  \\
$\beta^I_0$ & -3.000 & -3.338 & 0.233 & -3.841 & -2.924 \\ 
$\beta^N_0$ & 1.000 & 1.014 & 0.012 & 0.990 & 1.037 \\
\hline
\multicolumn{5}{l}{No recent purchase ($R_i > 5$)}\\
$\beta^I_1$ & 1.700 & 2.090 & 0.222 & 1.694 & 2.571 \\ 
$\beta^N_1$ & 1.600 & 1.604 & 0.021 & 1.564 & 1.645 \\ 
\hline
\multicolumn{5}{l}{Low prior treatment effect ($Q_i < 0$)} \\
$\beta^I_2$ & 0.000 & -0.046 & 0.135 & -0.311 & 0.212 \\ 
$\beta^N_2$  & 0.300 & 0.286 & 0.016 & 0.254 & 0.320 \\
\hline
$\mu^A_0$  & 4.600 & 4.600 & 0.010 & 4.580 & 4.619 \\ 
$\mu^A_1$& 4.700 & 4.695 & 0.013 & 4.669 & 4.722 \\ 
$\mu^I_1$ & 3.500 & 3.502 & 0.079 & 3.340 & 3.656 \\ 
$\sigma$  & 1.000 & 1.004 & 0.006 & 0.993 & 1.014 \\ 
\hline
ATE ($\tau_{c}$) & 0.063 & 0.058 & 0.007 & 0.046 & 0.072 \\ 
ATE (\$) & 3.291 & 3.133 & 0.479 & 2.196 & 4.091 \\
\hline
\end{tabular}
\end{table}
\linespread{1.5}


\newpage
\section{Distribution of purchase amounts for application in Section \ref{sec:applications}}
\setcounter{table}{0}
\setcounter{figure}{0}
\label{sec:app1_hists}

\linespread{0.1}
\begin{figure}[!ht]
\centering
\includegraphics[width=0.25\textwidth]{figures/expt_1_ctrl_hist.pdf}
\includegraphics[width=0.25\textwidth]{figures/expt_1_treat_hist.pdf}\\
\includegraphics[width=0.25\textwidth]{figures/expt_3_ctrl_hist.pdf}
\includegraphics[width=0.25\textwidth]{figures/expt_3_treat_hist.pdf}\\
\includegraphics[width=0.25\textwidth]{figures/expt_4_ctrl_hist.pdf}
\includegraphics[width=0.25\textwidth]{figures/expt_4_treat_hist.pdf}\\
\includegraphics[width=0.25\textwidth]{figures/expt_5_ctrl_hist.pdf}
\includegraphics[width=0.25\textwidth]{figures/expt_5_treat_hist.pdf}\\
\caption{Distribution of purchase amounts for application in Section \ref{sec:applications}}
\label{fig:app1_hists}
\end{figure}
\linespread{1.5}

\newpage
\section{Proofs}
\setcounter{table}{0}
\begin{proof}[Proof of Proposition \ref{prop:ranking}]
We first prove the following lemma:
\begin{lemma}
\label{lm:propensity}
Let $y_{it}$ be the purchase indicator and $z_{it}$ the exposure indicator for consumer $i$ at time $t$. Suppose that $1/2 >\pi_i^A>\pi_j^I$, and $1/2>\pi_i^I > \pi_j^I$. Then $Pr(y_{it}=1) > Pr(y_{jt}=1)$ and $Pr(y_{it}z_{it}=1) > Pr(y_{jt}z_{jt}=1)$.
\end{lemma}

\begin{proof}
Let $X_{it}^A=1$ iff the consumer is in stratum $A$ at time $t$ and $X_{it}^I=1$ iff the consumer is in stratum $I$ at time $t$. Let $p_z = Pr(z_{t}=1)$ be the unconditional probability of exposure in the population in each time period.

Then:
\begin{align}
    Pr(y_{it}=1) & = Pr((X_{it}^A+X_{it}^I)z_{it}+(X_{it}^A)(1-z_{it})=1) \\ & = Pr(X_{it}^A+X_{it}^I \cdot z_{it}=1) = Pr(X_{it}^A=1)+Pr(X_{it}^I \cdot z_{it}=1) \\
    & = \pi_i^A+\mathbb{E}[X_{it}^I \cdot z_{it}]
\end{align}
Where the penultimate equality follows from the mutual exclusivity of $X_{it}^A$ and $X_{it}^I$.
We notice that the exposure $z_{it}$ only influences purchase if $X_{it}^I=1$. Hence, only correlation between $z_{it}$ and $X_{it}^I=1$ might alter the expectation from one that assumes independence. Denote by $1 > \rho^I \ge 0$ the correlation between $X_{it}^I$ and $z_{it}$. Then $\mathbb{E}[X_{it}^I \cdot z_{it}] = \pi_i^A p_z + \rho^I \sqrt{\pi_i^I(1-\pi_i^I)p_z(1-p_z)}$

For any $\rho^I \ge 0$, if $\pi_i^I > \pi_j^I$ and $\pi_i^I<1/2$ , then $\rho^I \sqrt{\pi_i^I(1-\pi_i^I)p_z(1-p_z)} \ge \rho^I \sqrt{\pi_j^I(1-\pi_j^I)p_z(1-p_z)}$. This implies that $Pr(y_{it}=1) > Pr(y_{jt}=1)$ given that $\pi_i^A + \pi_i^I > \pi_j^A + \pi_j^I$.

Similarly:
\begin{align}
    Pr(y_{it}z_{it}=1) & = \mathbb{E}[y_{it}z_{it}] \\
    & = (\pi_i^A+\pi_i^I)p_z + \rho^A \sqrt{(\pi_i^A)(1-\pi_i^A)p_z(1-p_z)}+\rho^I\sqrt{(\pi_i^I)(1-\pi_i^I)p_z(1-p_z)}
\end{align}
where $1>\rho^A \ge 0$ is the correlation coefficient between $X^A_{it}$ and $z_{it}$. Using a similar argument to before, because $1/2 >\pi_i^A>\pi_j^A$, $Pr(y_{it}z_{it}=1) > Pr(y_{jt}z_{jt}=1)$.
\end{proof}

We can apply Lemma \ref{lm:propensity} and its proof to the responsiveness estimator $Q_i = \frac{\sum_t y_{it} z_{it}}{\sum_t z_{it}}-\frac{\sum_t y_{it} (1-z_{it})}{\sum_t (1-z_{it})}$ in the following way:
\begin{align}\mathbb{E}[Q_i] & = \mathbb{E}[2 y_{it} z_{it} - y_{it}]=2Pr(y_{it}z_{it}=1)-Pr(y_{it}=1)\\ &= 2\left(\pi_i^Ip_z + \rho^A \sqrt{(\pi_i^A)(1-\pi_i^A)p_z(1-p_z)}\right) +\pi_i^A p_z + \rho^I \sqrt{\pi_i^I(1-\pi_i^I)p_z(1-p_z)}
\end{align}
This implies that when $1/2>\pi_i^A>\pi_j^A$ and $1/2>\pi_i^I>\pi_j^I$, then $\mathbb{E}[Q_i]>\mathbb{E}[Q_j]$.

Turning to prove that $\mathbb{E}[R_i] < \mathbb{E}[R_j]$, we first prove the following lemma:
\begin{lemma}
Let $\pi_i$ and $\pi_j$ be the probability of a purchase by a consumer in each time period, and assume that $\pi_i > \pi_j$. Then $\mathbb{E}[R_i]<\mathbb{E}[R_j]$. 
\end{lemma}
\begin{proof}
The pmf of the recency is: $Pr(R=k|\pi) = \frac{\pi (1-\pi)^{k-1}}{1 - (1 - \pi)^T}$.

The expected value is: \[\mathbb{E}[R|\pi] = \frac{T}{(1-\pi)^T-1}+\frac{1}{\pi}+T-1\].

The derivative of the expected value with respect to $\pi$ equals:
\begin{equation}
\frac{\partial \mathbb{E}[R|\pi]}{\partial \pi}  = \frac{T^2 (1-\pi)^{T-1}}{\left((1-\pi)^T-1\right)^2}-\frac{1}{\pi^2}
\end{equation}
At $T=1$ the derivative is zero. The expression $\frac{T^2 (1-\pi)^{T-1}}{\left((1-\pi)^T-1\right)^2}$ is decreasing in $T$. Hence for every $T>1$, $\frac{\partial \mathbb{E}[R|\pi]}{\partial \pi} < 0$ which implies that $\mathbb{E}[R_i]<\mathbb{E}[R_j]$ when $\pi_i > \pi_j$.
\end{proof}

The first lemma shows that if $1/2 > \pi_i^I > \pi_j^I$ then $\pi_i=Pr(y_{it}=1)>Pr(y_{jt}=1)=\pi_j$. Hence, using the second lemma, it implies that $\mathbb{E}[R_i]<\mathbb{E}[R_j]$.
\end{proof}

\newpage
\section{Additional application}
\setcounter{table}{0}
\setcounter{figure}{0}
\label{sec:app2_details}

In a second application we analyze six catalog experiments conducted by a different multi-channel retailer with a similar data structure. The firm collected CRM data for the customers in the experiments for 19 months prior (see Figure \ref{fig:app2_timeline}). 
\begin{figure}[htbp]
\centering
\includegraphics[width=\textwidth]{figures/Application2Timeline.pdf}
\caption{Data timeline for additional application}
\label{fig:app2_timeline}
\end{figure}
The response is all-channel purchases within 30 days after the catalog mailing, which are tracked in the CRM system by name/address matching; this retailer matches more than 90\% of transactions back to existing customers in the CRM system. For each campaign the retailer selected a target population from the CRM system and then randomly assigned a small proportion of those customers to be held out from the catalog mailing. Table \ref{tab:app2_desc} shows the sample sizes for these 6 experiments and the standard difference-in-means estimates of the ATE.
\begin{table}[htbp]
\linespread{1}
\centering
\begin{tabular}{lrrrrrrr}
  \hline
& & & & & & \multicolumn{2}{c}{Diff-in-Means ($\widehat{\tau}_d$)}\\ 
\cline{7-8}
Expt & Month & $n_1$ & $n_0$ & $\overline{\log(Y_i(1)+1)}$ & $\overline{\log(Y_i(0)+1)}$ & mean & sd \\ 
  \hline
1 & Oct 2013 & 906 &  36 & 0.322 & 0.409 & -0.088 & 0.214  \\ 
2 & Nov 2013 & 858 &  36 & 0.427 & 0.428 & -0.005 & 0.246 \\ 
3 & Dec 2013 & 582 &  51 & 0.597 & 0.711 & -0.116 & 0.246  \\ 
4 & Jan 2014 & 783 &  32 & 0.316 & 0.376 & -0.058 & 0.224 \\ 
5 & Feb 2014 & 648 & 125 & 0.212 & 0.188 & -0.025 & 0.097 \\ 
6 & Mar 2014 & 723 &  49 & 0.274 & 0.582 & -0.310 & 0.172 \\ 
  \hline
\end{tabular}
\caption{Descriptive statistics for additional application} 
\label{tab:app2_desc}
\end{table}
The point estimates of the difference in mean are all negative; these experiments have small holdout groups and purchase incidence around 5\%, which results in very imprecise estimates of the ATE.\footnote{A previous version of this paper reported the differences in mean sales, which are mostly positive due to large purchases in the treatment group. Table \ref{tab:app2_desc} reports the difference in log-sales, which are mostly negative.}

To confirm that the distribution of log-sales is suitable for the principal stratification model, we plot histograms of the positive responses in Figure \ref{fig:app2_hists}.
\begin{figure}[!htbp]
\centering
\includegraphics[width=0.4\textwidth]{figures/expt_1_ctrl_hist_2.pdf}
\includegraphics[width=0.4\textwidth]{figures/expt_1_treat_hist_2.pdf}
\caption{Distribution of purchase amounts for Experiment 1 for additional application}
\label{fig:app2_hists}
\end{figure}
While the small sample size makes it difficult to assess these distributions, they are not inconsistent with the Normal and Normal mixture distributions (for purchases in the control and treatment groups, respectively) in the principal stratification model. 

The parameter estimates for the principal stratification model with recency and responsiveness as covariates are shown Table \ref{tab:app2_ps_w_covs} and are distinct from Application 1 in a few ways: 1) the $N$ stratum is larger, the $A$ stratum is smaller and the $I$ stratum is larger, 2) the average purchase amount for the $I$ stratum is higher than $A$ (with the exception of Experiment 6). In other words, this retailer's catalogs generate a larger lift in purchase incidence and those who only purchase when treated make relatively large purchases. This retailer is more luxury-oriented, which may explain this difference in response, and the principal stratification model helps to identify differences like these in the effect of a marketing treatment on sales. The estimates also show that low responsiveness ($Q_i<0$) is predictive of being in the $N$ stratum, i.e., $\beta_2^N$ is large; in the application reported in the paper, responsiveness is less predictive. 

Finally, Table \ref{tab:var_reduction_2} compares the estimates of the ATE for the benchmark models and shows a consistent reduction in the variance of the ATE for the principal stratification models.
\linespread{1.1}
\begin{sidewaystable}[!htbp]
\caption{Parameter estimates for principal stratification model (additional application)}
\label{tab:app2_ps_w_covs}
\centering
\begin{tabular}{lrrrrrrrrrrrr}
\hline
& \multicolumn{2}{c}{Expt 1} &  \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} &\multicolumn{2}{c}{Expt 4} & \multicolumn{2}{c}{Expt 5} & \multicolumn{2}{c}{Expt 6}\\
Model & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd \\
\hline
\multicolumn{11}{l}{Baseline strata probabilities (logit scale)}\\
$\beta_0^A$ & 0 & & 0 & & 0 & & 0 & & 0 & & 0\\
$\beta_0^I$ & -0.035 & 0.461 & -0.362 & 0.415 & -0.596 & 0.531 & -0.258 & 0.430 & -1.357 & 0.757 & -1.459 & 0.741 \\ 
$\beta_0^N$ & 2.188 & 0.324 & 2.079 & 0.253 & 1.591 & 0.267 & 1.948 & 0.286 & 3.003 & 0.357 & 2.737 & 0.321 \\ 
\hline
\multicolumn{11}{l}{No recent purchase ($R_i > 5$)}\\
$\beta_1^I$ & -2.401 & 0.643 & -1.790 & 0.612 & -1.221 & 0.703 & -1.749 & 0.686 & -0.438 & 0.809 & -0.634 & 0.845 \\ 
$\beta_1^N$ & 0.096 & 0.338 & 0.346 & 0.266 & 0.615 & 0.271 & 0.620 & 0.314 & 0.413 & 0.356 & 0.216 & 0.328 \\ 
\hline\multicolumn{11}{l}{Low prior treatment effect ($Q_i < 0$)}\\
$\beta_2^I$ & -1.046 & 0.747 & -1.560 & 0.698 & -1.076 & 0.715 & -1.064 & 0.776 & -0.680 & 0.899 & -0.341 & 0.846 \\ 
$\beta_2^N$ & 1.227 & 0.287 & 0.539 & 0.257 & 0.218 & 0.258 & 1.355 & 0.338 & -0.120 & 0.350 & -0.023 & 0.310 \\ 
\hline
$\mu^A_0$ & 4.917 & 0.521 & 5.150 & 0.453 & 4.530 & 0.319 & 4.027 & 0.456 & 4.688 & 0.310 & 5.700 & 0.308 \\ 
$\mu^A_1$ & 4.143 & 0.140 & 4.723 & 0.112 & 4.553 & 0.185 & 4.412 & 0.156 & 4.827 & 0.184 & 4.779 & 0.177 \\ 
$\mu^I_1$& 6.955 & 0.347 & 6.843 & 0.284 & 6.394 & 0.555 & 6.673 & 0.304 & 5.651 & 1.735 & 4.524 & 2.123 \\ 
$\sigma$ & 0.905 & 0.089 & 0.778 & 0.079 & 0.903 & 0.119 & 0.781 & 0.101 & 0.683 & 0.104 & 0.694 & 0.100 \\ 
\hline
ATE ($\tau_c$)& 0.039 & 0.042 & 0.073 & 0.045 & 0.157 & 0.063 & 0.122 & 0.039 & 0.050 & 0.032 & -0.004 & 0.046 \\ 
   \hline
\end{tabular}
\end{sidewaystable}

\begin{sidewaystable}[htbp]
\small
\caption{Comparison of ATE estimates (additional application)} 
\label{tab:var_reduction_2}
\centering
\begin{tabular}{lrrrrrrrrrrrr}
\hline
& \multicolumn{2}{c}{Expt 1} &  \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} &\multicolumn{2}{c}{Expt 4} & \multicolumn{2}{c}{Expt 5} & \multicolumn{2}{c}{Expt 6}\\
Model & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd\\
\hline
Diff-in-Means ($\widehat{\tau}_d$)  & -0.0883 & 0.2141 & -0.0053 & 0.2462 & -0.1159 & 0.2460 & -0.0584 & 0.2238 & 0.0253 & 0.0972 & -0.3096 & 0.1723 \\ 
Zero-Inflated ($\widehat{\tau}_{zi}$) & -0.1506 & 0.2142 & -0.1053 & 0.2353 & -0.1574 & 0.2364 & -0.1826 & 0.2324 & -0.0173 & 0.0987 & -0.3120 & 0.2140 \\ 
\hspace{0.1in} with Pos. Lift & 0.1110 & 0.0799 & 0.1689 & 0.0953 & 0.2024 & 0.0977 & 0.1637 & 0.0666 & 0.0986 & 0.0472 & 0.0418 & 0.0614 \\ 
PS w/o covs ($\widehat{\tau}_{ps}$) & 0.1075 & 0.0755 & 0.1733 & 0.0908 & 0.2356 & 0.0844 & 0.1853 & 0.0575 & 0.1038 & 0.0504 & 0.0508 & 0.0612 \\ 
\hline
\multicolumn{11}{l}{Reduction in variance for Principal Stratification versus Diff-in-Means}\\
StDev Reduction(\%) & & 64.8 &  & 63.1 &  & 65.7 &  & 74.3 &  & 48.1 &  & 64.5 \\ 
Var Reduction(\%) &  & 87.6 &  & 86.4 &  & 88.2 &  & 93.4 &  & 73.1 &  & 87.4 \\ 
\hline
Reg Adj & -0.0774 & 0.2153 & -0.0299 & 0.2470 & -0.1482 & 0.2457 & -0.0823 & 0.2295 & 0.0302 & 0.1012 & -0.3218 & 0.1770 \\ 
PS w/ covs ($\widehat{\tau}_{c}$)& 0.0389 & 0.0417 & 0.0726 & 0.0449 & 0.1567 & 0.0631 & 0.1220 & 0.0393 & 0.0497 & 0.0323 & -0.0039 & 0.0459 \\ 
\hline
\multicolumn{11}{l}{Reduction in variance for Principal Stratification versus Regression Adjustment}\\
StDev Reduction(\%) &  & 80.6 &  & 81.8 &  & 74.3 &  & 82.9 &  & 68.0 &  & 74.1 \\ 
Var Reduction(\%) &  & 96.2 &  & 96.7 &  & 93.4 &  & 97.1 &  & 89.8 &  & 93.3 \\ 
\hline
\end{tabular}
\end{sidewaystable}



\section{Introduction}

Advertising experiments (also called incrementality tests, lift tests or holdout experiments) gauge the effect of media by comparing sales for customers who are randomly assigned to receive a marketing communication to sales for customers who do not receive that communication, i.e., a control or ``holdout" group \citep{LewisRao2015, HobanBucklin2015, SahniZouChintagunta2016, ZantedeschiFeitBradlow2016, JohnsonReileyNubbmeyer2017, JohnsonLewisReiley2017}. The estimated sales lift obtained from such experiments can be compared to advertising costs to determine whether the advertising has positive returns. This approach to gauging the value of advertising is the ``gold standard" for causal inference \citep{gordonetal2019}, is increasingly popular among digital marketers, and has been implemented in tools like Facebook Conversion Lift and Google Campaign Experiments.

We focus on experiments where ad exposure is randomized at the individual level and the response variable is sales or profit for each customer. In practice, such experiments are often inconclusive because the response for individual customers typically has high variance and advertising effects are often small relative to this noise. This results in estimates of sales lift that are highly imprecise even for very large experiments, leading to imprecise estimates of marketing ROI \citep{LewisRao2015}. Compounding this problem, advertisers often use small control groups (e.g. 5-10\% of the total test size), to reduce the opportunity cost of the test \citep{FeitBerman2019}. 

We propose a new model for analyzing advertising experiments that leverages post-stratification. Post-stratification involves separating the subjects in the experiment into groups with different treatment effects and has two key benefits for marketing analysts: First, stratification provides estimates of heterogeneous treatment effects for each stratum that can be useful for targeting, and, second, the stratified estimate of the average treatment effect may be more precise than standard estimators \citep{ganju2011, deng2013, MiratrixSekhonYu2013}. The latter benefit is less well-known in the marketing literature, but potentially very important for advertisers; our applications show that the improvements in the variance of the estimated average treatment effect (ATE) can be substantial, even for experiments with large sample sizes. 

A key motivation for our approach to post-stratification is that many customers in advertising experiments do not purchase. Thus, the observed response can be decomposed into the purchase incidence and the amount purchased conditional on incidence.\footnote{This approach would not provide any benefit for experiments with a binary response (conversions, clicks).} We stratify customers in a marketing experiment into three groups based on their potential outcomes for purchase incidence: those who only purchase if exposed to the ad, those who purchase regardless of their exposure to the ad, and those who do not purchase regardless of treatment. This is distinct from most post-stratification approaches which rely on observed pre-randomization (baseline) covariates.

Since we do not observe both potential outcomes for each customer, these strata are latent, but the size of these groups and their average response can be estimated using a mixture model for the customers in the treatment group who are observed to purchase. 
This is a new adaptation of principal stratification \citep{FrangakisRubin2002}, which has been used previously to analyze experiments with treatment non-compliance \citep{RoyHoganMarcus2007, Gallopetal2009, barajas2016} and truncation by death \citep{rubin2006, ZhangRubinMealli2009, ding2011}.  The details of this model are laid out in Section \ref{sec:principal_stratification}.

The model divides customers into strata that have substantially different treatment effects. Specifically, it allows us to estimate the large number of customers who will not purchase regardless of treatment and therefore have a treatment effect of precisely zero for each customer. Typically, the other two groups will have larger treatment effects, with the greatest treatment effect for the group who only purchases when treated. When we re-combine the estimates for each stratum to estimate the overall ATE, the precision of the overall ATE is increased. In the applications to catalog tests reported in Section \ref{sec:applications}, the ATEs from the principal stratification model have on average 50\% smaller posterior variances than estimates from the standard difference-in-means model. 

The approach is distinct from previous applications of post-stratification where observed pre-randomization covariates for each customer are included in a regression as main effects \citep{deng2013, LewisRao2015} or as interactions with the treatment indicator \citep{ganju2011, lin2013,  MiratrixSekhonYu2013}. Empirically, using pre-randomization covariates in this way has produced somewhat modest increases in the precision of the ATE especially when demographics are used for stratification. One suggestion is to use pre-treatment measurements of the same response (in our case sales) as main effects in a regression \citep[the CUPED method of][]{deng2013}, although this does not work in all cases \citep{jones2017}. 

Although the principal stratification model does not require pre-randomization covariates, it can incorporate them as predictors of the principal strata, as we discuss in Section \ref{sec:covs}. Many marketers have rich observational data on each customer's purchase history and response to past marketing. We focus on two summaries of past customer behavior that are theoretically related to the effect of a marketing treatment. The first is a measure of the customer's responsiveness to similar marketing treatments in the past. While estimates of responsiveness are often biased due to targeting or treatment selection, the estimates still rank customers accurately, under some mild assumptions which we detail. The second pre-randomization covariate we focus on is the time since last purchase (i.e., recency), which is a predictor of latent attrition \citep{FaderHardieLee2005}. Customers with longer time since last purchase are more likely to have left the firm and therefore less likely to be responsive to the marketing treatment. As we will show, these two pre-randomization covariates are associated with stratum membership and are available to many marketers. 
Post-stratification is carried out in the analysis stage of the experiment, and requires no changes to the design of the experiment; the strategies we propose can be used to re-analyze experiments that have already been fielded. In the main application, we use the proposed model to re-analyze 5 catalog experiments conducted by a retailer and across all of these experiments we find that more than 75\% of customers have potential outcomes $Y(0) = Y(1) = 0$. That is, they will not purchase during the experiment regardless of treatment. Customers who have not purchased recently are even more likely to be in this zero-treatment effect group. Separating out these ``never-buy" customers results in a 36-57\% reduction in the variance of the estimate of the ATE. We also report a second application in Appendix \ref{sec:app2_details}.

To summarize, the principal stratification model we propose leverages the fact that many advertising experiments contain a large volume of zero responses, while the positive responses can be measured continuously. The decomposition into three strata provides a more precise estimate of the ATE and provides predictions about customer stratum membership. The reduction in the variance of the ATE is useful for computing a more accurate ROI estimate for marketing actions, and can also be used to lower the required sample size of experiments.\footnote{The sample size increases linearly with the expected variance of the ATE.} The allocation of customers into strata can also allow firms to target customers based on their responsiveness. 

\section{Principal stratification for advertising experiments}
\label{sec:principal_stratification}

In an advertising experiment $n$ customers indexed by $i = 1,\ldots,n$ are randomly assigned to treatment ($Z_i = 1$) or control ($Z_i = 0$) and each customer's total sales or profit, $Y_i(Z_i)$, is observed for some period after the exposure. We adopt the potential outcomes framework where each customer in the test has two potential outcomes, $Y_i(Z=0)$ and $Y_i(Z=1)$, and the analyst observes the outcome consistent with the customer's random assignment. The main goal of the experiment is to estimate the ATE: 
\begin{equation}
\tau = \mathbb{E} [Y_i(Z=1)-Y_i(Z=0)]
\label{eq:ATE_defn}
\end{equation}
Estimates of $\tau$ can be compared to costs to determine if the marketing treatment being tested has positive ROI for the population in the test. A standard way to estimate the ATE is by linear regression:
\begin{equation}
\label{eq:diff-in-means_est}
    Y_i = \alpha +\tau_{d}Z_i + \epsilon_i
\end{equation}
This results in nearly the same estimate of the ATE as the common difference-in-means estimator; the only difference is the pooled variance in the regression model. This estimate of the ATE is often imprecise due to the large variance of $Y_i$ in most marketing response data \citep{LewisRao2015}. 

Why is the variance of $Y_i$ high in advertising experiments? A major contributor is the fact that many observed values of $Y_i$ are zero because the customer did not purchase, while $Y_i$ is usually substantially larger than zero for a relatively small fraction of customers who purchase (see, for example, Figure \ref{fig:app1_hist}). The model we propose divides customers into three principal strata based on whether their potential outcomes are zero or positive:

\noindent \textbf{Always-buy ($A$)} - those who will buy regardless of treatment ($Y_i(1)>0$ and $Y_i(0)>0$).

\noindent \textbf{Influenced ($I$)} - those who will buy if treated and not buy otherwise ($Y_i(1)>0$ and $Y_i(0)=0$).

\noindent \textbf{Never-buy ($N$)} - those who will not buy regardless of treatment ($Y_i(1)=0$ and $Y_i(0)=0$).

The columns in Figure \ref{fig:principal_strata} depict these three strata. We denote the proportion of customers in each stratum as $\pi^A$, $\pi^I$ and $\pi^N$, while we denote the mean response in stratum $s\in\{A,I,N\}$ with exposure $Z=z$ as $\mathbb{E}[Y_i^s(z)]=\mu^s_z$.

\begin{figure}[htbp]
\centering
\includegraphics{figures/Principal_Stratification_no_C.pdf}
\caption{Customer groups under principal stratification}
\label{fig:principal_strata}
\end{figure}

The model assumes that there are no customers who purchase only if they are \emph{not} exposed to advertising. This is consistent with most other models of ad response that assume there can not be a negative response to ads (e.g., ad stock models) and improves the identification of the model \citep{feller2019}.  As we show formally in Proposition \ref{prop:var_reduction} below, this assumption it is not the primary driver of reductions in the posterior variance of the ATE. 

This model is similar to, but different from, principal stratification models used to analyze experiments with truncation-by-death \citep{ZhangRubinMealli2009, ding2011}. Truncation-by-death occurs when the focal response variable can only be measured for subjects who are alive, e.g., heart rate or quality-of-life. However, in truncation-by-death the focal response is undefined for subjects who die and so the ATE is only defined for the group which is alive under both treatments. By contrast, in our model for sales response customers who do not buy yield the firm zero revenue. Thus, the three strata have ATEs: 
\begin{align}
\tau^A &= \mathbb{E}[Y_i(1) - Y_i(0)|i \in A] = \mu^A_1-\mu^A_0\nonumber\\
\tau^I &= \mathbb{E}[Y_i(1) - Y_i(0)|i \in I] = \mathbb{E}[Y_i(1) - 0| i 
\in I] = \mu^I_0\nonumber\\
\tau^N &= \mathbb{E}[Y_i(1) - Y_i(0)|i \in N] = \mathbb{E}[0-0] = 0
\label{eq:strata_effects}
\end{align}
If the experimenter had full knowledge of stratum membership for each customer denoted by the indicators $X_i^A$, $X_i^I$ and $X_i^N$, they could incorporate these covariates in a regression with a full set of treatment:covariate interactions to estimate the post-stratified ATE $\tau_{ps}$:
\begin{equation}
\label{eq:ps_est}
    Y_i = \alpha + \tau_{ps} Z_i + \beta^A X_i^A + \beta^I X_i^I + \gamma^A Z_i (X_i^A - \overline{X}_i^A) + \gamma^I Z_i (X_i^I - \overline{X}_i^I)+\epsilon_i
\end{equation}
where $\overline{X}$ is the mean of covariate $X$ \citep{ganju2011,lin2013, MiratrixSekhonYu2013}. The mean-centering ensures that the coefficient $\tau_{ps}$ is the ATE. Not including the interaction terms (sometimes called ``regression adjustment'') introduces a bias in the estimates of the ATE \citep{lin2013} which becomes smaller with $n$.

In practice, the experimenter does not know the stratum membership of each observation, but if they can estimate $\pi^s$ and $\mu^s_z$ they can compute:
\begin{equation}
\widehat{\tau}_{ps} = \sum_{s \in \{A,I,N\}} \widehat{\pi}^s \widehat{\tau}^s = \widehat{\pi}^A (\widehat{\mu}^A_1 - \widehat{\mu}^A_0) + \widehat{\pi}^I \widehat{\mu}^I_1
\label{eq:ATE}
\end{equation}
We refer to this as the principal stratification ATE for advertising experiments. As we discuss in Section \ref{sec:applications}, $\pi^s$ and $\mu^s_z$ can be estimated from the experimental data where the $X_i^s$ are latent. However, before we discuss estimation, we first show the theoretical benefit of stratification assuming the $X^s_i$ are observed. 

\subsection{Benefit of principal stratification}
Using equations (1) and (5) of \cite{ganju2011} we can show that the gain from using post-stratification when the allocation to treatment and control is equal ($n_1=n_0$) is:
\begin{equation}
\label{eq:reduction}
\Delta = Var(\widehat{\tau}_{d}) - Var(\widehat{\tau}_{ps}) = \frac{4\sum_s \pi^s (\mu^s_1-\mu_1)(\mu^s_0-\mu_0)}{n}
\end{equation}
where $\mu_z = \sum_s \pi^s \mu^s_z$. 
In general, stratification may not always be beneficial for reducing the variance of the ATE; when $\Delta<0$, the benefit is negative. Intuitively, because $\Delta$ is proportional to the covariance in the treatment and control means for each stratum, $\Delta>0$ when the mean response within strata are positively correlated between treatment and control. 

In the principal stratification model the treatment and control means for each stratum are defined by $\mu^s_1$ and $\mu^s_0$, which can be substituted into \eqref{eq:reduction} to prove the following proposition:
\begin{prop}
\label{prop:var_reduction}
If stratum memberships are observed:
\begin{itemize}
\item The reduction in variance equals: $\Delta = Var(\widehat{\tau}_{d}) - Var(\widehat{\tau}_{ps})=\frac{4 \pi^A \mu^A_0 \left((1-\pi^A) \mu ^A_1 -\pi^I \mu^I_1\right)}{n}$.
\item There is a positive reduction in variance if and only if $\frac{\mu^A_1}{\mu^I_1} > \frac{\pi^I}{\pi^I+\pi^N}$.
\end{itemize}
\end{prop}
\begin{proof}
The expression for reduction in variance results from plugging in $\mu_0 = \pi^A \mu^A_0$ and $\mu_1 = \pi^A \mu^A_1 + \pi^I \mu^I_1$ into \eqref{eq:reduction}. The condition in the second item can be obtained from noticing that the expression inside the parenthesis needs to be positive for a positive reduction.
\end{proof}

The advertiser can expect to achieve a more-precise estimate of the ATE from principal stratification when the always-buyer group has high average sales under treatment compared to the influenced group, or when the never-buy group is large relative to the influenced group. Figure \ref{fig:ate-simul} illustrates the benefit through a simulation comparing the standard difference-in-means estimate from \eqref{eq:diff-in-means_est} to the post-stratified estimate from \eqref{eq:ps_est} using different sample sizes and treatment allocation proportions. The vertical bars indicate the sampling variance of the estimators, which are substantially smaller for the post-stratified estimator in all cases. 

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.48\textwidth]{figures/ate-n.pdf}
\includegraphics[width=0.48\textwidth]{figures/ate-p.pdf}\\
\begin{flushleft}
\footnotesize
Note: Data is simulated using the parameters estimated for Experiment 2 in Table \ref{tab:ps_wo_covs} with 500 replications. Vertical lines show the range of estimates under these true values. Treatment proportion in left panel is 0.5. Sample size in right panel is 10,000.
\end{flushleft} 
\caption{Simulated ATE estimates for difference-in-means and post-stratification}
\label{fig:ate-simul}
\end{figure}

Although $\Delta$ shrinks at a rate of $\frac{1}{n}$, the variances themselves also shrink at the same rate, and so the relative reduction in sampling variance $\left(Var(\widehat{\tau}_{d}) - Var(\widehat{\tau}_{ps})\right)/Var(\widehat{\tau}_{d})$ can be substantial even for large samples. Thus, stratification can improve estimates of advertising incrementality and ROI even for large tests. 

Proposition \ref{prop:var_reduction} and the previous work on post-stratification has assumed that each observation's stratum membership is observed by the experimenter, and not latent as in principal stratification. In our model, the strata sizes and membership are estimated, which may add additional noise and increase the variance $\widehat{\tau}_{ps}$. Research in the survey literature on such ``endogenous post-stratification'' estimators has shown that they are asymptotically consistent \citep{breidt2008,dahlke2013}. We use Bayesian inference to propagate these estimation errors into $\widehat{\tau}_{ps}$. The decrease in precision due to estimation error is an empirical question and our applications show that this is far outweighed by the increase in precision due to principal stratification.

Before we proceed to discuss the estimation of the model, we develop a summary statistic firms can compute from their data to determine if stratification may be beneficial.

\subsection{Quantifying the expected benefit from principal stratification}

Because the treatment assignment $Z$ is independent from the potential outcomes and because everyone who buys with $Z=0$ is in the always buyer group, the experimenter can estimate $\pi^A$ as the proportion in the control group that buys. Any buyer with $Z_i=1$ who is not an always-buyer is an influenced, hence $\pi^I$ can be estimated as the proportion of buyers in treatment minus the proportion of buyers in control. Thus, we can estimate the strata proportions with the following consistent estimators: 
\begin{align} 
\widehat{\pi}^A &= \sum_{i: Z_i=0} \frac{1}{n_0}I(Y_i>0)\\ \nonumber
\widehat{\pi}^I &= \frac{1}{n_1}\sum_{i: Z_i=1} I(Y_i>0)-\widehat{\pi}^A\\ \nonumber
\widehat{\pi}^N &= 1 - \widehat{\pi}^A - \widehat{\pi}^I
\end{align}
where $n_0$ and $n_1$ are the number of customers with $Z_i=0$ and $Z_i=1$ respectively.

The values of $\mu^A_1$ and $\mu^I_1$ cannot be identified without further modeling assumptions (which we layout in the next section), but a lower bound on the ratio $\mu^A_1/\mu^I_1$ can be calculated.
Using the average of the lowest $\lfloor n_1 \cdot \widehat{\pi}^A \rfloor$ observations from the set $\{Y_i(1) | Y_i(1)>0\}$ provides a lower bound on $\widehat{\mu}^A_1$ and the average of the remaining largest observations from $\{Y_i(1) | Y_i(1)>0\}$ will provide an upper bound on $\widehat{\mu}^I_1$. If we denote these averages as $\widehat{\mu}^{I\max}_1$ and $\widehat{\mu}^{A\min}_1$ then clearly $\widehat{\mu}^A_1/\widehat{\mu}^I_1 > \widehat{\mu}^{A\min}_1/\widehat{\mu}^{I\max}_1$.

Hence, based on Proposition \ref{prop:var_reduction} if the experimenter observes 
\begin{equation}
\frac{\widehat{\mu}^{A\min}_1}{\widehat{\mu}^{I\max}_1} > \frac{\widehat{\pi}^I}{\widehat{\pi}^I+\widehat{\pi}^N}
\end{equation}
they should expect a benefit from applying stratification.

How large can this benefit be? We can use the same approach to provide a lower bound on the gain:
\begin{align}
    \label{eq:lower-bound}
    \widehat{\Delta}_{\min} & = \frac{4 \widehat{\pi}^A \widehat{\mu}^A_0 \left((1-\widehat{\pi}^A) \widehat{\mu}^{A\min}_1 -\widehat{\pi}^I \widehat{\mu}^{I\max}_0\right)}{n}
\end{align}
By dividing these estimates with $Var(\widehat{\tau}_d)$, the experimenter can find a lower bound on the expected relative decrease in variance if they analyze the experiment using principal stratification.

Figure \ref{fig:var-simul} illustrates a simulation study of the differences between the lower bound in \eqref{eq:lower-bound} and the actual reduction in variance. Proposition \ref{prop:var_reduction} and the lower bound in \eqref{eq:lower-bound} both assume an equal allocation between treatment and control and $\widehat{\Delta}_{\min}$ underestimates the variance improvement for unequal treatment allocations.

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.48\textwidth]{figures/var-n.pdf}
\includegraphics[width=0.48\textwidth]{figures/var-p.pdf}\\
\begin{flushleft}
\footnotesize
Note: Data is simulated using the parameters estimated for Experiment 2 in Table \ref{tab:ps_wo_covs} with 500 replications. Treatment proportion in left panel is 0.5. Sample size in right panel is 10,000.
\end{flushleft}
\caption{Average decrease in variance of the post-stratified ATE relative to diff-in-means}
\label{fig:var-simul}
\end{figure}

In summary, the principal stratification model provides us an integrated accounting of the effect of the ad (i.e., the lift in purchase incidence, $\pi^I$, versus the increase in spend for those in stratum $A$,  $\mu^A_1 - \mu^A_0$) and has the potential to increase the precision of the estimate of the overall ATE. Moreover, analysts can compute a simple statistic prior to estimating the model to determine if the post-stratified estimate is likely to be more precise than the difference-in-means. 

\section{Estimation of principal stratification model}
\label{sec:applications}

We do not directly observe which principal strata each customer belongs to. We only observe that the customer belongs to one of the four groups shaded in Figure \ref{fig:principal_strata}: customers who are treated and buy,
customers who are treated and don't buy, customers in the control who buy, and customers in the control who do not buy. However, even without directly observing the principal strata, the model can be estimated based on the marginal likelihood of the observed data. 

Following the literature on principal stratification, the response for each strata and treatment combination is specified as: 
\begin{equation}
\log(Y_i(Z_i) + 1) \sim 
\begin{cases}
\mathcal{N}(\mu^A_0,\sigma) &\text{ if } i \in A \text{ and } Z_i=0\\
\mathcal{N}(\mu^A_1,\sigma) &\text{ if } i \in A \text{ and } Z_i=1\\
\mathcal{N}(\mu^I_0,\sigma) &\text{ if } i \in I \text{ and } Z_i=1\\
0 &\text{otherwise}\\
\end{cases}
\label{eq:specification}
\end{equation}
If the log-sales are not normally distributed, other distributions could be specified for the positive purchase amounts. 

Using the Normal specification in \eqref{eq:specification}, the marginal likelihood of a positive observation in the treatment group (upper left shaded area in Figure \ref{fig:principal_strata}) is:
\begin{equation}
\ell(Y_i(Z_i=1)>0) = \pi^A\mathcal{N}(\log(Y_i+1)|\mu^A_1,\sigma) + \pi^I\mathcal{N}(\log(Y_i+1)|\mu^I_0,\sigma)
\label{eq:treat_purch}
\end{equation}
The remaining customers in the treatment group who did not purchase (upper right shaded area of Figure \ref{fig:principal_strata}) must belong to stratum N, and this occurs with likelihood:
\begin{equation}
\ell(Y_i(Z_i=1)=0) = \pi^N
\label{eq:treat_nopurch}
\end{equation}
Similarly, customers in the control group who purchase (lower left shaded area in Figure \ref{fig:principal_strata}) must be in stratum $A$ and the likelihood of these observations is: 
\begin{equation}
\ell(Y_i(Z_i=0)>0) = \pi^A \mathcal{N}(\log(Y_i + 1)|\mu^A_0,\sigma)
\label{eq:ctrl_purch}
\end{equation}
\noindent and the customers in the control group who do not purchase (lower right shaded area in Figure \ref{fig:principal_strata}) are in strata $I$ or $N$ with likelihood: 
\begin{equation}
\ell(Y_i(Z_i=0)=0) = \pi^I + \pi^N
\label{eq:ctrl_nopurch}
\end{equation}

Equations \eqref{eq:treat_purch} -- \eqref{eq:ctrl_nopurch} form a marginal likelihood for the observed data which we can use for analyzing advertising experiments. Equations \eqref{eq:treat_nopurch}, \eqref{eq:ctrl_purch} and \eqref{eq:ctrl_nopurch} serve to identify $\pi^A$, $\pi^I$ and $\pi^N$ and equation \eqref{eq:ctrl_purch} identifies $\mu^A_0$ and $\sigma$. The mixture model in \eqref{eq:treat_purch} is identified except in the pathological case that $\mu^A_1 = \mu^I_0$ or $\pi^I$ or $\pi^A$ is zero; that is, the $A$ and $I$ strata must not be empty and there must be a difference in the mean purchase amount for treated customers in the $A$ stratum and the $I$ stratum. The parameters can be estimated using maximum likelihood, method of moments or Bayesian estimation. We opt for Bayesian estimation, because it allows us to propagate the noise in the estimates into an overall ATE calculation without making use of approximations. 

We estimate the model in \eqref{eq:treat_purch} -- \eqref{eq:ctrl_nopurch} using Hamiltonian Monte Carlo as implemented in Stan \citep{stan2018}. The priors are appropriately scaled to be relatively uninformative. The details of the priors, complete Stan code for the model and a parameter recovery study are included in Appendix \ref{sec:ps_details}.

\subsection{Application}
We estimate the model for 5 catalog lift tests that were conducted by a US multi-channel speciality retailer between September 2017 and February 2018. For each experiment, the retailer randomly selected approximately 140,000 customers from their active customer list and a high-end catalog was mailed to half this list at random. For each customer, all-channel purchases (net returns) in the 30 days after the experiment were tracked using the retailer's regular name/address/email matching process. (Credit-card usage at this retailer is high and more than 80\% of transactions are matched to an existing customer in the CRM system.) Our analysis focuses on the log-sales for each customer, $\log(Y_i + 1)$, as the key response.  

Table \ref{tab:app1_desc} contains basic summary statistics and the difference-in-means estimate of the ATE for each experiment.\footnote{Table \ref{tab:app1_desc} shows the posterior mean and standard deviation of the difference in log-sales for the model in equation \eqref{eq:diff-in-means_est}; OLS estimates of the mean and standard error are nearly identical.} For experiments 1 and 5, the estimated treatment effect $\widehat{\tau}_d$ is small and the posterior of the difference-in-means estimate straddles zero. For experiments 2, 3 and 4, which were conducted over the holiday season, the treatment effect is larger and the 95\% credible interval is positive.
\linespread{1.1}
\begin{table}[htbp]
\caption{Summary of experiments} 
\label{tab:app1_desc}
\centering
\begin{tabular}{c l *{2}{S[table-format=2.3,
         input-decimal-markers={,},
         output-decimal-marker={,}]} *{4}{S[table-format=1.4,
         input-decimal-markers={.},
         output-decimal-marker={.}]}}
\hline
& & & & & & \multicolumn{2}{c}{Diff-in-Means ($\widehat{\tau}_d$)}\\ 
\cline{7-8}
\multicolumn{1}{c}{Expt} & \multicolumn{1}{c}{Month} & \multicolumn{1}{c}{$n_1$} & \multicolumn{1}{c}{$n_0$} & \multicolumn{1}{c}{$\overline{\log(Y_i(1) + 1)}$} & \multicolumn{1}{c}{$\overline{\log(Y_i(0) + 1)}$} & \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd}\\
\hline
1 & Sept 2017 & 69,291 & 68,990 & 0.757 & 0.753 & 0.004 & 0.010 \\ 
2 & Oct 2017 & 69,268 & 68,959 & 0.772 & 0.747 & 0.025 & 0.010 \\ 
3 & Nov 2017 & 69,241 & 68,914 & 1.016 & 0.992 & 0.025 & 0.011 \\ 
4 & Dec 2017 & 69,238 & 68,900 & 1.103 & 1.078 & 0.025 & 0.011 \\ 
5 & Feb 2018 & 69,207 & 68,832 & 0.554 & 0.543 & 0.011 & 0.008 \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}

The principal stratification model specifies log-sales in the control group as Normally distributed (see equation \eqref{eq:ctrl_purch}) and log-sales in the treated group follow the mixture distribution in equation \eqref{eq:treat_purch}. Figure \ref{fig:app1_hist} shows histograms of the positive log-sales for Experiment 2, which are consistent with these assumptions. Histograms for other experiments are similar and are included in Appendix \ref{sec:app1_hists}.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{figures/expt_2_ctrl_hist.pdf}
\includegraphics[width=0.45\textwidth]{figures/expt_2_treat_hist.pdf}
\caption{Distribution of purchase amounts for Experiment 2}
\label{fig:app1_hist}
\end{figure}

\subsubsection{Parameter estimates}
Posterior estimates for the principal stratification model for all 5 experiments are shown in Table \ref{tab:ps_wo_covs}. Taking Experiment 2 as an example, we find that 16.2\% of customers are in stratum $A$ and 0.4\% of customers are in stratum $I$, consistent with the observed purchase rates in Figure \ref{fig:app1_hist}. The average sales amount for customers in stratum $I$ is $\exp(3.078 + 1.101^2/2)-1 = \$38.81$. For customers in the $A$ stratum, we see an increase in average purchase amount of ($\exp(4.691 + 1.101^2/2$) - 1) - ($\exp(4.616+1.101^2/2$) - 1) = \$14.43. Importantly, we estimate that 83.4\% of customers are in the never-buy stratum with a treatment effect of exactly zero. Results for other experiments are similar. 
\linespread{1.1}
\begin{table}[htbp]
\caption{Parameter estimates for principal stratification model}
\label{tab:ps_wo_covs}
\centering
\begin{tabular}{lrrrrrrrrrr}
\hline
& \multicolumn{2}{c}{Expt 1} & \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} & \multicolumn{2}{c}{Expt 4} & \multicolumn{2}{c}{Expt 5}\\
Par & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd\\
\hline
$\pi^A$ & 0.163 & 0.001 & 0.162 & 0.001 & 0.213 & 0.001 & 0.236 & 0.001 & 0.116 & 0.001 \\ 
$\pi^I$ & 0.003 & 0.001 & 0.004 & 0.001 & 0.005 & 0.001 & 0.006 & 0.002 & 0.003 & 0.001 \\ 
$\pi^N$ & 0.834 & 0.001 & 0.834 & 0.001 & 0.783 & 0.001 & 0.758 & 0.001 & 0.881 & 0.001 \\ 
$\mu^A_0$ & 4.575 & 0.010 & 4.616 & 0.010 & 4.660 & 0.009 & 4.551 & 0.008 & 4.648 & 0.012 \\ 
$\mu^A_1$ & 4.638 & 0.014 & 4.691 & 0.015 & 4.717 & 0.012 & 4.597 & 0.012 & 4.734 & 0.017 \\ 
$\mu^I_0$ & 3.016 & 0.245 & 3.078 & 0.239 & 3.056 & 0.213 & 3.296 & 0.238 & 3.089 & 0.220 \\ 
$\sigma$ & 1.091 & 0.006 & 1.101 & 0.006 & 1.088 & 0.005 & 1.061 & 0.005 & 1.099 & 0.007 \\ 
\hline
ATE ($\tau_{ps}$)& 0.021 & 0.006 & 0.026 & 0.007 & 0.027 & 0.007 & 0.031 & 0.009 & 0.021 & 0.006 \\ 
ATE (\$) & 1.980 & 0.556 & 2.535 & 0.601 & 2.573 & 0.681 & 2.160 & 0.622 & 2.142 & 0.523 \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}

The estimated within-strata response variance for Experiment 2 is $\sigma^2 = (1.101)^2 = 1.212$, which is substantially smaller than the variance estimated for the difference-in-means model, which is $(1.774)^2 = 3.147$. This implies that the variances in the individual treatment effects within each stratum are substantially smaller than the variance across all customers, which is precisely the condition under which post-stratification produces a more precise estimate of the average treatment effect than the difference-in-means estimator. 

\subsubsection{Improvement in posterior variance of the ATE} 
To illustrate the reduction in posterior variance of the ATE, we compare the principal stratification model to the standard difference-in-means model in equation \eqref{eq:diff-in-means_est}. We estimate it with Bayesian methods to facilitate comparison, assuming a Normally distributed error term and diffuse priors. 
As a second benchmark, we also compare to a model with zero-inflated Normals, to accommodate the large number observed zeros, specifically: 
\begin{align}
\log(Y_i(Z_i=0)+1) \sim \begin{cases}
    N(\alpha, \sigma) & \text{with probability } q_0\\
    0              & \text{otherwise}
\end{cases}\\
\log(Y_i(Z_i=1)+1) \sim
\begin{cases}
    N(\beta, \sigma) & \text{with probability } q_1\\
    0             & \text{otherwise}
\end{cases}  \nonumber
\label{eq:zi-diff-in-norms}
\end{align}
\noindent where the ATE is estimated as $\tau_{zi} = q_1 * \beta - q_0 * \alpha$.\footnote{We overload $\alpha$ and $\sigma$ to emphasize the relationships between models.} Since the principal stratification model only allows for an increase in purchase incidence, we also estimate a constrained version of this model where $q_1 > q_0$.

Table \ref{tab:var_reduction_1} and Figure \ref{fig:var_reduction_1} compare the ATE estimates from the principal stratification model versus the three benchmark models. The point estimates of the ATE are generally similar; the exception are Experiments 1 and 5 where the estimated ATE is different for models that are constrained to positive lift versus the unconstrained models. Principal stratification restricts the change in purchase incidence to be non-negative, which pushes the estimates up slightly particularly for Experiments 1 and 5, where the observed incidence in treatment and control are similar. Other changes in the estimates are due to the ATE being estimated for the sub-sample in each stratum, rather than from the whole sample. 

More importantly, the estimates of the ATE based on the principal stratification model have substantially lower posterior variance than all benchmark models, including the zero-inflated model that is constrained to have positive increase in incidence between treatment and control. Across the 5 experiments, we see reductions of 36.1\% to 57.4\% over the standard difference-in-means estimator ($\widehat{\tau}_d$).  The zero-inflated model provides very little reduction in variance, suggesting that simply accounting for zeros in the model is insufficient to reduce the variance. The zero-inflated model that is constrained to positive lift provides a substantial improvement in variance, but is still dominated by the principal stratification model. 
\begin{figure}[!htbp]
\centering
\includegraphics[width=\textwidth]{figures/ATE_by_model.pdf}
\caption{Comparison of ATE estimates for principal stratification versus benchmark models}
\label{fig:var_reduction_1}
\end{figure}
\begin{table}[!htbp]
\small
\caption{Comparison of ATE estimates for principal stratification versus benchmark models} 
\label{tab:var_reduction_1}
\centering
\scalebox{0.9}{
\begin{tabular}{lrrrrrrrrrr}
\hline
& \multicolumn{2}{c}{Expt 1} &  \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} &\multicolumn{2}{c}{Expt 4} &\multicolumn{2}{c}{Expt 5}\\
Model & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd\\
\hline
Diff-in-Means ($\widehat{\tau}_d$) & 0.0042 & 0.0094 & 0.0249 & 0.0095 & 0.0242 & 0.0108 & 0.0246 & 0.0108 & 0.0108 & 0.0083 \\ 
Zero-Inflated ($\widehat{\tau}_{zi}$) & 0.0041 & 0.0094 & 0.0248 & 0.0096 & 0.0248 & 0.0106 & 0.0253 & 0.0107 & 0.0107 & 0.0085 \\ 
\hspace{0.1in} with Pos. Lift & 0.0160 & 0.0064 & 0.0289 & 0.0086 & 0.0299 & 0.0097 & 0.0298 & 0.0098 & 0.0180 & 0.0065 \\ 
Principal Strat ($\widehat{\tau}_{ps}$) & 0.0206 & 0.0061 & 0.0255 & 0.0068 & 0.0268 & 0.0074 & 0.0307 & 0.0086 & 0.0210 & 0.0055 \\ 
\hline
\multicolumn{11}{l}{Reduction in variance for Principal Strat versus Diff-in-Means}\\
StDev Reduction (\%) & & 34.7 & & 28.7 & & 31.5 & & 20.1 & & 32.9 \\ 
Var Reduction (\%) & & 57.4 & & 49.2 & & 53.0 & & 36.1 & & 54.9 \\ 
\hline
\end{tabular}}
\end{table}

One way to realize the benefit of principal stratification is to consider how large the control group needs to be to detect a difference in the ROI given a fixed campaign size of 140K. For Experiment 2, testing a null hypothesis of $0\%$ ROI vs. an alternative of $25\%$ ROI with 90\% power and 5\% 1-sided confidence level,  requires 66K customers in the control group using the diff-in-means analysis (assuming a catalog cost of \$1 and using the average sales of \$9.31 per customer). Using principal stratification cuts this requirement in half to 33K and the difference translates to an increase of approximately \$77K in additional revenues when the alternative hypothesis is true.\footnote{The ROI is defined as $ROI = \frac{Y(1)-Y(0)-c}{c}$ when $c$ is the cost. It can be transformed to fit our model for estimating $ATE = \log(1+Y(1)-c(1+ROI)) - \log(1+Y(0))$.}

\section{Pre-randomization covariates}
\label{sec:covs}
In this section, we discuss how  pre-randomization covariates can be incorporated into the principal stratification model. There are several ways to do this: covariates could be used to predict stratum membership, or predict response for the groups with positive response, or both. We focus on pre-randomization covariates as correlates of stratum membership, because such covariates improve the identification of the mixture model in \eqref{eq:treat_purch}. For example, \cite{ding2011} shows that a discrete covariate with $3$ levels is enough to identify the treatment effect for the $A$ group in a non-parametric principal stratification model. Under specific assumptions, the proof can be extended to show that the ATE is nonparametrically identified with slightly more levels. This is an important advantage, as the identification of normal mixture models (even for simple ones) has been shown to be potentially problematic \citep{feller2019} and any posterior uncertainty in the mixture model will propagate into estimates of the ATE. However each additional level effectively creates substrata, which may lower the number of observations in each substratum which, in turn, will decrease the precision of ATE the estimate. 

Specifically, we extend the model in equations \eqref{eq:treat_purch} -- \eqref{eq:ctrl_nopurch} to allow the principal strata probabilities to depend on covariates. If $x_i$ is a vector of observed pre-randomization covariates for customer $i$ including a leading column of ones for the intercept, then we specify the conditional principal strata probabilities as a multinomial logit model:
\begin{equation}
\begin{aligned}
\pi^A|x_i &= \frac{1}{1 + \exp(x_i \beta^I) + \exp(x_i \beta^N)}\\
\pi^I|x_i &= \frac{\exp(x_i \beta^I)}{1 + \exp(x_i \beta^I) + \exp(x_i \beta^N)}\\ 
\pi^N|x_i &= \frac{\exp(\beta^N x_i)}{1 + \exp(x_i \beta^I) + \exp( x_i \beta^N)}\\ 
\label{eq:mnl}
\end{aligned}
\end{equation}
\noindent where the coefficient vectors $\beta^I$ and $\beta^N$ are estimated from the data in the experiment. We use $\tau_c$ to refer to the ATE produced by fitting the model defined by equations \eqref{eq:treat_purch} -- \eqref{eq:ctrl_nopurch} and \eqref{eq:mnl}. In addition to potentially improving the precision of the ATE estimate it can also provide insights into which customers are more likely to be in the $N$ stratum, who are obviously poor targets for treatment, versus the $A$ and $I$ strata, which typically have positive treatment effects.  

The key to applying this model is identifying pre-randomization covariates that are associated with stratum membership. The literature on latent attrition suggests that recency, which is the number of periods that have passed since the customer's last purchase, is a good predictor of whether a customer is active \citep{FaderHardieLee2005}.\footnote{Recency is a sufficient statistic for some latent attrition models.} If the customer is no longer active, then they are, by definition, a part of the $N$ stratum in the experiment. Thus, we expect the customer's recency at the time of the experiment, $R_i$, to be associated with increased probability of belonging to the $N$ stratum. Many advertisers have data on prior purchases for the customers in the test \citep[e.g., ][]{LewisRao2015}, so recency is available for many retention campaigns like the ones in our application. 

Most CRM databases that track purchases for individual customers also track prior exposures to marketing. Similarly, in online advertising settings records of past exposures are also frequently available. Such data may or may not have come from a randomized experiment, but it can be used to create a (potentially biased) individual-level estimate of past responsiveness to similar marketing, $Q_i$. If this estimate of past responsiveness is correlated with future behavior, than it will be predictive of belonging to the ``Influenced" stratum. 

Specifically, in each pre-experiment period $t$ let $y_{it}$ be an indicator of whether the customer made a purchase and let $z_{it}$ be an indicator for whether the customer was treated. The prior treatment response, $Q_i$, is the difference between the average probability of purchasing in periods with no ad exposure and purchasing in periods with an ad exposure:
\begin{equation}
    Q_i = \frac{\sum_t y_{it} z_{it}}{\sum_t z_{it}}-\frac{\sum_t y_{it} (1-z_{it})}{\sum_t (1-z_{it})}
\end{equation}
If responsiveness is static and past exposure $z_{it}$ is exogenous, the responsiveness metric is an unbiased estimate of the customer's likelihood of being in stratum $I$. However, it is not necessary for advertising exposure to be exogenous; the treatment $z_{it}$ may be correlated with stratum membership.
Despite the potential endogenity, we show that if customer $i$ is more likely to be in the $A$ or $I$ strata than customer $j$, then customer $i$ is expected to have both higher $Q_i$ and lower $R_i$: 
\begin{prop}
\label{prop:ranking}
Assume each customer $i$'s stratum membership in each prior period $t$ is determined by an independent draw from a categorical distribution with latent parameters $(\pi_i^A, \pi_i^I, \pi_i^N)$ which remains constant before and during the experiment. If for two consumers $i$ and $j$, $1/2 >\pi_i^A>\pi_j^A$ and $1/2>\pi_i^I>\pi_j^I$, then:
\begin{itemize}
    \item The expected responsiveness of $i$ is higher than that of $j$: $\mathbb{E}[Q_i] > \mathbb{E}[Q_j]$. 
    \item The expected recency of $i$ is lower than that of $j$: $\mathbb{E}[R_i] < \mathbb{E}[R_j]$.
\end{itemize}
These results hold even if past marketing exposures $z_{it}$ are positively correlated with the potential outcomes of each customer $i$.
\end{prop}
\begin{proof}
See appendix.
\end{proof}
The major implication of this result is that we can rank consumers by their past recency and responsiveness and divide them into groups that will be predictive of their future stratum membership. Thus, $Q_i$ and $R_i$ are reasonable candidates to include as covariates to stratum membership in the principal stratification model. 

It is notable that $\pi_i^A$ and $\pi_i^I$ must be less than $1/2$ for this identification approach to be valid. The reason is that when both of these values are high and the targeting is precise, it may be the case that the expected recency and responsiveness metrics are lower for consumers with high values of $\pi_i^A$ and $\pi_i^I$. If ad exposure was exogenous for the prior periods, then ranking based on past behavior predicts future behavior for any values of $\pi_i^A$ and $\pi_i^I$.

This approach to pre-randomization covariates is different from including the covariates in a regression for the response. Such ``regression adjustments" do not always provide meaningful gains in the precision of the ATE \citep{jones2017} and, as we show in the application, they do not improve the variance of the ATE as well as principal stratification. In the principal stratification model, these covariates are used to predict stratum membership and are theoretically justified for that purpose. Including these covariates may increase or decrease the precision of the ATE relative to principal stratification without covariates. However, regardless of the effect on the precision of the ATE, covariates also bring additional insight into which customers are likely to belong to the $N$ stratum. 

\subsection{Application with covariates}

For the application, we had access to CRM data indicating whether each customer received a catalog or made a purchase in each month for 13 months prior to the experiment (see Figure \ref{fig:app1_timeline}). 
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{figures/Application1Timeline.pdf}
\caption{Data timeline for application}
\label{fig:app1_timeline}
\end{figure}
Using this data, we computed $Q_i$ and $R_i$ for each customer. 

Table \ref{tab:ps_w_covs} presents the parameter estimates of the principal stratification model. We see that the response means, $\mu^A_0$, $\mu^A_1$ and $\mu^I_1$, are largely the same as in the model without covariates. The response variance $\sigma$ is also similar. Both recency and responsiveness are associated with substantial changes in the strata probabilities as shown by the large and significant $\beta$ coefficients in Table \ref{tab:ps_w_covs}. 
\linespread{1.1}
\begin{table}[htbp]
\caption{Parameter estimates for principal stratification model with pre-randomization covariates} 
\label{tab:ps_w_covs}
\centering
\footnotesize
\begin{tabular}{l *{10}{S[table-format=2.3,
         input-decimal-markers={.},
         output-decimal-marker={.}]}}
  \hline
& \multicolumn{2}{c}{Expt 1} & \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} & \multicolumn{2}{c}{Expt 4} & \multicolumn{2}{c}{Expt 5}\\
& \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd} & \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd}& \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd}& \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd}& \multicolumn{1}{c}{mean} & \multicolumn{1}{c}{sd}\\
\hline
\multicolumn{11}{l}{Baseline strata probabilities (logit scale)}\\
$\beta^A_0$ & 0 & & 0 & & 0 & & 0 & & 0\\
$\beta^I_0$ & -3.737 & 0.280 & -3.548 & 0.271 & -3.706 & 0.608 & -3.561 & 0.698 & -3.470 & 0.286 \\ 
$\beta^N_0$ & 1.005 & 0.012 & 0.997 & 0.012 & 0.653 & 0.012 & 0.530 & 0.012 & 1.413 & 0.013 \\
\hline
\multicolumn{11}{l}{No recent purchase ($R_i > 5$)}\\
$\beta^I_1$ & 1.332 & 0.297 & 1.684 & 0.271 & 0.767 & 2.040 & 0.569 & 2.381 & 1.246 & 0.292 \\ 
$\beta^N_1$ & 1.617 & 0.021 & 1.608 & 0.022 & 1.431 & 0.040 & 1.338 & 0.042 & 1.577 & 0.025 \\ 
\hline
\multicolumn{11}{l}{Low prior treatment effect ($Q_i < 0$)}\\
$\beta^I_2$ & -0.102 & 0.319 & 0.056 & 0.230 & 0.002 & 0.810 & -0.236 & 0.678 & -0.028 & 0.289 \\ 
$\beta^N_2$ & 0.268 & 0.016 & 0.318 & 0.016 & 0.373 & 0.017 & 0.373 & 0.015 & 0.307 & 0.018 \\ 
\hline
$\mu^A_0$ & 4.575 & 0.010 & 4.616 & 0.010 & 4.658 & 0.009 & 4.547 & 0.009 & 4.648 & 0.012 \\ 
$\mu^A_1$ & 4.653 & 0.014 & 4.711 & 0.014 & 4.720 & 0.039 & 4.597 & 0.039 & 4.748 & 0.017 \\ 
$\mu^I_1$ & 3.268 & 0.152 & 3.530 & 0.132 & 4.088 & 0.878 & 4.158 & 0.884 & 3.336 & 0.163 \\ 
$\sigma$ & 1.088 & 0.006 & 1.100 & 0.006 & 1.088 & 0.008 & 1.059 & 0.006 & 1.098 & 0.007 \\ 
\hline
ATE $\widehat{\tau}_c$ & 0.031 & 0.006 & 0.045 & 0.007 & 0.056 & 0.007 & 0.059 & 0.009 & 0.029 & 0.005 \\ 
ATE (\$) & 2.559 & 0.534 & 3.485 & 0.561 & 3.746 & 0.891 & 3.576 & 0.613 & 2.584 & 0.504 \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}

Since the multinomial logit parameters ($\beta$) can be difficult to interpret, Table \ref{tab:conditonal_pis} shows the conditional estimates of the strata probabilities. As expected, customers who have not made a recent purchase are much more likely to be in the $N$ stratum and less likely to be in the $A$ stratum. Customers who are less responsive are less likely to be in the $A$ or $I$ strata, although $R_i$ is a stronger predictor than $Q_i$. In Appendix \ref{sec:app2_details}, we report coefficients from a second application with a different retailer. For that retailer, $Q_i$ is a more important predictor with $\beta$ coefficients similar to $R_i$. 
\linespread{1.1}
\begin{table}[htbp]
\caption{Estimated conditional principal strata sizes and ATEs for Experiment 2} 
\label{tab:conditonal_pis}
\centering
\begin{tabular}{lccccc}
\hline
& Recent Purch. & Recent Purch. & No Recent Purch. & No Recent Purch. \\
& Responsive & Less Responsive & Responsive & Less Responsive \\
& $R_i<6, Q_i>0$ & $R_i<6, Q_i<0$ &$R_i\ge6, Q_i>0$ & $R_i\ge6, Q_i<0$ \\
  \hline
$\pi^A$ & 0.267 & 0.210 & 0.068 & 0.051 \\ 
$\pi^I$ & 0.008 & 0.007 & 0.011 & 0.009 \\ 
$\pi^N$ & 0.725 & 0.783 & 0.921 & 0.941 \\ 
ATE ($\tau_c$)& 0.054 & 0.043 & 0.044 & 0.035 \\ 
\hline
\end{tabular}
\end{table}
\linespread{1.5}

Finally, Table \ref{tab:var_reduction_covs} compares the variance of the ATE between the principal stratification model with covariates and a regression adjustment model where main effects and interactions with treatment for $R_i$ and $Q_i$ are included in a linear model for log-sales. The principal stratification estimates have 32.6\% to 57.9\% lower posterior variance, with the exception of Experiment 3, where both methods have similar posterior variance. In fact, the regression adjustment estimates provide little improvement in posterior variance versus the difference-in-means model without covariates (see Table \ref{tab:var_reduction_1}). 
\begin{table}[ht]
\caption{Comparison of ATE estimates for models with covariates}
\label{tab:var_reduction_covs}
\centering
\scalebox{0.85}{
\begin{tabular}{lrrrrrrrrrr}
& \multicolumn{2}{c}{Expt 1} &  \multicolumn{2}{c}{Expt 2} & \multicolumn{2}{c}{Expt 3} &\multicolumn{2}{c}{Expt 4} &\multicolumn{2}{c}{Expt 5}\\
Model & mean & sd & mean & sd & mean & sd & mean & sd & mean & sd\\
\hline
Regression Adj. & 0.0056 & 0.0092 & 0.0262 & 0.0092 & 0.0262 & 0.0102 & 0.0266 & 0.0106 & 0.0121 & 0.0081 \\ 
Principal Strat ($\widehat{\tau}_{c}$) & 0.0313 & 0.0060 & 0.0450 & 0.0065 & 0.0506 & 0.0105 & 0.0593 & 0.0087 & 0.0291 & 0.0054 \\ 
\hline
StDev Reduction (\%) &  & 35.1 &  & 29.4 &  & -2.6 &  & 17.9 &  & 32.5 \\ 
Var Reduction (\%) &  & 57.9 &  & 50.2 &  & -5.3 &  & 32.6 &  & 54.4 \\ 
\hline
\end{tabular}}
\end{table}

\section{Conclusion}
The recent popularity of advertising experiments to estimate incrementality has exposed many challenges that marketers face when fielding and analyzing experiments. Because advertising effects are often small and because consumer response is noisy, precisely estimating these effects is hard even with large samples \citep{LewisRao2015, berman2018p, azevedo2019}. 

The principal stratification model allows a firm to decompose consumer responses into purchase incidence and purchase amount. This, in turn, can provide a more precise estimate of the ATE by separating out the people who will never buy regardless of treatment and therefore have zero treatment effect. The estimate will be more precise when the share of customers who do not purchase in the experiment is large, which is often the case in advertising experiments. While we have focused on advertising lift testing here, the principal stratification model would be useful for any type of marketing experiments where the response is customer-level sales. 


A reliable reduction in the variance of the ATE, as is shown in both applications, may be useful for firms in several ways. First, when making decisions about advertising investments and budget allocations, precise ROI measurements will allow firms to better allocate their advertising spending. Second, a smaller variance in the ATE translates to a smaller sample size required to detect a positive effect in experiments. When designing experiments, the required sample sample size is linear in the variance of the ATE, and hence a 30\% reduction in variance translates to a 30\% reduction in required sample size. Depending on the advertising medium, smaller required sample sizes result in cost savings because 1) the budget required to purchase media for the test is smaller, 2) there is a shorter amount of time required to achieve the required sample size (i.e., faster learning), or 3) a smaller control group can be used to decrease the opportunity cost of the test.

Beyond the benefit of lowering the variance of the ATE, our method also lets firms understand how consumers are influenced by advertising. When $\pi^I$ is large, the advertising is convincing people to buy, while when $\mu^A_0-\mu^A_1$ is large, the advertising is encouraging those who would have bought to buy more. Marketers can test different ad creatives to learn how those creative affect both types of customers. When the analysis is combined with covariates about a customer's past behavior, the information can also be used to predict a customer's response stratum, and the result can be used for targeting.

In the application, the variance of the estimated ATE is consistently decreased by about 30-60\%. We also provide a summary statistic based on simple averages that analysts can calculate to predict if principal stratification will be useful without complex modeling. Moreover, the consistency of the results across experiments and time periods means that if firms are engaged in repeated advertising (such as retailers who use catalogs throughout the year), they can use the estimates from past experiments to inform the sample sizes for future experiments.

Our method is not without limitations. First, principal stratification is only useful for experiments with a continuous response that has a large number of observed zeros. This structure is typical of customer-level sales and could be applied to time-on-site data where there are typically a large proportion of session with zero time-on-site (i.e., bounces). However, conversion experiments where the outcome is binary will not fit this setup. Second, we make distributional assumptions in the estimation procedure. A non-parametric approach similar to \cite{ding2011} can be applied, but will require using covariates and other assumptions. Third, the model assumes there are no customers who would purchase only when they are \emph{not} exposed the the advertising. This assumption improves the identification of the principal stratification model and is consistent with other models of advertising response, e.g., the ad-stock model. 

The two covariates we focused on (recency and responsiveness) are commonly available to marketers. To formally prove their usefulness, we made the assumption that stratum membership for each customer is drawn from a static distribution. This model can be extended to allow stratum membership to follow a hidden Markov model (HMM) which could be estimated from panel data. Such an approach would allow us to incorporate past behavior of customers into the analysis of an experiment in a more structured way. We also encourage the investigation of other covariates to predict stratum membership or mean response within-stratum, particularly those based on consumers' prior engagements with the brand. 

Although we focused on the static analysis of experiments that have already been fielded, the Bayesian implementation of the principal stratification model can be readily adapted to dynamic inference in experiments, which is becoming the norm for many A/B testing platforms (e.g., sequential testing, online Bayesian inference). This would provide insights about the size and response for each stratum as the experiment is running. We believe such applications hold tremendous potential for future work that will combine principal stratification with dynamic experimental designs.

