

\section*{Appendices}
\section{Algorithm}
\label{appendix:algorithm}
In this appendix, we provide a detailed algorithm for implementing \textit{PatchUp}{}. As with most regularization techniques, \textit{PatchUp}{} also has two modes (either inference or training). It also needs the combining type (either \textit{Soft PatchUp}{} or \textit{Hard PatchUp}{}), $\gamma$, and $block\_size$. This algorithm shows how \textit{PatchUp}{} generates a new hidden representation from $(g_{k}(x_{i}), y_{i})$ and $(g_{k}(x_{j}), y_{j})$. Lines 4 to 9 in the algorithm\ref{alg:patchup} are the binary mask creation process used in both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{}.

\import{tables/}{algo.tex}

Figure~\ref{fig:block_selection} briefly illustrates and summarizes the binary mask creation process in \textit{PatchUp}{}. Lines 11 to 25 correspond to the interpolation and combination of hidden representations in the mini-batch in \textit{PatchUp}{}. Figure~\ref{fig:mask_cmp} compares the masks generated by \textit{PatchUp}{} and \text{CutMix}{}. 

\begin{figure}[htbp]
  \centering
\begin{subfigure}{0.47\linewidth}
	\label{fig:mixer_mask}
	\includegraphics[width=1.\textwidth,height=55pt]{pics/blockmix_mask.png}
	\caption{Mask sampling in \textit{PatchUp}{}.}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\label{fig:cutmix_mask}
	\includegraphics[width=1.\textwidth]{pics/cutmix_mask.png}
	\caption{Mask sampling in \text{CutMix}{}.}
\end{subfigure}
\caption{Mask sampling in \textit{PatchUp}{} is applied in the hidden state, compared to \text{CutMix}{} which is applied in the input space. Red areas show the blocks that should be altered.}
\label{fig:mask_cmp}
\end{figure}

\section{Why random $k$?}
\label{appendix:random_k_selction}
\textit{PatchUp}{} applies block-level regularization at a randomly selected hidden representation layer $k$. The Information Bottleneck (IB) principle, introduced by \citet{DBLP:Tishby}, gives a formal intuition for selecting $k$ randomly. First, let us encapsulate the layers of the network into blocks where each block could contain more than one layer. Let $g_k$ be the $k$-th block of layers. In this case, sequential blocks share the information as a hidden representation to the next block of layers, sequentially. We can consider this case as a Markov chain of the block of layers as follows:
\begin{equation}
x \to g_{1}(x) \to  g_{2}(x) \to  g_{3}(x).
\end{equation}
In this scenario, the sequential communication between the intermediate hidden representations are considered to be an information bottleneck. Therefore, 
\begin{equation}
I(g_{3}(x);\;g_{2}(x)) < I(g_{2}(x);\;g_{1}(x)) < I(g_{1}(x);\;x),
\end{equation}
where $I(g_{k}(x);\;g_{k-1}(x))$ is the mutual information between the $k$-th and $(k-1)$-th layer.

If $g_{k=3}(x)$ has enough information to represent $x$, then applying regularization techniques in $g_{k=3}(x)$ will provide a better generalization to unseen data. However, most of the current state-of-the-art CNN models contain residual connections which break the Markov chain described above (since information can skip the $g_{k=3}$ layer). One solution to this challenge is to randomly select a residual block and apply regularization techniques like ManifoldMixup or \textit{PatchUp}{}. 

\begin{figure}[tbp]
\centering
\begin{minipage}{.51\textwidth}
  \centering
  \includegraphics[width=1.\linewidth, height=75pt]{pics/max_pool2d.png} 
\caption {\textit{PatchUp}{} mask creation process ($block\_size = 5$). The left matrix shows the process of feature selection from feature maps. By using a $max\_pool2d$ function, we can create blocks around selected features. The $max\_pool2d$ function uses $stride = (1,\;1)$, $kernel\_size = (block\_size,\,block\_size)$, and $padding = (\frac{block\_size}{2}, \frac{block\_size}{2})$. Red and blue points are $1$ and $0$ in the generated binary mask, respectively.}
  \label{fig:block_selection}
\end{minipage}%
\hfill
\begin{minipage}{.47\textwidth}
  \centering
  \includegraphics[width=1.\textwidth]{pics/delta_theta.png} 
\caption{The comparison of $\rho$ for flattened hidden representations of a mini-batch of samples at the second residual block (layer $k = 3$) of WideResNet-28-10 with corresponding regularization method.}
\label{fig:sim_comparison}
\end{minipage}
\end{figure}

\section{\textit{PatchUp}{} Interpolation Policy Effect}
\label{appendix:sim_comparison}
Assume that $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are flattened hidden representations of two examples produced at layer $k$. And, $\mathcal{H}$ is the flattened interpolated hidden representation of these two paired samples at layer $k$. First, we calculate the cosine distance of the pairs ($\mathcal{H}_{2}$, $\mathcal{H}_{1}$), ($\mathcal{H}_{1}$, $\mathcal{H}$), and ($\mathcal{H}_{2}$, $\mathcal{H}$). Reversing the cosine of these cosine similarities give the angular distance between each pair of vectors denoted as $\angle\theta$, $\angle\alpha_{1}$, and $\angle\alpha_{2}$, respectively. There is always a surface that contains $\mathcal{H}_{2}$ and $\mathcal{H}_{1}$ denoted as $\mathcal{S}$. Mathematically, we have:
\begin{equation}
\begin{array}{l}
\label{eq:lsim_study}
    \angle \alpha_{1} = \cos^{-1}(\frac{\mathbf{\mathcal{H}_{1}} \cdot \mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{1}}\|\|\mathbf{\mathcal{H}}\|}) \And
    \angle \alpha_{2} = \cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}} \cdot \mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}}\|}) \And
    \angle \theta =
    \cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}} \cdot \mathbf{\mathcal{H}_{1}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}_{1}}\|}),
\end{array}
\end{equation}
Let us define \(\angle\rho=(\angle\alpha_{1}+ \angle\alpha_{2})-\angle\theta\) and \(\angle\rho'= |\angle\alpha_{1}-\angle\alpha_{2}|-\angle\theta\).
According to the triangle inequality principle, either $\angle\rho$ or $\angle\rho'$ will be zero if, and only if, $\mathcal{H}\in\mathcal{S}$.
Figure~\ref{fig:sim_vectors} illustrates three possible scenarios for two paired flattened hidden representations and their flattened interpolated hidden representations. $\angle\rho$ and $\angle\rho'$ are zero for the left and right figures, respectively. We try to empirically show that $\mathcal{H}_{2}$, $\mathcal{H}_{1}$, and $\mathcal{H}$ always lie in the same surface $\mathcal{S}$ and $\mathcal{H}$ lies between $\mathcal{H}_{2}$ and $\mathcal{H}_{1}$ in ManifoldMixup. This means that $\angle\rho = 0$ for ManifoldMixup because of its linear interpolation policy. The middle figure in~\ref{fig:sim_vectors} is the case that both $\angle\rho$ and $\angle\rho'$ are not equal to zero. This figure shows that one possible situation is that flattened interpolated hidden representation does not lie in the surface $\mathcal{S}$. Our goal is to produce the interpolated hidden representation that lies in all possible places towards all dimensions in the hidden space. 
\begin{figure}[H]
\centering
  \includegraphics[width=.32\textwidth]{pics/vector_1.png} \hfill 
  \includegraphics[width=.32\textwidth]{pics/vector_2.png} \hfill 
  \includegraphics[width=.32\textwidth]{pics/vector_3.png}
    \caption{$\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are the flattened hidden representations. $\mathcal{H}$ is the flattened interpolated hidden representation that can be produced by either ManifoldMixup, \textit{Soft PatchUp}{} or \textit{Hard PatchUp}{}.}
    \label{fig:sim_vectors}
\end{figure}
As discussed in section~\ref{para:patchup_vs_manifold}, ManifoldMixup can provide interpolated hidden representation only in a limited space. However, \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} can produce a wide variety of interpolated hidden representations towards different dimensions. To support that, in WideResNet-28-10, for a mini-batch of 100 samples, we calculated the $\angle\rho$ for the flattened interpolated hidden representation produced by ManifoldMixup, \textit{Hard PatchUp}{}, and \textit{Soft PatchUp}{} at the second residual block (layer $k=3$) with the same interpolation policy ($\lambda=.4$, $\gamma=.5$, and $block\_size=7$) for both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} for all samples in the mini-batch. The swarmplot~\ref{fig:sim_comparison} shows all $\angle\rho$ for the mini-batch are equal to zero in ManifoldMixup, which empirically supports our hypothesis. However, \textit{PatchUp}{} produces more diverse interpolated hidden representations towards all dimensions in the hidden space. It is worth mentioning that few $\angle\rho$ that are equal to zero in \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} belong to the interpolated hidden representation that was constructed from the pairs with the same labels. 

\section{\textit{PatchUp}{} Experiment Setup and Hyper-parameter Tuning}
\label{appendix:hyperparameters}
This section describes the hyper-parameters of each model in table~\ref{table:models_other_hp} following the hyper-parameter setup from ManifoldMixup \cite{manifold} experiments in order to create a fair comparison. First, we performed hyper-parameter tuning for the \textit{PatchUp}{} to achieve the best validation performance. Then we ran all the experiments five times, reporting the mean and standard deviation of errors and negative log likelihoods for the selected models. We let models train for defined epochs and checkpoint the best model in terms of validation performance during the training. 
In our study, we used PreActResNet18, PreActResNet34, and WideResNet-28-10 models. Table~\ref{table:models_other_hp} shows the hyper-parameters used for training the models.

\begin{figure}[bt!]
\centering
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/manifold_alpha_ht.png}
	\caption{Impact of $\alpha$ in ManifoldMixup approach.}
	\label{fig:hyp_manifold}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/cutmix_k_ht.png}
	\caption{Impact of $cutmix\_prob$ in \text{CutMix}{} approach.}
	\label{fig:hyp_cutmix}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/softpatchup_gamma_ht.png} 
	\caption{Impact of $\gamma$, $block\_size$ with $patchup\_prob$ as 1.0 for \textit{Soft PatchUp}{}.}
	\label{fig:hyp_soft}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/harpatchup_gamma_ht.png} 
	\caption{Impact of $\gamma$, $block\_size$ with $patchup\_prob$ as 0.7 for \textit{Hard PatchUp}{}.}
	\label{fig:hyp_hard}
	\end{subfigure}
\caption{Impact of hyper-parameters $\gamma$, $block\_size$ and $patchup\_prob$ on error rates in the CIFAR-10 validation set for PreActResNet18. We repeated each job three times to collect the mean and the standard deviation of errors. Marked points are the mean of the error rate in the validation set. And, the shadow shows the bootstrapping of results for each hyper-parameter setting.
The lower numbers on the y-axes correspond to better performance.}
\label{hyptune}
\end{figure}
{\renewcommand{\arraystretch}{1.1}
\begin{table*}[t]
\centering
    \captionsetup[subtable]{position = top}
    \captionsetup[table]{position=top}
    \caption{The hyper-parameters used for each model to compare the effect of each regularization technique. The learning rate is denoted as \textit{lr}. And, \textit{lr} is multiplied at each learning rate schedule step by the step factor.}
    \label{table:models_other_hp}
    \begin{subtable}{.7\linewidth}
        \resizebox{\linewidth}{!}{
	\begin{tabular}[h]{ ll rrr }
		\toprule
		{Model} & {\textit{lr}} & {\textit{lr} steps} & {step factor} & {Epochs}\\
        \midrule
		{PreactResnet18}  & $0.1$ & $500${-}$1000${-}$1500$ & $0.1$ & $2000$\\
		\cmidrule(lr){2-5}
		{PreactResnet34} & $0.1$ &$500${-}$1000${-}$1500$ & $0.1$ & $2000$\\
		\cmidrule(lr){2-5}
		{WideResnet-28-10} & $0.1$ & $200${-}$300$ & $0.1$ & $400$\\
        \bottomrule
	\end{tabular}}
\end{subtable}
\end{table*}}

\textit{PatchUp}{} adds $patchup\_prob$, $\gamma$ and $block\_size$ as hyper-parameters. $patchup\_prob$ is the probability that the \textit{PatchUp}{} operation is performed for a given mini-batch, i.e if there are $N$ mini-batches and $patchup\_prob$ is $p$, \textit{PatchUp}{} is performed in $p$ fraction of $N$ mini-batches. $\gamma$ and $block\_size$ are described in section~\ref{patchup}. 
We tuned the \textit{PatchUp}{} hyper-parameter on CIFAR-10 with the PreActResNet18. To create a validation set, we split $10\%$ of training samples into a validation set. We set $\alpha$ to $2$ in \textit{PatchUp}{}. For \textit{Soft PatchUp}{}, we set $patchup\_prob$ to $1.0$ and applied \textit{PatchUp}{} to all mini-batches in training. Then, we did a grid search by varying $\gamma$ from $0.45$ to $0.9$ and $block\_size$ from $3$ to $9$. We found that $\gamma$ of $0.75$ and $block\_size$ of $7$ work best for \textit{Soft PatchUp}{} as shown in figure~\ref{fig:hyp_soft}. Similarly, for \textit{Hard PatchUp}{}, we set $patchup\_prob$ to $0.7$ and performed a grid search by varying $\gamma$ from $0.2$ to $0.6$ and $block\_size$ from $3$ to $9$. We found that $block\_size$ of $7$ and $\gamma$ of $0.5$ yield the best results for \textit{Hard PatchUp}{} as shown in figure~\ref{fig:hyp_hard}. Figure~\ref{fig:hyp_manifold} shows that ManifoldMixup with ($\alpha = 1.5$) achieves the best validation performance. For cutout, we used the same hyper-parameters proposed in \cite{cutout}, setting cutout to $16$ for CIFAR10, $8$ for CIFAR100, and $20$ for SVHN following~\cite{cutout}.
Figure~\ref{fig:hyp_cutmix} shows that \text{CutMix}{} achieves its best validation performance in PreActResNet18 in CIFAR-10 with $cutmix\_prob = 0.4$. Furthermore, DropBlock achieves its best validation performance on this task by setting the block size and $\gamma$ to $7$ and $0.9$, respectively~\cite{dropblock}.

\section{ Generalization on Deformed Images}
\label{appendix:affine_transformation}
\import{tables/}{affine_wrn.tex}
We created the deformed test sets from CIFAR100, as described in Section~\ref{exp:deformed_images_test}. Table~\ref{tab:affine_result} shows improved quality of representations learned by a WideResNet-28-10 model regularized by \textit{PatchUp}{} on CIFAR-100 deformed test sets. The significant improvements in generalization provided by \textit{PatchUp}{} in this experiment shows the high quality of representations learned with \textit{PatchUp}{}.

\section{Robustness to Adversarial Examples}
\label{appendix:adversarial-attack}
The adversarial attacks refer to small and unrecognizable perturbations on the input images that can mislead deep learning models \cite{goodfellow2014FGSM, goodfellow2016deep}. 
One approach to creating adversarial examples is using the Fast Gradient Sign Method (FGSM), also known as a white-box attack~\cite{goodfellow2014FGSM}. FGSM creates examples by adding small perturbations to the original examples. Once a regularized model is trained then FSGM creates adversarial examples as follows \cite{goodfellow2014FGSM}:
\begin{equation}
\begin{array}{l}
\label{eq:fgsm_perturbation}
    x^{\prime}= x + \epsilon \times \operatorname{sign}\left(\nabla_{\boldsymbol{x}} J(\boldsymbol{\theta}, \boldsymbol{x}, y)\right).
\end{array}
\end{equation}
where $x^{\prime}$ is an adversarial example, $x$ is the original example, $y$ is the ground truth label for $x$, and $J(\boldsymbol{\theta}, \boldsymbol{x}, y)$ is the loss of the model with parameters of $\theta$. $\epsilon$ controls the perturbation. 

Our experiments show the effectiveness of \textit{Soft PatchUp}{} against the attacks in most cases. However, \textit{Hard PatchUp}{} performed well against the FGSM attack only on PreActResNet34 for CIFAR-100. Figure~\ref{fig:appendix_adv_attack} shows the comparison of the state-of-the-art regularization techniques' effect on model robustness against the FGSM attack.

\begin{figure}[bt!]
\centering
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_p18.png} 
	\caption{Comparison on PreActResNet18 for CIFAR-10.}
	\label{fig:appendix_attack_c10_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_p18.png} 
	\caption{Comparison on PreActResNet18 for CIFAR-100.}
	\label{fig:appendix_attack_c100_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_p34.png} 
	\caption{Comparison on PreActResNet34 for CIFAR-10.}
	\label{fig:appendix_attack_c10_p34}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_p34.png} 
	\caption{Comparison on PreActResNet34 for CIFAR-100.}
	\label{fig:appendix_attack_c100_p34}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_wrn.png} 
	\caption{Comparison on WideResNet28-10 for CIFAR-10.}
	\label{fig:appendix_attack_c10_wrn}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_wrn.png}
	\caption{Comparison on WideResNet28-10 for CIFAR-100.}
	\label{fig:appendix_attack_c100_wrn}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_svhn_wrn.png}
	\caption{Comparison on WideResNet28-10 for SVHN.}
	\label{fig:appendix_attack_svhn_wrn}
\end{subfigure}	
\caption{Robustness to the FGSM attack, known as a white-box attack. We repeated each test for five trained models to report the mean and the standard deviation of the accuracy of each method against the FGSM attack. The higher values on the y-axes show the robustness of the model against the attack. And, $\epsilon$ is the magnitude that controls the perturbation. }
\label{fig:appendix_adv_attack}
\end{figure}

\section{Analysis of \textit{PatchUp}{}'s Effect on Activations}
\label{appendix:activation_comparison}
In our implementation WideResNet28-10 has a conv2d module followed by three residual blocks. Figure~\ref{fig:appendix_activation_cmp} illustrates the comparison of ManifoldMixup, cutout, CutMix, \textit{Soft PatchUp}{}, and \textit{Hard PatchUp}{}. Figure~\ref{fig:appendix_activation_conv1},~\ref{fig:appendix_activation_layer1}, and~\ref{fig:appendix_activation_layer2} show that \textit{PatchUp}{} produces more variety of features in layers that we apply \textit{PatchUp}{} on. 
\begin{figure}[tb!]
\centering
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_conv1.png}
	\caption{Comparison on the first convolution module.}
	\label{fig:appendix_activation_conv1}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer1.png} \caption{Comparison on 1st Residual Block.}
	\label{fig:appendix_activation_layer1}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer2.png} \caption{Comparison on 2nd Residual Block.}
	\label{fig:appendix_activation_layer2}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer3.png}
	\caption{Comparison on 3rd Residual Block.}
	\label{fig:appendix_activation_layer3}
\end{subfigure}
\hfill
\caption{The effect of the state-of-the-art regularization techniques on activations in WideResNet28-10 for CIFAR100 test set. Each curve is the magnitude of feature activations, sorted by descending value, and averaged over all test samples for each method. The higher magnitude shows a wider variety of the produced features by the model at each block.}
\label{fig:appendix_activation_cmp}
\end{figure}
\section{Conclusion}
\label{conclusion}
We presented \textit{PatchUp}{}, a simple and efficient regularizer scheme for CNNs that alleviates some of the drawbacks of the previous mixing-based regularizers. Our experimental results show that with the proposed approach, \textit{PatchUp}{}, we can achieve state-of-the-art results on image classification tasks across different architectures and datasets. Similar to previous mixing based approaches, our approach also has the advantage of avoiding any added computational overhead. The strong test accuracy achieved by \textit{PatchUp}{}, with no additional computational overhead, makes it particularly appealing for practical applications. 



\section*{Acknowledgments}

We would like to acknowledge Compute Canada and Calcul Quebec for providing computing resources used in this work. 
The authors would also like to thank Damien Scieur, Hannah Alsdurf, Alexia Jolicoeur-Martineau, and Yassine Yaakoubi for reviewing the manuscript.
SC is supported by a Canada CIFAR AI Chair and an NSERC Discovery Grant.
\section{Relation to Other Methods}
\label{discussion}
\begin{figure}[htbp]
\centering
\begin{minipage}{.49\textwidth}
  \centering
  \includegraphics[width=.47\linewidth]{pics/manifoldmixup.png} \ \
    \includegraphics[width=.47\linewidth]{pics/blockmix.png}
    \caption{Left: ManifoldMixup interpolated samples for any combination of the three blue hidden states selected only from along orange line. Right: \textit{PatchUp}{} can produce interpolated hidden representations for these three hidden states in almost all possible places in all dimensions except the samples which lie directly on the orange lines.}
    \label{fig:diagram_mix_3d}
\end{minipage}%
\hfill
\begin{minipage}{.49\textwidth}
    \centering
\includegraphics[width=.481\linewidth]{pics/cat_2.png} \hfill \ \
  \includegraphics[width=.481\linewidth]{pics/dog_2.png}
    \caption{The two possible block selections from \text{CutMix}{} for two samples (cat and dog) with a large background. Swapping a similar part of the background or an essential element correlated to the label in the selected images can have a negative effect on the \text{CutMix}{} learning objective.}
    \label{fig:cat_dog}
\end{minipage}%
\end{figure}
\paragraph{\textit{PatchUp}{} Vs. ManifoldMixup:}\label{para:patchup_vs_manifold}Both \textit{PatchUp}{} and ManifoldMixup try to improve the generalization of a model by combining the latent representations of a pair of examples. ManifoldMixup combines two hidden representations by using the mixing operation defined in Equation~\ref{equ:mix_lam} which produces a new latent representation in a linear way for a pair of two hidden representations. \textit{PatchUp}{} uses a more complex approach to find a combination of two hidden representations, ensuring that a more diverse subspace of the hidden space gets explored. To understand the behaviour and the limitation that exist in the ManifoldMixup, assume that we have a 3D hidden space representation as illustrated in figure~\ref{fig:diagram_mix_3d}. Figure~\ref{fig:diagram_mix_3d} presents the possible combinations of hidden representations explored via ManifoldMixup and \textit{PatchUp}{}. Blue dots represent real hidden representation samples. ManifoldMixup can produce new samples that lie directly on the orange lines which connect the blue point pairs due to its linear interpolation strategy. On the other hand, \textit{PatchUp}{} can select various points in all dimensions, and can also select points extremely close to the orange lines. The proximity to the orange lines depends on the selected pairs and $\lambda$ sampled from the beta distribution. Appendix-\ref{appendix:sim_comparison} provides the mathematical and experimental justifications to show the comparative advantage of \textit{PatchUp}{} over ManifoldMixup.
\paragraph{\textit{PatchUp}{} Vs. \text{CutMix}{}:}The \text{CutMix}{} strategy is to cut and fill some parts of the selected pairs instead of using interpolation for creating a new sample in the input space. Therefore, the \text{CutMix}{} method has less potential for a manifold intrusion problem, however, \text{CutMix}{} may still suffer from a manifold intrusion problem. Figure~\ref{fig:cat_dog} shows two samples with small portions that correspond to their labels. \text{CutMix}{} cuts and fills the rectangular parts of the selected image randomly. In this example, if only the parts within the yellow bounding boxes are swapped, then the label does not change. However, if the parts within the white bounding boxes are swapped, then the entire label is swapped. In both scenarios, \text{CutMix}{} only learns the interpolated target based on the fraction of the images that is swapped. In contrast, these scenarios cannot occur in \textit{PatchUp}{} since it works in the hidden representation space. Another difference between \text{CutMix}{} and \textit{PatchUp}{} is how the masks are created. \textit{PatchUp}{} can create arbitrarily shaped masks while \text{CutMix}{} masks can only be rectangular. Figure~\ref{fig:mask_cmp} shows an example of \text{CutMix}{} Mask and \textit{PatchUp}{} mask in input space and hidden representation space, respectively.

Feature-CutMix applies \text{CutMix}{} in the latent space. According to \citet{cutmix}, Feature-CutMix is not as effective as \text{CutMix}{}. Both the learning objective of \textit{PatchUp}{}, as well as the binary mask selection are different from Feature-CutMix.   
\section{Experiments}
\label{experiments}
To evaluate the generalization improvements that \textit{PatchUp}{} can provide with either \textit{Hard} or \textit{Soft PatchUp}{}, we applied \textit{PatchUp}{} to image classification tasks on CIFAR-10, CIFAR-100~\cite{dataset:cifar}, and SVHN~\cite{netzer2011reading} datasets with PreActResNet18, PreActResNet34, and WideResNet-28-10 models\footnote{The code to reproduce all the results is available at \href{https://github.com/chandar-lab/PatchUp}{\textcolor{WildStrawberry}{https://github.com/chandar-lab/PatchUp}}.}. We used the same set of base hyper-parameters for all the models to be able to compare and evaluate the generalization improvements due to different regularization methods in a fair way. Appendix-\ref{appendix:hyperparameters} explains the experiment setup and the hyper-parameter tuning processes in detail. \textit{PatchUp}{} adds $patchup\_prob$, $\gamma$ and $block\_size$ to the set of hyper-parameters. $patchup\_prob$ is the probability that \textit{PatchUp}{} is performed for a given mini-batch. We set $\alpha$ to $2$ in \textit{PatchUp}{}. And, based on the hyper-parameter tuning described in Appendix-\ref{appendix:hyperparameters}, \textit{Hard PatchUp}{} yields the best performance with $patchup\_prob$, $\gamma$, and $block\_size$ as $0.7$, $0.5$, and $7$, respectively.
\textit{Soft PatchUp}{} achieves the best performance with $patchup\_prob$, $\gamma$, and $block\_size$ as $1.0$, $0.75$, and $7$, respectively.
\import{tables/}{cifar.tex}

\import{tables/}{svhn.tex}
\subsection{Generalization on Image Classification}
Table~\ref{tab:tbl-cifar} shows the comparison of the generalization performance of \textit{PatchUp}{} with five other state-of-the-art regularization techniques on the CIFAR-10 and CIFAR-100 datasets. Our experiments show that \textit{Hard PatchUp}{} leads to a lower test error for all the models on both CIFAR-10 and CIFAR-100. \textit{Soft PatchUp}{} has the second-best result for this task. From Table~\ref{tab:tbl-cifar}, we see that \textit{PatchUp}{} provides a significant improvement over previous state-of-the-art regularization techniques across different architectures. We observed that \textit{Soft PatchUp}{} has the second-best performance across all the architectures indicating the potential that \textit{Soft PatchUp}{} is beneficial in fine-grained classification. Table~\ref{tab:tbl-svhn} shows that \textit{Hard PatchUp}{} achieves the best top-1 error across the different models. Our experiments show that ManifoldMixup has the second-best performance for PreActResNet18 on SVHN. \textit{Soft PatchUp}{} also achieves the second-best performance for WideResNet-28-10 on SVHN. It is also worth noting that \textit{Soft PatchUp}{} performs reasonably well and is comparable to ManifoldMixup for PreActResNet34 on SVHN.
\import{tables/}{affine_p34.tex}

\subsection{Generalization on Deformed Images}
\label{exp:deformed_images_test}
Regularization methods aim to improve the generalization of a model to unseen data. Applying affine transformations on the test set can provide novel deformed data that can be used to evaluate and compare the minimality and sufficiency of the representations learned by models with state-of-the-art regularization techniques \cite{manifold}. We trained PreActResNet34 and WideResNet-28-10 on the CIFAR100 dataset. And then, we created deformed test sets from CIFAR100 by applying random rotations, random shearings, and different rescalings. Table~\ref{tab:affine_p34_result} shows that \textit{PatchUp}{} provides the best performance on affine transformed test sets and better generalization in PreActResNet34. Table~\ref{tab:affine_result} in Appendix-\ref{appendix:affine_transformation} illustrates that the quality of representations is improved by \textit{PatchUp}{} and it also shows better generalization in deformed test sets on WideResNet-28-10. Generalization is significantly improved by \textit{PatchUp}{}, as are the quality of representations learned by \textit{PatchUp}{} as demonstrated by this experiment.
\begin{figure}[tb]
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=.95\linewidth]{pics/attack_c100_p18.png} 
	\caption{Comparison on PreActResNet18 for CIFAR-100.}
	\label{fig:attack_c100_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=.95\linewidth]{pics/attack_svhn_wrn.png} 
	\caption{Comparison on WideResNet28-10 for SVHN.}
	\label{fig:attack_svhn_wrn}
	\end{subfigure}
\caption{Robustness to FGSM attack. Plots are based on the mean and standard deviation of the accuracy of five trained models for each method against FGSM attack. The x-axis represents $\epsilon$ which is the magnitude that controls the perturbation.}
\label{fig:adv_attack}
\end{figure}
\begin{figure}[tb]
\centering
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/act_cmp_layer1.png}
	\caption{Comparison on 1st Residual Block.}
	\label{fig:activation_layer1}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/act_cmp_layer2.png} 
	\caption{Comparison on 2nd Residual Block.}
	\label{fig:activation_layer2}
	\end{subfigure}
\caption{The effect of the state-of-the-art regularization techniques on activations in WideResNet28-10 for CIFAR100 test set on the first and second Residual Blocks. Each curve is the magnitude of the feature activations, sorted in descending value, and averaged over all test samples for each method. The higher magnitude indicates a wider variety of the produced features by the model at each block.}
\label{fig:activation_cmp}
\end{figure}
\subsection{Robustness to Adversarial Examples}
Since neural networks are trained based on Empirical Risk Minimization (ERM), slight changes in the data distribution have a significant effect on the model performance~\cite{szegedy_Zaremba, mixup}. Such unseen data used to confuse the models are known as adversarial examples. Certain data-dependent regularization techniques can alleviate such fragility to adversarial examples by training the models with interpolated data. Therefore, the robustness of a regularized model to adversarial examples can be considered as a criterion for comparison~\cite{mixup, manifold, cutmix}. To evaluate the robustness of \textit{PatchUp}{} against adversarial attacks, we compared the performance of PreActResNet18, PreActResNet34, and WideResNet-28-10 on CIFAR-10 and CIFAR-100 with adversarial examples created by the FGSM attack described in~\cite{goodfellow2014FGSM}. We compared the performance of WideResNet28-10 for SVHN against the FGSM attack. Figure~\ref{fig:adv_attack} shows the robustness of \textit{PatchUp}{} against FGSM attack. Figure~\ref{fig:appendix_adv_attack} in the appendix shows further comparisons. Based on the results, we can see that \textit{Soft PatchUp}{} is more robust to adversarial attacks when compared to other regularization methods. \textit{Hard PatchUp}{} and ManifoldMixup achieve the second-best performance in most experiments. While \textit{Hard PatchUp}{} achieves better performance in terms of classification accuracy, \textit{Soft PatchUp}{} seems to trade-off a slight loss of accuracy in order to achieve more robustness.

\subsection{Effect on Activations}
To study the effect of the state-of-the-art regularization techniques on the activations in the residual blocks, we compared the mean magnitude of feature activations in the residual blocks following~\cite{cutout} in WideResNet28-10 for the test set in CIFAR-100. We first train the models with regularization techniques and then calculate the magnitudes of activations in the validation set. The higher mean magnitude of features shows that the models tried to produce a wider variety of features in the residual blocks~\cite{cutout}. Our WideResNet28-10 has a conv2d module followed by three residual blocks. For this ablation study, we selected $k$ randomly such that $k\in \{0, 1, 2, 3\}$. Therefore, we apply the ManifoldMixup and \textit{PatchUp}{} in either input space, first conv2d, first residual block, or second residual block. Figure~\ref{fig:activation_cmp} illustrates the comparison of ManifoldMixup, cutout, CutMix, \textit{Soft PatchUp}{}, and \textit{Hard PatchUp}{}.
Figures~\ref{fig:activation_layer1} and~\ref{fig:activation_layer2} illustrate that \textit{PatchUp}{} produces more diverse features in the layers where we apply \textit{PatchUp}{}. In Appendix-\ref{appendix:activation_comparison}, figure~\ref{fig:appendix_activation_cmp} shows this ablation study results in first conv2d, first residual block, second residual block and third residual block. Since we are not applying the \textit{PatchUp}{} in the third residual block, the mean magnitude of the feature activations are below, but very close to, cutout and \text{CutMix}{}. This experiment also shows that producing a wide variety of features can be an advantage for a model. However, according to our experiments, a larger magnitude of activations does not always mean better performance. Figure~\ref{fig:activation_cmp} shows that for ManifoldMixup, the mean magnitude of the feature activations is less than other approaches. But, it performs better than cutout and \text{CutMix}{} in image classification, affine transformations, and FGSM attacks. 

\subsection{Significance of loss terms}
\label{appendix:patchup_loss}
\textit{PatchUp}{} uses the loss that is introduced in Equation~\ref{eq:swap_loss}. We can paraphrase the \textit{PatchUp}{} learning objective for this ablation study as follow:
\begin{equation}
\begin{aligned}
\label{eq:swap_losss}
    L(f) = &\expectation_{(x_i,\;y_i) \sim P}\,
    \expectation_{(x_j,\;y_j) \sim P}\,
    \expectation_{\lambda \sim \text{Beta}(\alpha,\;\alpha)}
    \expectation_{k \sim \mathcal{S}} ~~(L_1 + L_2) ,
\end{aligned}
\end{equation}
where
\begin{align}
L_{1} &= \text{Mix}_{p_{u}} [ \ell(f_{k}(\phi_k), y_i), \ell(f_{k}(\phi_k), Y)] ,\\
L_{2} &= \ell(f_{k}(\phi_k), W(y_i, y_j)) ,
\end{align}

This section is an ablation study to show the effect of $L_{1}$ and $L_{2}$ in \textit{PatchUp}{} loss. Table~\ref{tab:loss_study} shows the error rate on the validation set for WideResNet-28-10 on CIFAR-100. This study shows that the summation of the $L_{1}$ and $L_{2}$ reduces error rate by $.1\%$ in both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{}.

{\renewcommand{\arraystretch}{1.1}
\begin{table*}[hbt!]
\centering
    \captionsetup[subtable]{position = top}
    \captionsetup[table]{position=top}
    \caption{The error rate on the validation set on CIFAR-100 for WideResNet-28-10 with \textit{Hard PatchUp}{} and \textit{Soft PatchUp}{}. The result is the mean and standard deviation of the experiment for five runs. A smaller number indicates better performance.}
    \label{tab:loss_study}
    \begin{subtable}{.9\linewidth}
        \resizebox{\linewidth}{!}{
        \begin{tabular}{llll}
        \toprule
        Simple WideResNet-28-10& & & Error Rate:  \(23.256 \pm 0.586\) \\
        \toprule
        {}     & Error rates with $L1$ & Error rates with $L2$ & Error rates with $L (f)$ \\ \cmidrule(lr){2-4}
        \textit{Soft PatchUp} & \(16.856 \pm 0.666\)   & \(16.865 \pm 0.339\)   &  \(\mathbf{16.75 \pm 0.291}\)\\
        \cmidrule(lr){2-4}
        \textit{Hard PatchUp} & \(16.135 \pm 0.229\)   & \(16.79 \pm 0.457\)   &  \(\mathbf{16.02 \pm 0.358}\) \\
       
        \bottomrule
        \end{tabular}}
    \end{subtable}
\end{table*}
}

\section{Introduction}
\label{introduction}
Deep Learning (DL), particularly deep Convolutional Neural Networks (CNNs) have achieved exceptional performance in many machine learning tasks, including object recognition~\cite{krizhevsky}, image classification~\cite{krizhevsky,NIPS2015_Ren,Kaiming}, speech recognition~\cite{hinton} and natural language understanding \cite{seq2seq, transformers}. However, in a very deep and wide network, the network has a tendency to memorize the samples, which yields poor generalization for data outside of the training data distribution~\cite{arpit2017closer, goodfellow2016deep}. To address this issue, noisy computation is often employed during the training, making the model more robust against invariant samples and thus improving the generalization of the model~\cite{information_dropout}. This idea is exploited in several state-of-the-art regularization techniques. 

Such noisy computation based regularization techniques can be categorized into data-dependent and data-independent techniques~\cite{DBLP:Hongyu}. Earlier work in this area has been more focused on the data-independent techniques such as Dropout~\cite{article:dropout}, SpatialDropout~\cite{DBLP:spatial_dropout}, and DropBlock~\cite{dropblock}. Dropout performs well on fully connected layers~\cite{JMLR:v15:srivastava14a}. However, it is less effective on convolutional layers~\cite{jonathan}. One of the reasons for the lack of success of dropout on CNN layers is perhaps that the activation units in the convolutional layers are correlated, thus despite dropping some of the activation units, information can still flow through these layers. SpatialDropout~\cite{jonathan} addresses this issue by dropping the entire feature map from a convolutional layer. DropBlock~\cite{dropblock} further improves SpatialDropout by dropping random continuous feature blocks from feature maps instead of dropping the entire feature map in the convolutional layers.

Recent works show that data-dependent regularizers can achieve better generalization for CNN models. Mixup~\cite{mixup}, one such data-dependent regularizer, synthesizes additional training examples by interpolating random pairs of inputs $x_i$, $x_j$ and their corresponding labels $y_i$, $y_j$ as: 
\begin{equation}
\begin{array}{ll}{\tilde{x}=\lambda x_{i}+(1-\lambda) x_{j}} \text{\quad and \quad} {\tilde{y}=\lambda y_{i}+(1-\lambda) y_{j}} \end{array}
\end{equation}
where $\lambda \in [0,1]$ is sampled from a Beta distribution such that \(\lambda \sim Beta(\alpha, \alpha)\) and ($\tilde{x},\;\tilde{y}$) is the new example. By using these types of synthetic samples, Mixup encourages the model to behave linearly in-between the training samples.

The mixing coefficient $\lambda$ in Mixup is sampled from a prior distribution. This may lead to the \textit{manifold intrusion problem}~\cite{DBLP:Hongyu}: the mixed synthetic example may \textit{collide} (i.e. have the same value in the input space) with other examples in the training data, essentially leading to two training samples which have the same inputs but different targets. To overcome the manifold intrusion problem, \citet{mai2019metamixup} used a meta-learning approach to learn  \(\lambda\) with a lower possibility of causing such collisions. However, this meta-learning approach adds significant computation complexity. ManifoldMixup~\cite{manifold} attempts to avoid the manifold intrusion problem by interpolating the hidden states (instead of input states) of a randomly chosen layer at every training update.

Different from the interpolation based regularizers discussed above, cutout~\cite{cutout} drops the contiguous regions from the image in the input space. This kind of noise encourages the network to learn the full context of the images instead of overfitting to the small set of visual features. 
CutMix~\cite{cutmix} is another data-dependent regularization technique that cuts and fills rectangular shape parts from two randomly selected pairs in a mini-batch instead of interpolating two selected pairs completely. Applying \text{CutMix}{} at the input space improves the generalization of the CNN model by spreading the focus of the model across all places in the input instead of just a small region or a small set of intermediate activations. \text{CutMix}{} also improves the generalization performance of a very deep and wide CNN model such as PyramidNet. According to the \text{CutMix}{} paper, applying \text{CutMix}{} at the latent space, Feature \text{CutMix}{}, is not as effective as applying \text{CutMix}{} in the input space~\cite{cutmix}.    

In this work, we propose \textit{PatchUp}{} which is a regularization technique that operates in the hidden space by masking out contiguous blocks of the feature map of a random pair of samples, and then either mixes (\textit{Soft PatchUp}{}) or swaps (\textit{Hard PatchUp}{}) these selected contiguous blocks.
Our experiments verify that \textit{Hard PatchUp}{} achieves a better generalization performance compared to other state-of-the-art regularization techniques for CNNs such as Mixup, cutout, CutMix and ManifoldMixup on CIFAR-10, CIFAR-100, and SVHN datasets. \textit{Soft PatchUp}{} achieves the second-best performance on CIFAR-10, CIFAR-100 with PreactResnet18, PreactResnet34, and WideResnet-28-10 models while achieving comparable results to ManifoldMixup on SVHN with PreactResnet18 and PreactResnet34. Furthermore, \textit{PatchUp}{} provides significant improvements in the generalization on deformed images and better robustness against Fast Gradient Sign Method (FGSM) adversarial attack.  









\section{PatchUp}
\label{patchup}
\textit{PatchUp}{} is a hidden state block-level regularization technique that can be used after any convolutional layer in CNN models. Given a deep neural network $f(x)$ where $x$ is the input, let $g_k$ be the $k$-th convolutional layer. The network $f(x)$ can be represented as $f(x) = f_k(g_k(x))$ where $g_k$ is the mapping from the input data to the hidden representation at layer $k$ and $f_k$ is the mapping from the hidden representation at layer $k$ to the output \cite{manifold}. In every training step, \textit{PatchUp}{} applies block-level regularization at a randomly selected convolutional layer $k$ from a set of intermediate convolutional layers. Appendix-\ref{appendix:random_k_selction} gives a formal intuition for selecting $k$ randomly.
\subsection{Binary Mask Creation}
\label{binary_mask}
\begin{figure}[tbp]
\centering
	\includegraphics[width=\textwidth]{pics/patchup_process.png} 
\caption{\textit{PatchUp}{} process for two hidden representations associated with two samples randomly selected in the mini-batch ($a,\;b)$. $X_{1} = g_{k}^{(i)}(a)$ and $X_{2} = g_{k}^{(i)}(b)$ where $i$ is the feature map index. Right top shows \textit{Hard PatchUp}{} output and the right bottom shows the interpolated samples with \textit{Soft PatchUp}{}. The yellow continuous blocks represent the interpolated selected blocks.}
\label{fig:patchup_process}
\end{figure}
Once a convolutional layer $k$ is chosen, the next step is to create a binary mask $M$ (of the same size as the feature map in layer $k$) that will be used to \textit{PatchUp}{} a pair of examples in the space of $g_k(x)$. The mask creation process is similar to that of DropBlock~\cite{dropblock}. The idea is to select contiguous blocks of features from the feature map that will be either mixed or swapped with the same features in another example. To do so, we first select a set of features that can be altered (mixed or swapped). This is done by using the hyper-parameter $\gamma$ which decides the probability of altering a feature. When we alter a feature, we also alter a square block of features centered around that feature which is controlled by the side length of this square block, $block\_size$. Hence, the altering probabilities are readjusted using the following formula \cite{dropblock}:
\begin{equation}
    \label{equ:gamma}
    \gamma_{adj} = \frac{\gamma \times \textit{(feature map's area)}}{\textit{(block's area)} \times \textit{(valid region to build block)}}.
\end{equation}
where the area of the feature map and block are the $feat\_size^2$ and $block\_size^2$, respectively, and the valid region to build the block is $(feat\_size - block\_size + 1)^2$. 

For each feature in the feature map, we sample from $Bernoulli(\gamma_{adj})$. If the result of this sampling for feature $f_{ij}$ is 0, then $M_{ij} = 1$. If the result of this sampling for $f_{ij}$ is 1, then the entire square region in the mask with the center $M_{ij}$ and the width and height of the square of $block\_size$ is set to 0. Note that these feature blocks to be altered can overlap which will result in more complex block structures than just squares. The block structures created are called patches. Figure-\ref{fig:patchup_process} illustrates an example mask used by \textit{PatchUp}{}. The mask $M$ has 1 for features outside the patches (which are not altered) and 0 for features inside the patches (which are altered).
\subsection{\textit{PatchUp}{} Operation}

Once the mask is created, we can use the mask to select patches from the feature maps and either swap these patches (\textit{Hard PatchUp}{}) or mix them (\textit{Soft PatchUp}{}).

Consider two samples $x_i$ and $x_j$. The \textit{Hard PatchUp}{} operation at layer $k$ is defined as follows:
\begin{equation}
       \mathbf{\phi_{hard}}(g_k(x_i), g_k(x_j)) = \mathbf{M} \odot g_k(x_i) + (\mathbf{1}- \mathbf{M}) \odot g_k(x_j), 
\end{equation}
where $\odot$ is known as the element-wise multiplication operation and $\mathbf{M}$ is the binary mask described in section ~\ref{binary_mask}. 

To define \textit{Soft PatchUp}{} operation, we first define the mixing operation for any two vectors $a$ and $b$ as follows:
\begin{equation}
\label{equ:mix_lam}
\text{Mix}_\lambda(a,\;b) = \lambda \cdot a + (1 - \lambda) \cdot b,
\end{equation}
where $\lambda \in[0,1]$ is the mixing coefficient. Thus, the \textit{Soft PatchUp}{} operation at layer $k$ is defined as follows:
\begin{equation}
    \mathbf{\phi_{soft}}(g_k(x_i), g_k(x_j)) = \mathbf{M} \odot g_k(x_i) + \text{Mix}_\lambda[((\mathbf{1}- \mathbf{M}) \odot g_k(x_i)),\;((\mathbf{1}- \mathbf{M}) \odot g_k(x_j))].    
\end{equation}
where \(\lambda\) in the range of $[0,\;1]$ is sampled from a Beta distribution such that \(\lambda \sim Beta(\alpha, \alpha)\). $\alpha$ controls the shape of the Beta distribution. Consequently, it controls the strength of interpolation \cite{mixup}. Both \textit{PatchUp}{} operations are illustrated in Figure~\ref{fig:patchup_process}. 
\subsection{Learning Objective}

After applying the \textit{PatchUp}{} operation, the CNN model continues the forward pass from layer $k$ to the last layer in the model. The output of the model is used for the learning objective, including the loss minimization process and updating the model parameters accordingly. 

Again, consider the example pairs $(x_i, y_i)$ and $(x_j, y_j)$. Let $\phi_k = \phi(g_k(x_i), g_k(x_j))$ be the output of \textit{PatchUp}{} after the $k$-th layer. Mathematically, the CNN with \textit{PatchUp}{} minimizes the following loss function:
\begin{equation}
\begin{aligned}
\label{eq:swap_loss}
    L(f) = &\expectation_{(x_i,\;y_i) \sim P}\,
    \expectation_{(x_j,\;y_j) \sim P}\,
    \expectation_{\lambda \sim \text{Beta}(\alpha,\;\alpha)}
    \expectation_{k \sim \mathcal{S}}\\
    &~~~~\text{Mix}_{p_{u}} [ \ell(f_{k}(\phi_k),\;y_i) \; , \ell(f_{k}(\phi_k),\;Y)]\; + 
    \ell(f_{k}(\phi_k),\; W(y_i,\;y_j)).
\end{aligned}
\end{equation}
where $p_{u}$ is the fraction of the unchanged features from feature maps in $g_{k}(x_i)$ and $\mathcal{S}$ is the set of layers where \textit{PatchUp}{} is applied randomly. $\phi$ is $\phi_{hard}$ for \textit{Hard PatchUp}{} and $\phi_{soft}$ for \textit{Soft PatchUp}{}.

$Y$ is the target corresponding to the changed features. In the case of \textit{Hard PatchUp}{}, $Y = y_j$ and in the case of \textit{Soft PatchUp}{}, $Y = \textit{Mix}_{\lambda}(y_i, y_j)$. $W(y_i, y_j)$ calculates the re-weighted target according to the interpolation policy for $y_i$ and $y_j$. $W$ for \textit{Hard PatchUp}{} and \textit{Soft PatchUp}{} is defined as follows:
\begin{align}
W_{hard}(y_i, y_j) &= \text{Mix}_{p_{u}}(y_i, y_j).    \\
W_{soft}(y_i, y_j) &= \text{Mix}_{p_{u}}(y_i,\;\text{Mix}_{\lambda}(y_i, y_j)).
\label{eq:target_reweighted}
\end{align}

The \textit{PatchUp}{} loss function has two terms where the first term is inspired from the CutMix loss function and the second term is inspired from the MixUp loss function.
\subsection{\textit{PatchUp}{} in Input Space}
When $k = 0$, \textit{PatchUp}{} only gets applied to the input space. To apply \textit{PatchUp}{} to the input space, only the \textit{Hard PatchUp}{} operation is used since swapping in the input space provides better generalization compared to mixing~\cite{cutmix}. Only one random rectangular patch is selected in the input space (similar to CutMix) because the \textit{PatchUp}{} binary mask is too strong for the input space, which has only three channels, compared to hidden layers in which each layer has numerous channels.







