\section{Introduction}
Deep neural networks have shown tremendous success in image classification \cite{imagenet,clf}, object detection \citep{fasterrcnn,yolov3}, and semantic segmentation \cite{deeplab,semanticseg}. These
networks are data hungry with millions of parameters that make them prone to overfitting \cite{overfitting2,overfitting3,overfitting}.
To enhance model generalization, many approaches
have been suggested, for example, regularization \cite{regularization},
dropout \cite{dropout}, and data augmentations \cite{surveyimaug} including image rotation, random cropping, jittering and etc.
In addition to these techniques, a relatively new data
augmentation strategy that mixes and deletes different image regions has recently gained popularity. The
augmented images generated by this approach are present in the vicinity of original data distribution.
The model trained with this new data generalizes
well against data variations, avoids overfitting, and
achieves robustness against data corruptions. Overall, the model learns to focus on the complete structure of the object under consideration resulting in
improved generalization. \\
Mainly, image mixing and deleting are divided into
three categories, cut and delete, cut and mix, and
mixup. The taxonomy of image augmentation can be
seen in Figure \ref{flow_diagram}. Cut and delete \cite{cutout,hidenseek,gridmask,rerase} erases
the selected region, cut and mix \cite{cutmix,attentivecutmix,ricap,resizemix,saliencymix,snapmix} 
replaces the selected region with some region of other
image, whereas mixup \cite{mixup,manifold,smoothmix,comixup,samplepairing} mixes the
pixel intensity values of two images. The first two
categories are region-level augmentation, whereas the
third category is image-level augmentation. Within
these categories, many approaches rely completely on
random data augmentation strategies, whereas others consider salient information either keeping it or
deleting it in the generated image. \\
\begin{figure*}[t]
\centering
\includegraphics[width=1.75\columnwidth]{flow_diagram}
\caption{The categorical division of image mixing and deleting data augmentation}
\label{flow_diagram}
\end{figure*}
Although image mixing and deleting improves the
overall performance by adding noise and ambiguity
in training data and labels, it is also likely to reduce
the performance if performed naively. Therefore, carrying out image mixing carefully is important as it
should not reduce or delete the information in an image beyond a certain limit. Because in this scenario,
the model does not have enough information to learn
and distinguish among classes, and the label of the
image becomes completely irrelevant and noisy for
the model. To avoid this situation, many approaches
are suggested in the literature that emphasizes keeping salient information in the augmented image. \\
The generalization power of image mixing or deleting
is not limited to image classification only. It has been
proven to increase accuracy for fine-grained image
recognition and object detection tasks as well. The
augmentation approaches suggested for image classification are transferrable to object detection without any modifications and shown to improve accuracy over the baseline models. Whereas, for fine-grained image recognition, particular augmentation strategies by image mixing are proposed. Apart from this, image mixing or deleting can easily be used along with regularization \cite{regularization}, dropout \cite{dropout} or conventional image augmentations \cite{surveyimaug} based on affine transformations to further enhance the performance. \\
The rest of the paper contains the review of the various image augmentation techniques in section 2, performance comparison in section 5 followed by conclusion in section 6. The review is focused entirely on the proposed hand-crafted image mixing and deletion data augmentation techniques. It does not discuss any of the approaches that perform neural augmentation search \cite{autoaugment,fastaugment}.
\section{Image Mixing and Deleting}
We divide image mixing and deleting-based augmentation schemes into three main categories. This
includes 1) erasing image patches, 2) cut image region
and replacing with some other image region, and 3)
mixing multiple images. All the proposed approaches
within these categories are discussed in detail in this
section.
\subsection{Cut and Delete}
This section discusses data augmentation by deleting image patches randomly or semantically. The purpose of erasing image patches is to allow the network to learn in case of occlusions. This way model is forced to understand and focus on the partially visible image or object properties. This kind
of dropout is different from conventional dropout because it drops contiguous image regions, whereas values in traditional dropout work at non-contiguous locations. Here, much of the image region is provided to the network to build connections semantically among various parts of the image. Below, we discuss proposed approaches falling in this category.
\begin{figure*}[ht]
\centering
\includegraphics[width=2\columnwidth]{cutout_gridmask}
\caption{Comparison of different cut and delete approaches, cutout first column, random erasing second column, hide and
seek third column and gridmask last column, courtsey \cite{gridmask}}
\label{grid_mask_fig}
\end{figure*}
\subsubsection{Cutout}
Cutout \cite{cutout} removes constant size square patches
randomly by replacing them with any constant value.
The selection of region is performed by selecting a
pixel value randomly and placing a square around it.
In this procedure, cutout region lying outside the borders is allowed and proven to improve performance.
Patch deletion is applied on all the training images
where optimal patch size is selected by performing
grid search on training data. An example is shown in
figure \ref{grid_mask_fig}.

\subsubsection{Random Erasing}
Random Erasing \cite{rerase} deletes contiguous rectangular
image regions similar to cutout \cite{cutout} with minor differences in region selection procedure. Opposite to
cutout, where deletion is applied on all the images,
here it is performed with a probability of either applying it or not. In every iteration, region size is defined
randomly with upper and lower limits on region area
and aspect ratio. The algorithm also takes care of the region to be inside the image boundaries. Additional to this, random erasing provides region-aware deletion for object detection and person identification tasks. Regions inside the object bounding boxes are randomly erased to generate occlusions. Figure \ref{grid_mask_fig} shows augmented images.

\subsubsection{Hide and Seek}
Hide and seek \cite{hidenseek} divides an image into a specified number of grids and turns on and off each grid with an assigned probability. This way various image regions are deleted, and they can be connected or disconnected from each other, as shown in figure \ref{grid_mask_fig}. Values in turned-off regions are replaced with the average of all the pixel values in the entire dataset.

\subsubsection{GridMask}
Another simple approach is GridMask \cite{gridmask}. The algorithm tries to overcome drawbacks of Cutout, Random Erasing, and Hide and Seek that are prone to deleting important information entirely or leaving it untouched without making it harder for the algorithm to learn. To handle this, GridMask creates multiple blacked-out regions in evenly spaced grids to
maintain a good balance between deletion and retention of critical information. The number of masking grids and their sizes are tuneable. The paper provides a comparison between erasing a single region vs multiple regions, showing GridMask is less likely to hide out foreground information completely. Even with
the large box size, it retains a sufficient amount of
important information for the model to learn. Gridmask generated examples are shown in figure \ref{grid_mask_fig}.
\begin{figure*}[ht]
\centering
\includegraphics[width=2\columnwidth]{attentivecutmix}
\caption{The figure shows attentive cutmix image augmentation procedure, attentive regions from one image are overlayed on
other image, image is taken from \cite{attentivecutmix}}
\label{attentive_cutmix}
\end{figure*}
\subsubsection{Cut and Delete for Object Detection}
This section describes cut and delete specifically designed for object detection.
\subsubsubsection{Adversarial Spatial Dropout for Occlusion}
This technique \cite{ASDN} drops region pixels to generate hard positives for object detection by learning key image regions. Within the proposed region only 1/3 pixels are dropped after sorting based on magnitudes. Dropped values are non-contiguous here as compared to previously discussed methods. Salient image regions are learned using a separate network. The mask generator is penalized based on the performance of the detector similar to generative adversarial networks \cite{GANs}.

\subsection{Cut and Mix}
This section discusses data augmentation strategies
where instead of deleting a patch, the patch is replaced with some other image region. By this approach, an image shares multiple class labels, whereas the major class label belongs to the original class label. Hence, the model learns to differentiate between two classes within a single image.
\subsubsection{CutMix}
In CutMix \cite{cutmix} images are augmented by sampling
patch coordinates, $x, y, h, w$ from a uniform distribution. The selected patch is replaced at the corresponding location with a patch from the other randomly picked image from the current mini-batch during training. CutMix updates image labels according
to replaced patch proportion as defined in equation
\ref{cutmix_eq}, where $M$ is the image mask, $x_a$ and $x_b$ are images,
$\lambda$ is the proportion of label, and $y_a$ and $y_b$ are the
labels of images.
\begin{equation}
\begin{aligned}
x_{new} = M.x_a + (1-M).x_b \\
y_{new} = \lambda.y_a + (1-\lambda).y_b
\end{aligned}
\label{cutmix_eq}
\end{equation}

\subsubsection{Attentive CutMix}
Attentive CutMix \cite{attentivecutmix} builds up on CutMix. Instead of random pasting, it identifies attentive patches for
cutout and pastes them at the same location in the other image. This avoids the problem of selecting a background region not important for the network and updating the label information accordingly, as in equation \ref{cutmix_eq}. A separate pre-trained network is employed to extract attentive regions. The attention output is mapped back onto the original image. Image is divided into a grid of patches where highly activated six responses are pasted onto the training image. These image pairs are selected randomly in every training iteration. This procedure is shown in figure \ref{attentive_cutmix}.

\subsubsection{Attribute Mix}
Attribute Mix \cite{attributemix} augments images based on semantically extracted image attributes. It trains an attribute classifier by extracting $k$ attributes (e.g., leg, head, and wings of a bird) from each image. The attribute mining procedure for every image is performed repetitively $k$ times, whereas for each iteration, an attribute is masked out from the original image based on the most discriminative region in the attention map. With these images, an attribute-level classifier is trained to generate new images for the actual classification model. For the given two images, the attribute-level classifier identifies attributes masks, these masks are randomly picked to create a
new training image. The procedure of label update
is the same as in equation \ref{cutmix_eq}.
\subsubsection{RICAP}
Random image cropping and patching (RICAP) \cite{ricap}
performs data augmentation by cropping four regions from randomly sampled images and augmenting them to create a new image. The generated image has mixed labels proportional to the pasted
area. The area of cropped regions in the output
image is determined by sampling through uniform distribution. The paper suggests three methods for common boundary points in the augmented image: anywhere-RICAP (origin can be anywhere), center-RICAP (origin can only be in the middle of the image), and corner-RICAP (origin can only be in corners). Corner-RICAP has shown the best performance because a larger region of one image is visible to the network to learn. An example of RICAP strategy is shown in figure \ref{ricap_fig}.
\begin{figure}[H]
\centering
\includegraphics[width=1\columnwidth]{ricap}
\caption{An example of RICAP image augmentation procedure, courtesy \cite{ricap}}
\label{ricap_fig}
\end{figure}

\subsubsection{Mixed Example}
This method \cite{mixedexample} experimented with 14 different
types of augmentation approaches. The output image is generated using the following techniques: vertical concatenation, horizontal concatenation, mixed concatenation, random 2x2, VH-mixup (vertical concatenation, horizontal concatenation, and mixup), VH-BC+ (vertical concatenation, horizontal concatenation, and between-class), random square, random column interval, random row interval, random rows, random columns, random pixels, random elements, and noisy mixup. From all these approached, VHmixup has the best performance.
\subsubsection{CowMask}
CowMask \cite{cowmask} is used in semi-supervised learning
where original and augmented images are brought
closer during training. CowMask suggests two types
of mixing approaches 1) erasing and 2) mixing two
images similar to cutmix. The mask here is of irregular shape rather than rectangular and generated by masking or keeping a proportion of image pixels through thresholding. Gaussian filter is applied to remove noise before thresholding. Pixel values below the threshold are either erased or replaced by the
pixel values of the randomly selected image at the
corresponding locations.
\subsubsection{ResizeMix}
ResizeMix \cite{resizemix} performs random image cropping
and pasting. It empirically proves to perform better than salient and other non-salient image mixing
methods, as shown in table 1. ResizeMix solves the
random region cropping problem that misallocates
the output image label in certain cases where the
pasted region does not contain any object information. The labels of the output image are updated as per the equation \ref{cutmix_eq}. ResizeMix handles this by scaling down (scale rate is sampled from a uniform distribution) the selected image completely and pasting it randomly on the target image, as shown in Figure \ref{resizemix_fig}. This way, modified labels of the output image are always accurate and proportionate to the mixing.
\begin{figure}[H]
\centering
\includegraphics[width=1\columnwidth]{resizemix}
\caption{An example of augmented image generated using
resizemix, figure taken from \cite{resizemix}}
\label{resizemix_fig}
\end{figure}
\subsubsection{SaliencyMix}
Instead of random cut and mix, SaliencyMix \cite{saliencymix}
extracts salient regions and pastes them on the corresponding location in the target image. The salient region is extracted around the maximum intensity pixel location in the saliency map. Augmented image labels are updated as per the equation \ref{cutmix_eq}.
\subsubsection{Intra-Class Part Swapping}
Intra-Class Part Swapping \cite{intraPartSwap} replaces most attentive regions of one image by the other. Attentive
regions are extracted using a classification activation
map (CAM), thresholded for the most prominent region. The attentive region in the source image is scaled and translated according to the attentive region of the target image for region replacement. The label information of the output is similar to the target image as this approach relies on augmenting similar
class images.
\subsubsection{SnapMix}
SnapMix \cite{snapmix} augments training images by extracting and merging random image regions of different sizes, where the region size is drawn through the beta distribution for both the images. Generated image label is assigned based on semantic composition from normalized (sum to one) CAMs. The procedure of
label generation is similar to equation \ref{cutmix_eq}. However,
the summation of label coefficients can exceed beyond one depending on the semantic composition of
the output image.
\subsubsection{KeepAugment}
Similar to SaliencyMix and attentive cutmix, KeepAugment \cite{keepaugment} performs augmentation based on
salient region. However, this approach does not use two images for the augmentation. KeepAugment identifies the salient area in an image and assures the image generated by the augmentation strategies, for example, Cutout, RandAugment \cite{randaugment}, CutMix \cite{cutmix} or AutoAugment \cite{autoaugment}, contains salient region in it.

\begin{figure*}[ht]
\centering
\includegraphics[width=2\columnwidth]{augmix}
\caption{Augmix \cite{augmix} data augmentation procedure is shown in image, multiple transformations are applied randomly on
image in separate branches that are later combined to generate one image, example is from \cite{augmix}}
\label{augmix_fig}
\end{figure*}

\subsubsection{Cut and Mix for Object Detection}
This section discusses cut and mix approaches
specifically designed for object detection.

\subsubsubsection{Visual Context Augmentation}
This augmentation strategy \cite{VCAugment1} learns to place object instances at an image location depending on the surrounding context. A neural network is trained for this purpose. The training data is prepared to generate a context image with the masked-out object inside it. From an image, 200 context sub-images are generated surrounding the blacked-out bounding box. The neural network learns to predict the category (object or background) in masked pixels. During testing, the context network performs the identification of plausible object bounding boxes. The object instances are placed inside the selected boxes to generate a new
training image. These newly created images are used
as training data for the other network, for example,
object detectors.
\subsubsubsection{Cut, Paste and Learn}
This simple approach \cite{cutpasteandlearn} generates new data by extracting object instances and pasting them on randomly selected background images. Instances are blended with various blending approaches, for example, gaussian blurring and poison blending, to reduce
pixel artifacts around the augmented object boundaries. Added instances are also rotated, occluded, and truncated to make the learning algorithm robust. This simple data mixing approach improves object detection performance.
\subsection{MixUp}
This type of data mixing is different from Cutout
and CutMix such that it mixes pixel values of two
images instead of cropping or erasing image regions.
MixUp regularizes neural networks by forcing them
to learn linear interpolations between training images. The network trained with MixUp has shown robustness to adversarial attacks with improved performance on the test set.

\begin{figure}[H]
\centering
\includegraphics[width=1\columnwidth]{smoothmix}
\caption{An example of network training using smoothmix
strategy, courtsey \cite{smoothmix}}
\label{smoothmix_fig}
\end{figure}

\subsubsection{MixUp}
MixUp [83] generates a new image by mixing pixel
values of two randomly selected images. The proportion of pixel strength during data mixing is decided
by the mixing factor $\lambda$ $\in$ [0-1] as shown in equation
\ref{mixup_eq}. The values for $\lambda$ are drawn from the beta distribution.

\begin{equation}
\begin{aligned}
x_{new} = \lambda.x_a + (1-\lambda).x_b \\
y_{new} = \lambda.y_a + (1-\lambda).y_b
\end{aligned}
\label{mixup_eq}
\end{equation}

\subsubsection{Manifold MixUp}
Opposite to mixup, manifold mixup \cite{manifold} mixes feature values generated from intermediate neural network layers. For two random images, inputs are feedforward up to k layer of the network where the output feature maps are mixed based on equation \ref{mixup_eq}. The mixed feature maps are given input to the next layer and forward propagated up to the last layer. After the forward propagation, backward propagation is performed in the standard way with updated labels (similar to mixup).

\subsubsection{AugMix}
Augmix \cite{augmix} performs data mixing using the input
image itself. It transforms (translate, shear, rotate
and etc) the input image and mixes it with the original image. Image transformation involves series of randomly selected augmentation operations applied with three parallel augmentation chains. Each chain has a composition of operations that could involve applying, for example, translation on input image followed by shear and so on. The output of these three chains is three images mixed to form a new image. An example is shown in figure \ref{augmix_fig}. This new image is later mixed with the original image to generate the final augmented output image, as per equation
\ref{mixup_eq}. The transformations in AugMix are selected from AutoAugment \cite{autoaugment}.
\begin{figure*}[ht]
\centering
\includegraphics[width=2\columnwidth]{comixup}
\caption{Image augmentation performed by co-mixup is shown in figure, courtesy \cite{comixup}}
\label{comixup_fig}
\end{figure*}


\subsubsection{SmoothMix}
SmoothMix \cite{smoothmix} is a mask-based approach, matching closely with the Cutout and CutMix techniques. However, it has few differences: (1) the mask has soft edges with gradually decreasing intensity, and (2) the mixing strategy is the same as in equation 2. The augmented image has mixed pixel values depending on the strength of the mask, as shown in figure \ref{smoothmix_fig}.
The paper has suggested various square and circular
masks containing smooth edges. Frome these circular masks are the best performing, generated using Gaussian distribution. The value of $\lambda$ is decided by the following equation \ref{smooth_mix_eq}:
\begin{equation}
\lambda = \frac{\sum_{i=1}^{W}\sum_{j=1}^{H}G_{ij}}{WH}
\label{smooth_mix_eq}
\end{equation}
where $G_{ij}$ is the pixel value of mask $G$, $H$ is height,
and $W$ is width. With the given mask, the equation
to mix two images becomes:
\begin{equation}
\begin{aligned}
x_{new} = G.x_a + (1-G).x_b \\
y_{new} = \lambda.y_a + (1-\lambda).y_b
\end{aligned}
\end{equation}


\subsubsection{Co-Mixup}
Co-Mixup \cite{comixup} performs salient image mixing on a
batch of input images to generate a batch of augmented images. This technique maximizes saliency in
output images by penalizations to ensure local data
smoothness and diverse image regions. An example
of generated images by Co-Mixup is shown in figure
\ref{comixup_fig}.

\subsubsection{Sample Pairing}
This simple technique \cite{samplepairing} merges two images by averaging their pixel intensities. The new image has
the same training image label opposite to MixUp and
other approaches where labels are updated according
to the proportion of image mixing.

\subsubsection{Puzzle Mix}
Puzzlemix \cite{puzzlemix} learns to augment two images optimally based on saliency. The algorithm employs the following procedure 1) Images are divided into regions for the mixup, 2) The algorithm learns to transport the salient region of one image such that the output image has the maximized saliency from
both images. An example of a transported salient
region in an augmented image is shown in figure \ref{puzzlemix_fig}.

\begin{figure}[H]
\centering
\includegraphics[width=1\columnwidth]{puzzlemix2}
\caption{Example of puzzlemix, depicting how maximum
saliency in output image is attained with region transport,
courtesy \cite{puzzlemix}}
\label{puzzlemix_fig}
\end{figure}

\subsubsection{ReMix}
Remix \cite{remix} addresses the issue of class imbalance by
generating mixed images for minority classes. The
process of remix is identical to equation 2 for data
mixing, but in the case of label assignment, it sets
the label of the output image to the minority class.

\subsubsection{Mixup for Object Detection}
Similar to Cutout and CutMix, MixUp is also used
to enhance the performance of object detection algorithms. One of the approaches in \cite{mixupobj} MixUp image data to update object labels and locations. Instead of employing the fundamental MixUp strategy, this algorithm identifies object locations, mixes object pixels at corresponding locations in two images,
and updates the object box-coordinates in output image depending on the prominent object and the image
label.

\section{Why Image Mixing and Deleting performs
well?}
Understanding the underlying cause of why these techniques improve the performance of baselines is important. This builds the foundation to improve the existing approaches. The various methods are adopted to draw analysis between standard training
procedures and training with augmentations. Cutout
\cite{cutout} and CutMix \cite{cutmix} argues that hindering image regions enforces the classifier to learn from the partially visible objects and understand the overall structure. CutMix verifies this argument by showing enhanced focus towards the target class in CAMs, shown in figure \ref{cam_fig}. Opposite to this, MixUp has shown to improve classifier’s calibration and reduced prediction uncertainity in \cite{icapu}. To prove this, authors draw a comparison between mean of predictions vs accuracy where the confidence distribution for MixUp trained model is evenly distributed against the standard model whose distribution is towards higher conficence i.e. over-confidence. Similarly, the loss contours obtained for a network trained with MixUp are
smooth as compared to sharp contours in standarad
training \cite{undermixup}. This proves \cite{icapu,undermixup} the better generalization and robustness of MixUp against adversarial
attacks.


\section{Applications}
The benefits of image mixing and deleting are not
limited to only image classification or object detection, for which they were designed initially. These algorithms, when used in conjunction, have been shown to enhance the performance of other systems. A few examples are semi-supervised learning, unsupervised learning, adversarial training, privacy mixing, etc.
The subsequent sub-sections provide a brief overview, how the incorporation of image mixing and deleting with minor tweaks have improved the performance in other applications.


\begin{figure}[H]
\centering
\includegraphics[width=1\columnwidth]{cam}
\caption{Class activation maps (CAMs) for the augmented
images, image is taken from \cite{cutmix}}
\label{cam_fig}
\end{figure}
\subsection{Semi-Supervised}
Semi-supervised learning (SSl) leverages unlabelled
data to train strong classifiers. One of the semisupervised learning approaches, named as MixMatch
\cite{SSL2} suggested weighted MixUp, given in equation 5,
to ensure augmented image is closer to the original
image. This approach is successfully employed in
other SSL techniques \cite{remixmatch,dividemix,focalmix}. Another SSL approach \cite{fixmatch} make use of Cutout in association with
RandAugment \cite{randaugment}14\ naming it control theory augment
(CTAugment).

\begin{equation}
\begin{gathered}
\lambda \sim \beta(\alpha, \alpha) \\
\acute{\lambda} = max(\lambda, 1 - \lambda) \\
x_{new} = \acute{\lambda}.x_a + (1-\acute{\lambda}).x_b \\
y_{new} = \acute{\lambda}.y_a + (1-\acute{\lambda}).y_b
\end{gathered}
\end{equation}

\subsection{Unsupervised Learning}
Unsupervised learning aims at learning better data
representations without requiring human annotations. Recent success in self-supervised and contrastive learning \cite{mocov2,simclr,byol}, that learns to bring positive samples closer and push away negative samples, has attracted researchers to investigate image mixing and deleting for better representation learning in the unsupervised domain. The literature shows unsupervised algorithms using CutMix in \cite{centerwise} and MixUp in \cite{imix} improve the vanilla model performances simply by additional image mixing data augmentation. Another approach Unmix \cite{unmix}61], employs MixUp for prediction smoothness in an unsupervised setting. Instead of keeping the distance of positive pairs equal to zero, unmix generates a mini-batch of mixtures of positive pair augmented images. The distance between pairs is lowered and equaled to the mixing factor either  $\lambda$ or $1 - \lambda$ based on the mixing order in a
mini-batch. \\
Unsupervised domain adaptation (UDA), a sub-area
of unsupervised learning utilizes MixUp to introduce
linear prediction behavior across domains. For efficient domain adaptation, from the source domain (has data labeled) to target domain (has data unlabelled), approaches cite{fixbi,virtualmix,domainadaptation} mixes images from both the domains. For UDA, images and labels $x_a$ and $y_a$ are collected from the source domain, $x_b$ and $y_b$ (pseudo label) are taken from the target domain.

\subsection{Adversarial Training}
Adversarial training is about training the neural
networks for robustness against adversarial examples. However, to achieve adversarial robustness neural networks performance becomes worse for standard classification. To reduce the impact of this phenomenon, approaches are proposed to incorporate
adversarial training with MixUp. Adversarial vertex mixup \cite{avm} trains the network with the MixUp of clean image x and adversarial vertex image $x_{av}$ generated by the equation \ref{vm_eq}, where perturbations $\delta_x$ are generated by Projected Gradient Descent (PGD) \cite{pgd}. Similarly, interpolated adversarial training \cite{iat}, performs the MixUp of clean and perturbed images generated by PGD for enhanced robustness.
\begin{equation}
x_{av} = x + \gamma\delta_x
\label{vm_eq}
\end{equation}

\subsubsection{Backdoor and Targeted Attacks}
Backdoor attacks poison data by inserting a trigger patch (generally a small noise patch), whereas
targeted attacks modify the image such that the network labels the input image as the targeted label. These attacks lose their effectiveness when the model is trained with the CutMix and MixUp data augmentations as in \cite{backdooratt,dpinstahide}.

\subsection{Privacy Preserving}
Training on edge devices produces less accurate
models due to data scarcity. Additional to this, the
edge devices have insufficient computing resources
both for training and inference. To overcome these
problems, private data has to be transferred to the
cloud. The cloud is prone to be malicious where the
attacker can exploit the user’s private data. This inspired the researchers to devise solutions to preserve data privacy. Within the proposed approaches, few employ MixUp to achieve data privacy \cite{instahide,datamix,xormixup}. InstaHide \cite{instahide} mixes private images with public data k times that are later passed through random pixel sign flipping. DataMix \cite{datamix} is a privacy preserving inference algorithm. This algorithm mixes two private images multiple times and transmits the data to the cloud for inference. The predictions are transferred back to the edge device that performs demix operation and generates true prediction probabilities. The mixing and demixing coefficients are only known to the edge devices. XOR MixUp \cite{xormixup} employs iterative MixUp of private and dummy images followed by XOR for data encoding. The data is decoded on the server-side using dummy data present on the server by performing another XOR operation. The model is trained using decoded data whose parameters are shared with all the devices after model convergence.

\subsection{Text Classification}
The applications of MixUp are also found in text
classification. Due to the discrete nature of the text,
mixing is not performed on the input, but words or
sentence embeddings \cite{mintext,seqmix,augmentingtext}. The performance of transformers for text classification improves significantly with MixUp on transformers generated embeddings \cite{mixuptransformer}. Another approach \cite{nonlinearmixup} suggests non-linear mixup on embeddings space of words, where every dimension of word embedding (multi-dimensional embedding) has a different mixing parameter rather than the global value as in equation \ref{mixup_eq}.


\begin{table*}[ht]
\centering
\hspace*{-1.85cm} 
\begin{tabular}{cclcccccc}
\hline
 & Code & Method & \begin{tabular}[c]{@{}c@{}}Cifar-10\\ Accuracy (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Cifar-10\\ (Best Model)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Cifar-100\\ Accuracy (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Cifar-100\\ (Best Model)\end{tabular} & \begin{tabular}[c]{@{}c@{}}ImageNet\\ Accuracy (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}ImageNet\\ (Best Model)\end{tabular} \\ \hline
Cutout & y & Random & 97.04 & WRN-28-10 & 81.59 & WRN-28-10 & 77.1 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Random\\ Erasing\end{tabular} & y & Random & 96.92 & WRN-28-10 & 82.27 & WRN-28-10 & - & - \\ \hline
\begin{tabular}[c]{@{}c@{}}Hide and\\ Seek\end{tabular} & y & Random & 96.94 & WRN-28-10 & 78.13 & R-110 & 77.20 & R-50 \\ \hline
GridMask & y & Random & 97.24 & WRN-28-10 & - & - & 77.9 & R-50 \\ \hline
Cutmix & y & Random & 97.10 & WRN-28-10 & 83.40 & WRN-28-10 & 78.6 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Attentive\\ Cutmix\end{tabular} & n & Salient & 95.86 & ENet-B7 & 78.52 & ENet-B7 & - & - \\ \hline
RICAP & y & Random & 97.18 & WRN-28-10 & 82.56 & WRN-28-10 & 78.62 & WRN-50-2 \\ \hline
\begin{tabular}[c]{@{}c@{}}Mixed\\ Example\end{tabular} & y & Random & 96.2 & ResNet-18 & 80.3 & R-18 & - & - \\ \hline
CowMask & y & Random & 97.44 & \begin{tabular}[c]{@{}c@{}}WRN-28-96 \\ x2d\end{tabular} & 84.27 & \begin{tabular}[c]{@{}c@{}}WRN-28-96 \\ x2d\end{tabular} & 73.94 & R-152 \\ \hline
\begin{tabular}[c]{@{}c@{}}Resize-\\ Mix\end{tabular} & n & Random & 97.60 & WRN-28-10 & 84.31 & WRN-28-10 & 79.00 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Saliency-\\ Mix\end{tabular} & y & Salient & 97.24 & WRN-28-10 & 83.44 & WRN-28-10 & 78.74 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Keep-\\ Augment\end{tabular} & n & Salient & 97.8 & WRN-28-10 & - & - & 79.1 & R-50 \\ \hline
MixUp & y & Random & 97.3 & WRN-28-10 & 82.5 & WRN-28-10 & 77.9 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Manifold\\ Mixup\end{tabular} & y & Random & 97.45 & WRN-28-18 & 81.96 & WRN-28-10 & 78.7 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Smooth-\\ Mix\end{tabular} & y & Random & 97.02 & PNet-200 & 85.53 & PNet-200 & 77.66 & R-50 \\ \hline
\begin{tabular}[c]{@{}c@{}}Sample\\ Pairing\end{tabular} & n & Random & 93.07 & GNet & 72.1 & GNet & 70.99 & GNet \\ \hline
AugMix & y & Random & - & - & - & - & 77.6 & R-50 \\ \hline
PuzzleMix & y & Salient & - & - & 84.05 & WRN-28-10 & 77.51 & R-50 \\ \hline
Co-Mixup & y & Salient & - & - & 80.85 & WRN-16-8 & 77.61 & R-50 \\ \hline
\end{tabular}
\caption{Performance comparison for various image augmentation approaches for image classification problem. In table, R
refers to ResNet \cite{RNet}, PNet is PyramidNet \cite{PNet}, GNet is GoogleNet \cite{GNet}, and ENet is EfficientNet \cite{ENet}}
\label{img_clf_tab}
\end{table*}

\section{Performance Comparison}
This section compares the augmentation approaches
performance for main-stream tasks, image classification, object detection, and fine-grained image recognition on publicly available datasets.

\subsection{Image Classification}
Performance for image classification is evaluated on
three datasets, Cifar-10 \cite{cifar}, Cifar-100 \cite{cifar} and ImageNet \cite{imagenet}. For consistency of reported results among different approaches, we try to collect results for a similar base model from respective publications. The WideResnet-28-10 is largely common for the Cifar-10 and Cifar-100, and ResNet-50 for ImageNet. We can conclude about best performing approaches even for the evaluations with other models based on performance and number of parameters of these models. From the Table \ref{img_clf_tab}, we can conclude ResizeMix \cite{resizemix}, SaliencyMix \cite{saliencymix} and KeepAugment \cite{keepaugment} performs best for Cifar-10, Cifar-100 and ImageNet respectively. Overall, the performance gap between salient and random mixing techniques is not very high, and one can prefer random mixing approaches for data augmentation due to its low computation requirements.


\begin{table*}[]
\resizebox{\textwidth}{!}{%
\begin{tabular}{ccccccl}
\hline
 & \begin{tabular}[c]{@{}c@{}}Faster-RCNN\\ Backbone\end{tabular} & \begin{tabular}[c]{@{}c@{}}Baseline\\ (mAP)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Augmented\\ (mAP)\end{tabular} & Train Set & Test Set & \multicolumn{1}{c}{\begin{tabular}[c]{@{}c@{}}Experimentation\\ Design\end{tabular}} \\ \hline
Cutout & R-50 & 76.71 & 77.17 & VOC 07+12 & VOC 07 & \begin{tabular}[c]{@{}l@{}}Backbone is trained on ImageNet using cutout\\ augmentation strategy. The Faster-RCNN backbone\\  is initialized with this newly trained model\\ and fine-tuned on train set as in \cite{fasterrcnn}\end{tabular} \\ \hline
CutMix & R-50 & 76.71 & 78.31 & VOC 07+12 & VOC 07 & Same as above \\ \hline
MixUp & R-50 & 76.71 & 77.98 & VOC 07+12 & VOC 07 & Same as above \\ \hline
SaliencyMix & R-50 & 76.71 & 78.38 & VOC 07+12 & VOC 07 & Same as above \\ \hline
GridMask & R-50-FPN & 37.4 & 38.3 & COCO 17 & COCO 17 & \begin{tabular}[c]{@{}l@{}}Backbone trained on ImageNet is used for\\ initialization. During object detector training,\\ gridmask is applied on input after baseline\\ augmentation strategy as in \cite{fasterrcnn}\end{tabular} \\ \hline
KeepAugment & R-50 & 38.4 & 39.5 & COCO 17 & COCO 17 & \begin{tabular}[c]{@{}l@{}}Pre-trained backbone on ImageNet with\\ keepaugment is used. The object detector\\ is fine-tuned using strategy as in \cite{fasterrcnn}\end{tabular} \\ \hline
Random Erasing & VGG-16 & 74.8 & 76.2 & VOC 07+12 & VOC 07 & \begin{tabular}[c]{@{}l@{}}Pre-trained backbone with ImageNet weights.\\ The detector is trained with combined image\\ and object random erasing strategies keeping\\ everything similar to \cite{fasterrcnn}\end{tabular} \\ \hline
ResizeMix & R-50 & 38.1 & 38.4 & COCO 17 & COCO 17 & Same as above \\ \hline
\end{tabular}%
}
\caption{The table shows the improvements in accuracy for object detection using various image mixing and deleting augmentation techniques. In table, R refers to ResNet \cite{RNet} and FPN is feature pyramid network \cite{FPN}, whereas VGG is from
\cite{VGG}}
\label{obj_det_tab}
\end{table*}

\subsection{Object Detection}
Backbones trained with these data mixing techniques are also evaluated for object detection. The
evaluation procedure involves replacing the existing
backbone with a newly trained model and fine-tune
it with the object detection task, as per the standard
methods mentioned in the literature. \\
The performance comparison for this task is shown in
Table \ref{obj_det_tab}. The methods in Table \ref{obj_det_tab} are evaluated for the faster-RCNN \cite{fasterrcnn} object detection. Every approach shows the comparison between the standard backbone and the backbone trained with data augmentation techniques. Here, SaliencyMix outperforms for VOC 07 \cite{VOC} dataset, whereas KeepAugment is the best among all for the COCO 17 \cite{MSCOCO} dataset.
\begin{multicols}{2}

\begin{table*}[]
\resizebox{0.5\textwidth}{!}{%
\begin{tabular}{|c|c|l|c|c|c|}
\hline
 & Code & Model & CUB (\%) & Cars (\%) & Aircraft (\%) \\ \hline
\begin{tabular}[c]{@{}c@{}}Attribute\\ Mix\end{tabular} & n & R-50 & 88.4 & 94.9 & 92.0 \\ \hline
\begin{tabular}[c]{@{}c@{}}Intra-Class \\ Part Swapping\end{tabular} & n & R-50 & 87.56 & 94.59 & 92.65 \\ \hline
SnapMix & y & R-50 & 87.75 & 94.30 & 92.80 \\ \hline
\end{tabular}%
}
\caption{Comparison of data augmentation strategies designed \\
for fine-grained image classification. In table, abbreviation of \\
ResNet is R}
\label{fine_grained_tab}
\end{table*}

\end{multicols}
\subsection{Fine-Grained Image Recognition}
Fine-grained image classification emphasizes classifying images based on sub-class categories. For example, the class bird is further split into multiple categories based on species. More information can be seen for Caltech-UCSD Birds 200 (CUB), Cars
and aircraft in \cite{CUB}, \cite{cars} and \cite{aircraft} respectively. \\
Table \ref{fine_grained_tab} shows the performance comparison for a few
of the specific image augmentation techniques designed for fine-grained image recognition. All of these approaches are designed with saliency information. AttributeMix is well ahead for the CUB and Cars dataset, whereas SnapMix wins for aircraft.



\section{Conclusion}
Image augmentation with mixing and deletion has
shown promising results, improving the accuracy of
CNN’s over baselines. The improvement is not limited to few tasks but also extended and proven for other tasks, for example, object detection, semisupervised learning, adversarial training, etc. This paper has reviewed recently introduced techniques in
image mixing and deleting. The performance comparison among various approaches for different tasks is shown in tables. The comparative analysis will help researchers in selecting suitable data augmentation techniques in their research to improve performance. Overall, the margin of improvement lies within 1-3\%
range with these augmentation techniques.

