
\section{Experiments}
\label{sec:exp}
In this section, to demonstrate the efficacy of our training framework, we carry out experiments on five action recognition datasets, including Kinetics-400~\cite{kay2017kinetics}, Something-Something-v2~\cite{goyal2017something}, Moments-in-Time~\cite{monfort2019moments}, Activitynet~\cite{caba2015activitynet} and Kinetics-700~\cite{carreira2019short}.
The action recognition task is defined to be a classification task given a trimmed video clip.
Unlike previous works~\cite{likhosherstov2021polyvit,zhang2021co}, we do not initialize our model using ImageNet~\cite{deng2009imagenet} since it consumes more computation. Please refer to the supplementary material for detailed comparison between train from scratch recipe and from ImageNet.
In the experiments, we aim to showcase
that our method can achieve significant performance improvement with minimal computation overhead compared to baselins.

\subsection{Experimental Setup}
\noindent\textbf{Datasets.}
We evaluate our method on five datasets.
Kinetics-400~\cite{kay2017kinetics} (K400) consists of about 240K training videos and 20K validation videos in 400 human action classes. 
The videos are about 10 seconds long. 
Kinetics-700~\cite{carreira2019short} (K700) extends the action classes to 700 with 545K training and 35K validation videos.
The Something-Something-v2 (SSv2)~\cite{goyal2017something} dataset contains person-object interactions, which emphasizes temporal modeling. 
SSv2 includes 168K videos for training and 24K videos for evaluation on 174 action classes.
The Moments-in-Time (MiT) dataset is one of the largest action dataset with 727K training and 30k validation videos.
MiT videos are mostly short 3-second clips.
The ActivityNet dataset~\cite{caba2015activitynet} (ActNet) originally contains untrimmed videos with temporal annotations of 200 action classes.
We cut the annotated segments of the videos into 10-second long clips and split the dataset into 107K training and 16K testing.
Following previous works~\cite{feichtenhofer2019slowfast,zhang2021co}, we follow the standard dataset split and report top-1/top-5 classification accuracy on the test split for all datasets.
We conduct two sets of experiments, namely, ``K400, MiT,  SSv2, ActNet'', and ``K700, MiT, SSv2, ActNet''.

\textbf{Implementation.}
Our backbone model utilizes MViTv2 as described in Section~\ref{sec:mvit}.
Our models are trained from scratch with random initialization, without using any pre-training (same as in ~\cite{feichtenhofer2019slowfast} and different from previous works~\cite{zhang2021co,likhosherstov2021polyvit} that require large-scale image dataset pre-training like ImageNet-21K~\cite{deng2009imagenet} or JFT-3B~\cite{DBLP:journals/corr/abs-2106-04560}). 
We follow standard dataset splits as previous works~\cite{li2021improved,feichtenhofer2019slowfast,yan2022multiview}.
See more details in the supplementary material.
\begin{comment}
For both sets of experiments, we start our training from the corresponding Kinetics-trained model checkpoints to reduce training time.
The Kinetics pre-training follows ``train from scratch'' protocol of MViTs~\cite{fan2021multiscale,li2021improved}.
We use a learning rate of 2e-4 and cosine learning rate decay. 
We use synchronized SGD with 128 V100/A100 GPUs to train for 50 epochs with a batch size of 512, with a linear warm-up from 2e-6 for the first 10 epochs.
Each experiment takes 2-4 days to finish.
We use a weight decay of 0.05 and the adamw~\cite{loshchilov2017decoupled} optimizer is used. 
We use 32x3 video frames (number of frames times sampling rate) as inputs except for SSv2.
For SSv2, following the MViT paper~\cite{fan2021multiscale}, we uniformly sample frames from the whole videos instead of getting fix-length clips.
For all three datasets, we use random $224\times224$ crops unless mentioned otherwise and horizontal flipping from a video clip (except for videos from SSv2), which is randomly sampled from the full-length video and resized to a shorter edge side of randomly sampled in [256, 320] pixels.
We utilize Mixup~\cite{zhang2017mixup} and Rand Augment~\cite{cubuk2020randaugment} for data augmentation during training.
Following previous works~\cite{feichtenhofer2019slowfast,fan2021multiscale}, we sample $3\times3$ (unless mentioned otherwise) clips for each video during testing: we uniformly sample 3 clips for the temporal domain and 3 spatial crops of size $224\times224$ after the shorter edge size are resized to 256 pixels.
We average the softmax scores across all clips for final prediction.
We utilize 2-layer MLPs with 2048 hidden size and a GELU activation for the expander and the classification heads.
A single-layer linear projection head is added for each pair of datasets during training, and they are discarded during inference (the original classification head outputs are used as the final prediction.)
\end{comment}

\textbf{Baselines.} 
PolyViT~\cite{likhosherstov2021polyvit} utilizes multi-task learning on image, video and audio datasets to improve vision transformer performance. The backbone they used are based on ViT-ViViT~\cite{arnab2021vivit}.
Similarly, VATT~\cite{akbari2021vatt} utilizes additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets. The backbone network is based on ViT~\cite{dosovitskiy2020image}.
CoVER~\cite{zhang2021co} is a recently proposed co-training method that includes training with images and videos simultaneously. Their model backbone is based on TimeSFormer~\cite{bertasius2021space}.
We also compare our method with other recent models trained using large-scale image datasets. 
See Table~\ref{tab:k400} and Table~\ref{tab:k700} for the full list.

\subsection{Main Results}

\input{table/k400}


\input{table/k700}

We summarize our method's performance in Table~\ref{tab:k400} and Table~\ref{tab:k700}.
We train our model jointly on MiT, SSv2, ActNet and two versions of the Kinetics datasets.

We first compare our method with the original MViTv2 backbone in Table~\ref{tab:k400}.
``MViTv2 w/ abs. pos.'' means MViTv2 model with absolute positional embedding, which is taken from Table A.6 of MViTv2 paper~\cite{li2021improved} and it is (almost) the same as our model implementation. 
We can not achieve the same accuracy with the same recipe as MViTv2, which may be due to differences in the Kinetics dataset (missing some videos, etc. See supplementary material for full dataset statistics).
PolyViT~\cite{likhosherstov2021polyvit} is trained jointly with multiple image, audio and video datasets.
We list the larger ones.
We train our baseline model on the training set of each dataset to investigate the baseline performance.
As we see, after adding robust joint training proposed in this paper, performance on each dataset has increased by 2.1\%, 3.1\%, 1.9\% and 5.9\% on K400, MiT, SSv2, ActivityNet, respectively in terms of top-1 accuracy.
Note that our method achieves such improvement withtout large-scale image pre-training and additional inference computational cost.

We then compare our method with state-of-the-art on these datasets.
We train a higher resolution model with larger spatial inputs (312p) and achieves better performance compared to recent multi-dataset training methods, CoVER~\cite{zhang2021co} and PolyVit~\cite{likhosherstov2021polyvit}, on Kinetics-400, and significantly better on MiT and SSv2, as shown in Table~\ref{tab:k400}.
Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines.
We also show that our performance boost does not come from the additional training dataset of ActivityNet in Table~\ref{tab:ablation}.

Our method also achieves competitive results compared to state-of-the-art models trained with large-scale image dataset (ImageNet-21K~\cite{deng2009imagenet}).
Compared to a recent method, MTV-B~\cite{yan2022multiview}, our method is able to achieve significantly better top-1 accuracy across Kinetics-400, MiT, SSv2 by 0.8\%, 1.4\%, 0.8\%, respectively, at half of the computation cost and without large-scale pre-training.
Note that our model is parameter-efficient, while multiple MTV-B models need to be trained and tested on these datasets separately.
Our method can achieve better performance with a deeper base backbone or larger resolution inputs but we have not tested due to limitation of computation resources.



We then compare our method on the Kinetics-700, MiT, SSv2 and ActivityNet training with baselines.
Our parameter-efficient model can achieve better performance than MTV-B~\cite{yan2022multiview} at one-fifth of the computation cost.
With a larger resolution model at 312p, we achieves significantly better performance than the baseline across Kinetics-400, MiT, SSv2 by 2.2\%, 4.9\%, 3.4\%, respectively.


\subsection{Ablation Experiments}
In this section, we perform ablation studies on the K400 set.
To understand how action models can benefit from our training method,
we explore the following questions (results are shown in
Table~\ref{tab:ablation}):

\input{table/ablation}

\noindent \textbf{Does our proposed robust loss help?}
We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss.
The vanilla model is trained from a K400 checkpoint as ours.
As shown in Table~\ref{tab:ablation}, we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule. 
As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method.
Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
In terms of performance on ActivityNet, we observe that both training methods achieve good results, which might be because ActivityNet classes are highly overlapped with Kinetics-400 (65 out of 200). 

\noindent \textbf{How important is the informative loss?}
We then experiment with removing the informative loss (Section~\ref{sec:robust}) during multi-dataset training. 
It seems that the feature embedding of the model collapse and the model is not trained at all.
We further investigate why ``w/o informative Loss'' completely fails but ``Vanilla'' seems to work by running an experiment of "w/o informative Loss \& w/o projection add", which means we remove the projected logits addition in Eq. 5 and directly compute classification loss on the projected logits. 
Therefore we can consider this run as adding additional projection branches to the vanilla architecture. The results are slightly better than "Vanilla" on K400 and much better on MiT/SSv2. It indicates that adding projected logits to the original branch without informative loss would prevent the model from converging (the total loss does not go down).

\noindent \textbf{How important is the projection loss?}
We then experiment with removing the projection heads (Section~\ref{sec:robust}) during multi-dataset training. 
The model is trained with the original cross-entropy loss and the informative loss.
As shown in Table~\ref{tab:ablation}, the performance on MiT and SSv2 suffers by a large margin, indicating that the projection design helps boost training by better utilizing multi-dataset information.

\input{table/projection}
\noindent \textbf{What does the cross-dataset projection layer learn?}
We analyze the cross-dataset projection weights of the K400/312p model and list top 5 concepts for each pair of datasets in Table~\ref{tab:proj}.
We make two observations. First, the top projections are visually similar actions, which confirms our intuition that there are intrinsic relations in the datasets that the model can mine to improve performance. For example, “bending metal” in K400 and “bending” in MIT, “parkour” in K400 and “Capoeira” in Activitynet. Interestingly, “Wiping something off of something” in SSv2 and “cleaning windows” in K400.
Second, the action with the same name may not have the highest weights. In “mit to kinetics”, the “sneezing” action ranks 5th in the projection weights, suggesting that there might be discrepancies of the same concept in different datasets.
These observations are interesting and one may compare the learned weights with textual semantic relations (like those in ConceptNet). We leave this to future work. 

\noindent \textbf{Does the additional ActivityNet data help?}
In previous methods like CoVER and PolyViT, the ActivityNet dataset has not been used.
In this experiment, we investigate the important of the ActivityNet dataset by removing it from the training set.
From Table~\ref{tab:ablation}, we can see that the performance across all datasets drop by a small margin, indicating our superior results compared to CoVER (see Table~\ref{tab:k400} and Table~\ref{tab:k700}) come from the proposed robust training paradigm rather than the additional data.





\subsection{Discussion}
\label{sec:discussion}
By multi-dataset training transformers on various datasets, we obtain competitive results on multiple action datasets, without large-scale image datasets pre-training.
Our method, \fancyname, is parameter-efficient and does not require hyper-parameter tuning.
Current limitations of our experiments are that we have not tried co-training with image datasets such as ImageNet-21K~\cite{deng2009imagenet}. 
Hence we do not know how much performance gain that would entail.
We plan to explore this in future work.
In addition, we have not tried training larger model with FLOPs on par with state-of-the-art or other backbone architectures (e.g. CNNs) due to limitation of our computational resources.
Hence we are not sure how our algorithm would behave with these models.
We have not explicitly explored how temporal modeling could benefit from multi-action-dataset training, which we leave for future work.
Although our model is trained on multiple datasets, potential dataset biases can still cause negative societal impact in real-world deployment, as the datasets we have do not fully represent all aspects of human actions.
\section{Introduction}


Human vision can recognize video actions efficiently despite the variations of scenes and domains. 
Convolutional neural networks (CNNs) \cite{cnn-lecun-eccv10,cnn-dtran-iccv15,cnn-Carreira-Zisserman-cvpr17,DBLP:journals/corr/SigurdssonDFG16,DBLP:conf/nips/FeichtenhoferPW16,liang2020spatial} 
effectively 
exploit 
the power of modern computational devices and employ spatial-temporal filters to recognize actions, which
considerably outperform
traditional models such as oriented filtering in space-time (HOG3D) \cite{DBLP:conf/bmvc/KlaserMS08}.
However, due to the high variations in space-time, the
state-of-the-art accuracy of action recognition is still far from being
satisfactory, compared with the success of 2D CNNs in image recognition \cite{he2016deep}.

Recently, vision transformers
such as 
ViT \cite{dosovitskiy2020image}, and MViT \cite{fan2021multiscale} that are based on the self-attention~\cite{vaswani2017attention} mechanism
were 
proposed to tackle the problems of image and video recognition,
and achieved impressive performance. 
Instead of modeling pixels
as 
CNNs, transformers apply attentions on top of visual tokens. 
The inductive bias of translation
invariance 
in CNNs makes it require less training data than
attention-based transformers in general. 
In contrast, 
transformer has the advantage that it 
can better leverage  `big data', leading to improved accuracy than CNNs. 
We have
witnessed 
a rapid growth in video datasets \cite{kay2017kinetics} in recent years, which would make up for the shortcomings of data-hungry transformers.
The video data has not only grown in quantity from hundreds to millions of videos~\cite{monfort2019moments},  but also evolved from simple actions such as handshaking to complicated daily activities from the Kinetics-700 dataset~\cite{carreira2019short}.
Meanwhile, transformers combined with low-level convolutional operations have been proposed \cite{fan2021multiscale} to further improve the
efficiency and accuracy. 


Due to the data-hungry nature of transformers, most transformer-based models for action recognition requires large-scale pre-training with image datasets such as ImageNet-21K~\cite{deng2009imagenet} and JFT-3B~\cite{DBLP:journals/corr/abs-2106-04560} to achieve good performance.
This pre-training and fine-tuning training paradigm is time-consuming and it is not parameter-efficient, meaning that for each action dataset, a new model need to be trained end-to-end.
Different from large image datasets such as ImageNet-21K that covers a wide range of object classes, currently the most diverse action dataset, Kinetics-700, only contains 700 classes.
Each action dataset may be also limited to a certain topic or camera views.
For example, Moments-in-Time~\cite{monfort2019moments} only contains short actions that happen in three  seconds and Something-Something-v2~\cite{goyal2017something} focuses on close-up camera view of person-object interactions.
These dataset biases might hinder  models trained on a single dataset to generalize and be used in practical applications.
These challenges in action datasets make learning a general-purpose action model very difficult.
An ideal model should be able to cover a wide range of action classes meanwhile keeping the computation cost low.
However, \textit{simply combining all these datasets to train a joint model does not lead to good performance} \cite{likhosherstov2021polyvit}.
In previous work~\cite{zhang2021co}, 
the authors 
have shown the benefit of training a joint model using multiple action datasets but their method requires large-scale image datasets such as ImageNet-21K~\cite{deng2009imagenet} and JFT-3B~\cite{DBLP:journals/corr/abs-2106-04560}, which is not available to the research community.


\begin{figure}[t!]
	\centering
		\includegraphics[width=1.051\textwidth]
 
  {figures/fig1a-crop.pdf}
	\caption{Overview of our multi-dataset training framework. We propose to utilize the intrinsic relations between classes across different action datasets. As we can see, the two video examples from Kinetics and Moments-in-Time datasets, respectively, show that samples from these two classes can be used to train both classification heads.
	The videos from multiple action datasets are input to the MViTv2 (Section~\ref{sec:mvit}) backbone and the model is trained jointly.
	The informative loss is applied to maximize the information content of the embedding from the backbone and the projection loss is applied to learn the intrinsic relations (Section~\ref{sec:robust}).
	}
	\label{fig:overview}
\end{figure}

In this paper,
we propose a general training paradigm for \textbf{Multi}-dataset \textbf{Train}ing of robust action recognition models, \fancyname.
Our method is designed to learn robust and informative feature representations in a principled
approach, using the informative loss for regularization.
We do not assume the availability of large-scale image datasets pre-training (although one can certainly take advantage of that).
Since there are intrinsic relations between different classes across different action datasets (See Fig.~\ref{fig:overview} for examples of similar classes from two datasets), we propose a projection loss to mine such relations such that the whole network is trained to avoid over-fitting to certain dataset biases.
Finally, all proposed loss terms are weighted using learned parameters. Thus, no hyper-parameter tuning is needed.
Our empirical findings as shown in Table~\ref{tab:k400} indicate that our robust training method can consistently improve model backbone performance across multiple datasets.
We show that our model can achieve competitive results compared to state-of-the-art methods, even without large-scale image dataset pre-training, and with a lower  computational cost. 

The main contributions of this paper are thus three-fold:   
\begin{itemize}
\itemsep 0cm
\item
To our knowledge, 
this is the first work to introduce informative representation regularization into multi-dataset training for improving action recognition.
\item We propose an effective
approach 
to mine intrinsic class relations in multi-dataset training by introducing the projection loss.
\item Our method requires
negligible computation 
overhead during training and no additional computation during inference to the backbone network. Extensive experiments on 
various 
datasets suggest our method can 
 consistently 
improve performance.
\end{itemize}
\section{Method}

Our method is
built upon the backbone of 
the improved Multi-scale Vision Transformers (MViTv2)~\cite{li2021improved,fan2021multiscale}. 
Note that 
our approach
works with any action recognition backbones.
Given videos from multiple datasets during training, the model backbone takes the video frames and produces feature embeddings for each video.
The same number of Multi-layer Perceptron (MLP) as the datasets are constructed as model heads to predict action classes for each dataset.
To facilitate robust cross-dataset training, we propose two loss terms, namely, \textit{the informative loss and projection loss}.
The informative loss aims to maximize the embeddings' representative power.
The projection loss, with the help of multiple cross-dataset projection layers,  guides the model to learn intrinsic relations between classes of different dataset, hence the model heads can be trained jointly.
See Fig.~\ref{fig:overview} for an overview of our framework.
In this section, we first briefly describe the MViTv2 backbone design, and then present our proposed robust cross-dataset training paradigm.

\begin{comment}
\begin{figure}[t!]
	\centering
		\includegraphics[width=0.47\textwidth]{figures/mvitv2_block.drawio.png} 
	\caption{The MViTv2 Block. The residual connection for pooled query tensor (red arrow) and the residual 3D convolution operation outside the Multi-head Pooling Attention block are additions to the MViTv1~\cite{fan2021multiscale} design. The linear layer in the residual connection of the MLP block is only needed when the output embedding dimension is different.
	Compared to the MViTv2 paper~\cite{li2021improved}, we do not use the decomposed relative embedding.
	}
	\label{fig:mvit}
\end{figure}

\end{comment}

\subsection{The MViTv2 Backbone}
\label{sec:mvit}
Our model is based on the improved multi-scale vision transformers (MViTv2)~\cite{fan2021multiscale,li2021improved}, which learns a hierarchy from dense (in space) and simple (in channels) to coarse and complex features.
The series of work of vision transformers~\cite{dosovitskiy2020image} (ViTs) follows the basic self-attention architecture~\cite{vaswani2017attention} originally proposed for machine translation. 
\begin{comment}
In contrast to natural language which can be directly tokenized into words, given the input video $\mathbf{V} \in \mathbb{R}^{T \times H \times W \times 3}$, ViTs extract tokens by splitting the video into 
$N = \lfloor T/t \rfloor \times \lfloor H/h \rfloor \times \lfloor W/w \rfloor$
non-overlapping patches, $\{ \mathbf{v_1}, \cdots, \mathbf{v_N} \in \mathbb{R}^{t \times h \times w} \}$.
Each patch is then projected into a patch embedding by a 3D convolution operator $E$.
All patch embeddings are then concatenated into a sequence, and separate learnable spatial-temporal positional embeddings $\mathbf{p_s}, \mathbf{p_t}$ are also
added to this sequence. 
The patch embedding process is denoted by:
\begin{equation}
    \mathbf{X_0} = [ \mathbf{E}\mathbf{v_1} \cdots  \mathbf{E}\mathbf{v_N}] + P(\mathbf{p_s}, \mathbf{p_t}) \in \mathbb{R}^{N \times d_{p}}
\end{equation}
The $P$ function extends the separate position embedding into the length of the sequence by repeating at the same spatial or temporal location.
$d_{p}$ is the dimension of the patch embedding.
\end{comment}
The key component of the MViTv1 model~\cite{fan2021multiscale} is the Multi Head Pooling Attention (MHPA), which pools the sequence of latent tensors to reduce the spatial or temporal dimension of the feature representations.
In MViTv2~\cite{li2021improved}, a residual connection in MHPA for the pooled query tensor and a decomposed relative position embedding~\footnote{We did not implement this part as the code were not available at the time of writing (March 2022).} are added.
In this paper, we use 3D convolution as the pooling operation.
Please refer to supplemental material for a visualization of the MViTv2 block.
Each MViTv2 block consists of a multi-head pooling attention layer (MHPA) and a multi-layer perceptron (MLP), and the residual connections are built in each layer.
The feature of each MViTv2 block is computed by:
\begin{equation}
\begin{split}
    X_1 &= {\rm MHPA}(LN(X)) + Pool(X) \\
    Block(X) &= {\rm MLP}(LN(X_1)) + X_1,
\end{split}
\end{equation}
where $X$ is the input tensor to each block. 
Multiple MViTv2 blocks are grouped into stages to reduce the spatial dimension while increase the channel dimension. 
The full backbone architecture 
is listed in supplementary material.



\textbf{Classification head.} 
For the action recognition problem, the model produces C-class classification logits by first averaging the feature tensor from the last stage along the spatial-temporal dimensions (we do not use the [CLASS] token in our transformer implementation), denoted as $\mathbf{z} \in \mathbb{R}^{d}$. 
A linear classification layer is then applied on the averaged feature tensor to produce the final output, $\mathbf{y} = \mathbf{W_{\rm out}}\mathbf{z} \in \mathbb{R}^{C}$.

\begin{comment}
\textbf{Pre-training and finetuning.}
In the standard training paradigm for action recognition, models are pretrained using image datasets (ImageNet~\cite{deng2009imagenet} or large-scale datasets like JFT-3B~\cite{DBLP:journals/corr/abs-2106-04560}) and then finetune on the target action recognition dataset.
For CNN-based backbones, model weight inflation~\cite{kay2017kinetics} is utilized to adapt the model trained on 2D image data to 3D video input.
For transformer-based backbones, as the inputs are tokenized into a sequence, to adapt the model from image pretraining, the positional embeddings are interpolated to account for the additional temporal dimension before the finetuning.
\end{comment}

\textbf{Multi-dataset training paradigm.}
In general, to facilitate multi-dataset training of $K$ datasets, the same number of classification heads are appended to the feature embeddings.
The $k$-th dataset classification output is defined as
$\mathbf{Y}_k = h_k(\mathbf{Z}; \mathbf{W}_{k}) \in \mathbb{R}^{B \times C}$, 
where $h_k$
can 
be a linear layer or a MLP and $\mathbf{W}_{k}$ is the layer parameter.

\subsection{  {\textbf MultiTrain}: Robust Multi-dataset Training}
\label{sec:robust}





Our training process fully leverages different action recognition datasets by enforcing an \textbf{informative loss} to maximize the expressiveness of the feature embedding and a \textbf{projection loss} for each dataset that mines the intrinsic relations between classes across other datasets.
We then use uncertainty to weight different loss terms without the need for any hyper-parameters.

\textbf{Informative loss.} Inspired by the recently proposed VICReg~\cite{bardes2021vicreg} and Barlow Twins~\cite{zbontar2021barlow} methods  for self-supervised learning in image recognition, we propose to utilize an informative loss function with two terms, \textbf{variance and covariance}, to maximize the expressiveness of each variable of the embedding.
This loss is applied to each mini-batch, without the need for batch-wise nor feature-wise normalization.
Given the feature embeddings of the mini-batch, $\mathbf{Z} \in \mathbb{R}^{B \times d}$, an expander (implemented as a two-layer MLP) maps the representations into an embedding space for the informative loss to be computed, denoted as $\mathbf{Z'} \in \mathbb{R}^{B \times d}$.
The \textbf{variance loss} is computed using a hinge function and the standard deviation of each dimension of the embeddings by:
\begin{equation}
   \mathcal{L}^v = \frac{1}{d}\sum^{d}_{j=1} \text{max}\Bigl( 
        0, 1 - \sqrt{\frac{\sum(\mathbf{Z'}_{ij} - \bar{\mathbf{Z'}}_{:j})}{d-1} + \epsilon} 
   \Bigr),
\end{equation}
where $:$ is a tensor slicing operation that extracts all elements from a dimension, and $\bar{\mathbf{Z'}}_{:j}$ is the mean over the mini-batch for $j$-th dimension. $\epsilon$ is a small scalar preventing numerical instabilities.
With random sampling videos across multiple datasets for each batch, this criterion encourages the variance of each dimension in the embedding to be close to $1$, preventing embedding collapse~\cite{zbontar2021barlow}. 
The \textbf{covariance loss} $c(\mathbf{Z'})$ is defined as:
\begin{equation}
\begin{split}
    C(\mathbf{Z'}) &= \frac{1}{n - 1}\sum^{n}_{i=1} (\mathbf{Z'}_i - \bar{\mathbf{Z'}})(\mathbf{Z'}_i - \bar{\mathbf{Z'}})
    ^{ 
    T 
    } \;,\; 
    \text{where} \; \bar{\mathbf{Z'}} = \frac{1}{n}\sum^{n}_{i=1}\bar{\mathbf{Z'}}_{i} \\
    \mathcal{L}^c &= \frac{1}{d}\sum_{i \neq j}[C(\mathbf{Z'})]^2_{i,j}
\end{split}
\end{equation}
Inspired by VICReg~\cite{bardes2021vicreg} and Barlow Twins~\cite{zbontar2021barlow}, we first compute the covariance matrix of the feature embeddings in the batch, $C(\mathbf{Z'})$, and then define the covariance term $\mathcal{L}^c$ as the sum of the squared off-diagonal coefficients of $C(\mathbf{Z'})$, scaled by a factor of $1/d$.


\textbf{Projection Loss.} In previous works~\cite{zhang2021co,likhosherstov2021polyvit}, the intrinsic relations between classes from across different datasets have been mostly ignored during training. 
We believe that samples in one dataset can be utilized to train the classification head of other datasets. As shown in Fig.~\ref{fig:overview}, the ``Clean and jerk'' video sample from Kinetics can be considered as a positive sample for ``Weightlifting'' in Moments-in-Time as well (but not vice versa).
Based on this intuition, we propose to add a directed projection layer for each pair of datasets for the model to learn such intrinsic relations.
One can also initialize the projection using prior knowledge but it is out-of-scope for this paper.
Given the output from the $k$-th dataset classification output, the projected classification output is defined as:
\begin{equation}
    \mathbf{Y'}_k = \mathbf{Y}_{k} + \sum^{K-1}_{i \neq k} \mathbf{W}^{proj}_{ik}\mathbf{Y}_{i} \in \mathbb{R}^{C_k},
\end{equation}
where $C_k$ is the number of classes for the $k$-th dataset and $\mathbf{W}^{proj}_{ik}$ is the learned directed class projection weights from $i$-th to $k$-th dataset. 
In this paper we only consider a linear projection function.
We then use the ground truth labels of the $k$-th dataset to compute standarad cross-entropy loss:
\begin{equation}
    \mathcal{L}_k = - \sum^{C_k}_{c=1} \hat{\mathbf{Y}}_{k,c} \log(\mathbf{Y'}_{k,c}),
\end{equation}
where $\hat{\mathbf{Y}}_{k,c}$ is the ground truth label for the $c$-th class from the $k$-th dataset.

\textbf{Training.} We jointly optimize the informative loss and the projection loss during multi-dataset training. 
To avoid tuning loss weights of different terms, we borrow the weighting scheme from multi-task learning~\cite{kendall2018multi} and define the overall objective function as:
\begin{equation}
   
   
    %
    %
    \mathcal{L}(\sigma) = \mathcal{L}^v + \mathcal{L}^c + \sum^{K}_{k=1} \frac{1}{2\sigma_k^2}\mathcal{L}_k+\log\sigma_k,
\end{equation}
where $\sigma$ is a vector of learnable parameters of size $K$ (the number of datasets) for each projection loss term. This avoids the need to manually tune loss weights for different datasets.







\section{Related Work}
We review some work that is closest to ours. 

\noindent\textbf{CNNs and Vision Transformers.}
CNNs work as the standard backbones throughout computer vision tasks for image and video. Various effective convolutional neural architectures have been raised to improve the precision and efficiency (\textit{e.g.}, VGG \cite{simonyan2014very}, ResNet \cite{he2016deep} and DenseNet \cite{huang2017densely}). Although CNNs are still the primary models for computer vision, the Vision Transformers
have already
shown their enormous potential. 
Vision Transformer (ViT \cite{dosovitskiy2020image}) directly applies the architecture of Transformer on image classification and gets encouraging performance. ViT and its variants~\cite{arnab2021vivit,liang2022stargazer,bertasius2021space,fan2021multiscale,liu2021video,yan2022multiview} 
achieve outstanding results in both image and video processing in recent years.

\noindent\textbf{Action Recognition$/$Classification.}
The research of action recognition has advanced with both new datasets and new models. 
One of the largest modern benchmarks for action recognition is the Kinetics dataset \cite{kay2017kinetics}. 
The Kinetics dataset proposes a
large 
benchmark with more categories and more videos (\textit{e.g.}, 400 categories 160,000 clips in  \cite{kay2017kinetics} and 700 categories in ~\cite{carreira2019short}) as 
more challenging 
benchmarks  compared to previous datasets like UCF-101~\cite{ucf101}. 
The Moments-in-Time~\cite{monfort2019moments} (MiT) dataset provides a million short video clips that cover 305 action categories.
Note that it is
infeasible 
for 
Kinetics and MiT datasets
to 
cover 
all the possible actions in all possible scales. 
For example, surveillance actions~\cite{chen2019minding,chang2019mmvg} are missing in the two datasets. 
Many new approaches ~\cite{tran2018closer,zhao2018trajectory,lin2019tsm,feichtenhofer2019slowfast,yang2020temporal,liang2020spatial,chen2019minding,chang2019mmvg} have been carried out on these datasets, of which the SlowFast network \cite{feichtenhofer2019slowfast} and MViT \cite{fan2021multiscale} obtain
promising 
performance. 
We can see that the trend of action recognition in the last two decades is to collect 
larger 
datasets (\textit{e.g.}, Kinetics) 
and 
build
models with a larger capacity.

\noindent\textbf{Multi-dataset Co-Training.}
Previously, multi-dataset co-training has been explored in the image domain such as detection~\cite{zhou2022simple,wang2019towards} and segmentation~\cite{lambert2020mseg}.
Several works~\cite{munro2020multi,chen2019temporal, Song_2021_CVPR,hinami2018multimodal}
were proposed
to combine multiple video datasets for training. 
Larger datasets
often deliver 
better results. Combining
multiple 
datasets to boost data size, and
improve the final performance \cite{ghadiyaram2019large}, and the simultaneous use of multiple datasets
is 
also likely to 
alleviate 
the damaging impact of dataset bias. OmniSource \cite{duan2020omni} utilizes 
web images as part of the training dataset to expand the diversity of the training data to reduce dataset bias.
VATT~\cite{akbari2021vatt}
uses 
additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets.
CoVeR~\cite{zhang2021co} combines image and video training even during the finetuning stage and reports significant performance boost compared to single-dataset training.
PolyViT~\cite{likhosherstov2021polyvit} further extends to training with image, video and audio datasets using different sampling procedures.
In this paper, we propose a simple yet effective way (no multi-stage training, no complex dataset-wise sampling and hyper-parameter tuning) for multi-action-dataset training, without the use of any image or additional data from other modality.

\noindent\textbf{Video Domain Generalization (VDG).}
Our work is also related but different from video domain generalization~\cite{zhou2022domain}.
The key distinction is that our goal is to train a single model on multiple related tasks (multiple action datasets) such that the model performs well on the same set of tasks, whereas VDG aims to generalize a model to unseen out-of-distribution target domain~\cite{yao2021videodg,chen2019temporal,choi2020shuffle,da2022dual,tzeng2017adversarial}. 
These models still suffer from problem of parameter-inefficiency, meaning that separate models are needed for different target datasets.







\section{Conclusion}
In this paper, we present \fancyname, a robust multi-dataset training approach that maximizes information content of representation and learns intrinsic relations between individual datasets.
Our method can train parameter-efficient models that perform well across multiple datasets.

{
\bibliographystyle{plainnat.bst}







