\section{Introduction}
\noindent We benchmark contemporary action recognition models (TSN~\cite{wang2018_TemporalSegmentNetworks}, TRN~\cite{zhou2017_TemporalRelationalReasoning}, and TSM~\cite{lin2018_TSMTemporalShift}) on the recently introduced EPIC-Kitchens dataset~\cite{damen2018_ScaleingEgocentricVision} and release pretrained models on GitHub\footnote{\href{https://github.com/epic-kitchens/action-models}{\texttt{github.com/epic-kitchens/action-models}}} for others to build upon.
In contrast to popular action recognition datasets like Kinetics~\cite{kay2017_KineticsHumanAction}, Something-Something~\cite{goyal2017_SomethingSomethingVideo}, UCF101~\cite{soomro2012_UCF101DatasetHuman}, and HMDB51~\cite{kuehne2011_HMDBLargeVideo}, EPIC-Kitchens is shot from an egocentric perspective and captures daily actions in-situ.
In this report, we aim to understand how well these models can tackle the challenges present in this dataset, such as its long tail class distribution, unseen environment test set, and multiple tasks (verb, noun and, action classification).
We discuss the models' shortcomings and avenues for future research.


\section{Models}

\noindent We benchmark 3 models: Temporal Segment Networks~(TSN)~\cite{wang2018_TemporalSegmentNetworks}, Temporal Relational Networks (TRN)~\cite{zhou2017_TemporalRelationalReasoning}, and Temporal Shift Module (TSM) based networks~\cite{lin2018_TSMTemporalShift}, including a variety of their variants.
These models are evaluated under a uniform training and testing regime to ensure the results are directly comparable.
TSN is the earliest model of the three and both TRN and TSM can be viewed as evolutionary descendants of TSN, integrating temporal modelling.
In the following paragraphs, we provide an explanation of how network inputs are sampled and a brief summary of the design of each network.

\paragraph{Sampling} Inputs to the models, \textit{snippets}, are sampled according to the TSN sampling strategy. An action clip is split into $n$ equally sized \textit{segments} and a snippet is sampled at a random position within each of these.
For an RGB network, the input is a single frame and for a flow network it is a stack of 5 $(u,v)$ optical flow pairs (proposed in the two-stream CNN~\cite{simonyan2014_TwoStreamConvolutional}).

\paragraph{TSN~\cite{wang2018_TemporalSegmentNetworks}} Temporal Segment Networks propagate each snippet through a 2D CNN backbone and aggregate the class scores across segments through average or max pooling.
As a consequence, TSN is unable to learn temporal correlations across segments.
TSN is typically trained on RGB and optical flow modalities and combined by late-fusion.

\paragraph{TRN)~\cite{zhou2017_TemporalRelationalReasoning}} Temporal Relation Networks propagate snippets through a 2D CNN, like in TSN, up to the pre-classification layer. These produce features rather than class confidence scores.
In order to support inter-segment temporal modelling, these segment-level features are then processed by a modified relational module~\cite{santoro2017_SimpleNeuralNetwork} sensitive to item ordering.
Two variants of the TRN module exists: a single scale version which computes a single $n$-segment relation, and a multi-scale (M-TRN) variant which computes relations over ordered sets of segment features of size 2 to $n$.
Once the relational features have been computed, they are summed and fed to a classification layer.

\paragraph{TSM~\cite{lin2018_TSMTemporalShift}} These networks functionally operate just like TSN, snippets are sampled per segment, propagated through the backbone, and then averaged.
However, unlike TSN, the backbone is modified to support reasoning across segments by shifting a proportion of the filter responses across the temporal dimension.
This opens the possibility for subsequent convolutional layers to learn temporal correlations.

\section{Experiments}

\noindent In this section, we examine how a variety of factors impact model performance such as backbone choice, input modality, and temporal support.
We analyse model performance across tasks from the perspective of the more defining characteristics of the dataset: the long-tail class distribution, and the domain gap between the seen and unseen kitchen test sets.


\subsection{Experimental details}
\paragraph{Tasks} EPIC-Kitchens has three tasks within the action recognition challenge: classifying the verb, noun, and action (the verb-noun pair) of a given trimmed video.
We follow the approach in~\cite{damen2018_ScaleingEgocentricVision}, and  replace the classification layer of each model with two output FC layers, one for verbs $v$ and one for nouns $n$.
The models are trained with an averaged softmax cross-entropy loss over each classification layer: $\mathcal{L} = 0.5 (\mathcal{L}_n + \mathcal{L}_v)$.
We obtain action predictions from verb and noun predictions assuming the tasks are independent.
Later, we examine the impact of integrating action priors computed from the training set for action classification.
Performance on these tasks are evaluated on two test sets: seen kitchens (S1) and unseen kitchens (S2).
The unseen kitchens test set contains videos from novel environments, whereas the seen kitchens split contains videos from the same environments used in training.

\input{table-backbone-comparison}

\paragraph{Training}
We train all models with a batch size of 64 for 80 epochs using an ImageNet pretrained model for initialisation.
SGD is used for optimisation with momentum of 0.9.
A weight decay of $5\times 10^{-4}$ is applied and gradients are clipped at 20.
We replace the backbone's classification layer with a dropout layer, setting $p = 0.7$.
We train RGB models with an initial learning rate (LR) of 0.01 for ResNet-50 based models and 0.001 for BN-Inception models.
flow models are trained with an LR of 0.001.
These LRs were the maximum we could achieve whilst maintaining convergence.
The LR is decayed by a factor of 10 at epochs 20 and 40.
\paragraph{Testing}
Models are evaluated using 10 crops (center and corner crops as well as their horizontal flips) for each clip.
The scores from these are averaged pre-softmax to produce a single clip-level score.
Fusion results are obtained by averaging the softmaxed scores obtained for each modality.

\subsection{Results}


\paragraph{Backbone choice}
To choose a high performing backbone, we compare BN-Inception~\cite{ioffe2015_BatchNormalizationAccelerating,szegedy2015_GoingDeeperConvolutions} to ResNet-50~\cite{he2016_DeepResidualLearning} across the 3 models, training and testing with 8 segments.
We did not test TSM with BN-Inception as the authors state that the shift module is harmful unless placed in a residual branch~\cite{lin2018_TSMTemporalShift}.
The top-1/5 accuracy across tasks is reported in \cref{tab:backbone-comparison} where the results show ResNet-50 to be superior to BN-Inception in 14/18 cases when examining top-1 action accuracy across both test sets.

\paragraph{Aggregate performance}
We now compare models with ResNet-50 backbones across tasks in \cref{tab:backbone-comparison} using top-1/5 accuracy.
On the verb task, an intrinsically more temporal problem than classifying nouns, both M-TRN and TSM out perform TSN, especially when operating on RGB frames instead of flow.
This can be explained by TSN's inability to learn inter-segment correlations as only average or max pooling is used in aggregating class scores across segments.
TSN flow models outperform their RGB counterparts; this can be attributed to the network being passed temporal information in the form of stacked optical flow frames.
The 2D convolutions inside the network can learn temporal relations within the stack.
Both \mbox{(M-)}TRN and TSM flow models outperform TSN flow showing that inter-segment reasoning is complimentary to intra-segment reasoning.

Unlike verb classification, noun classification does not rely on temporal modelling \textit{as much} since objects can be recognised from a single frame.
TSM and TSN perform best on this task, with TRN models lagging 2--3\% points behind.
A possible explanation for the observed drop is that the relation module within TRN places heavy emphasis on extracting temporal relational information, which is of little relevance in recognising objects.
Noun performance drops considerably across models when switching from RGB to flow as the former is a much better modality for recognising objects.
Unexpectedly, TSM improves top-1 noun accuracy by 1\% point over TSN.
Additionally, we find all fusion models improve over the RGB models alone.
We hypothesise that the temporal information here is helping disambiguate the action relevant object from those that are simply present in the environment.

Classifying actions, the joint task of classifying both verb and noun, is clearly very challenging, with the best top-1 accuracy on actions being 29.9\% and 17.9\% for the seen and unseen test set respectively.
Even at top-5, the best results are 49.8\% and 32.8\%.
Despite flow's superior results on verb classification on the unseen test set, the inferior noun performance drags flow models below RGB models on both test sets.


An enduring approach, pioneered by the 2SCNN~\cite{simonyan2014_TwoStreamConvolutional}, has been to ensemble networks trained on different modalities through late fusion at test-time.
Averaged across all model variants, fusing both modalities results in a 2.9\%, 5.8\%, and 9.7\% relative improvement over the best performing single modality model for verb, noun, and action classification respectively.
The best model on the seen test set is TSM fusion, followed by M-TRN fusion. On the unseen test set, the trend is reversed with M-TRN out-performing TSM.

\paragraph{Novel environment robustness}
It is interesting to examine the relative drop in model performance from the seen to unseen test set to determine the models' ability to generalise to new environments.
\cref{tab:backbone-comparison} shows that flow models are more robust to the domain gap between the seen kitchens and unseen kitchens test sets only suffering an average 22\% relative drop in top-1 action accuracy compared to a 44\% drop for RGB models, and 39\% for fused models.
The domain gap on fused models suggests that the RGB model's predictions dominates those of the flow model.
We find that flow models consistently outperform RGB models for verb classification on the unseen test set.
We hypothesis this is due to the absence of appearance information in optical flow, forcing flow models to focus on motion.
Motion is more environment-invariant and salient to the classification of verbs than the visual cues the RGB models will use.


\paragraph{Class performance analysis}
To further understand the differences between models, we look at confusion amongst the top-20 most frequent classes in training in \cref{fig:long-tail-confusion-matrices}.

The verb classification results show the top-3 verbs (accounting for 53\% of the actions in training) dominate predictions due to the dataset imbalance, with this effect being especially pronounced in the unseen test set.
Classes outside the top-20 are rarely correctly classified and instead are classified into one of the majority classes.
The fine-grained nature of the verbs seems to pose challenges, particularly in the unseen test set, with similar classes being confused, such as `move' with `put'/`take', `turn' with `mix', and `insert' with `put'.
TSN shows increased confusion between classes that differ primarily in their temporal aspects (e.g. `put' vs `take'), compared to TSM and TRN.
The generic class `move' is hardest to classify, for all models.

For noun classification, the confusion matrices show the models don't struggle as much to classify less frequent classes compared to verb classification. 
This is likely as a result of the models benefiting from pretraining on the large-scale ImageNet dataset.
However, when fine-tuned, some overfitting to seen environments is observed, as the unseen test set matrices demonstrate that the models generalise less well to new objects.
Like the verb results, the fine-grained classes pose a challenge with confusion between similar objects like `fork' with `spoon', and `bowl' with `plate' occurring.
Another interesting contrast between the verb and noun tasks is that the top-20 verbs almost never get misclassified into any of the classes outside the top-20, whereas for nouns, there are more misclassifications of top-20 nouns into the long-tail.

\input{figure-confusion-matrices}

\input{figure-qualitative-examples-temporal}

For action classification, the models perform well on frequent actions, but suffer more misclassifications into the long tail than nouns (as evidenced by the confusion into `other' classes).
Confusion within the top-20 actions highlight an issue not visible from the verb and noun matrices: semantically identical classes like `turn-on tap' are confused with `open tap'.
Whilst these are different classes in the dataset, they refer to the same action.
This highlights an issue with the open vocabulary annotation process employed by the dataset: annotators may use different phrases for describing the same action.


We provide qualitative examples in \cref{fig:qualitative-results} where TSM and M-TRN correctly classify the actions, but TSN fails.
In the top example TSN confuses `put' and `take' as a result of averaging the scores across segments, and thus discarding temporal ordering.
M-TRN and TSM show a much larger disparity between the scores of these classes indicating they have better learnt the difference.
In the bottom example, TSN again struggles to correctly classify the action.
The temporal bounds are quite wide and capture frames just after someone has picked up a bowl, they then open the cupboard and are about to place the bowl.
M-TRN and TSM, through their ability to draw correlations across segments, are able to disambiguate the correct class from the action which came before and comes after.


\paragraph{Temporal support}
How many frames/optical flow snippets does the network need to see before performance saturates?
We examine the answer to this question by training models with different numbers of segments, presenting results in \cref{fig:temporal-support-performance}.
Overall, flow models benefit more from increasing temporal support, showing monotonically increasing performance, unlike RGB models whose performance saturates at 8 frames, even dropping for the action task when using 16 frames.
Curiously, the RGB TSM model is severely harmed by using 16 segments instead of 8, unlike its flow counterpart whose performance improves moving from 8 to 16 segments.
This is in contrast to the authors results on Kinetics and Something-something which show an improvement in using 16 frames over 8.
This drop was consistently observed across varying LRs suggesting this is not due to a suboptimal learning rate.


\input{figure-temporal-support-performance}
\paragraph{Action priors}
In the previous sections, action predictions have been computed assuming independence between verbs and nouns
\begin{equation}
  P(A = (v, n)) = P(V = v)P(N = n),
\end{equation}
however this is na\"ive as verb-noun combinations aren't all as equally likely.
For example, it is much more probable to observe `cut onion' than `cut chopping board'.
In Long-term Feature Banks~\cite{wu2019_LongTermFeature}, the authors propose leveraging the prior knowledge of verb-noun co-occurrence in the training set $\mu(v, n)$ to weight the action prediction, \ie{}
\begin{equation}
P(A = (v, n)) \propto {\mu(v, n)}{P(V=v)}{P(N=n)}.
\label{eq:action-prior}
\end{equation}
The method in Eq.~\ref{eq:action-prior} does not allow zero-shot learning of unseen verb-noun combinations.
To remedy this, we apply Laplace smoothing to $\mu$ to avoid eliminating the possibility of recognising unseen actions.
We evaluate the relative benefit of using action priors in \cref{tab:action-prior}, finding it provides little benefit on the seen test set, but improves performance on the unseen test set by $\sim1\%$ point for top-1 accuracy.

\input{table-action-prior}

\section{Released Models}
All models required to reproduce the results in \cref{tab:backbone-comparison} are made available.
We release both RGB and flow models whose predictions can be combined to produce fusion results. To reproduce or compare to these results, the test set predictions should be submitted to the EPIC-Kitchens leaderboard\footnote{\url{https://epic-kitchens.github.io/2019\#challenges}} to calculate the performance.

The complexity of the models using ResNet-50 backbone is compared in \cref{tab:model-complexity},  

\input{table-model-params-flops.tex}

\section{Conclusion}

We have benchmarked 3 contemporary models for action recognition and analysed their performance, highlighting areas of good and poor performance.
TSM is competitive with M-TRN, and both outperform TSN.
These results highlight the necessity for temporal reasoning to recognise actions in EPIC-Kitchens.
Yet, the relatively low scores for top-1 accuracy show the challenge is far from solved.
Particular issues common to all models are the long-tailed nature of the dataset, fine-grained classes, and difficulty in generalising to unseen environments where we observe a significant drop across all metrics.


{\small
\bibliographystyle{ieee-fullname}








