
\section{Introduction}\label{sec:intro}







One major gap between artificial intelligence and human intelligence lies in their abilities to generalize and perform well on new tasks with limited annotations. 
Recent advances in large-scale pre-trained generative language models~\citep{raffel2019t5, brown2020gpt3, zhang2022opt,lewis2019bart} have shown promising few-shot capabilities~\citep{zhao2021calibrate,peng2020few,pcia} %
in understanding natural language. However, few-shot video-language understanding is still in its infancy. 
A particular limitation of most recent video-language pretraining frameworks~\citep{ li2020hero, ClipBERT, xu2021videoclip, zellers2021merlot, zellers2022merlot, li2021alpro, yang2022code} is that they are encoder-only, which means they do not have the ability to generate text from videos for purposes such as captioning~\citep{xu2016msrvtt,wang2019vatex}, question answering~\citep{xu2017video}, and future prediction~\citep{lei2020vlep}. 
Meanwhile, unified video-language models~\citep{luo2020univl, seo2022mvgpt} that are capable of language decoding still rely heavily on finetuning using a large number of manually annotated video-text pairs, therefore cannot adapt quickly to unseen tasks. %
Few-shot video-to-text decoding is challenging because %
the natural language supervision for learning video-language representation is typically based on subtitles and automatic speech recognition (ASR) transcripts~\citep{miech2019howto100m,zellers2021merlot}, which differ significantly from downstream tasks in terms of distribution  and may have poor semantic alignment across vision and text modalities. 

We propose to address this problem by harnessing the few-shot power of large-scale language models, such as GPT-3~\citep{brown2020gpt3} and InstructGPT~\citep{instruct-gpt3}. Our inspiration is derived from the fact that humans are excellent visual storytellers~\citep{huang2016visual},  %
with the ability to piece together a coherent story from a few isolated images.
To mimic this, we propose \textbf{\ours}, a few-shot \textbf{Vid}eo-language Learner via \textbf{I}mage and \textbf{L}anguage models, to use image models to provide information about the visual content in the video (as well as optionally use ASR to represent speech), and then we instruct language models to interpret a video-based summary, answer, or other target output {for diverse video-language tasks}. 
\begin{wrapfigure}{r}{0.5\textwidth}
  \begin{center}
    \includegraphics[width=0.48\textwidth]{image/intro_overview.pdf}
  \end{center}
  \caption{Multiple levels of information in videos.
  }
  \vspace{-10pt}
  \label{fig:overview}
\end{wrapfigure}

Videos contain rich semantics and temporal content at multiple granularities. Similar to static images, videos depict objects, attributes, and events. However, the sequence of frames further conveys object state changes, actions, and events. %
For example, in Figure~\ref{fig:overview}, the frame captions describe static visual features such as \textit{``a person holding a green object in hand''} for the first frame. %
In contrast, the video clip could be correctly captioned as \textit{"a woman makes realistic looking leaves and flowers for a cake"}, or represented as a collection of the objects and events that occur at different timestamps in the video clip, such as \textit{cutting mat} and \textit{flowered design}. Hence, to inform video-level description and queries, we need to represent all of this information and its temporal ordering.

To address these requirements, we propose to decompose a video into three levels: the video output, frame captions, and visual tokens (including object, event, attribute).
One major benefit from this hierarchical video representation is that we can separate the visual and temporal dimensions of a video. We leverage well-trained image-language foundational models at lower levels to collect salient visual features from the sparsely sampled frames. 
Specifically, we leverage pretrained image-language contrastive model CLIP~\citep{clip} to perform visual tokenization, based on the similarity score between frames and tokens of objects, events and attributes. The tokenization is done under the guidance of semantics role labeling~\citep{gardner2018allennlp}, which provides us with candidate events with involved objects and related attributes. Next, in order to capture the overall semantics at the frame level, we employ the pretrained image captioner in image-language model BLIP~\citep{li2022blip} to obtain frame captions. %
We then instruct pretrained language models using in-context learning~\citep{instruct-gpt3, gao2020making, tam2021improving, schick2020exploiting} to interpret visual tokens and frame captions into the target text content. In detail, we temporally order visual tokens and frame captions using specially designed prompts such as \textit{``First...Then...Finally''}, to instruct the pretrained language model to track the changes of objects, events, attributes and frame semantics along the temporal dimension.  %

We show that our approach outperforms both video-language and image-language state-of-the-art baselines on few-shot video captioning and question answering tasks. Moreover, on video-language event prediction, our approach significantly outperform fully-supervised models while using only 10 labeled examples. We further demonstrate that our generative model can benefit broader video-language understanding tasks, such as text-video retrieval, via pseudo label generation. Additionally, we show that our model is highly flexible in adding new modalities, such as ASR transcripts.





\section{Method}
We propose a hierarchical video representation framework which decomposes a video into three levels, i.e., \textbf{visual token level}, \textbf{frame level} and \textbf{video level}. The motivation is to separate the spatial and temporal dimension of a video in order to leverage image-language and language-only foundation models, such as CLIP~\citep{clip} and GPT-3~\citep{brown2020gpt3}. All three levels use a unified textual representation which enables us to leverage the powerful few-shot ability from pretrained language models. %

\subsection{Frame Level: Image Captioning} \label{sec:method_frame_level}
Following \cite{ClipBERT} we first perform sparse sampling to obtain several video frames. Unless otherwise specified, we sample 4 frames for frame level and 8 frames for visual token level. We then feed each frame into a pre-trained image-language model to obtain frame level captions. An example can be found in the \textcolor{framelevel}{blue} part of Figure~\ref{fig:framework}. In our experiments, we use BLIP~\citep{li2022blip}, a recent image-language framework containing both image-grounded encoder and decoder, for generating frame captions. We follow \cite{li2022blip} to do both captioning and filtering on each frame. However, as mentioned in Section~\ref{sec:intro}, videos contain rich semantics and temporal contents at multiple granularities.
It is not enough to generate video-level target text such as video captions solely based on frame captions. Thus, we further perform visual tokenization for each frame to capture features at a finer granularity.

\subsection{Visual Token Level: Structure-Aware Visual Tokenization} \label{sec:method_token_level}
At this level, we aim to extract the textual representations of salient visual token types, such as objects, events and attributes. We found that pre-defined classes for classification, such as those in ImageNet~\citep{deng2009imagenet}, are far from enough for covering the rich semantics in open-domain videos.
Thus, instead of using classification-based methods for visual tokenization as in previous work~\citep{lin2021vx2text, pcia}, we adopt a retrieval-based visual tokenization approach by leveraging pre-trained contrastive image-language models. Given a visual token vocabulary which contains all candidate object, event, and attribute text phrases, we compute the image embedding of a frame and the text embeddings of the candidate visual tokens using a contrastive multi-modal encoder, CLIP~\citep{clip}. We then select top 5 visual tokens based on the cosine similarity of the image and text embeddings. An example of the extracted object tokens can be found in the \textcolor{tokenlevel}{green} part of Figure~\ref{fig:framework}.

Unlike in images where objects and attributes already cover most visual features, events are more informative in videos. In order to discover events from video frames, we construct our own event vocabulary by extracting event structures from Visual Genome~\citep{visualgenome} synsets\footnote{We use the keys in Visual Genome~\citep{visualgenome} object synsets which contains frequent <verb,object> pairs.} using \textbf{Semantic Role Labeling}. Specifically, we first select the phrases that contains at least one verb and one argument as \textbf{events}. Then we remove highly similar events based on their sentence similarity using SentenceBERT~\citep{sbert} embeddings. For object vocabulary, we adopt OpenImage~\citep{openimage} full classes ($\sim$20k), instead of using the visually groundable subset ($\sim$600) as in concurrent work~\citep{zeng2022socratic}. We found that using \textit{large but noisy} vocabulary is more effective than using \textit{small but clean} vocabulary in our retrieval-based setting with CLIP. For attribute vocabulary, we adopt visual genome attribute synset. 
In Section~\ref{sec:ablation}, we provide ablation study on the impact of different types of visual tokens. The statistics of visual token vocabulary can be found in Appendix.
\begin{figure}[thb]
    \centering
\includegraphics[width=\textwidth]{image/temporal_prompt_example.pdf}
    \caption{Temporal-aware prompt successfully distinguishes the \textbf{Sunset} and \textbf{Sunrise} scenario from the temporal ordering change of objects and frame captions, while the static prompt fails. }
    \label{fig:temporal_prompt_example}
\end{figure}

\subsection{Video Level: Temporal-Aware Few-shot Prompting}\label{sec:method_video_level}
Once we obtain the textual representation from frame level and visual token level, the final step is to put the pieces together to generate a video level target text. 
The goal is to build a model that can be quickly adapted to any video-to-text generation task with only a few examples.
To this end, we propose to leverage large-scale pre-trained language models, such as  GPT-3~\citep{brown2020gpt3}, with a temporal-aware few-shot prompt. As shown in Figure~\ref{fig:framework}, our framework can be readily applied to various video-to-text generation tasks, such as video captioning and video question answering, with a shared prompt template.  The proposed prompting strategy enables a language model to attend to the lower level visual information as well as taking into account the temporal ordering. 

Here, we use the video captioning task depicted in Figure~\ref{fig:framework} to illustrate the details. 
The few-shot prompt consists of three parts: \textbf{instruction}, \textbf{few-shot context}, and \textbf{task query}. The \textbf{instruction} is a concise description of the generation task, e.g., \textcolor{prompttext}{\texttt{"Generate a video caption based on the objects, events, attributes and frame captions. Example:"}}, which is proved to be effective in zero-shot and few-shot settings~\citep{brown2020gpt3, wei2021finetuned}. 
The \textbf{few-shot context} contains the selected in-context examples as well as the test video instance. Each video instance is represented by the aggregated visual tokens\footnote{To obtain video level visual tokens, the visual tokens extracted from each frame are further ranked and ordered based on frequency and frame index.
}, e.g., \textcolor{prompttext}{\texttt{"\textcolor{prompttext}{Objects}: \textcolor{prompttext}{First,} bath toy. \textcolor{prompttext}{Then,}..."}}, the frame captions, e.g., \textcolor{prompttext}{\texttt{\textcolor{prompttext}{"Frame Captions:} \textcolor{prompttext}{First,} a toddler playing in a bathtub filled with toys. \textcolor{prompttext}{Then,}..."}}, and the ASR inputs if available, e.g., \textcolor{prompttext}{\texttt{"Subtitle:<ASR Transcript>"}}. Finally, the \textbf{task query} is a task-specific suffix indicating the target text format, e.g. \textcolor{prompttext}{\texttt{"Video Caption:"}}. For in-context examples (omitted here for simplicity), the task query is followed by ground truth annotation, while for the test instance, the generation starts at the end of the task query.

Formally, we denote the instruction line as $\mathbf{t}$, few-shot context as $\mathbf{c}$, the task query as $\mathbf{q}$, and the target text as $\mathbf{y}$, where $\mathbf{y} = (y_1, y_2, ... ,y_L)$. The generation of the next target token $y_l$ can be modeled as:
\begin{align}
    y_l = \argmax_{y} p(y | \mathbf{s},\mathbf{c},\mathbf{q},y_{<l})  
\end{align}
In order to capture the temporal dynamics between frames and visual tokens, we further propose to inject  \textbf{temporal markers} to the prompt. As shown in the few-shot context in Figure~\ref{fig:framework}, each visual token and frame caption is prefixed with a natural language phrase indicating its temporal ordering, e.g., \texttt{"\textcolor{prompttext}{First},"},\texttt{"\textcolor{prompttext}{Then},"}, and \texttt{"\textcolor{prompttext}{Finally},"}. We found adding the temporal marker can make the language model condition on not only the literal but also the temporal information of the context. We show an example in Figure~\ref{fig:temporal_prompt_example}, where we compare our \textit{temporal-aware prompt} with a \textit{static prompt} on video captioning using InstructGPT. Again, the in-context examples are omitted here, which can be found in Appendix. In this example, the only difference between the two context is the ordering of the visual tokens and the frame captions. For the context on the left, where \textcolor{prompttext}{\texttt{"sun moving"}} appears before \textcolor{prompttext}{\texttt{"night sky"}}, we are expected to see a story talking about \textbf{sunset}, while for the context on the right, we are expected to see \textbf{sunrise}. We can see the static prompt generates captions about sunset for both contexts, while the temporal-aware prompt can capture the temporal ordering correctly and generate sunrise for the context on the right. 





\section{Experiments}
\subsection{Experimental Setup}
\label{sec:experimental_setup}
To comprehensively evaluate our model, we show results on four video-language understanding tasks in few-shot settings: video captioning, video question answering (QA), video-language event prediction, and text-video retrieval. 
We compare our approach with state-of-the-art approaches on five benchmarks, i.e, MSR-VTT~\citep{xu2016msrvtt}, MSVD~\citep{msvd}, VaTeX~\cite{wang2019vatex}, YouCook2~\citep{YouCook2}, and VLEP~\cite{lei2020vlep}. %


\input{table/data_stat}

\paragraph{Implementation Details.} We use CLIP-L/14\footnote{\url{https://huggingface.co/openai/clip-vit-large-patch14}} as our default encoder for visual tokenization. We adopt BLIP captioning checkpoint\footnote{\url{https://github.com/salesforce/BLIP\#finetuned-checkpoints}} finetuned on COCO~\citep{coco} for frame captioning. We use InstructGPT~\citep{instruct-gpt3} as our default language model for generating text conditioned on the few-shot prompt. To construct event vocabulary, we use the semantic role labeling model from AllenNLP\footnote{\url{https://docs.allennlp.org/models/main/models/structured_prediction/predictors/srl/}}. 
The experiments are conducted on 2 NVIDIA V100 (16GB) GPUs. 

\paragraph{In-context Example Selection.} From our preliminary experiments, we found that the generation performance is sensitive to the quality of in-context examples. For example, for QA tasks such as MSVD-QA where the annotations are automatically generated, the <question, answer> pair in randomly selected in-context examples can be only weakly-correlated with the video context. Thus, instead of using a fixed prompt for each query, we dynamically filter out the irrelevant in-context examples. %
Specifically, given a randomly sampled \textit{M}-shot support set from the training set, we select a subset of \textit{N}-shots as in-context examples based on their SentenceBERT~\citep{sbert} similarities with text queries. %
For QA tasks, we choose the most relevant in-context examples by comparing with questions. While for captioning task, we compare with frame captions.
If not otherwise specified, we use \textit{M=10} and \textit{N=5}, which we consider as 10-shot training.


\subsection{Few-shot Video Captioning}
\label{sec:captioning}
We report BLEU-4~\citep{bleu}, ROUGE-L~\citep{rougeL}, METEOR~\citep{meteor}, and CIDEr~\citep{cider} scores on three video captioning benchmarks covering both open-domain (MSR-VTT, VaTeX) and domain-specific (YouCook2) videos.
We compare with both state-of-the-art video captioner (UniVL~\citep{luo2020univl}) and image captioner (BLIP~\citep{li2022blip}).
In order to implement the BLIP baseline for few-shot video captioning, we extend the approach used for text-video retrieval evaluation in \cite{li2022blip} to video-language training. Specifically, we concatenate the visual features of sampled frames and then feed them into the image-grounded text-encoder to compute the language modeling loss. This is equivalent to stitching the sampled frames into a large image and then feeding it to BLIP for image captioning. We found that this simple approach results in very strong baselines. 

\input{table/caption_result}
\vspace{-7pt}

As shown in Table~\ref{tab:caption_result}, existing methods have strong bias on certain datasets. For example, UniVL performs well on YouCook2 but fails on MSR-VTT and VaTeX, while BLIP performs the opposite. This is because UniVL is pretrained on HowTo100M which favors instructional videos, i.e., YouCook2, while BLIP is pre-training on image-caption pairs which favors description-style captions, i.e., MSR-VTT and VaTeX. On the contrary, our model performs competitively on both open-domain and instructional videos, and significantly outperforms the baselines in the average CIDEr score across all three benchmarks. This indicates that by leveraging language models, we can maintain strong few-shot ability regardless of the video domain and or the target caption distribution. 

As discussed in Section~\ref{sec:intro}, video captions describes the content in various semantic levels. The N-gram based metric may not fairly reflect the models' performance in capturing the video-caption alignment. We further verifies this hypothesis in Section~\ref{sec:retrieval}. Thus, in addition to the automatic metrics, we include qualitative examples illustrated in Figure~\ref{fig:qualitative_examples}. More examples are in Appendix.

Additionally, for most existing methods and also concurrent work, e.g., Flamingo~\citep{alayrac2022flamingo}, adding a new modality often requires a dedicated model redesign and or retraining. However, the nature of our framework, where we use a unified textual representation for each level, making it highly flexible for incorporating new modalities. As shown in row 5 and 6 in Table, our model can benefit from the added modality more effectively than the baselines.



\begin{figure}[htb]
    \centering
    
    \includegraphics[width=\textwidth]{image/qualitative_examples_3.pdf}
    \caption{Qualitative examples on video captioning. \textcolor{boxgrey}{Grey boxes} contains part of the video representation from our model. \textcolor{boxblue}{Blue boxes} contains caption generation from different models.  \textcolor{boxgreen}{Green boxes} contains ground truth annotations.  \textcolor{textgreen}{\textbf{Bold green text}} highlights the correct information that is not captured in baseline outputs which can be reasoned from our visual tokens and frame captions.
    }
    \label{fig:qualitative_examples}
\end{figure}
\vspace{-7pt}






\subsection{Few-shot Video Question Answering}
\label{sec:qa}
We compare the test accuracy of our approach with few-shot pretrained BLIP, BLIP$_{VQA}$~\citep{li2022blip}, and concurrent work Flamingo~\citep{alayrac2022flamingo} on two video question answering benchmarks, MSR-VTT\_QA and MSVD\_QA. BLIP$_{VQA}$ represents finetuned BLIP on VQA~\citep{VQA} dataset, which is the previous SOTA on zero/few-shot video question answering. In order to have fairer comparison with BLIP$_{VQA}$, we reduce the shot number to 5 and report the average accuracy on three sets of randomly selected 5-shot examples.
As shown in Table~\ref{tab:qa_result}, our method outperforms previous SOTA by a large margin. Comparing with concurrent work Flamingo, which is post-pretrained on a large number of video-text data, our model is training-free and did not observe any video data. However, with only image-language and language-only knowledge, our 5-shot model is able to beat 8-shot Flamingo-3B and achieve on-par performance with 4-shot Flamingo-80B. 

\input{table/qa_result}


\subsection{Few-shot Video-Language Event Prediction}
In this section, we show that our model not only can answer questions about the video visual features but also answering "What is more likely to happen next?". Given a video with associated subtitle transcript as premise, the video-language event prediction (VLEP) task is to predict the most likely future event. The original VLEP~\citep{lei2020vlep} paper formulates the problem as a binary classification problem where the model will be chosen from two possible future event candidates. Instead, we formulate this problem as another video-to-text generation problem to fit into our framework. Figure~\ref{fig:vlep} depicts an example with the same format as in Figure~\ref{fig:framework}. Similar to the evaluation setting in QA, the generated free-form text will first be mapped to one of the two candidate answers using SentenceBert~\citep{sbert}, and then calculate the accuracy. 
In Table~\ref{tab:vlep_result}, we report accuracy on the hidden test set of VLEP~\citep{lei2020vlep}. To our surprise, our 10-shot model beats state-of-the-art fully-supervised baseline, i.e., MERLOT~\citep{zellers2021merlot}, by a large margin ($\sim4\%$). This is showing that our model has strong few-shot ability not only in video-language understanding but also in prediction. Since event prediction tasks rely heavily on temporal ordering, we show that with the proposed temporal-aware prompting, language models can be guided to capture temporal dynamics between historical and future events. 

\begin{figure}[thb]
    \centering
    \includegraphics[width=0.95\textwidth]{image/vlep.pdf}
    \vspace{-4pt}
    \caption{\footnotesize Prompt for VLEP task.}
    \label{fig:vlep}
\end{figure}


\input{table/retrieval_result}

\subsection{Semi-supervised Text-Video Retrieval}
\label{sec:retrieval}

In addition to video-to-text generation tasks, we show that a broader range of video-language tasks can benefit from our few-shot video captioner from a data perspective. 
Here, we consider a low-budget semi-supervised setting where we only have a few labeled video-captain pairs and a large amount of unlabeled videos. The idea is to leverage our video captioner to generate \textbf{pseudo labels} for training any given vision-language models. As a case study, we evaluate on two text-video retrieval benchmarks, i.e., MSR-VTT and VaTeX. We use greedy decoding to generate pseudo caption for each video in the training set. We then train an identical base model, i.e., BLIP, using different pseudo labeled data as well as ground truth annotations. We report Recall @ 1 and 5 for both video-to-text and text-to-video retrieval. Table~\ref{tbl:retrieval_results} shows that through training on our pseudo labels, we can achieve significant improvements compared with zero-shot BLIP. We also show that the performance gain is not simply a result of training on more data, since finetuning on the pseudo labels generated by other baselines (UniVL, BLIP) is less effective and can even hurt the performance. Furthermore, on MSR-VTT Recall @ 5 we can even achieve comparable performance against BLIP model finetuned on full ground truth annotations.

Another interesting observation is that, compared with the video captioning results in Table~\ref{tab:caption_result}, we found that the gain of our model over baselines on text-video retrieval is more visible than on captioning. A key factor in performing well on text-video retrieval tasks is to learn a good video-text multi-modal alignment. This result shows that our pseudo labels capture richer video-text alignment that can benefit the retrieval-style downstream task.  The N-gram based generation metrics, e.g., BLEU, may not be able to fully reflect the alignment information, due to the variety of semantic levels in video captions. 
Furthermore, from a data perspective, our video captioner can be viewed as a data augmentation tool which is capable of generating or augmenting any open-domain video-language pretraining datasets with minimal human effort. As a result, we can potentially improving video-language pretraining by constructing a cleaner and more diverse video-text corpus.




\input{table/ablation_results}

\subsection{Ablation Studies}
\label{sec:ablation}
We perform comprehensive ablation studies on our few-shot prompt including the impact of different video representation, the number of shots and the in-context selection. All the ablation results are evaluated on MSVD\_QA validation set, we report the mean and standard deviation of each setting on three sets of randomly sampled shots. For the cases \textbf{with} in-context example selection, we further select $5$ examples as in-context examples from the sampled shots, while for the cases \textbf{without} in-context selection, all shots will be feed into the prompt. In Table~\ref{tab:ablation_representation}, we show adding visual tokens consistently improves not only the model accuracy but also the model variance. A lower standard deviation indicates that the model is less sensitive to the few-shot sampling.

In Table~\ref{tab:ablation_shots}, we first show that, with the same context length, namely, $5$ examples, in-context example selection significantly increases the performance as well as the robustness. At $10$-shot, and $20$-shot, directly fitting more shots into the prompt, i.e., w/o in-context selection, results in better performance than selecting from a larger support set while keeping the context length unchanged. Interestingly, at $30$-shot, the situation reversed again. We observe that selecting $5$ examples from a support set of size $30$ results in better performance than adding all $30$ shots as in-context example. This is showing that in-context selection can help our model utilize a larger number video examples. However, we still observe that the benefit of adding more shots saturated at around $20$ to $30$ shots, even if with in-context selection. we view this as a remaining challenging on how to make language models benefit from longer contexts. 






\section{Related Work}


\subsection{Image-Language Models and Their Applications on Video-Language Tasks}
Large-scale image-language pretraining models optimize image-text matching through contrastive learning such as CLIP~\citep{clip} or by learning image-text alignments~\citep{yu2021ernie, li2021align, wang2021simvlm, yuan2021florence,lu2019vilbert,tan2019lxmert,chen2020uniter,li2020oscar,zhou2020unified,zhang2021vinvl,jia2021scaling,kim2021vilt,huang2021seeing}. Recently, BLIP~\citep{li2022blip} proposes an image captioner pretrained with filtering in order to achieve optimal image-text alignment and has shown promising performance on various image-language tasks. 
However, video-language pretraining~\citep{li2021alpro,luo2020univl, li2020hero,miech2020end,alayrac2020self,akbari2021vatt,patrick2020support} is still hindered by noisy and domain-specific video datasets~\citep{lei2018tvqa,miech2019howto100m}. 
Naturally, researchers start to explore transferring the rich knowledge from image models to videos. 
Different from the traditional way of representing videos by 3D dense features~\citep{feichtenhofer2019slowfast}, recent work~\citep{ClipBERT, li2021alpro} proves that sparse sampling is an effective way to represent videos, which facilitates applying pre-trained image-language models to video-language tasks~\citep{Luo2021CLIP4Clip, fang2021clip2video}. 
Specifically, the image-language model BLIP~\citep{li2022blip} sets new state-of-the-art on zero-shot retrieval-style video-language tasks, such as video retrieval and video question answering. However, for generation-style tasks such as domain-specific video captioning, video-language model UniVL~\citep{luo2020univl} still leads the performance but highly rely on fine-tuning. In this work, we extend the idea of leveraging image-language models to a wide variety of video-to-text generation tasks. We further connect image-language models (ILM) with language models (LM) which empowers our model with strong generalization ability. We show that the knowledge from both image-language pretraining and language-only pretraining can benefit video-language understanding in various aspects. 

\subsection{Unifying Multi-Modal Tasks with Language Models}

Connecting different modalities with a unified representation has been paid much attention to recently. Text-only generation models, such as T5~\citep{raffel2020exploring}, has been extended to vision-language tasks by text generation conditioned on visual features \cite{cho2021unifying, tsimpoukelli2021multimodal, lmcansee, zhu2021uni, wang2022OFA}. In order to fully leverage the generalization power from pretained language models, \cite{pcia} represents images using text in a fully symbolic way. \cite{lin2021vx2text} includes more modalities such as video and audio, but requires annotated video-text data to jointly training the language model with the video and audio tokenizer. In this work, we propose a temporal-aware hierarchical representation for describing a video textually. To our knowledge, we are the first work to leverage prompting a frozen language model for tackling few-shot video-language tasks with a unified textual representation. Concurrent work Socratic~\citep{zeng2022socratic} uses a zero-shot language-based world-state history to represent long videos with given time stamps, while our model can quickly adapt to different video and text distributions with few examples. Furthermore, we show that by injecting \textit{temporal markers} to the prompt we can make a pre-trained language model understand fine-grained temporal dynamics in video events.  
Compared with the concurrent work Flamingo~\citep{alayrac2022flamingo}, which requires dedicated vision-language post-pretraining, our framework does not require to pretrain on any videos. Our framework is simple and highly modulated where all the components are publicly available. Additionally, our framework is more flexible on adding new modalities, e.g., automatic speech recognition, without the need for complex redesigning.

\section{Conclusions, Limitations and Future Work}
This paper proposes \ours, a few-shot \textbf{Vid}eo-language Learner via \textbf{I}mage and \textbf{L}anguage models. It demonstrates the strong ability of large-scale language models on performing video-to-text tasks when frame features are provided as unified text representations using image-language models. 
We propose a temporal order aware prompt by decomposing videos into a hierarchical structure, which is able to plug in multiple levels of frame features, along with speech transcripts. 
Without pretraining on videos, our model outperforms vision-language models learned from large-scale video datasets on a variety of few-shot tasks, such as domain-specific captioning, question answering, and future event prediction. 
One limitation is that the model is sensitive to the quality of in-context examples, as discussed in ablation studies. 
Future work will focus on leveraging large-scale language models to assist in video pretraining, such as its script knowledge for long video understanding and storytelling. For broader impact on the society, please refer to the supplemental material.
\section{Additional Qualitative Examples}
Additional qualitative examples on MSR-VTT, YouCook2 and VaTex captioning can be found in Figure~\ref{fig:add_qualitative_exp_1},\ref{fig:add_qualitative_exp_2}. We show that our framework can capture important video semantics (shown in \textcolor{textgreen}{\textbf{bold green text}}), such as objects, events and attributes, that are missing in the captions generated by baselines.
\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/additional_qualitative_examples_1.pdf}
    \caption{Additional qualitative examples on MSR-VTT Captioning.}
    \label{fig:add_qualitative_exp_1}
\end{figure}

\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/additional_qualitative_examples_2.pdf}
    \caption{Additional qualitative examples on YouCook2 and VaTex Captioning.}
    \label{fig:add_qualitative_exp_2}
\end{figure}




\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/msrvtt_captioning_prompt.pdf}
    \caption{An example of few-shot prompts for MSR-VTT captioning.}
    \label{fig:msrvtt_captioning_prompt}
\end{figure}
\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/youcook2_with_asr_prompt.pdf}
    \caption{An example of few-shot prompts with ASR input for YouCook2 captioning.}
    \label{fig:youcook2_with_asr_prompt}
\end{figure}
\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/msvd_qa_prompt.pdf}
    \caption{An example of few-shot prompts for MSVD question answering.}
    \label{fig:msvd_qa_prompt}
\end{figure}
\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/vlep_prompt.pdf}
    \caption{An example of few-shot prompts for video-language event prediction (VLEP) task.}
    \label{fig:vlep_prompt}
\end{figure}
\begin{figure}[thb]
    \centering
    \includegraphics[width=\textwidth]{image/temporal_vs_static_prompt.pdf}
    \caption{Full prompt of the "Sunrize" scenario for the example shown in Figure 3 in the main body. We show the impact of temporal-aware prompt on capturing temporal dynamics in videos. The sentences in \textcolor{textblue}{blue} following "Video Caption:" are generated by GPT-3. Text marked in \textcolor{textgreen}{green} indicates the generated caption is semantically coherent with the given objects and frame captions, while text marked in \textcolor{textreddark}{red} indicates incorrectness.  }
    \label{fig:temporal_vs_static_prompt}
\end{figure}


\section{Few-shot Prompt Examples}
We show a full view of the few-shot prompts used in video captioning (Figure~\ref{fig:msrvtt_captioning_prompt}, \ref{fig:youcook2_with_asr_prompt}), video question answering (Figure~\ref{fig:msvd_qa_prompt}) and video-language event prediction (Figure~\ref{fig:vlep_prompt}). Additionally, in Figure~\ref{fig:temporal_vs_static_prompt}, we show the omitted in-context examples for Figure 3 in the main body.

\section{Additional Implementation Details}
\input{table/visual_token_vocab_stat}
\paragraph{Statistics of Visual Token Vocabulary} We construct our visual token vocabulary based on OpenImage~\citep{openimage} v6 class names\footnote{\url{https://storage.googleapis.com/openimages/v6/oidv6-class-descriptions.csv}}, visual genome~\citep{visualgenome} object synsets\footnote{\url{https://visualgenome.org/static/data/dataset/object_synsets.json.zip}} and visual genome attribute synsets\footnote{\url{https://visualgenome.org/static/data/dataset/attribute_synsets.json.zip}}. The statistics can be found in Table~\ref{tab:vis_tok_stat}. Visual genome synsets are <key, value> pairs, where the keys are noisy natural language phrases and the values are the mapped WordNet synsets~\citep{miller1995wordnet}. For object vocabulary, we perform minimum cleaning by removing fictional character names such as \texttt{"robin (fictional character)"}, which we found are highly biased by the CLIP~\citep{clip} model on video frames. For attribute vocabulary, we clean up attribute synset keys by removing phrases with a cosine similarity larger than 0.9 using SentenceBert~\citep{sbert} embedding, such as \texttt{"facing upward"} and \texttt{"facing upwards"}. For event vocabulary, we select phrases containing <verb,object> structures from the object synset keys by running semantic role labeling\footnote{\url{https://docs.allennlp.org/models/main/models/structured_prediction/predictors/srl/}}. We then remove semantically similar entries with a threshold of 0.9 based on SentenceBert embeddings.

\paragraph{Implementation Details for Few-shot Video Captioning Baselines}
In order to finetune the pretrained baselines (UniVL~\citep{luo2020univl}, BLIP~\citep{li2022blip}, BLIP$_{cap}$~\citep{li2022blip}) with few annotated examples on video captioning, we set the learning rate to be small and the warm-up steps to be high. Specifically, for UniVL, we set the number of epoches to be $50$ and the linear warmup steps to be $40$. We use a learning rate of 1$e$-6 for captioning task without ASR input and 3$e$-6 with ASR input. 
For BLIP and BLIP$_{cap}$, we set the number of epoches to be $5$ with a learning rate of 5$e$-7. For each video, we sample 4 frames (each with a size of 224) at training time and 8 frames at test time. We set all batch size to be the same as the few-shot number, i.e., $10$. 

\paragraph{Implementation Details for Semi-supervised Text-Video Retrieval}
We use pretrained BLIP with Vit-B/16\footnote{\url{https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth}} as our base model for training on different pseudo labeled datasets as well as ground truth annotations for text-video retrieval. We train the model for one epoch using a batch size of 16 and a learning rate of 5$e$-6. For each video, we sample 4 frames (each with a size of 224) at training time and 8 frames at test time. We follow \cite{li2022blip} to first select $k$ candidates based on the video-text feature similarity, where the video features are represented by concatenated frame features. We then rerank
the selected candidates based on their pairwise Image-Text Matching (ITM) score. We set $k=64$ for both MSR-VTT and VaTex retrieval.


\section{Broader Impact}
An open-domain few-shot video-language learner has a wide range of beneficial applications for society, such as automatically detecting violent or mature content in videos and helping people with vision impairment understand videos. 
However, since the langauge model is pretrained on massive internet-scale text data, there might be unexpected output that can have potential negative impact on the society, such as bias against people of a certain gender, race or sexuality.
Future work and dedicated collaboration from the community are needed to alleviate the potential negative societal impact of large language models.




\section*{Acknowledgement}
We thank the anonymous reviewers helpful suggestions. 
This research is based upon work supported by U.S. DARPA AIDA Program No. FA8750-18-2-0014, U.S. DARPA KAIROS Program No. FA8750-19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

{
\small






