\subsection{Pre-training Settings}\label{sec:pre-training-setting}
We follow the architecture of BART~\cite{lewis2019bart}, which is made up of a 6-layer Transformer encoder and a 6-layer decoder with the dimension of 768 and 12 heads ($\sim$140M parameters). We adopt the parameters of PLBART~\cite{ahmad2021unified} to initialize the pre-trained model and add an additional segment embedding layer to encode different segment identifiers. We also adopt the vocabulary set of PLBART, which contains 50K subtokens and further add 5 different segment identifiers into the vocabulary set to represent different components in a commit. We set the maximum input sequence length to 512 and use the mixed precision of FP16 to accelerate the pre-training process. We set the batch size to 512 and employ the AdamW optimizer to update the model parameters with a learning rate of 2e-4. We pre-train the model on one DGX server, which has 8 NVIDIA Tesla V100 with 32GB memory. The total steps are set to 80K where the category of denoising objectives, cross-modal generation and contrastive learning, each takes up 60\%, 30\%, 10\% respectively. One category has two different pre-training tasks and each of them accounts for half of the steps. The total time for the pre-training process is about 60 hours. We follow Guo et al.~\cite{guo2020graphcodebert} to sample each batch from the same programming
language according to a distribution $\{q_i\}_{i=1...N}$, where $n_i$ is the number of examples for $i$-th programming language and $\alpha=0.7$ to alleviate the bias towards high-resource languages.

\begin{equation}
q_i=\frac{p^\alpha_i}{\sum_{j=1}^{N}p^\alpha_j} \ , \ p_i=\frac{n_i}{\sum_{k=1}^{N}n_k}
\end{equation}


\subsection{Fine-tuning Settings}\label{sec:fine-tune-setting}
\subsubsection{Understanding Task}
For the understanding task of security patch identification, we directly utilize the released dataset from Wu et al.~\cite{wu2022enhancing}, which consists of 26500 training samples, 3301 validation samples and 3294 test samples for evaluation. We employ AdamW optimizer to fine-tune {CommitBART}\xspace with a 5e-5 learning rate and the batch size 32 for 4K steps. We set the maximum input sequence length to 512 and the target sequence length to 5, which includes the start token ``[CLS]'', the task prefix ``security patch'', the label ``True'' or ``False'' and the end token ``[EOS]''. 

\subsubsection{Generation Tasks}
For the generation tasks of commit message generation and updated code snippet generation, we directly utilize the fine-tuning dataset from our benchmark (See Table~\ref{tab:data-statistics}) with train-validation-test with the ratio of 75\%:10\%:15\% for evaluation. For the task of positive code statements generation, we extract the consecutive modification samples from the constructed fine-tuning dataset for evaluation. The statistics of the train, validation and test are shown in Table~\ref{tab:fine-data-statistics}. For each task, we utilize AdamW optimizer to fine-tune {CommitBART}\xspace with a 5e-5 learning rate of batch size 32 for 10K steps. Furthermore, we set the early stop based on the validation loss. The maximum input sequence length is set to 512 for these tasks, while the target sequence is set to 150 for commit message generation, 512 for updated code snippet generation and 300 for positive code statements generation respectively.


\begin{table*}[ht!]
\centering
  \caption{The statistics of fine-tuning dataset for commit message generation, updated code snippet generation and positive code statements generation. The first row is the dataset used for first two tasks while second row is the dataset for the last task.}
  \label{tab:fine-data-statistics}
  \begin{tabular}{crrrrrrrr}
    \hline
    Fine-tune&C & CSharp & Java & JavaScript & PHP & Python & Typescript & Total\\
    \hline
    Train & 51,504& 45,425 & 63,189 & 68,338 &78,450  &70,137 & 51,526  &428,569 \\
    Validation &8,902 & 5,501 & 7,512  &8,100  &9,239  & 8,326& 6,172 & 53,752\\
    Test &  11,518& 10,976 &  10,425&  10,626& 11,541 & 11,039& 10,064 & 76,189\\ \hline
    Positive-Train & 20,746 &  19,726& 27,619 & 31,809 &35,015 & 28,198 & 26,325& 189,438\\
    Positive-Validation & 2,320 &  2,561& 2,580 & 2,655 & 2,490 & 2,363& 2,957 & 17,926\\
    Positive-Test & 2,321 & 2,562 & 2,580  & 2,656 &  2,491& 2,364&  2,958& 17,932 \\
    \hline
  \end{tabular}
\end{table*}



\subsection{Case Study}
In this section, we provide more examples to demonstrate the effectiveness of {CommitBART}\xspace. 
\subsubsection{Commit Message Generation}
We further provide one example for commit message generation. From Figure~\ref{fig:case_study_two}, we can see that its commit message is to delete an unused variable, hence there is only the deleted statement (i.e., ``const bool is\_gles = glext.version.is\_es'') without the added statement. By comparing the results produced by different models, we find that {CommitBART}\xspace can generate a better result. 



\begin{figure}[tp]
     \centering
     \includegraphics[width=\columnwidth]{figures/example_2_msg.pdf}
    \caption{One example from C for the task of commit message generation where the commit id is
    \href{https://github.com/Kinoli7/taisei/commit/6072a720465d60ca84d1a9825b132a1f138457d5}{8457d5}.}
     \label{fig:case_study_two}
\end{figure}



\subsubsection{Positive Code Statements Generation}
We provide two examples for the task of positive code statements generation in Figure~\ref{fig:single_example_1} and Figure~\ref{fig:single_example_2} respectively. The input consists of the file path, the commit message and the previous code snippet. We ask the model to generate the positive code statements marked as ``+'' in a commit. We can see that both commits aim to add a security check to avoid code crash. Compared with the results produced by different models, {CommitBART}\xspace generates accurate positive code statements. 

\begin{figure*}[htp]
     \centering
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/single_generation_example_1.pdf}
         \caption{The commit id is \href{https://github.com/abhirocks1211/CardStackView/commit/86094473ab4a3562587235f243cf95c0f1054122}{054122}.}
         \label{fig:single_example_1}
     \end{subfigure}
     \hfill
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/single_generation_example_2.pdf}
         \caption{The commit id is \href{https://github.com/renxuetao/FileDownloader/commit/447ea4dc7c19491b2b8bb2bd92418cbcad99cfb9}{99cfb9}.}
         \label{fig:single_example_2}
     \end{subfigure}
        \caption{Two examples from Java for the task of positive code statements generation.}
        \label{fig:single_examples}
\end{figure*}


\subsubsection{Updated Code Snippet Generation}
We provide two examples for the task of updated code snippet generation. Similar to the task of positive code statements generation, the input includes the file path, the commit message and the previous code snippet. For the first example, the input is presented in Figure~\ref{fig:code_gen_source} and the ground-truth for this example is presented in Figure~\ref{fig:code_gen_ground}. By Figure~\ref{fig:code_gen_source}, we can get that the updated code snippet needs to add a variable safety check to avoid ``ph'' as empty. The results produced by different models are presented in Figure~\ref{fig:code_gen_codebert} to Figure~\ref{fig:code_gen_commitbart} accordingly. We can observe that PLBART and Incr-PLBART produce better results that other baseline models for this example, however they are both misunderstand the semantics of the commit message and produce the exact opposite results (i.e., ``if (tPh.isEmpty())'' and ``if (tPh.length() == 0)''). In contrast, {CommitBART}\xspace can capture the semantics well and produce the same code snippet with the ground-truth. For another example, which is shown in Figure~\ref{fig:code_gen_2}, we can see that this commit removes the variable ``isContinuousIntegration'' and further deletes its usage in the following statement. The modification is non-consecutive. The results produced by CodeBERT and CodeT5 only copy the original statements while the results produced by the remaining baselines remove the variable declaration, but fail to delete its usage. In contrast, {CommitBART}\xspace can produce the same code snippet with the ground-truth. 




\begin{figure*}
     \centering
         \centering
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_source.pdf}
         \caption{The input for models to generate the updated code snippet.}
         \label{fig:code_gen_source}
     \end{subfigure}
     \hfill
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_ground.pdf}
         \caption{The updated code snippet (i.e., ground-truth).}
         \label{fig:code_gen_ground}
     \end{subfigure}
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_codebert.pdf}
         \caption{The result provided by CodeBERT.}
         \label{fig:code_gen_codebert}
     \end{subfigure}
    \hfill
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_plbart.pdf}
         \caption{The result provided by PLBART.}
         \label{fig:code_gen_plbart}
     \end{subfigure}
        \hfill
     \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_codet5.pdf}
         \caption{The result provided by CodeT5-base.}
         \label{fig:code_gen_codet5}
     \end{subfigure}
         \hfill
        \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_unixcoder.pdf}
         \caption{The result provided by UniXcoder.}
         \label{fig:code_gen_unixcoder}
     \end{subfigure}
         \hfill
          \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_inc_codebert.pdf}
         \caption{The result provided by Incr-CodeBERT.}
         \label{fig:code_gen_incr_coderbert}
     \end{subfigure}
         \hfill
        \begin{subfigure}{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_incrplbart.pdf}
         \caption{The result provided by Incr-PLBART.}
         \label{fig:incr_plbart}
     \end{subfigure}
         \hfill
     \begin{subfigure}[b]{0.49\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_commitbart.pdf}
         \caption{The result provided by CommitBART.}
         \label{fig:code_gen_commitbart}
        \end{subfigure}
         \caption{One example from Java for the task of updated code snippet generation where the commit id is \href{https://github.com/marytts/marytts/commit/0f5b3efb61a6762fed204b776f7e1730c20aa027}{0aa027}.}
        \label{fig:code_gen}
\end{figure*}



\begin{figure*}
     \centering
     \begin{subfigure}{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_source_2.pdf}
         \caption{The input for models to generate the updated code snippet.}
         \label{fig:code_gen_source_2}
     \end{subfigure}
     \hfill
     \begin{subfigure}{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_ground_2.pdf}
         \caption{The updated code snippet (i.e., ground-truth).}
         \label{fig:code_gen_ground_2}
     \end{subfigure}
     \begin{subfigure}{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_codebert_2.pdf}
         \caption{The results provided by CodeBERT and CodeT5-base.}
         \label{fig:code_gen_codebert_2}
     \end{subfigure}
    \caption{Another example from Java for the task of updated code snippet generation where the commit id is \href{https://github.com/Netflix/genie/commit/735d8c29809e997b5ed5978951729b1c33e640a3}{e640a3}.}
        \label{fig:code_gen_2}
\end{figure*}
    \begin{figure*}[ht]\ContinuedFloat
         \centering
     \begin{subfigure}{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_plbart_2.pdf}
         \caption{The results provided by PLBART, UniXcoder, Incr-CodeBERT and Incr-PLBART.}
         \label{fig:code_gen_plbart_2}
     \end{subfigure}
         \hfill
     \begin{subfigure}[b]{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/code_generation_commitbart_2.pdf}
         \caption{The result provided by CommitBART.}
         \label{fig:code_gen_commitbart_2}
        \end{subfigure}
         \caption{Another example from Java for the task of updated code snippet generation where the commit id is \href{https://github.com/Netflix/genie/commit/735d8c29809e997b5ed5978951729b1c33e640a3}{e640a3} (cont.).}
        \label{fig:code_gen_3}
\end{figure*}









\subsection{Model Architecture}
{CommitBART}\xspace follows the architecture of BART~\cite{lewis2019bart} and we adopt the parameters of PLBART~\cite{ahmad2021unified} to accelerate the training process. Since a commit usually consists of multiple different components such as the commit message, the updated code statements marked with ``+'', the deleted code statements marked with ``-'' and their context statements. To distinguish each component, we introduce some additional segment identifiers. Specifically, we define the identifier ``[MSG]'' at the start of a commit message $M=\{m_0,m_1,...,m_i\}$ and ``[FILE]'' for file path $F=\{f_0,f_1,...,f_j\}$ where $i$ and $j$ denote the number of $M$ and $F$ word tokens respectively. Similarly, we also define the identifier ``[CODE]'' at the start of the code to distinguish from the message and file path. In addition, to distinguish the deleted code statements and the updated statements, we further add the identifiers (``[NEG]''/``[END]'') and (``[POS]''/``[END]'') between the start and end of these deleted/updated statements for distinction. Hence, generally, the code in the commit can be represented as follows: 
\begin{align*}
C=\{c_0...[\mathrm{NEG}]n^{i}_0...[\mathrm{END}][\mathrm{POS}]p^{j}_0...[\mathrm{END}]c_k...\\
[\mathrm{NEG}]n^{i^{'}}_0...[\mathrm{END}][\mathrm{POS}]p^{j^{'}}_0...[\mathrm{END}]...c_l\}
\end{align*}
where $c$ denotes the context statement and $c_0$ is the first token in $c$. Furthermore,
$n^{i}/n^{i^{'}}$ and $p^{j}/p^{j^{'}}$ are the deleted and updated code statements respectively. We utilize the segment embedding to embed these segments for ensuring the model understand and incorporate information well. Hence, the input embedding representation for the model is constructed by summing the corresponding token, segment and positional embeddings. 

\subsection{Pre-training Tasks}
We introduce six pre-training tasks that can be divided into three categories for pre-training. We use the example from Figure~\ref{fig:commit_example} for illustration. Specifically, the denosing and cross-modal generation tasks are presented in Figure~\ref{fig:four_tasks}, the contrastive pre-training tasks are presented in Figure~\ref{fig:contrastive_tasks}.
\subsubsection{Denoising Objectives}
Denoising pre-training, which aims at generating the original sequence given the noisy input, is an effective technique for pre-training encoder-decoder models~\cite{lewis2019bart, raffel2020exploring}. We also use it to pre-train {CommitBART}\xspace. Specifically, we apply a noisy function $N$ to the input sequence $X=\{[\mathrm{CLS}][\mathrm{MSG}]M[\mathrm{FILE}]F[\mathrm{CODE}]C\}$ to get the noisy input defined as $N(X)$, where the noisy function is to corrupt the input sequence by some strategies such as token masking, tokens infilling. The learning process is to ask the model to recover the original sequence from the noisy input and the loss function can be calculated as follows:
\begin{equation}
     \mathcal{L}_{\mathrm{Denoise}}(\theta) = \sum_{t = 1}^{|X|}-\mathrm{log}P_{\theta}(X_t|X_{<t},N(X))
\end{equation}
where $P$ is a generator that generates the $t$-th token given the noisy input $N(X)$, the original sequence before t (i.e., $X_{<t}$) and the model parameters $\theta$. We adopted two strategies to corrupt the semantics of the input sequence (i.e., $X$). 

\textbf{Text Infilling.} It randomly masks spans with a single masked token ``[MASK]'' and then we ask the model to recover the original sequence. Specifically, we set the corruption rate as 15\% in the input sequence and ensure the average span length to 3, followed by a Poisson distribution~\cite{lewis2019bart}. A simple example is illustrated in Figure~\ref{fig:four_tasks} (a).



\textbf{Graph-guided Token Masking (GTM).} Inspired by Yasunaga et al.~\cite{yasunaga2020graph}, we introduce a graph-guided token masking task to enhance the relations between different components in a commit. A simple example is shown in Figure~\ref{fig:four_tasks} (b). Specifically, by our preliminary observation, we find that there are some critical tokens (e.g., variable names, keywords, file names) that may be shared in commit message $M$ and changed code $C$. Hence, to enhance the semantic mapping, we construct a commit-graph, which links shared tokens between the commit message $M$, file path $F$ and changed code $C$. Then we randomly mask half of the nodes in the graph and require the model to predict these masked nodes by learning the relations from connected neighbors. Through this task, {CommitBART}\xspace learns to predict the critical tokens in the commit message from the connected nodes in the changed code or vice versa, which further improves the model learning capacity and alleviates the semantic gap between natural language and code. 







\subsubsection{Cross-Modal Generation}
We design two bidirectional conversion generation tasks for the message and code to improve model in generating a coherent commit message or correct changed code. We formulate both cross-modal generation tasks as follows:


\begin{equation}
    \mathcal{L}_{\mathrm{Gen}}(\theta) = \sum_{t = 1}^{|Y|}-\mathrm{log}P_{\theta}(Y_t|Y_{<t},X_{\mathrm{Gen}})
\end{equation}
where $X_{\mathrm{Gen}}$ is the input sequence and $Y$ is the target sequence for generation.

\textbf{PL2NL Generation.}
As shown in Figure~\ref{fig:four_tasks} (c), this task takes the changed code (i.e. $X_{\mathrm{Gen}} = \{\mathrm{[CLS}][\mathrm{FILE}]F[\mathrm{CODE}]C\}$) as input and requires model to generate its corresponding commit message $Y=M$ to ensure model produce high-quality natural language texts for downstream natural language generation tasks. 



\textbf{PLNL2PL Generation.}
This task aims to generate the updated code snippet based on its commit message and the previous code snippet. Specifically, the previous code snippet before the modification can be expressed as $C^{-} = \{c_0...[\mathrm{NEG}]n^{i^{'}}_0...[\mathrm{END}]...c_l\}$. Hence, the input sequence for this task can be expressed as $X_{\mathrm{Gen}} = \{[\mathrm{CLS}][\mathrm{MSG}]M[\mathrm{FILE}]F[\mathrm{CODE}]C^{-}\}$. The output $Y= \{c_0...[\mathrm{POS}]p^{j^{'}}_0...[\mathrm{END}]...c_l\}$, which is the updated code snippet for the model to generate. We incorporate this task to improve the model in generating better code snippets. 


\subsubsection{Contrastive Learning}
Contrastive learning is an effective pre-training task which is used in some pre-trained code models~\cite{jain2020contrastive, wang2021syncobert, guo2022unixcoder}. Generally, it aggregates the similar sequence representation (i.e., $\bm{\widetilde{h}}_i^+$) while pushes away dissimilar representations after encoding the input to an encoder. The loss function is:

\begin{equation}
        \mathcal{L}_{\mathrm{Contra}}(\theta) = \sum_{i=0}^{b-1}-\mathrm{log}\frac{e^{\mathrm{cos}(\bm{\widetilde{h}}_i,\bm{\widetilde{h}}_i^+)/\tau}}{\sum_{j=0}^{b-1}e^{\mathrm{cos}(\bm{\widetilde{h}}_i,\bm{\widetilde{h}}_j)/\tau}}
\end{equation}
where b is batch size, $\tau$ is a temperature hyper-parameter~\cite{wu2018unsupervised} and cos($*$) is cosine similarity between two vector representations. 







\textbf{NLPL Alignment.}
As shown in Figure~\ref{fig:contrastive_tasks} (a), NLPL Alignment task takes the vector representations of commit message $M$ with its code $C$ embedded by {CommitBART}\xspace encoder as a pair of similar sample $(\bm{\widetilde{h}}_i, \bm{\widetilde{h}}_i^+)$ while takes the other representations in the batch as dissimilar samples to further enhance the model capacity of aligning the paired commit message and code.

\textbf{SimCSE.}
 We follow Gao et al.~\cite{gao2021simcse} to put the same input $X=\{[\mathrm{CLS}][\mathrm{MSG}]M[\mathrm{FILE}]F[\mathrm{CODE}]C\}$ with different dropout mask to {CommitBART}\xspace encoder to get vector representations as the similar samples $(\bm{\widetilde{h}}_i, \bm{\widetilde{h}}_i^+)$ and use other representations in the same batch as dissimilar samples for pre-training. An example is shown in Figure~\ref{fig:contrastive_tasks} (b).



\subsection{Fine-tuning}
We transfer {CommitBART}\xspace to commit-related tasks at fine-tuning phase. Generally, the fine-tuning tasks can be categorized into two classes: understanding tasks and generation tasks. For understanding tasks, we directly encode the source sequence to the encoder and ask the decoder to generate its predicted label. For generation tasks, {CommitBART}\xspace can be naturally adapted with its encoder-decoder framework to different commit-related generation tasks.



\subsection{GitHub Commits}
A commit usually consists of the changed code with its commit message to describe the purpose of the current changed code in natural language. We present a commit to illustrate each component in Figure~\ref{fig:commit_example}. The upper rectangle contains the content of the commit message, which summarizes this commit in natural language (e.g., ``Bugfix: Pass threshold to binarizer''). The lower rectangle is the changed code namely one chunk, which contains the file path (i.e., ``tpot/tpot.py'') and the modified code from line 1025 to line 1031. The first line in this chunk starting with ``@@'' consists of the start line number (i.e., 1025) in the file, the total line statements (i.e., 7) and the function name (i.e., ``\_binarizer''). The changed code in this chunk is marked with ``+'' in the updated version of the code with its previous version marked with ``-''. The remaining content is the context around the changed code to reveal its contextual information. Hence, in summary, for this commit in Figure~\ref{fig:commit_example}, we can obtain that there is one line of code at line 1028 in the file of ``tpot.py'' changing the statement from ``...copy=False)'' to ``...copy=False, threshold=threshold)''. The other statements (i.e. from line 1025 to 1027 and from line 1029 to 1031) are the context of the statement at line 1028. Note that we just utilize a simple commit, which only has one chunk with single-line statement modification as the example for better illustration. In some cases, a commit may only have the updated statements or the deleted statements and they may not appear in pairs. Furthermore, a commit can change multiple places of code in a chunk or even change multiple files. Since commits play a critical role in the software development cycle, many researchers conduct broad research such as commit message generation~\cite{jiang2017automatically, liu2020atom, jung2021commitbert}, security patch identification~\cite{zhou2021spi, zhou2021finding, wu2022enhancing}. Compared with these existing works, which design a specialized neural model for commit-related tasks, we are the first to propose a pre-trained model based on our large-scale benchmark and further unify these tasks with one model.





\begin{figure}[tp]
     \centering
     \includegraphics[width=0.99\columnwidth]{figures/commit.pdf}
    \caption{A commit from GitHub with its commit id \href{https://github.com/rsumner33/tpot/commit/dbec56b8f813733bf24e9947747a242af3bd7d14}{dbec56}.}
     \label{fig:commit_example}
     \vspace{-6mm}
\end{figure}




\subsection{Pre-trained Models in Code}
Pre-trained models for code have attracted widespread attention due to their superior performance against the supervised techniques. For example, Svyatkovskiy et al.~\cite{DBLP:conf/sigsoft/SvyatkovskiyDFS20} proposed GPT-C, which utilizes the Transformer decoder~\cite{DBLP:conf/nips/VaswaniSPUJGKP17} in a left-to-right manner for code completion. CodeBERT~\cite{feng2020codebert} and CuBERT~\cite{DBLP:journals/corr/abs-2001-00059} pre-trained a bidirectional Transformer encoder on source code for some code-related understanding tasks. To further improve the model learning capacity, PLBART~\cite{ahmad2021unified} and CodeT5~\cite{wang2021codet5} are proposed to pre-train a Transformer encoder-decoder model based on BART and T5 architecture to support both the understanding and generation tasks. In addition, Guo et al.~\cite{guo2022unixcoder} proposed UniXcoder, a unified cross-modal pre-trained model based on a multi-layer Transformer for ``code intelligence''. Specifically, it utilized mask attention matrices with prefix adapters for code understanding and generation. Apart from the aforementioned works, there are also some pre-trained works for code~\cite{guo2020graphcodebert, DBLP:conf/uai/JiangZLLL21, DBLP:conf/nips/LuGRHSBCDJTLZSZ21, wang2021syncobert, buratti2020exploring, liu2020multi}. Compared with these pre-trained models for code, in this paper, we propose a large pre-trained model based on BART for GitHub commits and further unify a ``commit intelligence'' framework with different commit-related tasks. 


\subsection{Compared with Baselines}
We compare the results of these evaluation tasks with the baselines in Table~\ref{tab:patch-identification}, Table~\ref{tab:message-generation}, Table~\ref{tab:code-snippet} and Table~\ref{tab:single-line} respectively. We can conclude the following findings: 1) Compared with the supervised techniques such as Transformer, the pre-trained models improve the performance significantly over these tasks. 2) PLBART and CodeT5 perform better than CodeBERT. We attribute the gains brought by PLBART and CodeT5 using an additional pre-trained decoder to improve the performance. Furthermore, since CodeT5 and UniXcoder design a cross-modal generation (NL-PL) pre-training task, while PLBART only uses denoising tasks for pre-training, CodeT5 and UniXcoder have better performance on these generation tasks except for the results of CodeT5 on updated code snippet generation. We investigate the reason and conjecture that it may be caused by the specificity of this task. The input of different models for this task consists of the message (NL) and the previous code snippet (PL). This task requires the model to generate the updated code snippet, which is different from the code-related tasks that CodeT5 used. 3) The incremental models trained on CodeBERT and PLBART with our collected commit dataset have better performance than their original models (i.e., Incr-CodeBERT vs CodeBERT, Incr-PLBART vs PLBART), we believe the improvements are from the incremental models learn more domain-related (i.e., commit-related) knowledge at the pre-training phase compared with the original models. 4) {CommitBART}\xspace outperforms these baselines especially on the generation tasks significantly. The improvements come from two aspects. One is a large amount of commit data. We use over \textbf{7} million commits to pre-train a model for commit-related tasks. Hence, our model is domain-specific. Another reason is the designed pre-training tasks. We utilize three categories for six pre-training tasks to improve the model performance on different types of downstream tasks. 



\vspace{-2mm}
\subsection{Model Analysis}
We further conduct an ablation study to investigate each pre-training task to the final performance across these fine-tuning tasks and the results are reported in the last line of Table~\ref{tab:patch-identification}, Table~\ref{tab:message-generation}, Table~\ref{tab:code-snippet} and Table~\ref{tab:single-line} respectively. Through analysing the results, we have the following findings: 1) Segment embedding is useful in {CommitBART}\xspace, although it is not widely used in code pre-trained models. We believe it is due to the commit usually having more complex components (e.g., commit message, deleted/updated statements), hence segment embedding helps the model incorporate each component well. 2) Two cross-modal generation tasks (i.e., PL2NL and PLNL2PL) provide significant improvements to the generation tasks. Taking the commit message generation task in Table~\ref{tab:message-generation} as an example, after removing the PL2NL, the overall smoothed BLEU-4 drops from 20.60 to 18.39 across the seven programming languages. 3) Apart from the cross-modal generation tasks, graph-guided token masking (GTM) is more critical than the remaining tasks on three generation tasks. It demonstrates that by constructing a commit-graph to predict the masked nodes by their connected nodes in the graph, the semantic gap between the message (NL) and changed code (PL) is effectively alleviated. 4) Both contrastive learning pre-training tasks are effective in improving performance. Overall, we conclude that each pre-training task improves {CommitBART}\xspace. 


\vspace{-2mm}

\subsection{Case Study}
We also conduct a case study including an example from commit message generation with the generation results from different models to intuitively demonstrate the effectiveness of {CommitBART}\xspace and it is shown in Figure~\ref{fig:case_study_one}. More examples of different models for these commit-related generation tasks can be found in Appendix. From the changed code in this example, we find that this commit is to add a package namely ``xray.backends''. Its commit message (i.e., ``Add xray.backends to setup.py'') further confirms this. The results produced by UniXcoder and Incr-PLBART are better than other baselines. However, both models fail to generate the accurate package name (i.e.,  ``xray.backends''). In contrast, {CommitBART}\xspace produces the same output with the ground-truth and the package name is accurately generated. We attribute the improvement to the designed graph-guided token masking pre-training task, which helps the model capture the critical semantic information in a commit. 



\begin{figure}[tp]
     \centering
     \includegraphics[width=\columnwidth]{figures/example_1_msg.pdf}
    \caption{One example from Python for the task of commit message generation where the commit id is
    \href{https://github.com/thadncs/https-github.com-pydata-xarray/commit/a68e10e866f24b014dc0ef8a670a552cf5966b15}{966b15}.}
     \label{fig:case_study_one}
     \vspace{-6mm}
\end{figure}
\subsection{Evaluation Tasks}
We select one understanding task (i.e., security patch identification) and three generation tasks (i.e., commit message generation, positive code statements generation, and updated code snippet generation) as  fine-tuning tasks. 

\noindent \textbf{Security Patch Identification.} The task aims to identify whether a commit fixing a software vulnerability or not and this task has been extensively researched with some neural models (e.g., SPI~\cite{zhou2021spi}, E-SPI~\cite{wu2022enhancing}, PatchRNN~\cite{wang2021patchrnn}). We use the same dataset provided by E-SPI~\cite{wu2022enhancing} with the same train-validation-test split for evaluation. 

\noindent  \textbf{Commit Message Generation.} The task targets generating a commit message to summarize the changed code in natural language. Although there are some open-source datasets such as ATOM~\cite{liu2020atom}, Jung et al.~\cite{jung2021commitbert}, we also construct a fine-tuning dataset from our benchmark (See Table~\ref{tab:data-statistics}) across \textbf{7} languages with the train-validation-test split of the ratio of 75\%:10\%:15\% for evaluation.

\noindent \textbf{Positive Code Statements Generation.} We propose this new commit-related task, which targets generating positive statements marked by ``+'', that takes the commit message, file path with the code snippet before modification as input. We only consider the commits that have consecutive statement modification for evaluation. Specifically, if a commit in our constructed fine-tuning dataset from Table~\ref{tab:data-statistics} has multiple non-consecutive modification (a simple example is shown in Appendix Figure~\ref{fig:code_gen_source_2}, which removes two non-consecutive lines of statements), we remove these samples. This task is analogous to the task of bug fixing to generate repaired statements to fix buggy code~\cite{yasunaga2020graph} in the program repair scenario. 


\noindent \textbf{Updated Code Snippet Generation.} We further propose a more challenging task that requires the model to generate the completed updated code snippet. Compared with positive code statements generation, this task needs to locate the position of the updated statements first and then generate the completed code snippet. In this task, we also include these commits that have multiple inconsecutive modification (see an example in Appendix Figure~\ref{fig:code_gen_source_2}) for experiments. This task is valuable for software developers to generate the code that meets the requirement based on the previous code and natural language description. We use our fine-tuning dataset in Table~\ref{tab:data-statistics} for evaluation.




The statistics of the used datasets for these fine-tuning tasks are presented in Appendix. 


\subsection{Baselines}
For the understanding task of security patch identification, since we use the dataset from E-SPI~\cite{wu2022enhancing} with the same train-validation-test split, we directly report the best results from their paper as one of our baselines. For generation tasks, we add one supervised baseline Transformer model~\cite{DBLP:conf/nips/VaswaniSPUJGKP17}, which is trained on fine-tuning commit dataset without pre-training to verify the effectiveness of pre-training techniques. Furthermore, we compare {CommitBART}\xspace with four state-of-the-art pre-trained models for code (i.e., CodeBERT~\cite{feng2020codebert}, PLBART~\cite{ahmad2021unified}, CodeT5-base~\cite{wang2021codet5} and UniXcoder~\cite{guo2022unixcoder}), which are trained on large corpus of code data to validate the effectiveness of utilizing commit data to pre-train a model for commit-related tasks. We directly employ these released pre-trained models with default configuration for comparison. 
In addition, to confirm the effectiveness of our pre-training tasks, we further add two baselines, Incr-CodeBERT and Incr-PLBART, which are incrementally trained on CodeBERT and PLBART on our collected commit benchmark using masked language modeling (MLM) and text infilling respectively.









\section{Introduction}
\input{files/introduction}
\section{Background}
\input{files/related}
\section{A Large-scale Benchmark for Commits}
\input{files/benchmark}
\section{CommitBART}
\input{files/model}
\section{Experimental Setup}
\input{files/setup}
\section{Experimental Results and Analysis}
\input{files/results}
\section{Conclusion}
\input{files/conclusion}

\newpage
