\section{Introduction}
The later a defect is identified in the software development life cycle, the higher its impact and maintenance cost~\cite{planning2002economic}. To identify and prevent defects early on, Just-in-time (JIT) defect identification~\cite{mockus2000predicting}\cite{kamei2012large} is proposed as it can identify a risky commit at check-in time. Identifying risky commits at commit-time allows practitioners to dedicate their limited testing resources to review and fix the riskiest commits while changes are still fresh in the authors' minds.

\textit{Explainability} is a desired property of a prediction model designed for a real life development environment. For a JIT approach to be called \textit{explainable}, the developers shall be able to trace back predictions to their originating documents. An example would be highlighting parts of a change that look similar to a past buggy change. The state-of-the-art JIT defect prediction approaches such as DeepJIT~\cite{hoang2019deepjit} and CC2Vec~\cite{hoang2020cc2vec} are not \textit{explainable} because of reliance on a complex underlying learning model such as a deep neural network model.
Deep learning models are not only cost-prohibitive due to the time and resources needed for training but inherently lack \textit{explainability}~\cite{dam2018explainable}.

To address these limitations of prior work, in this paper, we propose IRJIT, an \textit{explainable} information retrieval (IR) approach for JIT defect identification. IRJIT classifies a new commit into clean or buggy based on its similarity with past commits. To compute similarity, IRJIT exploits BM25 algorithm~\cite{BM25}.
Next, for each identified buggy commit, IRJIT estimates its buggy lines using tf–idf term weights~\cite{salton1978term}. By relying on source code token frequencies instead of hand-crafted features such as code change or review metrics~\cite{mcintosh2017fix}, we make our approach readily deployable for any project with an active version control system and an issue tracker. Our approach offers several advantages over recent JIT approaches~\cite{hoang2019deepjit}~\cite{hoang2020cc2vec}:
\begin{itemize}
	\item IRJIT works in an \textbf{online} manner which means it adapts over time by learning from new changes, as soon as they arrive, without the costs associated with re-training models. Being able to update online and quickly react to a change is desirable for the adoption of a JIT approach~\cite{wan2018perceptions}.
	
	\item IRJIT is \textbf{explainable}~\cite{pornprasit2021jitline}\cite{dam2018explainable}. Developers can inspect source code documents that were used in prediction to understand why a commit was classified as buggy. These characteristics improve transparency and increase a practitioners' trust in the prediction~\cite{dam2018explainable}, while also making the fixing of an actual bug easier.
	
	\item IRJIT is \textbf{simple} since it draws its conclusions by matching against the past source code documents that developers are already familiar with. Moreover, it does not rely on a frequent and extensive retraining process which can slow down the overall prediction process. 
	
\end{itemize}

To establish the strengths and weaknesses of our approach we compare it with a recent machine learning (ML) approach JITLine~\cite{pornprasit2021jitline}.
JITLine is a sophisticated approach that relies on code and review based metrics~\cite{mcintosh2017fix} and bag of word tokens. JITLine engages in extensive tuning through SMOTE and Differential Evolution to directly optimize metrics and word vectors for AUC. We believe that such an approach might delay the \textit{online} prediction process~\cite{tan2015online}\cite{cabral2019class}. Furthermore, the review metrics used by JITLine are extracted from Gerrit review databases~\cite{gerrit} which are not frequently used by open-source projects, hence limiting the usability of JITLine to only well-documented and well-reviewed systems. We believe that the excellent performance of JITLine might be specific to OpenStack and Qt project due to the use of review metrics. Lastly, JITLine, despite being more sophisticated than prior approaches, leverages SMOTE~\cite{chawla2002smote} to create new training examples, as a consequence of which it cannot retrieve originating commits that can be blamed for a bug. In other words, it cannot explain the predictions by linking to the original documents causing the classification, although it can rank lines by defectiveness. 

Our case study consists of an existing dataset containing $37,524$ commits from  OpenStack and Qt projects, and an extended dataset of $6$ diverse open-source projects containing $140,344$ commits. Through a realistic evaluation, we show that our approach correctly classifies a new commit with a median F-score of $0.40$ and AUC of $0.85$. IRJIT outperforms JITLine at ranking buggy lines of a buggy commit. The contributions of this work include: 
\begin{itemize}
	\item{a JIT defect prediction approach based on IR to detect buggy commits at commit check-in time;}
	
	\item{an approach to identify associated buggy lines for each identified buggy commit;}
	
	\item{a realistic evaluation of our approach on OpenStack, Qt, and $6$ other open-source projects using various metrics, and its comparison with a mature machine learning baseline, JITLine.}
	
\end{itemize} 
\section{Related Work}
\label{sec:1}
\textit{JIT defect prediction} emphasizes the identification of risky changes just-in-time, i.e., before they reach repository. Existing JIT approaches detect bugs based on historical information such as change histories~\cite{buggyorclean} or by relating past defects to software metrics~\cite{commitguru}\cite{kamei2012large}.
CLEVER~\cite{clever} detects new defects by comparing new code to known cases of buggy code. It uses clone detection to improve the detection accuracy of risky changes identified through a metric-based model. Unlike CLEVER we do not use a metric-based model to identify buggy changes. DeepBugs\cite{deepbugs} detects incorrect code by reasoning about identifier names, and training a binary classifier on correct and incorrect examples. We leverage code similarity between past buggy and clean changes to \textit{explain} why a new change is classified as buggy. This line of work is inspired by the success of code search and recommendation engines~\cite{aroma}\cite{facoy} which employ IR techniques. IR-based methods work by matching shared vocabulary between query and corpus documents, and are effective for code retrieval~\cite{sourcerer2009}\cite{lancer} and bug localization~\cite{rahman2018improving}.
 
Due to the repetitive nature of source code~\cite{hindle2012}, statistical language models can also spot defective code effectively~\cite{ray2016}\cite{campbell2014syntax}\cite{santos2018syntax}. Yan \emph{et al.}~\cite{yan2020JIT} use several change level features and software naturalness with the n-gram model to localize buggy changes. The proposed framework ranks buggy lines in order of suspiciousness. 
Recently, Pornprasit \emph{et al.}~\cite{pornprasit2021jitline} showed that their ML model outperforms n-gram model in prioritizing defective lines. Their model also beats sophisticated DL approaches like DeepJIT~\cite{hoang2019deepjit} and CC2Vec~\cite{hoang2020cc2vec}, which are computationally expensive and also lack explainability. A recent survey by Wang \emph{et al}~\cite{wan2018perceptions} suggests that developers prefer simple, explainable and online defect prediction approaches. In this paper, we design and evaluate a simple IR approach for JIT defect prediction. Our approach improves the explainability of JIT defect prediction by showing past examples that a prediction is based on while also being online and requiring limited training. 


\section{IRJIT Overview}
\label{sec:methodology}

IRJIT can be integrated into git through a commit hook~\cite{hooks}, or a pull request bot, which is triggered when a developer initiates a code commit. Before this new commit reaches the project central repository, IRJIT will analyze the source code modified by the developer and compare it against past changes in the project's version control system. If the new commit matches a past buggy commit, the developer is prompted to inspect their code. In this way, IRJIT can prevent the bugs from reaching the main branch. 
Figure~\ref{fig:methodology} shows the backend design of IRJIT framework. IRJIT extracts past buggy and clean commits (Section~\ref{sec:2a}) of a project and their associated source code changes (Section~\ref{sec:2b}). The source code changes are stored into an inverted index for querying (Section~\ref{sec:2c}). For each newly intercepted commit, IRJIT queries the index to identify matches of the newly committed change, and determine if the new changes are similar to past buggy changes. If a newly committed change matches a known buggy change in the indexed corpora, it is flagged as risky (Section~\ref{sec:2d}). Lastly, for each buggy change, IRJIT ranks source code lines that are part of the change according to their riskiness (Section~\ref{sec:2e}).


\begin{figure}
	\centering
	\includegraphics[scale=0.7]{MethodologyFigure-IR-JITLine.pdf}
	\caption{An overview of IRJIT operation. IRJIT first extracts and indexes past changes. When a new commit arrives it is classified based on its similarity with the past matches retrieved from the index. Finally, changed lines are ranked according to bugginess.
	}
	\label{fig:methodology}
\end{figure}

\subsection{Extracting buggy and clean commits}
\label{sec:2a}
As a first step, IRJIT builds a corpus of buggy and clean commits. A commit contains one or more lines of code modified by a developer and spread across one or multiple files. To obtain buggy commits from a project's history researchers use SZZ algorithm~\cite{szz2005,szz2006}. 
SZZ works on the information available in the version control and issue tracking system. In the first step, the algorithm identifies a defect-fixing commit. A commit is \textit{fixing} if it fixes a defect, i.e., it refers to an issue labeled as \textsc{fixed} in the issue tracking system. In the second step, SZZ traces back through the version control to identify the associated \textit{buggy} commit i.e. the one responsible for inducing the fix. For this purpose, SZZ algorithm leverages \textit{diff} and \textit{annotate/blame} functionality. The \textit{diff} helps identify lines that changed between a defect-fixing commit and its immediately previous commit. These are the lines that fixed the bug. The \textit{annotate/blame} functionality identifies commits that originally introduced or modified those lines in the past. SZZ flags those original changes as defect-introducing or buggy if the commit was made before filing the issue in project issue tracker. Once algorithm completes its execution, all buggy and fixing commits are identified, but there are other commits that are unmarked by SZZ because they are neither buggy nor fixing. Following prior work we extract and include them in our analysis~\cite{yan2020JIT}\cite{ISSTA2021}. From here onwards, we only use the terms \textit{buggy} and \textit{clean} to refer to commits.


\subsection{Extracting source code changes from commits}
\label{sec:2b}
A \textit{change} refers to modifications made in a single file in one commit. The source code changes introduced by a commit may be spread across multiple files modified in that commit. 
Each change contains several hunks, i.e., a continuous group of lines added to or deleted from each changed file associated with a given commit~\cite{bugmemories}. IRJIT extracts hunks using the \textit{git diff} utility and pre-processes them to remove blank lines.
The processed source code changes that belong to a single file are then stored in its corresponding JSON format document along with the necessary metadata which includes \texttt{lines added}, \texttt{lines deleted}, commit hash, file name, and label.
Currently, IRJIT uses \texttt{lines added} by default, because the probability of introducing a bug is higher with the addition of new lines, as suggested by Purushothaman \emph{et al}~\cite{purushothaman2005toward}, and hence spotting bugs might be easier with only \texttt{lines added} field. 


\subsection{Indexing past source code changes}
\label{sec:2c}
To make the classification of a new commit scalable and efficient, IRJIT builds inverted indexes using Elasticsearch, which is a real-time full-text search engine built on top of the Apache Lucene library~\cite{lucene}. Elasticsearch indexes complex data represented as JSON documents. 

After creating an index, the past changes or JSON documents are added to the index, but the new changes (test data) are used for querying, so those are not indexed. In practice a single index per project can serve the purpose of storing all past changes. The index can be updated by incrementally adding new changes when they arrive, without the need of re-indexing any previous documents. Unlike existing approaches that require retraining the model every time some new data is added, IRJIT's approach allows developers to query against the new changes right away. The ability to remain online allows IRJIT to be deployed for repositories of any size.

As mentioned earlier, the indexed documents contain \texttt{lines added} and \texttt{lines deleted} fields which are tokenized, indexed, and analyzed depending on the user specified index settings. IRJIT relies on various tokenizers and filters to build custom Elasticsearch analyzers. After experimenting with multiple analyzers, we found that they exhibit similar performance for our use-case with minimal to no difference. Therefore, IRJIT uses only camel-case analyzer~\cite{pattern} and shingle analyzer~\cite{shingle} because together these two slightly outperformed other analyzers. Table~\ref{table:analyzers} shows how an example source code line is processed by camel-case and shingle analyzer. The camel-case analyzer with whitespace tokenizer~\cite{whitespace} is designed to split words into tokens when it encounters a case change. The shingle analyzer with a custom tokenizer based on \textit{javalang} analyzer~\cite{javalang} is designed to produce word n-grams of size 4 as min\_shingle\_size=4 and max\_shingle\_size=4. The choice of n-gram size was inspired by Siamese~\cite{siamese} and allows capturing sequences as well
as small modifications within statements.


\begin{table}[tbp]
	\centering
	\caption{Elasticsearch analyzers used in IRJIT}
	\begin{tabular}{ll}
		\hline
	
		\textbf{Analyzer} & \textbf{Tokens for input line:} \textit{if(tbl != null \&\& !isExternal(tbl))}\\ 
		\hline
		
		camel-case &  \textit{[ if(tbl , if , tbl , != , null , \&\& , !isExternal(tbl)) , is , }\\
			&\textit{  External , tbl ]}\\ 
		
		shingle & \textit{[ !=null\&\&! , !isexternal(tbl , \&\&!isexternal( , (tbl!=null ,}\\
			&\textit{	 (tbl)) , if(tbl!= ,
		 		isexternal(tbl) , null\&\&!isexternal ,}\\
		&\textit{	 tbl!=null\&\& ]}	\\
		
		\hline
	\end{tabular}
	\label{table:analyzers}
\end{table}

\subsection{Classifying an incoming commit}
\label{sec:2d}
\subsubsection{Querying to find matches}
\label{sec:MLT}
When a new (test) commit is checked-in to a version control system, IRJIT classifies it as buggy or clean by finding matches of test commit from the past project changes. In order to classify a test commit, IRJIT first extracts source code changes from that commit and saves each change separately. Next, it formulates a query against each change using the \texttt{lines added} field of corresponding change as input. Lastly, it executes the query and retrieves matches from the previously created index. Since one query corresponds to only one change, IRJIT may issue multiple queries for a given test commit.
Specifically, IRJIT formulates a \textit{more\_like\_this} (MLT) query~\cite{MLT}, which finds indexed training documents that are ``like" a given test document. MLT query utilizes corpus-level and document-level term statistics, i.e., term frequencies and inverse document frequencies (tf–idf) to compute similarity between a query and documents in the corpus. By default Elasticsearch uses BM25 algorithm\cite{BM25} to compute similarity. Moreover, it uses the same analyzers during the querying process that were originally used for indexing the documents. Elasticsearch also allows users to configure a different similarity algorithm than BM25, but the default one was sufficient for the purpose of our study. Based on BM25 similarity results, Elasticsearch returns Top-K most similar training documents for a given test document. Elasticsearch also returns a relevance score against each match as shown in Figure~\ref{fig:defectpred}. The matching documents from all MLT queries are combined into a set called candidate matches and sorted according to relevance score. The Elasticsearch scoring formula~\cite{scoring}
is originally based on tf–idf, but contains adjustments to allow for comparison of scores across different queries~\cite{campbell2016unreasonable}. Next, IRJIT's classifier determines a final label for the test commit.

\subsubsection{Determining label from matches}
IRJIT currently implements two classifiers; a K-nearest neighbor (\texttt{KNN}) classifier and a threshold-based (\texttt{Threshold}) classifier. These take candidate matches from the last step as input and return either a \textit{buggy} or a \textit{clean} label for the test commit.

\paragraph{K-nearest neighbor classifier}
The K-nearest neighbor classifier takes K closest candidate matches and uses their majority label to make predictions. If the majority of K neighbors are buggy, the test commit is labeled buggy; otherwise, it is considered clean. The value of K might be tuned for each project to optimize performance. Alternatively, the user can provide it as input. Currently, IRJIT uses a default value of $K=3$ which was set empirically after experimenting on $K={1,3,5,7,9}$.

\paragraph{Threshold classifier}
The candidate matches returned for the test commit contain both buggy and clean changes from the past. Intuitively, a test commit may be considered buggy if there is at least one buggy change in the candidate matches. However, such a model would be considered biased towards buggy changes. To counteract this bias, we only accept a \textit{buggy} prediction if sufficient confidence can be established. Our threshold classifier uses a threshold value, T$_{score}$, to determine when to accept a buggy prediction. 
The threshold score is applied to the relevance score returned by Elasticsearch.
If there is at least one buggy change in the candidate matches with a relevance score greater than T$_{score}$, IRJIT classifies the commit as buggy. In case of more than one buggy change, the top match, i.e., the match with the highest relevance score, is used. In the example shown in Figure~\ref{fig:defectpred}, the incoming commit is predicted buggy based on the candidate match M${_2}$ which is the top relevant buggy match and also has a relevance score $(104)$ greater than T$_{score}$. 

The T$_{score}$ is project specific, and its higher values favor precision whereas the lower values favor recall. To ensure the best performance, T$_{score}$ can be tuned for any performance metric such as precision, recall, F1, or AUC using a validation dataset. Our work uses the training data for validation. In order to find an optimal T$_{score}$ value, IRJIT iterates over the set of relevance scores returned from the validation dataset evaluation. It then picks a T$_{score}$ value that maximizes performance across all queries for the AUC metric. Unlike SMOTE that tunes on metrics and word vectors of each commit, we tune once for each model. Our analysis suggests that the identified T$_{score}$ will be relevant unless more than 50\% of the data changes. In practice T$_{score}$ will only have to be recalculated after several months.

\paragraph{Ensemble classifier}
Ensemble learning is known for reducing bias and improving performance by combining the output of a set of base classifiers. This motivated us to incorporate an ensemble classifier in IRJIT because it does not slow down an online prediction process while still offering enhanced performance. An ensemble classifier can use one of the various meta-algorithms such as bagging, boosting or stacking on top of base classifiers to combine predictions. For classification problems the output of each base model might be combined using a hard or soft voting mechanism, which leverages discrete class labels and probabilities respectively. Liu \emph{et al}~\cite{liu2010evolutionary} proposed a validation and voting scheme similar to boosting whereby the output is decided based on the majority vote of classifiers. On the other hand Panichella \emph{et al}~\cite{panichella2014cross} used the outputs of multiple classifiers as predictors of a new prediction model, which is trained using Logistic Regression.
Yang \emph{et al}.~\cite{yang2017tlel} proposed TLEL, an ensemble approach for JIT defect prediction, that trains several Random Forest models using bagging and then combines them using stacking. We adopt the concept used in TLEL whereby all classifiers are given equal weight and their predictions are summed to get a final result. 
Recently, Zeng \emph{et al}~\cite{ISSTA2021} achieved great performance by training a simple Logistic Regression classifier for JIT defect prediction using a single feature, i.e., the number of lines added to a commit (la) and no optimization. Their \texttt{la} classifier has good AUC and F1 but a low recall, which inspired us to ensemble our KNN and Threshold classifier with their \texttt{la} classifier. Altogether, we create four ensemble models following TLEL approach; first using \texttt{Threshold} and \texttt{KNN}, second using \texttt{Threshold} and \texttt{la}, third using \texttt{KNN} and \texttt{la}, and fourth using \texttt{Threshold}, \texttt{KNN} and \texttt{la}. Next, we report the results of our base classifiers and the best performing ensemble classifier.


\begin{figure}
	\centering
	\includegraphics[width=0.8\linewidth]{defectpred_v2}
	\caption{ Figure shows set of candidate matches M$_{1}$, M$_{2}$, M$_{3}$ upto M$_{n}$ with relevance score 157, 104, 78 and 60 respectively corresponding to a commit, C1.  \texttt{KNN} classifier with \textit{K=3} predicts commit C1 is \textbf{clean} because majority, i.e., 2 out of top-3 matches are clean.  \texttt{Threshold} classifier with \textit{T$_{score}$=100} predicts C1 is \textbf{buggy} because relevance score of top buggy match M2 is 104 which is greater than \textit{T$_{score}$}}
	\label{fig:defectpred}
\end{figure}

\subsection{Ranking buggy lines of a commit}
\label{sec:2e}
If a change is identified as buggy, the last step is to rank the lines modified in that change according to their bugginess. Prior work~\cite{hindle2012, ray2016, yan2020JIT} exploits n-gram models to find most entropic source code tokens. However, we adopt a slightly different approach to estimate the buggy tokens, i.e., tokens that contribute towards overall bugginess of the document. We rely on tf–idf term weights obtained while executing \textit{more\_like\_this} query (Section~\ref{sec:MLT}). The terms that get highest weight when a document is predicted buggy are considered buggy tokens. After extracting buggy tokens we rank lines based on the number of occurrences of buggy tokens in each line. Lastly, we use ranked buggy lines to calculate our performance metrics.

\section{Evaluation Setup}
\label{sec:eval}
In this section, we present research questions, details of the evaluated datasets, evaluation context, and performance measures.
\subsection{Research Questions}

\textbf{RQ1:} How effective is IRJIT in identifying new buggy commits?

The goal of this question is to investigate if our IR-based approach accurately predicts buggy commits or not. 
 
 
\textbf{RQ2:} Does the performance of IRJIT change if trained only on the recent data?
 
This research question aims to evaluate the effect of training models on recent data on IRJIT's performance.
 
\textbf{RQ3:} How effective is IRJIT in comparison to state-of-the-art machine learning approach JITLine?
 
JITLine is an effective ML approach that outperformed CC2Vec~\cite{hoang2020cc2vec}, DeepJIT~\cite{hoang2020cc2vec} and EARL~\cite{kamei2012large} on the basis of multiple performance measures. Although the goal of our paper is not to beat JITLine by numbers, comparing against it gives us a fair idea of the strengths and weaknesses of our approach.
 
 
 
\textbf{RQ4:} How well does IRJIT perform in predicting buggy lines of a buggy commit?
 
Prior work~\cite{ray2016,yan2020JIT} suggests that focusing on more entropic or unnatural lines can save inspection effort. The goal of this evaluation is to find out if IRJIT can effectively rank buggy lines in the correctly predicted buggy commits or not. Such a ranking saves developers' time because they only need to inspect the top few lines to identify the error. 
 
\subsection{Dataset}
We begin our evaluation with OpenStack and Qt dataset, which was originally collected by McIntosh and Kamei~\cite{mcintosh2017fix} and later used in several studies~\cite{pornprasit2021jitline,hoang2020cc2vec,hoang2019deepjit}. This dataset contains code change and review properties extracted from the version control system and Gerrit code review database respectively. These are further explained in~\cite{mcintosh2017fix} as well as our replication package.

To extend our evaluation, we selected 6 additional open-source projects from GitHub following criteria similar to prior work~\cite{yan2020JIT}\cite{cabral2019class}. While OpenStack and Qt are mainly written in python and C++, we selected open-source Java projects to cover a diverse range of applications from different programming languages. Our selection criteria were that the project should use a Jira issue tracker, should have a rich history ($>$$10k$ commits), and should be popular ($>$$1000$ stars). Following this criteria we found $886$ projects out of which we randomly picked $6$ projects. These projects were also used in prior software engineering studies~\cite{neto2018impact}~\cite{yan2020JIT}~\cite{cabral2019class}~\cite{tabassum2020investigation}. Table~\ref{table:dataset} presents the distribution of commits for each evaluated project.

We collected the buggy commits for these projects using an open-source SZZ implementation, SZZUnleashed~\cite{szzunleashed}.
Our approach uses source code changes which we extracted those from each project's GitHub repository using GitPython. For calculating code change metrics for the baseline approach (JITLine) we wrote our own scripts. 

\subsection{Training and test set}
JIT defect prediction is useful if done in a timely manner using only past data available at commit time~\cite{ISSTA2021}\cite{cabral2019class}. Within-project defect prediction, uses past data from the same project, whereas cross-project defect prediction borrows training data from other projects~\cite{tabassum2020investigation}. This paper only evaluates the within-project context, so each model is trained and tested on the same project. We strictly avoid time-travel~\cite{bangash2020time} in our evaluation by not violating the chronological order of data, and our test data always comes after the training data. Another important consideration is that a JIT model should be able to adapt over time by learning from new data as soon as it becomes available. Such models are referred to as \textit{online}~\cite{cabral2019class}\cite{tabassum2020investigation}. One way of updating an \textit{online} model is to use new data along with any prior data for retraining the model. Another way is to discard old data and retrain the model using only new data. We consider both evaluation settings and refer to them as increasing-model and constant-model evaluation. 

\underline{\textit{Increasing-models}} are trained using \textit{all} data that is available prior to the test period. As the test period moves farther in time the training set increases, hence the name increasing models. 

\underline{\textit{Constant-models}} are trained using data from a constant number of time periods prior to the test period. In this paper we use just \textit{one} time period prior to test period for training.

Finally, when making predictions, we only evaluate against the past data that has already received its correct labels. Leaving a gap between the train and test set allows time for bugs to be identified and commits to get their correct label. This gap is referred to as verification latency in literature~\cite{cabral2019class}. Tan et al.~\cite{tan2015online} suggested that the sum of gap and test set should be close to typical bug-fixing time, i.e., the time from when a bug is introduced to when it is fixed. However, since bug-fixing time varies for different projects and bugs, this gap can also change with the dataset and is empirically set using the median values of \textit{Bug-fix delay} in Table~\ref{table:dataset}. 

\subsection{Baseline Approach} 
Our baseline is a state-of-the-art JIT approach, called JITLine~\cite{pornprasit2021jitline} proposed by Pornprasit \emph{et al} in 2021. The approach predicts buggy commits using a ML model
built from code change and review properties. To address the data imbalance problem caused by a low percentage of buggy commits, JITLine  invents new documents with the help of SMOTE technique that is optimized using a Differential Evolution (DE) algorithm. 

We chose JITLine as our baseline because of its effectiveness as it has been shown to outperform several deep-learning approaches such as CC2Vec~\cite{hoang2020cc2vec}, and DeepJIT~\cite{hoang2019deepjit} as well as an n-gram based approach of Yan \emph{et al}~\cite{yan2020JIT}. JITLine's replication package provides a Python implementation along with OpenStack and Qt dataset. 
As mentioned earlier, we anticipate that JITLine's great performance might be specific to OpenStack and Qt because those are well-documented and well-reviewed projects. We argue that such projects are rare, therefore, we extend our evaluation by including 6 projects that do not employ a Gerrit review database. For these projects \textit{review} and \textit{reviewer} metrics could not be extracted. We, therefore, modified JITLine to work without the aforementioned classes of metrics, and the modified version is referred to as review-free JITLine, JITLine$_{rf}$. 


\subsection{Performance Metrics}
\label{evalmeasure}
Our evaluation is based on metrics that were used by JITLine~\cite{pornprasit2021jitline}, and are also commonly used in other JIT studies~\cite{ISSTA2021,kamei2012large,mcintosh2017fix}. \textbf{\textit{Precision}} measures the relevant documents in the results, which in our case is correctly predicted buggy commits out of all commits predicted as buggy. \textbf{\textit{Recall}} measures the relevant documents retrieved out of all possible relevant documents, i.e., buggy commits. It does not consider the irrelevant documents, i.e., clean commits retrieved along. \textbf{\textit{F-score (F1)}} is the harmonic mean of precision and recall. \textbf{\textit{FAR}} or false alarm rate measures the ratio of incorrectly predicted buggy commits to the number clean commits. A lower FAR value indicates that developers need to review fewer incorrectly predicted commits and developer effort is not wasted. \textbf{\textit{AUC}} is the area under the ROC curve which plots true positive rate against the false positive rate. It is a threshold-independent measure and evaluates an approach's ability to distinguish between classes.  \textbf{\textit{Distance-to-Heaven (d2h)}} combines recall and FAR metrics. Smaller values of d2h are better. A d2h value of 0 suggests that the approach correctly predicts all buggy commits without any false positives. A high d2h value indicates that an approach is far from perfect, i.e., it achieves a high recall value but also has a high FAR value and vice versa. 

Again, for the line-level evaluation we use the metrics used by JITLine~\cite{pornprasit2021jitline}. \textbf{\textit{TopK}} accuracy measures the ratio of actual buggy lines to the Top \textit{K} lines. A high ratio suggests that many of the buggy lines are ranked at the top. \textbf{\textit{Recall@20\%LOC}} measures the ratio of actual buggy lines that can be found given only top 20\% changed lines of code. \textbf{\textit{Effort@20\%Recall}} measure the amount of effort made by the developers to identify top20\% of buggy lines in a commit. A higher value means that more effort is needed. \textbf{\textit{Initial False Alarm (IFA)}} measures the amount of clean lines of a commit that need to be inspected before identifying an actual buggy line.


\newcommand{\specialcell}[2][c]{%
	\begin{tabular}[#1]{@{}c@{}}#2\end{tabular}}

\begin{table*}
	\centering
	\caption{Statistics of evaluated projects.}
	\label{table:dataset}
	
	\begin{tabular}{l||lrrr||rr||rr}
		\toprule
		
		\multicolumn{1}{l}{Project}& \multicolumn{1}{l}{Lang.}&
		
		\multicolumn{1}{c}{Commits} & \multicolumn{1}{c}{Buggy} &
		
		\multicolumn{1}{c}{\specialcell{Bug-fix \\delay}} &
		
		\multicolumn{2}{c}{\specialcell{No. of commits }}&
		
	
		\multicolumn{2}{c}{\specialcell{No. of commits }}\\ 
			
		\cmidrule(lr){1-1}
		\cmidrule(lr){2-2}
		\cmidrule(lr){3-3}
		\cmidrule(lr){4-4}
		\cmidrule(lr){5-5}
		\cmidrule(lr){6-7}
		\cmidrule(lr){8-9}
			\multicolumn{1}{l}{}&
		\multicolumn{1}{l}{}&
		
		\multicolumn{1}{c}{} &
		\multicolumn{1}{c}{} &
		
		\multicolumn{1}{c}{} &
		
		\multicolumn{1}{c}{Min}&
		\multicolumn{1}{r}{Max}&
		
		\multicolumn{1}{c}{Min}&
		\multicolumn{1}{r}{Max}\\ 
		
		\cmidrule(lr){6-7}
		\cmidrule(lr){8-9}
		
		
	
		activemq  & Java   & 10,144	 & 16.4\% &  428  & 206   & 10,144 & 206   & 1,849\\
		camel     & Java   & 49,473	 & 8.7\%  &  261  & 1,173 & 49,473 & 1,173 & 4,442\\
		hadoop    & Java   & 18,179	 & 5.2\%  &  277  & 204   & 18,179 & 204   & 3,099\\
		hbase     & Java   & 18,266	 & 21.5\% &  187  & 258   & 18,266 & 258   & 2,075\\
		hive      & Java   & 15,100  & 35.3\% &  158  & 96    & 15,100 & 96    & 2,352\\
		lucene    & Java   & 29,182  & 9.5\%  &  382  & 66    & 29,182 & 66    & 4,203\\
		OpenStack & Py.    & 13,304  & 12.2\% &  280  & 616   & 13,303 & 616   & 6,291\\
		Qt        & C++    & 25,704  & 7.1\%  &  526  & 3,564 & 25,704 & 3,564 & 9,281\\
		\bottomrule
	\end{tabular}
\end{table*}


\section{Evaluation and Results}




\subsection{Commit-level performance of IRJIT (RQ1)}
\underline{\textit{Approach:}}
We ran IRJIT on each project shown in Table~\ref{table:dataset} using a within-project setting. For evaluation at any point in time, the project's past forms the training set, whereas the future forms test set. For an online learning problem~\cite{tabassum2020investigation} such as JIT defect prediction, this kind of evaluation is more realistic than an 80:20 split. It aims to replay an online setting where changes arrive over time and predictions are only based on past changes. Newly proposed JIT approaches should run their evaluation in a similar manner to ours. In this way, performance estimates will be closer to what developers might observe in a real-life setting.

\underline{\textit{Results:}}
Figure~\ref{fig:combinedplotrerankersIC} presents the performance of IRJIT on each evaluated project using an \textit{increasing} evaluation setup.
Each boxplot represents the range of performance values obtained by evaluating various models. Each model uses training and test data from a different period during a project's evolution and runs using two independent classifiers from IRJIT; a \texttt{KNN} classifier and a \texttt{Threshold} classifier. A quick look at Figure~\ref{fig:combinedplotrerankersIC} shows that there is no clear winner between classifiers. The \texttt{Threshold} classifier takes the lead in terms of recall, but it comes at the cost of mis-classifications which affects the AUC. One reason for the sudden performance variations of \texttt{Threshold} classifier is the quality of chosen T$_{score}$. Since we chose a T$_{score}$ that maximizes AUC, there is no guarantee that all other metrics will achieve their best values. 

On the other hand, the \texttt{KNN} classifier shows a more moderate and relatively consistent performance across different metrics and projects.
The highest median value of AUC and F1 were also achieved by the \texttt{KNN} classifier and were $0.69$ and $0.66$ for the \textit{hbase} and \textit{hive} project respectively. 
Across all evaluated projects the median AUC is $0.65$ and $0.38$ for \texttt{KNN} classifier and $0.65$ and $0.37$ for \texttt{Threshold} classifier. The best performance was achieved by the \texttt{Ensemble} (of \texttt{KNN} and \texttt{la}) classifier as it got a median AUC of $0.85$ and median F1 score of $0.40$. 

In our evaluation, the \textit{increasing} models (Figure~\ref{fig:combinedplotrerankersIC}) accumulate changes over time, which means that as time progresses, more commits arrive, and each subsequent model is trained on a larger number of changes than the model prior to it. This, however, can affect the model in two ways; either the model learns to make better predictions by training on an increased volume of data, or it loses its power due to concept drift~\cite{tabassum2020investigation} and more noise which affects all online learning problems~\cite{hoens2012learning}~\cite{cabral2019class}. Due to being trained on a higher number of changes, \textit{increasing} models show a limited variation in performance per project, as a consequence of which the AUCs are generally above $0.6$ and FAR and d2h dropped. However, since the new source code changes might be irrelevant to very old changes, they just cause noise which prevents \textit{increasing} models from achieving their true potential.

\findingbox{
	\textit{Finding 1: IRJIT can identify buggy commits across 8 projects with a reasonable performance obtaining a median AUC of 0.65 for both the Threshold and the \texttt{KNN} classifier and 0.85 for the ensemble classifier. The median F1 scores were 0.37, 0.38 and 0.40 respectively for Threshold, KNN and ensemble classifier respectively.}
}





\subsection{Effect of recent changes on performance of IRJIT (RQ2)}
\underline{\textit{Approach:}}
We now evaluate the performance of IRJIT by only using recent changes. Instead of using the entire past for training, the changes used as part of training set now come from time period closest to the test period. This evaluation shows the performance of IRJIT when entire past data is not available, or developers do not want to use it for any reason.

\begin{figure*}
	\centering
	\includegraphics[width=0.95\linewidth]{combinedplot_rerankers_ensemble_IC}
	\caption{Performance of IRJIT increasing-models across 8 different projects and 6 different performance metrics. }
	\label{fig:combinedplotrerankersIC}
\end{figure*}


\begin{figure*
	\centering
	\includegraphics[width=0.95\linewidth]{combinedplot_rerankers_ensemble_CC}
	\caption{Performance of IRJIT constant-models across 8 different projects and 6 different performance metrics. \textbf{can be removed}}
	\label{fig:combinedplotrerankersCC}
\end{figure*}

\underline{\textit{Results:}}
The threshold classifier achieves an AUC of $0.65$ and F1 of $0.36$, whereas the \texttt{KNN} classifier achieved an AUC of $0.63$ and an F1 score of $0.35$. The \texttt{Ensemble} classifier remains on top with an AUC of $0.85$ and F1 of $0.38$ which is a bit lower than RQ1. We compared the classifier results from constant-models and increasing-models using a pairwise Wilcoxon signed-rank test and did not find any significant differences between classifiers for any metric.

Similar to \textit{increasing-models} (RQ1), there are large performance variations within each project, which are a consequence of the changing size and characteristics of commits used for evaluating each model. Although, one might think that data size is relatively less of an issue for \textit{constant-models} evaluation because they train only on data from the latest period. However, we observed large variations in the number of commits coming from each time period. For example, we found that commits arrive with a high frequency during the early development stages of a project and later they become sparse. Consequently, earlier time periods have a different distribution of commits than the later ones which can affect performance. Similarly, the number of buggy instances present in the data of each time period also vary drastically. Most bugs appear early in the life cycle, and once the project matures, the frequency of bug occurrences also reduces. These two factors lead to significant variations in performance across periods. Based on their knowledge about the project, the developers can better decide how much past data might be most helpful in building their JIT models.

\findingbox{
	\textit{Finding 2: There is a slight drop in the performance of IRJIT when models are trained only on the recent changes. However, this difference is project specific, i.e., across periods, and the cumulative effect across projects is negligible. According to Wilcoxon signed-rank the differences are not statistically significant.}
}




\subsection{Comparison of IRJIT with baseline JITLine (RQ3)}
\underline{\textit{Approach:}}
Before we compare IRJIT and JITLine we wanted to understand how the missing categories of \textit{review} and \textit{reviewer} metrics will impact JITLine's performance. So we reran original JITLine (JITLine$_{orig.}$) and JITLine$_{rf}$ following a constant-models evaluation setup for the OpenStack and Qt project. After that, we evaluate JITLine$_{rf}$ using both \textit{increasing} and \textit{constant} evaluation setup from RQ1 and RQ2.

\underline{\textit{Results:}}
JITLine$_{rf}$, which is our simplified review-free version of JITLine, performs similar to original JITLine on the OpenStack and Qt dataset. The F1 and AUC remains unchanged for OpenStack, but in case of Qt, F1 increased by $0.03$ whereas AUC dropped by $0.02$, as shown in Table~\ref{table:jitline}. The $\Delta$ shows magnitude and direction of change for each metric. Contrary to what is expected, the review and reviewer metrics do not change prediction results significantly.

\findingbox{\textit{Finding 3: According to the pairwise Wilcoxon signed-rank test, there is no difference between JITLine$_{rf}$ and JITLine$_{orig.}$, which means JITLine is expected to maintain its good performance on remaining projects even without \textit{review} metrics.}
}

\begin{table}[tbp]
	\centering
	\caption{JITLine$_{orig.}$ vs JITLine$_{rf}$ median values from constant-model evaluation }
	\begin{tabular}{ll|rrrrrr}
		
		\hline
		& Project& Prec. & Rec. & F1 & AUC & FAR & d2h \\ 
		\hline
		
	
	
	
	
	
	
	
	
		
		
	
	
	
	
	
	
	
	
		
		JITLine$_{orig.}$ & OpenStack & 0.47  & 0.06  & 0.10 &  0.78  & 0.01 & 0.67\\
		JITLine$_{rf}$ & OpenStack & 0.50  & 0.05  & 0.10 &  0.78  & 0.01 & 0.67\\
		${\Delta}$     & OpenStack & +0.03 & -0.01 & 0.00 &  0.00  & 0.00 & 0.00\\
		\hline
		JITLine$_{orig.}$  & Qt &  0.43  & 0.12 & 0.19  & 0.78  & 0.01  & 0.62\\
		JITLine$_{rf}$     & Qt &  0.31  & 0.20 & 0.22  & 0.76  & 0.03  & 0.57\\
		${\Delta}$         & Qt & -0.12  & +0.08& +0.03 & -0.02 & +0.02 &-0.05\\
		
		\hline
	\end{tabular}
	\label{table:jitline}
\end{table}

\begin{figure*}
	\centering
	\includegraphics[width=0.9\linewidth]{combinedplot_rerankers_ensemble_JITLine_IC_nogap}
	\caption{Comparison of IRJIT vs JITLine for increasing-models across 8 projects and 6 metrics}
	\label{fig:combinedplot_irjit_jitlineIC}
\end{figure*}

\begin{figure*}
	\centering
	\includegraphics[width=0.9\linewidth]{combinedplot_rerankers_ensemble_JITLine_CC_nogap}
	\caption{Comparison of IRJIT vs JITLine for constant-models across 8 projects and 6 metrics}
	\label{fig:combinedplot_irjit_jitlineCC}
\end{figure*}


Figure~\ref{fig:combinedplot_irjit_jitlineIC} and  Figure~\ref{fig:combinedplot_irjit_jitlineCC}, compares IRJIT and JITLine results for increasing-models and constant-models. Our approach outperforms JITLine$_{rf}$ for the OpenStack and Qt project. There is an improvement of $3.4\%$ - $6.1\%$ in terms of the AUC metric when JITLine is compared with \texttt{Ensemble} classifier. However, this trend does not hold for the remaining $6$ projects, where JITLine$_{rf}$ mostly performs better than IRJIT with occasional drops in performance. The median values of different metrics presented in Table~\ref{table:SD} are also close, but JITLine generally achieves higher values than the \texttt{Ensemble} classifier especially for F1.


\findingbox{
	\textit{Finding 4: IRJIT performs better than JITLine for the OpenStack and Qt project. Across the remaining $6$ projects, it also achieves a reasonable performance, but it does not outperform JITLine.}
}


To statistically evaluate the differences between JITLine$_{rf}$ and our approach, we used pairwise Wilcoxon signed-rank test~\cite{shull2007guide}, which is a non-parametric test to compare two samples, along with \textit{Bonferroni} correction~\cite{abdi2007bonferroni}. The \textit{null hypothesis} of the test is that there is no difference between the two groups. We could not reject the \textit{null hypothesis} for AUC and d2h but for precision, F1, and FAR we were able to reject the  \textit{null hypothesis} in favour of JITLine. For recall the hypothesis was rejected in favour of IRJIT.
Additionally, we use Cliff’s delta $|\delta|$ ~\cite{shull2007guide} to measure the effect sizes of the observed differences. The effect sizes are \textit{negligible} for $|\delta|$ $<$ $0.147$, \textit{small} for $0.147$ $\le$ $|\delta|$ $<$ $0.33$, \textit{medium} for $0.33$ $\le$ $|\delta|$ $<$ $0.474$, and \textit{large} for $|\delta|$ $\ge$ $0.474$. The effect sizes were \textit{small} for AUC, F1 and recall, \textit{negligible} for d2h but \textit{large} for precision and FAR.

\findingbox{
	\textit{Finding 5: According to Wilcoxon signed-rank test there is no statistically significant difference between IRJIT and JITLine in terms of AUC metric. However, there is a difference in terms of F1 which according to Cliff's delta is small.}
}


Table~\ref{table:SD} presents standard deviation (SD) across different runs of IRJIT and JITLine. The variation is not specific to a particular JIT approach. It further reiterates the variation-related discussion from RQ1 and confirms that there will be some variation in performance across evaluations, whether models are trained on the latest data or the entire past. However, complementing models with data from old periods helps mitigate variation to some extent, e.g., see the drop in SD column for JITLine from $0.24$ to $0.20$.


\begin{table}[tbp]
	\centering
	\caption{Median (M) and standard deviation (SD) of IRJIT ensemble classifier and JITLine for constant (C) and increasing (I) model evaluation}
	\begin{tabular}{l||rr|rr||rr|rr}
		\hline
		\multicolumn{1}{c}{}& \multicolumn{4}{c}{JITLine}&
		\multicolumn{4}{c}{IRJIT}\\ 
		\cmidrule(lr){2-5}
		\cmidrule(lr){6-9}
		
		\multicolumn{1}{c}{}&
		\multicolumn{2}{c}{Constant}&
		\multicolumn{2}{c}{Increasing}&
		\multicolumn{2}{c}{Constant}&
		\multicolumn{2}{c}{Increasing}\\
		
		\cmidrule(lr){2-3}
		\cmidrule(lr){4-5}
		\cmidrule(lr){6-7}
		\cmidrule(lr){8-9}
		
		Metrics     & M & SD  & M & SD & M & SD & M & SD \\	\hline
		AUC      	& 0.86 & 0.06 & 0.87 & 0.06 & 0.85 & 0.04 & 0.85 & 0.04\\
		F1  	    & 0.44 & 0.24 & 0.48 & 0.20 & 0.38 & 0.19 & 0.40 & 0.19\\
		Prec. 	    & 0.50 & 0.21 &	0.50 & 0.19 & 0.30 & 0.17 &0.28  & 0.18\\
		Recall 	    & 0.43 & 0.27 & 0.60 & 0.24 & 0.57 & 0.23 & 0.62 & 0.20\\
		FAR      	& 0.04 & 0.08 &	0.06 & 0.08 & 0.25 & 0.20 & 0.28 & 0.17\\
		d2h         & 0.41 & 0.17 & 0.29 & 0.15 & 0.37 & 0.12 & 0.35 & 0.10\\
		\hline
	\end{tabular}
	\label{table:SD}
\end{table}





\subsection{Line-level performance of IRJIT (RQ4)}
\underline{\textit{Approach:}} 
For this evaluation, we use the line-level ground-truth data provided by the JITLine authors for OpenStack and Qt project. For the remaining $6$ projects we collected line-level data following JITLine's methodology. We obtained the diffs for each clean commit and labeled the modified or deleted lines of that commit as buggy, and the remaining lines were labeled clean. After collecting data, we rank buggy lines of each predicted buggy commit using the tf–idf term weights or importance score of tokens obtained from Elasticsearch Explain API. We parse the Explain API output to identify the tokens that contributed to the prediction of a buggy commit. Once identified, the lines modified in a commit were searched for all the tokens and ranked according to the number of occurrences of distinct tokens found in each line. The ranked lines are then returned to the developer for inspection.

\underline{\textit{Results:}}
Figure~\ref{fig:linelevelCC} presents the results of line-level evaluation for 4 metrics. The median Top-10 accuracy of JITLine is $0.30$ which is lower than our approach which achieves $0.70$. Our approach can identify buggy lines with a median of Recall@20\%LOC of
$0.25$ which is $31\%$ higher than JITLine. Similarly, our approach uses $61\%$ less effort than JITLine to find top $20\%$ buggy lines. IRJIT also has a median IFA of $0$ indicating that a buggy line is always ranked as the top location. The Wilcoxon signed rank test confirms that there is a statistically significant difference between all line-level metrics. Cliff's delta suggests that these differences are \textit{large} for all metrics.

\findingbox{\textit{Finding 6: For line-level evaluation, IRJIT outperforms JITLine with a statistically significant difference for all metrics.}}


\begin{figure*}
	\centering
	\includegraphics[width=0.95\linewidth]{linelevel_CC}
	\caption{Comparison of IRJIT vs JITLine constant-models for ranking buggy lines of an identified buggy commit.}
	\label{fig:linelevelCC}
\end{figure*}

\section{Discussion}
In this section, we discuss implications of our work, its limitations and opportunities for future work.

\subsection{Implications}
\textit{Researchers should focus on designing simple, online, and explainable approaches for JIT defect prediction.} IRJIT is one such approach that developers can use for identifying risky commits and ranking source code lines by bugginess. Our results show that IRJIT is similar to state-of-the-art approach JITLine in terms of AUC, and, both approaches can be applied to any project but there are some obvious trade-offs. 

IRJIT is \textit{simple} compared to state-of-the-art approach JITLine because it only uses source code changes along with basic IR methods. These changes are available in the project's source code repository so IRJIT can be easily deployed with limited expertise and effort. Our work demonstrates that IR-based methods are competitive out of the box, suggesting that further tuning and using them in combination with other project-specific features (e.g., metrics) are likely to bridge the gap while requiring significantly less training and allowing the use of explainable models. 

IRJIT is \textit{explainable} so a developer using IRJIT can see past commits that lead to a buggy prediction, whereas a developer relying on JITLine cannot because the commits are invented using SMOTE and are not real documents. For line-level prediction, JILine uses LIME~\cite{ribeiro2016should} to learn a model locally around the prediction which is an additional cost on top of the original model.
IRJIT improved over JITLine by eliminating the need to tune extensively through SMOTE and train an altogether new model such as LIME for line-level prediction. 
IRJIT uses a single IR model to identify a buggy commit and rank its lines by defectiveness. It then retrieves past changes responsible for the prediction and associated fixes for those changes from the corpus. In short, anything flagged by IRJIT would have a concrete, context relevant example. In future work, we plan to conduct a user study to assess the usefulness of IRJIT for developers. We also plan to improve fixes keeping in view the contextual factors.


\textit{Researchers should make sure that their JIT approach is suitable for online deployment without considerable retraining.} We show that IRJIT works in an \textit{online} fashion, and that our best performing models do not require any tuning or retraining but just updating indexes when new commits come. We investigated JIT in an online scenario where new changes arrive over time and can be used for updating the JIT models. Our approach supports incremental updates without retraining the prediction model, and hence it is very adaptive and scalable. Contrarily, the baseline approach, JITLine relies on extensive retraining of their models through SMOTE and Differential Evolution. Therefore, JITLine can be more computationally expensive for online deployment, and might slow down the overall prediction process. However, JITLine might offer higher precision compared to IRJIT so a developer who wants very precise models and has resources for expensive retraining may choose JITLine over IRJIT. 

\textit{Researchers should conduct a realistic evaluation of JIT approaches.} Prior studies evaluated their JIT defect prediction approaches using the traditional 80:20 split for the data. Through an evaluation at different points in time, we show that both IRJIT and JITLine exhibit varying performance along time, and an 80:20 evaluation could overestimate (or underestimate) performance. Therefore, we insist that future studies should be evaluated following an evaluation methodology that represents a realistic application of JIT from a development perspective.







\subsection{Limitations and Opportunities for Future Work}
\subsubsection{\textbf{Impact of tuning}} Our best performing model is an ensemble classifier of \texttt{KNN} and \texttt{la} and it does not involve any tuning. However, our \texttt{Threshold} classifier relies on a threshold T$_{score}$ that is optimized for AUC using the training dataset. We observed a decent improvement in our AUC values after tuning for T$_{score}$. The T$_{score}$ is project-specific because of the differences in the size of changes and underlying vocabulary, but it is relatively quick and easy to tune for T$_{score}$.  One can also train for other metrics such as F1 or precision. In future work, we plan to investigate more advanced methods of adjusting the threshold. We will also examine the impact of changes in training data on the stability of the threshold. 

As for the \texttt{KNN} classifier we test for different K values but we observed  a drop in performance for higher values of K. The best results were achieved for $K=3$, therefore, IRJIT uses $K=3$ by default.

\subsubsection{\textbf{Impact of gap}} To avoid evaluating on data that has not received its correct labels, we leave a gap between our training and test sets while evaluating all projects except OpenStack and Qt. OpenStack and Qt dataset was curated by following strict quality criteria which accounts for such problems. For remaining projects, we chose a gap interval following recommendations of prior work ~\cite{yan2020JIT}\cite{tan2015online}\cite{ISSTA2021}\cite{cabral2019class}.
This gap means we waste some of the data that could be useful to our models leading to an underestimate of performance for both IRJIT and JITLine. 


\subsubsection{\textbf{Reliance on SZZ}} We rely on SZZ algorithm to obtain the ground truth dataset for evaluation. We as part of the software engineering community are aware of the shortcomings of SZZ algorithm that it only identifies large defects that get reported to the issue tracking system and therefore
are important enough to be fixed. Other kinds of datasets such as ManySStuBs4J~\cite{manysstubs} are not widely available so we had to rely on SZZ for data extraction. In the area of defect prediction, researchers commonly rely on SZZ e.g., it was used to extract OpenStack and Qt dataset and the recent JIT work ~\cite{ISSTA2021}\cite{pornprasit2021jitline}\cite{cabral2019class}\cite{tabassum2020investigation} also used SZZ. We therefore think that it is out of the scope of our paper to design a new way of identifying buggy commits. Instead, we implore the software engineering community to come together and take up the challenge of improving the accuracy of SZZ algorithm or to propose new ways of identifying representative buggy commits. 


\subsection{Threats to Validity}
\paragraph{Internal validity}
We used an open-source implementation~\cite{szzunleashed} of the original SZZ algorithm~\cite{szz2005} along with the later enhancements proposed by Williams and Spacco~\cite{spacco2008szz}. They used the DiffJ tool to make SZZ aware of different types of format changes such as reordering and renaming of parameters. This implementation is used by many existing works~\cite{sohn2021leveraging}~\cite{vandehei2021leveraging}, but despite this we cannot rule out that the SZZ data we obtained contains errors and omissions. 

\paragraph{External validity}
The \textit{external validity} of our study is hampered due to the use of a limited number of projects for evaluation. However, to improve the external validity of our study, we made sure that the chosen projects have different languages, sizes and complexities. 



\paragraph{Conclusion validity}

To keep our comparison consistent with the baseline approach, we chose the same performance measures as JITLine. Furthermore, we optimized T$_{score}$ for AUC because it measures the discriminatory power of a model — ability to differentiate between clean and
buggy commits on varying thresholds. A threshold-independent measure is more stable in the presence of imbalance, and since JIT datasets are inherently imbalanced, many prior studies used AUC to measure model performance.



\section{Conclusion}
In this paper, we proposed IRJIT which is an explainable, and deterministic information retrieval approach for JIT defect prediction. IRJIT classifies a new commit-level change as buggy or clean, and shows past changes that can be blamed for the predicted bug. IRJIT does not require extensive retraining and can be updated in an online fashion making it suitable for use in real-world scenarios. We empirically evaluated our approach on OpenStack, Qt, and 6 other open-source projects using an evaluation setting that represents realism. Our comparison with a state-of-the-art ML approach, JITLine shows that IRJIT outperforms JITLine at line-level prediction. However, at commit level it only does better than JITLine for OpenStack and Qt project. Despite, IRJIT is more practical and can be appropriately integrated into existing development processes to facilitate JIT defect identification.


\vspace{12pt}
