

\section{Related Work}
\label{sec:related_work}

This work studies AIOps projects on GitHub and analyzes the quality of the projects using SonarQube. Thus, we discuss the related work on the following three aspects.


\noindent \textbf{AIOps solutions.}
In recent years and with the emergence of AIOps, more and more studies have been conducted in this field. Prior works have come up with different solutions for various problems. 
Anomaly detection \citep{he2018identifying, lim2014identifying}, failure prediction~\citep{zhao2021predicting, zhao2020real, lin2018predicting, li2020predicting}, task and job failures~\citep{gao2020task, islam2017predicting, el2017learning, rosa2015catching}, ticket management \citep{xue2016managing, xue2018spatial}, self-healing \citep{lou2017experience, lou2013software, ding2014mining}, and issue diagnosis \citep{luo2014correlating} are among the most studied topics in AIOps solutions. For example, Gao et al.~\citep{gao2020task} attempt to predict the task failures in cloud data centers. They use Google cluster trace \citep{reiss2012heterogeneity} and can achieve a 93\% accuracy in task failure prediction. Similarly, Islam et al.~\citep{islam2017predicting} characterize the failures of the Google cluster trace and then uses LSTM (Long Short-Term Memory) to predict the task failures. 
There also exists studies focusing on hardware failures~\citep{pinheiro2007failure, schroeder2007understanding, mahdisoltani2017proactive, xu2018improving, botezatu2016predicting}, 
where they perform large-scale studies on disk failures. 
Different from these studies, in this work, we perform an empirical study to understand the practices and characteristics of real-world AIOps projects on GitHub.









\noindent \textbf{Characterizing GitHub projects.}
As the biggest hosting service for open-source software, GitHub has been studied remarkably in the past years. Some studies focus on the best approaches for finding the most prominent repositories on GitHub~\citep{kalliamvakou2014promises, kalliamvakou2016depth, dabic2021sampling}. Many studies also have used GitHub as a source for mining software repositories~\citep{vadlamani2020studying, subramanian2020analyzing, horschig2018java, guzman2014sentiment, manes2021studying, lopes2017dejavu, kallis2021predicting, coelho2018identifying, businge2019studying, wessel2018power}. 
For example, Vadlamani et  al.~\citep{vadlamani2020studying}, along with  Subramanian et al.~\citep{subramanian2020analyzing} and Horschig et al.~\citep{horschig2018java} try to characterize the developers of open-source software. As another example, Kallis et al.~\citep{kallis2021predicting} try to predict the issue types of projects.
More similar to our study, Ghrairi et al.~\citep{ghrairi2018state} study the state of Virtual Reality (VR) projects extracted from GitHub. In another work, Coppola et al.~\citep{coppola2019characterizing} characterize the popularity of Kotlin in Android projects by analyzing 1,232 applications on GitHub. 
However, as far as we know, no studies have been conducted yet on the subject of AIOps. Our work is the first study that uses GitHub to characterize and analyze AIOps projects.




\noindent \textbf{Analyzing code quality using SonarQube.}
SonarQube is one of the most popular static code analyses used both in academia~\citep{lenarduzzi2017analyzing, lenarduzzi2018survey} and industry~\citep{vassallo2020developers}. Previous research has conducted many analyses on the code quality of open-source software projects using SonarQube. Businge et al.~\citep{businge2019studying} use SonarQube and analyses the source code of 119 applications to find their number of bugs. In another work, Tan et al.~\citep{tan2018towards} study 9 Apache software systems written in Python to investigate their technical debt. Lenarduzzi et al.~\citep{lenarduzzi2019diffuseness} also investigate the technical debt of 33 Apache systems written in Java. They find that the amount of code smell is much higher than bugs or vulnerabilities in their set of projects. They also report that the code smells with the severity level \textsc{major} take the longest time to get fixed. 
Compared to the mentioned studies, our paper analyzes the quality of 6,101 projects, which is much higher than in the previous studies, making our results more confident and robust.


\section{Conclusion and Future Work} 
\label{sec:conclusion}

This work studies the characteristics of AIOps projects on GitHub and compares them with two baselines (ML and General projects). We combine both quantitative and qualitative analyses to understand the current state of AIOps solutions. Specifically, we illustrate the state of AIOps projects on GitHub in RQ1, determine the most common input data, techniques, and goals of AIOps solutions in RQ2, and identify the most common issues in terms of code quality in RQ3.
We observe that AIOps projects are relatively new and growing fast in recent years. However, the quality of the AIOps projects is poorer compared to the baselines. 
Furthermore, we uncover the common patterns that exist in the input data, analysis techniques, and goals of the AIOps projects. Practitioners and researchers can learn from these patterns and adopt AIOps solutions that are optimal for their specific application scenarios. 
Our findings shed light on future efforts to improve AIOps practices, for example, to pay more attention to the weak aspects (e.g., self-healing), or reduce the \textit{ad-hoc}ness of coding to improve AIOps quality.



Our study could be extended in different ways. First, we study the AIOps projects publicly available on GitHub. However, there might be differences between open-source projects and closed-source projects which are implemented for private datasets in companies. Thus, one extension to our study could be to gather and analyze the industrial AIOps solutions and report any similarities and differences with AIOps projects on GitHub. Second, based on our results regarding the inputs, techniques, and goals of AIOps solutions, future work may develop a pipeline to automate or augment the development of AIOps solutions. Finally, we discussed the possibility of adopting available quality assurance techniques in machine learning systems into AIOps solutions. Hence, future work may actually implement these techniques for AIOps solutions and report their effects in terms of improving AIOps code quality. 

\section{Introduction}
\label{intro}
With the prevalence of
generated data in large-scale systems, monitoring this data and transforming it into practical insights is becoming a complex challenge. Artificial Intelligence for IT Operations (AIOps) has been introduced to cope with this challenge. It combines big data, machine learning (ML) approaches, and other advanced analysis techniques (e.g., statistical analysis)
to analyze
system performance patterns to be able to
improve service quality and reduce operational costs~\citep{dang2019aiops,gartnerwebsite}. By utilizing AIOps techniques, organizations are now able to collect and combine different sources of system data and use them to perform various tasks (e.g., anomaly detection or failure prediction) in their DevOps or operations environment~\citep{dang2019aiops,gartnerwebsite}.

A considerable amount of research has been performed on the topic of AIOps.
Prior work has proposed AIOps solutions for various maintenance and operations tasks, such as predicting node failures~\citep{lin2018predicting, li2020predicting}, predicting task or job failures~\citep{gao2020task, el2017learning, rosa2015catching}, anomaly detection~\citep{he2018identifying, lim2014identifying}, and self healing~\citep{ding2014mining, ding2012healing}.
However, no work has systematically studied AIOps practices in real-world projects (e.g., public GitHub projects).

Studying AIOps practices in real-world projects is important and has several benefits, including (1) helping researchers and practitioners understand the current status of AIOps solutions and the characteristics of AIOps projects; (2) providing guidance for researchers and practitioners to adopt best performing
AIOps solutions for their application scenarios; 
and (3) identifying problems in AIOps practices and shedding lights on future research opportunities.

\begin{comment}
\begin{itemize}
\item Help researchers and practitioners understand the current status of AIOps solutions.
\item Identify the existed gap and issues and recommend improvement opportunities.
\item Provide insights for future work.
\end{itemize}
\end{comment}

Therefore, this work identifies and studies a set of AIOps projects publicly available on GitHub. We analyze these projects from different aspects (e.g., repository characteristics, goals, and code quality) and perform both quantitative and qualitative analyses to comprehend the practices of AIOps in real-world projects. We also compare the selected AIOps projects with two baselines: traditional ML projects and General purpose projects.
We specifically aim to answer the following three research questions (RQs).







\begin{description}
\item[RQ1] \textbf{\textit{What are the characteristics of AIOps projects in terms of their GitHub metrics?}}
We analyze the GitHub metrics
of AIOps and baseline projects
to understand the current status of AIOps projects and also compare them with baselines in terms of their GitHub metrics such as the programming languages and the number of stars.
We observe that AIOps solutions are being developed with a faster growth rate compared to the baselines.
AIOps projects also have a higher distribution of popularity metrics (e.g., number of stars and forks), and also more pull requests and issues compared to baselines.

\item[RQ2] \textbf{\textit{What are the characteristics of AIOps projects in terms of their input data, analysis techniques, and goals?}}
In order to understand the characteristics of AIOps projects (i.e., their input data, analysis techniques, and goals), we manually investigate each of these projects.
We find that monitoring data (e.g., logs and performance metrics) is the most used input data, classical machine-learning techniques are the most adopted analysis techniques, and anomaly detection is the primary goal of many AIOps projects.

\item[RQ3] \textbf{\textit{What is the code quality of AIOps projects?}}
We further analyze the source code of AIOps and baseline projects to identify any interesting patterns related to their quality. We find that AIOps projects have higher issue 
rate, specifically in terms of bugs, code smells, and technical debt, than the baselines. 

\end{description}

We share our replication package on GitHub~\footnote{\url{https://github.com/AIOpsstudy/understanding\_aiops}} so that future work can reproduce or extend our study. Our work makes several important contributions:



\begin{enumerate}
    \item As the first study on AIOps practices in real-world projects, our work helps practitioners and researchers understand the status of AIOps from a practical point of view.
    \item Our qualitative analysis of the input data, analysis techniques, and goals of the AIOps projects can help practitioners and researchers consider and adopt AIOps solutions that fit into their specific application scenarios.
    \item Our work identifies problems in AIOps practices (e.g., code quality) and sheds light on future research opportunities in AIOps.
\end{enumerate}
 
The rest of the paper is organized as follows. Section~\ref{sec:experiment-setup} describes the experiment setup of our study, including the collection and preparation of the AIOps project data used for answering our research questions. Section~\ref{sec:rqs&results} presents our approach and results for answering the research questions. Section~\ref{sec:discussion} provides further discussions of our results. Section~\ref{sec:threats} discusses the threats to the validity of our findings. Section~\ref{sec:related_work} summarizes prior research related to our work, and finally, Section~\ref{sec:conclusion} concludes our paper.
\section{Experiment Setup} \label{sec:experiment-setup}
This section describes our approach for collecting the AIOps and baseline projects. We first present the overview of our study, then describe the steps for collecting and verifying AIOps and baseline projects, respectively.
\subsection{\textbf{Overview of our study}}
Figure~\ref{fig:overview} presents an overview of our approach for studying the characteristics of AIOps projects. We select our projects from GitHub which is the largest code hosting site with over 330 million repositories. 
We start by searching the projects with the keyword ``AIOps''.
Then, through manual verification (e.g., removing non-AIOps projects), keyword expansion (i.e., through pattern mining), second-round search and manual verification, and threshold-based filtering (e.g., by the number of stars), we collect a total of 97 AIOps projects that are used to answer our research questions. 
In order to better understand the characteristics of AIOps projects in a bigger context, we also compare our identified AIOps projects with two baselines:
\begin{enumerate}
\item randomly sampled machine learning (ML) projects;
\item randomly sampled general projects
\end{enumerate}
We choose the ML baseline because most of the AIOps projects leverage ML techniques. We choose the General baseline to compare our AIOps projects with general software applications on GitHub. Finally, we perform qualitative and quantitative analyses on the collected data to answer our research questions. 
Below, we describe the details of our data collection. 
The detailed approaches for answering our research questions are presented in Section~\ref{sec:rqs&results}.

\begin{figure*}
  \centering
  \includegraphics[width=1\textwidth]{overview_approach_v3.pdf}
\caption{An overview of our study.}
\label{fig:overview}      
\end{figure*}


\subsection{\textbf{Collecting AIOps Projects}}

Through two rounds of searching AIOps related keywords on GitHub, we collect a total of 888 candidate projects. Through filtering and manual verification, we end up with 97 of them as our final set of AIOps projects. 
The methodology we use to select these projects follows the systematic approach recommended by~\citep{basili1986experimentation} and is described in the following sections. 

\subsubsection{Search AIOps projects (first round)} \label{sec:first-round-search}
In the first step of finding a good set of projects, we search for the repositories that specifically indicate themselves as an AIOps project. For this purpose, we use the GitHub interface and look for projects that match the keyword ``AIOps''. Our search looks at four sections of a repository and checks if the searched phrase exists in these sections. These four sections are the repository name, the ``about'' section, the ``topics'' section, and the ``readme'' file. After searching for the keyword ``AIOps'', we find 505 repositories.

\subsubsection{Manual verification (first round)} \label{sec:manual-first-round}
Based on existing definitions of AIOps~\citep{gartnerwebsite,dang2019aiops}, we consider AIOps projects as: any project that uses IT Operations-related data, utilizes advanced analysis technologies such as machine-learning or statistical analysis, to reach valuable insights or enhance the system’s quality by actions such as monitoring and automation.

Therefore, not all the 505 discovered repositories are good AIOps candidates and suitable for our study. We hence select our subject projects based on three criteria: 
\begin{enumerate}
\item The project should be about AIOps, 
not similar topics such as Machine Learning Operations (MLOps)\footnote{A set of practices to maintain and deploy machine learning models.} or Development and Operations (DevOps)\footnote{A set of practices that aim to shorten the system development life cycle while preserving high quality.}. Therefore, we delete the projects that are mainly about other topics. 
\item The projects should contain sufficient code and be mature. Therefore, we delete the repositories that do not have code
and are a collection of papers, slides, or other repositories. However, we accept the projects that have created a dataset so that other developers and researchers can use it in their work.  
\item The project should not be a toy project: we delete the projects that are homework assignments or university projects.
\end{enumerate}

To select the desirable projects based on the explained criteria, the first two authors of the paper (i.e., coders) jointly perform a coding process, adding a YES (AIOps projects) or NO (non-AIOps projects) tag to each project. We perform the coding process as below.

\textbf{Step 1: Coding.} Each coder analyzes all the 505 repositories and solely decides if each project should be added to the final list of projects.

\textbf{Step 2: Discussion.} The coders share their responses and discuss their approaches for selecting a project. The discussion session's goal is to reach the same understanding of the inclusion criteria among the coders.

\textbf{Step 3: Revision.} Based on the discussion, each coder revises their responses from step 1. 

\textbf{Step 4: Resolving disagreements.} In the last step, the coders discuss any conflicts that may remain and try to resolve them. If an agreement can not be reached, the third author would analyze the project and a vote is performed.

After performing the manual verification process, we obtain a total of 84 candidate AIOps projects from the 505 projects derived from the search results; which corresponds to a selection rate of 17\%. 

\subsubsection{Keyword extraction (pattern mining)}

To have a comprehensive view of real-world AIOps projects, limiting our search to one keyword (i.e., ``AIOps'') is not thorough.
This is mainly because the term “AIOps” was introduced in 2018 \citep{gartnerwebsite}, and some projects might exist before introducing this terminology that are implementing AIOps solutions but are not using the exact term.
Hence, we expand our keywords to find more projects that do not explicitly indicate themselves as AIOps repositories but actually are. For doing so, we first extract all the topics
of each of the 84 AIOps projects resulted from the previous step. The topics of each project are available in the ``topic'' section of each GitHub repository.

Then we use frequent pattern mining~\citep{han2007frequent}
which is an analytical process to find the associations and patterns in a data set. Specifically, we utilize the frequent pattern growth method~\citep{han2004mining, han2000mining} to find the most common topics of GitHub repositories.
We set the support of the method to 2, meaning that the pattern should appear in at least two projects. In total, we find 194 patterns between the topics. Next, we perform a discussion session between all three authors to decide which patterns might be good candidates for finding more AIOps projects. We select four pairs of two-item patterns: ``anomaly detection'' and ``log analysis'', ``log analysis'' and ``machine learning'', ``anomaly detection'' and ``machine learning'', as well as ``machine learning'' and ``metrics''. All the selected keyword pairs
are among the most frequently used topics in the projects. We use these four sets of keywords to find more AIOps repositories. 

\subsubsection{Search AIOps projects (second round)} 
With the extracted four pairs of keywords from the previous step, we use the GitHub interface to search for additional projects related to AIOps.
We follow the same process as described in Section~\ref{sec:first-round-search}. 
Our search results identifies a total of 383 projects after removing duplicated projects contained in the first-round search results.

\subsubsection{Filtering}
Based on the knowledge that we have gained from our first-round manual verification, we apply a filtering phase to remove the toy projects. 
We use two criteria to filter the projects: the number of stars and forks. We only consider the projects with stars and forks greater than or equal to 1 (stars: >=1 \& forks: >=1). The purpose of removing the toy projects is to have relatively mature projects and not soil our results with small repositories~\citep{munaiah2017curating}. 

We apply the filtering to both the projects from the first round and the second round of searches. 
For the projects from the first round, we apply the filtering to the projects after the manual verification step. We intentionally manually analyze all the projects from the first-round search before the filtering to gain a more comprehensive understanding of the status of the AIOps projects. After the filtering process, the 84 manually verified projects from the first round are reduced to 40 projects.
For the projects from the second round search, we apply the filtering before the manual verification process, in order to save some unnecessary manual efforts in verifying the projects that are eventually excluded by the filtering criteria. 

\subsubsection{Manual verification (second round)}
As explained in Section~\ref{sec:manual-first-round}, not all the searched repositories are suitable for our study. To select the projects that fit the study, we repeat the process described in Section~\ref{sec:manual-first-round} to perform the manual verification. The coders follow the same procedure to decide whether each of these 383 projects is a real AIOps project or not. After separate coding, discussion session, revision, and resolving disagreements, we find 57 projects from the expanded keywords; which corresponds to a selection rate of 15\%.

Finally, we combine the 40 projects identified using the ``AIOps'' keyword with the 57 projects identified using the expanded keywords.
The final set of 97 AIOps projects is used to answer our research questions. 
The approaches for answering our research questions are detailed in Section~\ref{sec:rqs&results}

\subsubsection{Measuring the reliability of our manual verification
}
Reliability is vital to ensure the validity of the coding results \citep{artstein2008inter}. The coding results are reliable if there exists a certain level of agreement between the coders, known as inter-coder agreement. In this study, we use Cohen’s kappa \citep{cohen1960coefficient} to measure the reliability of the agreements between two coders. Cohen’s kappa is one of the most common approaches to measure the inter-coder agreement \citep{artstein2008inter}. Table~\ref{tab:cohen} indicates the relation between the value of Cohen’s kappa and the level of agreement \citep{mchugh2012interrater}.

\begin{table}[!ht]
    \centering
    \caption{Interpretation of Cohen’s kappa.}
    \resizebox{0.6\textwidth}{!}
    {
    \small
    \begin{tabular}{cc}
        \toprule
        Value of Cohen’s k &  Level of
        Agreement  \\

        \midrule
        0-.20 & None\\
        .21-.39 & Minimal \\
        .40-.59 & Weak \\
        .60-.79 & Moderate \\
        .80-.90 & Strong \\
        .90-1 & Almost Perfect 
    \end{tabular}
    }
    \label{tab:cohen}
\end{table}

Our manual verification for choosing the proper AIOps projects achieves a Cohen’s kappa of 0.68 before the discussion session. After the discussion session between the coders, the kappa score increased to 0.81. As shown in Table~\ref{tab:cohen}, kappa $\geq$ 0.80 indicates a strong agreement.

\subsection{\textbf{Collecting Baseline Projects}}
\label{sec:Baselines}



To understand how AIOps projects differ from traditional software projects,
we create two baselines and compare the AIOps projects with them. We select machine learning (ML) projects as our first baseline, and General
projects as our second baseline.

\subsubsection{Machine Learning projects}
As our first baseline, we choose ML projects. We select ML for the first baseline because AIOps can be considered as an application domain of machine learning.
Similar work typically searches for specific frameworks to gather ML projects on GitHub. For example, \citep{zhang2018empirical} select their set of projects based on searching the keyword ``TensorFlow''. Also, \citep{islam2019comprehensive} gather their projects based on searching for different keywords such as ``TensorFlow'' and ``Keras''. Since we are not focusing on any specific framework, we use more general keywords. Hence, to gather the set of ML baseline, we use two keywords:
``machine learning'' and ``deep learning''.

To compare the AIOps projects with the baselines in a similar context, we filter the baseline projects in the same way as filtering the AIOps projects.
Therefore, we only extract the projects with at least one star and one fork. Furthermore, because the ML projects have a longer history than AIOps projects, we apply another criteria: date of creation. In our set of AIOps projects, the first and last creation dates are 2012/12/25 and 2021/08/19, respectively. Thus, we apply the same dates to extract the ML baseline.
After performing the search and deletion of duplicate projects in two queries, we obtain 75,198 unique repositories.
Due to time and computational resource limitations, extracting and analyzing all these projects is not feasible. Hence, following~\citep{chen2020comprehensive} and~\citep{zhang2019empirical}, we randomly sample 383 projects from the pool of 75,198 ML repositories using a confidence level of 95\% and a confidence interval of 5\%. To ensure the robustness of our results, we repeat this process 10 times. Hence, for our ML baseline, we have 3,830 (10*383) projects.





\subsubsection{General projects}
As our second baseline, we choose general projects from GitHub, meaning that we do not specify any particular topic in our search. We also do not limit the General baseline to a specific language (e.g., Python) because AIOps projects are not limited to only one language as well.
In this way, the General baseline captures the general characteristics of all GitHub projects. We perform the same filtering phase as for the ML baseline; the difference is that we do not indicate any specific topic.
After applying the filtering, we obtain 3,796,841 public and available repositories. Following the same approach for creating the ML baseline (confidence level of 95\% and confidence interval of 5\%), we sample 385 projects. Then, for consistency and robustness, we repeat the sampling process 10 times. In total, we obtain 3,850 (10*385) projects for the General baseline. 

As shown in Figure~\ref{fig:overview}, we compare our AIOps projects with the baseline projects in RQ1 where we study the repositories statistics of the projects, as well as in RQ3 where we study the code quality of the projects. In RQ2, we perform a qualitative study for the AIOps projects only, since the research question is specifically about AIOps projects (i.e., the input data, analysis techniques, and goals of AIOps solutions).

\section{Research Questions and Results} \label{sec:rqs&results}
This section presents the details of our research questions (RQs) and the results. We organize each RQ by its motivation, approach, and results.

\input{5-rq1}
\input{6-rq2}
\input{7-rq3}


\subsection{\textbf{What are the characteristics of AIOps projects?}}


\subsection{\textbf{RQ1. What are the characteristics of AIOps projects in terms of their GitHub metrics?}}

\subsubsection{\textbf{Motivation}}
Prior studies proposed AIOps solutions that leverage AI technologies to support various software operation efforts~\citep{li2022cdx, li2020predicting, di2021prometheus}. 
However, no work has investigated real-world AIOps projects and their characteristics. 
Thus, this RQ bridges the gap to study the characteristics of AIOps projects and compare them with the baseline projects in terms of their GitHub metrics. With this comparison, we can understand the similar and different patterns between the characteristics of AIOps projects and the baselines.
Our findings can help AIOps researchers and practitioners understand the state of AIOps in practice.



\subsubsection{\textbf{Approach}}



In this RQ, we analyze the repository characteristics of AIOps projects and compare them with the baseline projects. 
We use 
GitHub REST API~\citep{GitHubSearchAPI:2022} to retrieve the repository characteristics of these projects.
In particular, we analyze the characteristics of AIOps projects and the baseline projects from three perspectives: growth of repositories, programming languages, and repository metrics.



\noindent\textbf{Growth of repositories.}
To understand the evolution of the population of AIOps projects, we analyze the distribution of the AIOps projects based on their creation time. We also compare the creation time distribution with that of the baseline projects. 

\begin{figure*}
\centering
\includegraphics[width=0.75\textwidth]{date.pdf}
\caption{The cumulative distribution of the creation time of AIOps and baseline projects.}
\label{fig:creation-date}    
\end{figure*}

\begin{table}[!h]
    \centering
    \caption{The top-5 languages of AIOps and baseline projects.}
    \resizebox{1\textwidth}{!}
    {
    \small
    \begin{tabular}{ll|ll|ll}
        \toprule
        \multicolumn{2}{c|}{\textbf{AIOps}} &  \multicolumn{2}{c|}{\textbf{ML}} &
        \multicolumn{2}{c}{\textbf{General}} \\
        Language & Usage $(\%)$ & 
        Language & Usage $(\%)$ & 
        Language & Usage $(\%)$ \\

        \midrule
        Python & 73.6 & Python & 77.4 & Python & 19.9\\
        Java & 12.1 & HTML & 3.3 & JavaScript & 16.6 \\
        HTML & 4.4 & C++ & 2.7 & Java & 9.2 \\
        Go & 3.3 & JavaScript & 2.3 & C++ & 5.7 \\
        JavaScript & 2.2 & R & 2.0 & PHP & 5.3 
    \end{tabular}
    }
    \label{tab:languages}
\end{table}

\noindent\textbf{Programming languages.}
Developers may use different programming languages for AIOps projects. Understanding the distribution of the programming languages can provide insights for future work to support AIOps project development. Each project may use multiple programming languages. In this work, we extract and present the primary language of each project.

\noindent\textbf{Repository metrics.}
We study the repository metrics of the AIOps projects, including the number of stars, forks, commits, contributors, releases, pull requests, issues, size, 
and the status of being archived.
For the pull requests, we sum the open and closed pull requests.
Similarly, for the issues, we sum the open and closed issues.


\begin{table*}[h!]
    \caption{The mean and median of GitHub metrics of AIOps projects and two baselines. The mean and median are represented by $\mu$ and $\widetilde{\chi}$, respectively. 
    }
    \resizebox{1\textwidth}{!}
    {
    \begin{threeparttable}
    \centering
    \small
    \begin{tabular}{l|cc|cc|cc|cc|c}
        \toprule
         & \multicolumn{2}{c|}{\textbf{Stars}} 
         & \multicolumn{2}{c|}{\textbf{Forks}} 
         & \multicolumn{2}{c|}{\textbf{Commits}} 
         & \multicolumn{2}{c|}{\textbf{Contributors}} 
         & \textbf{Archived} \\
        & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ \\
        \midrule
        \textbf{AIOps} & 122.7 & 19.0 & 32.1 & 7.0 & 492.7 & 25.0 & 4.7 & 2.0 & 1.0\% \\
        \textbf{ML} & 121.8 & 6.0 & 30.3 & 3.0 & 133.4 & 23.0 & 3.8 & 1.0 & 1.7\% \\
        \textbf{General} & 34.3 & 3.0 & 10.6 & 2.0 & 885.6 & 23.0 & 7.8 & 1.0 & 3.7\% \\
        
        \midrule
        \rule{0pt}{2ex}
        & \multicolumn{2}{c|}{\textbf{Releases\tnote{*}}} 
        & \multicolumn{2}{c|}{\textbf{Pull requests\tnote{*}}} 
        & \multicolumn{2}{c|}{\textbf{Issues\tnote{*}}} 
        & \multicolumn{2}{c|}{\textbf{Size (mb)}} \\
        & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ \\
        \midrule
        \textbf{AIOps} & 2.6 & 0.0 & 79.4 & 0.0 & 89.9 & 1.0 & 77.8 & 7.8 \\
        \textbf{ML} & 0.8 & 0.0 & 20.8 & 0.0 & 15.8 & 0.0 & 59.4 & 4.4 \\
        \textbf{General} & 1.7 & 0.0 & 14.8 & 0.0 & 9.6 & 0.0 & 29.9 & 0.5 \\
        \midrule
        
    \end{tabular}
    \begin{tablenotes}
        \small
        \item[*] The median number of 0 for the releases, pull requests, and issues indicate that more than half of the projects do not have any release, pull request, or issue.
    \end{tablenotes}
    \end{threeparttable}
    }

    \label{tab:github_metrics}
    \vspace{-1mm}
\end{table*}


\subsubsection{\textbf{Results}}
\label{rq1_results}



\noindent\textbf{Compared to the ML and General baselines, AIOps projects are relatively new and exhibit rapid growth in recent years.}
Figure~\ref{fig:creation-date} represents the percentage of projects created in and before each year for the AIOps and baseline projects.
As mentioned in Section \ref{sec:Baselines}, the creation date of all the projects is between 2012/12/25 and 2021/08/19. 
As shown in Figure~\ref{fig:creation-date}, in the first few years (from 2012 to 2017), the number of AIOps projects is very small. In recent years (from 2017 to 2021), the AIOps projects experience a faster growth compared to the ML and General baselines. As can be seen, the ML projects also exhibit a faster growth than the General baseline projects.



\noindent\textbf{Like in ML projects, Python is the dominant programming language in the AIOps projects; however, unlike in ML projects, Java is also a major programming language used in AIOps projects.}
As shown in Table \ref{tab:languages}, Python is the most used language among all the groups; however, the usage of Python is much higher in AIOps and ML repositories than in the General baseline (73.6\% and 77.4\% in contrast to 19.9\%). Another interesting finding is the relatively high usage of Java in AIOps projects (12.1\%), while Java is not among the top-5 popular languages in the ML baseline.



\noindent\textbf{On average, AIOps projects are more popular and active than the baselines.}
Table~\ref{tab:github_metrics} represents the mean and median of our selected repository metrics for the AIOps projects and the baselines. Since the data does not follow a normal distribution, we mainly compare the metrics using their median values. However, we consider the mean where the median is zero (e.g., pull requests). 

Considering the median number of stars and forks, AIOps projects are more popular than the baselines, having 6 times more stars and 3 times more forks than the General baseline. Regarding the commits, contributors, and the number of releases, AIOps projects are more collaborative (median of 2 contributors for AIOps projects, while 1 in ML and General baselines), resulting in more frequent releases that show the activeness of these projects. 

Comparing the number of pull requests and issues between AIOps projects and the baselines, we notice that AIOps projects experience more pull requests and issues. This may be explained in three ways. First, the AIOps projects are more active and popular (i.e., more developers proactively use them), leading to a more significant number of pull requests and issues. The second interpretation might be that AIOps projects are in the first stages of formation and not mature enough, having more defects and flaws and more developers trying to fix these problems. The third explaination is that AIOps projects are on average larger than the two baseline projects (as shown in Table~\ref{tab:github_metrics}), which may lead to more pull requests and issues. 
In addition, the state of the projects indicates that only 1\% of AIOps projects have been archived, while this amount for ML and General baseline is 1.7\% and 3.7\%, respectively.
Overall, taking into consideration all the mentioned metrics, AIOps projects seem to be more active and popular than both baselines.











\newenvironment{mybox}[1]{%
    \begin{tcolorbox}[title={Summary of RQ1}]%
    }{ 
    \end{tcolorbox}
}
\begin{mybox}{}
On average, AIOps projects are receiving more attention than the ML and General baselines. The primary language used in them is Python, followed by Java. They are growing faster than the baselines in recent years, demonstrating the growing needs and active practices in this area.
The size of AIOps projects is larger than the baselines, and focusing on other GitHub metrics such as number of stars, forks, and releases, AIOps projects seem to be more popular and active than both baselines. 
\end{mybox}
\subsection{\textbf{RQ2. What are the input data characteristics, analysis techniques, and analysis goals in AIOps projects?}}
\subsection{\textbf{RQ2. What are the characteristics of AIOps projects in terms of their input data, analysis techniques, and goals?}}
\label{sec:rq2}

\subsubsection{\textbf{Motivation}}
AIOps researchers and practitioners leverage different techniques to analyze different types of operational data and achieve different goals. However, it is unclear how real AIOps projects leverage data and technologies to achieve the goals. 
In this RQ, we qualitatively analyze our set of AIOps projects to understand the characteristics of these projects' input data, analysis techniques, and goals.
Our results can help researchers and practitioners further understand the status of AIOps practices and the characteristics of AIOps projects.
Our results can also provide insights for future work to provide support for different AIOps application scenarios. 

\subsubsection{\textbf{Approach}}
We manually examine each AIOps project to understand its input data, analysis techniques, and goals.
For each project, we manually investigate four sources of information; the ``about'' section, the ``readme'' file,
the source code, and the additional documentations if available.
Figure~\ref{fig:flow} illustrates the three key concepts of our manual analysis (input data, analysis techniques, and goals) and their relationship.


\begin{itemize}
    \item \textbf{Input data:}
    The types of data (e.g., log data) that an AIOps project takes as inputs to achieve its objectives.
    
    \item \textbf{Analysis techniques:}
    The main analysis techniques (e.g., machine learning techniques) that an AIOps project adopts to analyze the input data and achieve its objectives.
    
    \item \textbf{Goals:}
    The objectives (e.g., anomaly detection) that an AIOps project aims to achieve through its input data and analysis techniques.
\end{itemize}

\begin{figure*}
\centering
\includegraphics[width=0.7\textwidth]{schema.pdf}
\caption{The three key concepts of our manual analysis.}
\label{fig:flow}    
\end{figure*}

\noindent\textbf{Manual coding process.}
We use open coding approach~\citep{khandkar2009open} 
to extract the information related to the three key concepts shown in Figure~\ref{fig:flow}.
Open coding is widely used among software engineering researchers to conclude a high-level abstraction from lower-level data~\citep{wohlin2015towards, stol2016grounded}. 
To label the projects, the first two authors of the paper (i.e., coders) jointly perform a coding process, determining each project's input data, analysis techniques, and goals. We perform a five-step coding process as follows.

\begin{figure*}
\centering
\includegraphics[width=0.75\textwidth]{inputs.pdf}
\caption{The categorization of the input data used in AIOps projects. The high level categories are highlighted in dark.}
\label{fig:inputs}    
\end{figure*}

\begin{figure*}
\centering
\includegraphics[width=1\textwidth]{techniques.pdf}
\caption[caption]{The categorization of the analysis techniques used in AIOps projects. The high level categories are highlighted in dark.\footnotemark}
\label{fig:techniques}    
\end{figure*}

\begin{figure*}
\centering
\includegraphics[width=0.8\textwidth]{goals.pdf}
\caption[Caption]{The categorization of the goals of the AIOps projects. The high level categories are highlighted in dark.}
\label{fig:goals}    
\end{figure*}



\textbf{Step 1: Coding.} Each coder analyzes the 97 AIOps projects and assign labels for each concept (input data, techniques, and goals) of each project.
Multiple labels can be assigned to a concept of a project.
This step takes a few days for each coder to complete.

\textbf{Step 2: Discussion.} The coders share their responses and discuss the created labels. The main goal of the discussion session is to obtain a common understanding of the labels for the input data, techniques, and goals. Based on the separate labels of the coders, we join related labels together and take apart some high-level labels into smaller ones. After this session, we finalize the labels for each concept (input data, techniques, and goals).

\footnotetext{LSTM approaches can be used in a supervised or unsupervised manner~\citep{chen2021experience}. In our set of projects and primarily for anomaly detection, projects use LSTM in the unsupervised form.}

\textbf{Step 3: Revision.} Based on the results of the discussion session and the agreed-upon labels, each coder revises his responses from Step 1. 

\textbf{Step 4: Resolving disagreements.} The coders compare their final results from step 3 and discuss any conflict that may remain. The coders try to resolve the conflicts, but if an agreement can not be reached, the third author analyzes the project, and the final decision is made.

\textbf{Step 5: Final revision.} In the final stage, we create a mind map from all the produced
labels. We then discuss the labels and form an hierarchy, change some labels’ names for clarity, and merge some small categories to be cohesive.


\subsubsection{\textbf{Results}}


Figures~\ref{fig:inputs},~\ref{fig:techniques}, and~\ref{fig:goals} present our categorization of the input data, analysis techniques, and goals of the AIOps projects,
respectively. We further define our coding labels in Tables~\ref{tab:input_definitions},~\ref{tab:techniques_definitions}, and~\ref{tab:goals_definitions}.
It is important to note that each project may have multiple input data, analysis techniques, or goals. \\

\noindent\textbf{Input data: Monitoring data (e.g., logs, network-traffic data, and performance metrics) is the dominant type of input data of the AIOps projects, with logs being the most commonly used input data type.}
The categorization of the input data is illustrated in Figure~\ref{fig:inputs}.
We divide the input data of AIOps projects into five main categories: monitoring data, multi-media, source code, technical question-answer (QA) data, and alarm. We define these categories and provide an example
for each in Table~\ref{tab:input_definitions}. Among them, monitoring data is the most popular, used in 86\% of the projects. The monitoring data is divided into 6 sub-categories: log, network-traffic data, performance metric, Key Performance Indicator (KPI), Internet of Things (IoT), sensor data, and trace. Among them, log data is the most commonly used.
We find two interesting input data types for the AIOps projects: network traffic and IoT sensor data. A considerable proportion of the projects (18\%) use network traffic data as their input. Also, 8\% of projects use IoT sensor data. 

\begin{table*}[h!]
\caption{Different types of input data, their definitions, and examples.}

    \resizebox{1\textwidth}{!}
    {
    \centering
    \small
    \begin{tabular}{lll}
        \midrule
        Input data & Definition & Repository example \\
        \midrule
        \textbf{Monitoring Data} & \textbf{Different types of data that} \\
        & \textbf{record the runtime status of a system.} \\
        
        \rule{0pt}{4ex} 
        \quad Log & \quad System-generated data that records & Repository~\href{https://github.com/logpai/loglizer}{(58811148)} uses log data as  \\
        & \quad runtime events that have happened. & input to perform anomaly detection. \\
        
        \rule{0pt}{4ex} 
        \quad Network-traffic data & \quad Monitoring data that records  & Repository~\href{https://github.com/alexamanpreet/Network-Log-and-Traffic-Analysis}{(79239275)} uses network- \\
        & \quad network activities. &  traffic data to identify malicious \\
        & & behaviors and attacks. \\

        \rule{0pt}{4ex} 
        \quad Performance metric & \quad Quantitative measurements used to & Repository~\href{https://github.com/jixinpu/aiopstools}{(160285839)} uses \\
        & \quad track the performance of a system. & different performance metrics such as \\
        & \quad These metrics are often operational, & CPU and memory usage to perform \\
        & \quad such as CPU usage. & various tasks such as anomaly \\
        & & detection and time series forecasting. \\
        
        \rule{0pt}{4ex} 
        \quad KPI & \quad Measurements related to key business & Repository~\href{https://github.com/logpai/Log3C}{(142442484)} uses different \\
        & \quad goals of a system. These metrics are & KPIs in time intervals \\ 
        & \quad often strategic.  & to identify impactful system problems. \\

        \rule{0pt}{4ex} 
        \quad IoT sensor data & \quad Data collected by devices connected & Repository~\href{https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot}{(142325304)} uses real-time \\
        & \quad to an IoT network. & IoT sensor data to detect anomalies.\\

        \rule{0pt}{4ex} 
        \quad Trace & \quad A specialized use of logging to record & Repository~\href{https://github.com/CloudWise-OpenSource/GAIA-DataSet}{(397983735)} gathers a \\
        & \quad information about a system's & dataset of traces that can be used \\
        & \quad execution with comprehensive details. & to analyze operations problems.\\
        \midrule
        
        \textbf{Multi-media} & \textbf{Different types of multi-media data, } & Repository~\href{https://github.com/whylabs/whylogs}{(287642401)} generates \\
        & \textbf{including images, videos, and} & summaries of data types including \\
        & \textbf{audios.} & image data for monitoring purposes \\
        \midrule
        
        \textbf{Source code} & \textbf{Source code of software systems.} & Repository~\href{https://github.com/keikoproj/active-monitor}{(201529303)} can be \\
        & \textbf{} & installed on Kubernetes source code \\
        & \textbf{} & and provide self-monitoring and \\
        & & self-healing. \\
        \midrule
        
        \textbf{Technical Q/A data} & \textbf{Data collected from} & Repository~\href{https://github.com/OS-ABC/AIOps-Event-Graph-WebData}{(345320486)} extracts \\
        & \textbf{technical Q/A websites such as } & information from Stack Overflow to \\
        & \textbf{Stack Overflow.} & find fast solutions for faults in their \\
        & \textbf{} & platform. \\
        \midrule
        
        \textbf{Alarm} & \textbf{Alarms generated} & Repository~\href{https://github.com/jixinpu/aiopstools}{(160285839)} uses alarms \\
        & \textbf{during system run time.} & to find the association rules between \\
        & & them. \\
        \midrule
 

 \end{tabular}
    }
    \begin{tablenotes}
    \small
      \item To find a GitHub repository with its ID, one can either click the hyperlink or use the link \textit{https://api.github.com/repositories/\{ID\}} where \textit{\{ID\}} is replaced by a specific repository ID.
    \end{tablenotes}
    \label{tab:input_definitions}
\end{table*}

\begin{table*}[h!]
\caption{Different types of analysis techniques, their definitions, and examples.}
    \resizebox{1\textwidth}{!}
    {
    \centering
    \small
    \begin{tabular}{lll}
        \midrule
        Analysis techniques & Definition & Repository example \\
        \midrule
        \textbf{Classic machine} & \textbf{Classic machine learning techniques such as}  \\
        \textbf{learning} & \textbf{Logistic Regression and Decision Tree.} & \\
        
        \rule{0pt}{4ex} 
        \quad Supervised-learning & \quad A leaning technique that uses & Repository~\href{https://github.com/shimo85/2019AIOps_ai}{(165321356)} uses various \\
        & \quad labeled datasets. & classic supervised approaches such as \\
        & & Random Forest and Decision Tree to \\
        & & detect anomalies in KPI data. \\
        
        \rule{0pt}{4ex} 
        \quad Unsupervised-learning & \quad An approach that system learns & Repository~\href{https://github.com/logpai/loglizer}{(58811148)} implements \\
        & \quad without using labeled datasets. &  multiple unsupervised approaches \\
        & & such as Isolation Forest and Invariant \\
        & & Miner to detect anomalies. \\
        
        \midrule
        \textbf{Deep learning} & \textbf{A subfield of machine learning that} & \\
        & \textbf{uses artificial neural networks with multiple} & \\
        & \textbf{layers (i.e., deep neural networks).} & \\
        
        \rule{0pt}{4ex}
        \quad Supervised-learning & \quad A leaning technique that uses & Repository~\href{https://github.com/Shauqi/Attack-and-Anomaly-Detection-in-IoT-Sensors-in-IoT-Sites-Using-Machine-Learning-Approaches}{(187774599)} uses Artificial \\
        & \quad labeled datasets. & Neural Network to find anomalous \\
        & & behavior on IoT sensor data. \\
        
        \rule{0pt}{4ex}
        \quad Unsupervised-learning & \quad A technique that learns & Repository~\href{https://github.com/donglee-afar/logdeep}{(246569386)} uses Long \\
        & \quad patterns from unlabelled data. & Short-Term Memory to detect \\
        & & anomalies from log data. \\
        
        \midrule
        \textbf{Time series model} & \textbf{Techniques that aim to model time} & Repository~\href{https://github.com/dreamhomes/TroubleShooter}{(302842095)} applies different \\
        & \textbf{series data, mainly used for finding} & time series models such as Auto \\
        & \textbf{trends and forecasting.} & Regression and Holt-winters to detect \\ 
        & & anomalies and analyze root causes. \\
        
        \midrule
        \textbf{Statistical analysis} & \textbf{Analytical techniques to} & \\
        & \textbf{understand, analyze and interpret } & \\
        & \textbf{the input data.} & \\
        
        \rule{0pt}{4ex}
        \quad Descriptive analysis & \quad Describing the features of data and & Repository~\href{https://github.com/IBM-Cloud/iot-device-phone-simulator}{(114942949)} uses descriptive \\
        & \quad summarize data in a quantitatively & analysis such as measuring minimum and \\
        & \quad manner. & maximum values and plotting scatter \\
        & & figures to interpret their IoT sensor data. \\
        
        \rule{0pt}{4ex}
        \quad Statistical outlier & \quad Applying statistical tests such as & Repository~\href{https://github.com/ThirdEyeData/Anomaly-Detections-Apache-Spark}{(156308650)} uses z-score \\
        \quad detection & \quad z-score to identify outlier values. & test to detect anomalies on CPU usage \\
        & & data. \\
        
        \rule{0pt}{4ex}
        \quad Exploratory analysis & \quad Exploring data to identify new & Repository~\href{https://github.com/BBVA/Tarkin}{(123162193)} uses exploratory \\
        & \quad connections, inspect missing data, & analysis such as plotting and describing \\
        & \quad or check hypotheses. & their log data to inspect and understand \\
        & & their input data. \\
        
        \midrule
        \textbf{Associate rule mining} & \textbf{Finding frequently occurring} & Repository~\href{https://github.com/jixinpu/aiopstools}{(160285839)} uses Apriori \\
        & \textbf{patterns and associations between} & algorithm to perform associate mining \\
        & \textbf{variables.} & in their input alarms. \\
        
        \midrule
        \textbf{Natural language} & \textbf{Techniques that aim to analyze} & Repository~\href{https://github.com/Ohou-csu/AIOps-Learning-and-Exploration}{(316407231)} leverages \\
        \textbf{processing} & \textbf{and model text data.} & Bidirectional Encoder Representations \\
        & & from Transformers (BERT), a pre-trained \\
        & & language model as one of their techniques \\
        & & to perform anomaly prediction. \\
        
        \midrule
        \textbf{Others} & \textbf{Other techniques that could not be} & Repository~\href{https://github.com/NCBI-Hackathons/Semantic-search-log-analysis-pipeline}{(146802240)} uses fuzzy \\
        & \textbf{categorized in previous categories.} & matching to classify web queries. \\
        
        \midrule
        \textbf{Unknown} & \textbf{Projects for which we could not find} &  \\
        & \textbf{any specific techniques.} & \\
        
        \midrule
        \textbf{N/A} & \textbf{Projects that only provide} & Repository~\href{https://github.com/NetManAIOps/KPI-Anomaly-Detection}{(238914477)} is a dataset of \\
        & \textbf{datasets or do not use any} & KPI data and does not use any \\
        & \textbf{analysis techniques.} & analysis techniques. \\
        \midrule

 \end{tabular}
    }
    \begin{tablenotes}
    \small
      \item To find a GitHub repository with its ID, one can either click the hyperlink or use the link \textit{https://api.github.com/repositories/\{ID\}} where \textit{\{ID\}} is replaced by a specific repository ID.
    \end{tablenotes}
    \label{tab:techniques_definitions}
\end{table*}

\begin{table*}[h!]
\caption{Different types of goals, their definitions, and examples.}
    \resizebox{1\textwidth}{!}
    {
    \centering
    \small
    \begin{tabular}{lll}
        \midrule
        Goal & Definition & Repository example \\
        \midrule
        
        \textbf{Anomaly detection} & \textbf{Identifying anomalies that deviate} & Repository~\href{https://github.com/alexfrancow/A-Detector}{(134266587)} uses log data \\
        & \textbf{from the normal behavior.} & and analyze them to find anomalous \\
        & & behavior. It then display anomalies \\
        & & using dynamic graphics. \\
        \midrule
        
        \textbf{Monitoring} & \textbf{Collecting and observing the} & Repository~\href{https://github.com/opendistro-for-elasticsearch/anomaly-detection}{(221989665)} provides data \\
        & \textbf{real-time stream of data to} & monitoring and alerting.\\
        & \textbf{understand system runtime status.} & \\
        \midrule
        
        \textbf{Anomaly prediction} & \textbf{Analyzing historical data to } & Repository~\href{https://github.com/geekidharsh/predicting-harddrive-failures-using-ml}{(169132015)} uses metric \\
        & \textbf{forecast future anomalies.} & data of hard drives to predict the \\
        & & failures in the near future. \\
        \midrule
        
        \textbf{Root cause analysis} & \textbf{Identifying the root causes of} & Repository~\href{https://github.com/afritzler/oopsie}{(238914477)} analyzes the \\
        & \textbf{faults or problems.} & logs of Kubernetes containers to find \\
        & & the root causes of issues. \\
        \midrule
        
        \textbf{Dataset} & \textbf{Providing datasets to be used} & Repository~\href{https://github.com/logpai/loghub}{(60705895)} collects various \\
        & \textbf{in other projects.} & system log datasets that can be used \\
        & & for log analysis.\\
        \midrule
        
        \textbf{AIOps infrastructure} & \textbf{Providing infrastructure support} & Repository~\href{https://github.com/nailo2c/spellpy}{(244678163)} provides \\
        & \textbf{or utility functions such as log} & automated parsing of raw logs. \\
        & \textbf{parsing.} & \\
        \midrule
        
        \textbf{Classification} & \textbf{Classifying different input data} & Repository~\href{https://github.com/NCBI-Hackathons/Semantic-search-log-analysis-pipeline}{(238914477)} classifies web \\
        & \textbf{instance based on their similarities.} & queries to find similar important \\
        & & information and trends. \\
        \midrule
        
        \textbf{Knowledge extraction} & \textbf{Extracting and summarizing} & Repository~\href{https://github.com/OS-ABC/AIOps-Event-Graph-WebData}{(345320486)} extracts \\
        & \textbf{knowledge from datasets} & information from Stack Overflow to  \\
        & \textbf{or websites.} & find fast solutions for faults in their \\
        & & platform. \\
        \midrule

        
        \textbf{Self healing} & \textbf{Conducting health checks and} & Repository~\href{https://github.com/keikoproj/active-monitor}{(238914477)} aims to provide \\
        & \textbf{automatically fixing the issues.} & software systems with monitoring and \\
        & & self-healing. \\
        \midrule

 \end{tabular}
    }
    \begin{tablenotes}
    \small
      \item To find a GitHub repository with its ID, one can either click the hyperlink or use the link \textit{https://api.github.com/repositories/\{ID\}} where \textit{\{ID\}} is replaced by a specific repository ID.
    \end{tablenotes}
    \label{tab:goals_definitions}
\end{table*}

The second popular category of input data for AIOps approaches is multi-media (image, video, or audio) (6\%), followed by source code (5\%), technical Q/A data (2\%), and alarm (1\%). We define the input of a project as ``source code'' if it analyzes the source code of other software. We also define the input as ``technical Q/A data'' if the project analyzes data from Q/A websites like Stack Overflow.

\bigskip
\noindent\textbf{Analysis techniques: Classical machine learning models are the most commonly used analysis techniques, followed far behind by deep learning, time series, and statistical analysis techniques.}
We present our categorization of the applied analysis techniques of AIOps projects in detail in Figure~\ref{fig:techniques}. 
We derive 9 high-level categories: classic machine learning (50\%), deep learning (14\%), time series model (11\%), statistical analysis (11\%), others (5\%), unknown (3\%), language model (2\%), associate rule mining (2\%), and N/A  (not applicable) (2\%). Definitions and examples for each category can be found in Table~\ref{tab:techniques_definitions}.

Both the classic machine learning and deep learning categories are further divided into supervised and unsupervised learning algorithms.
Regarding classic machine learning approaches, supervised algorithms are used more than unsupervised ones (60\% to 40\%). The top-3 supervised algorithms are random forest (23\%), support vector machine (SVM) (19\%), and decision tree (17\%). The top-3 unsupervised algorithms are isolation forest (23\%), k-means (13\%), and local outlier factor (LOF) (12\%). Regarding the deep learning approaches, unsupervised techniques are more popular than supervised algorithms (63\% compared to 37\%). Long short-term memory (LSTM), autoencoder (AE), and self-organizing map (SOM) are the most used unsupervised algorithms, while artificial neural networks (ANN) is the dominant supervised technique. 


Considering our results, we could not find a strong correlation with~\citep{dang2019aiops}, where they state that in \textit{many} AIOps cases, only unsupervised machine learning models are practical. In our set of projects, there is no substantial difference between supervised and unsupervised approaches.

The following two common techniques are time series models and statistical analysis. As most of the input data is time series (e.g., logs, network traffic data, performance metrics), it is not surprising that some projects use time series techniques to model their data. Autoregressive Integrated Moving Average (ARIMA) is the most common time-series technique. 

We also find three major statistical analysis techniques: descriptive analysis, outlier detection, and exploratory analysis. We categorize the technique of a project as descriptive analysis if it analyzes the data numerically, and we group it as exploratory if it uses visualization to analyze the data. Usually, the exploratory analysis will be done after performing the descriptive analysis.

Finally, we can count Natural Language Processing (NLP) as one of the lowest-used techniques in the AIOps projects (2\%). However, with the recent developments in the NLP field, for example generating language models such as BERT~\citep{devlin2018bert} and CodeBERT~\citep{feng2020codebert}, we believe AIOps solutions can also benefit more from NLP techniques. We discuss this point in more detail in Section~\ref{sec:discussion}.

\begin{figure*}
\centering
\includegraphics[width=0.7\textwidth]{bubble.pdf}
\caption{The relation between the input data and goals of the AIOps projects. The sizes of the circles are proportional to the number of projects that use a certain input for a certain goal.}
\label{fig:bubble}    
\end{figure*}

\bigskip
\noindent\textbf{Goals:
Anomaly detection is the most popular goal of the studied AIOps projects, followed by monitoring, anomaly prediction, and root cause analysis.}
We find 9 categories
for the goals of the AIOps projects which are shown in Figure~\ref{fig:goals} and are described in Table~\ref{tab:goals_definitions}. Anomaly detection (58\%) and monitoring (15\%) are the most common reasons for using AIOps solutions.
Anomaly prediction, root cause analysis, and dataset are the next main goals of our set of projects. We categorize the projects as dataset if they mainly provide public AIOps data that can be used by other practitioners or researchers.

As shown in Figure~\ref{fig:goals}, only 1\% of projects do self-healing as their final goal. It means, in the other projects, after achieving the final goal, for example, anomaly detection, an agent (e.g., developer) should decide what to do with the founded anomalies. This not completely automated procedure will lead to a loss of time and resources. We discuss this point in more detail in Section~\ref{sec:discussion}. 

The relation between the input data and the goals of the AIOps projects is shown in Figure~\ref{fig:bubble}. 
As shown in this figure, log data and network traffic data are the most used data types for achieving the goal of anomaly detection, while performance metric data is the most used data type for anomaly prediction. Log data and performance metric data are also the most common data sources for the goal of root cause analysis. 






\newenvironment{mybox2}[1]{%
    \begin{tcolorbox}[title={Summary of RQ2}]%
    }{Logs are the most commonly used input data in the studied AIOps projects, followed by network-traffic data and performance metrics. Classical machine learning techniques are the most used analysis techniques, followed (far behind) by deep learning, time series, and statistical analysis.
    The most popular goals of the AIOps projects are anomaly detection, followed by monitoring, anomaly prediction, and root cause analysis. 
   
    \end{tcolorbox}
}
\begin{mybox2}{}

\end{mybox2}

\subsection{\textbf{RQ3. What is the code quality of AIOps projects?}}

\subsubsection{\textbf{Motivation}}
Ensuring the quality of AIOps projects is vital for the software operation tasks they are designed for. Otherwise, the insights derived from their analysis would not be reliable. Thus, in this RQ, we aim to understand the code quality of the AIOps projects. We analyze different metrics in order to come to a comprehensive conclusion. We also compare the code quality of AIOps projects with our baselines to find similar patterns or differences.
Our results can provide insights for future work to improve the quality of AIOps projects.

\begin{table*}[h!]
\caption{The code quality metrics and their definitions.}
    \resizebox{1\textwidth}{!}
    {
    \centering
    \small
    \begin{tabular}{ll}
        \midrule
        Metric & Definition \\
        \midrule
        
        \textbf{Size} & \textbf{The metrics that represents the size of a project.} \\
        
        \rule{0pt}{4ex} 
        \quad Lines of Code (LOC) & \quad Number of lines that contain at least one character which is not a whitespace, \\
        & \quad a tabulation, or part of a comment. \\
        
        \rule{0pt}{4ex} 
        \quad Comment lines & \quad Number of lines containing comments. Non-significant comments (empty comment \\
        & \quad lines, comment lines only having special characters, etc.) are not considered. \\
        
        \rule{0pt}{4ex} 
        \quad Density of comments & \quad The amount of lines of comments compared to lines of code. It is calculated\\
        & \quad based on the following formula. \\
        & \quad Density of comments = Comment lines / (Lines of code + Comment lines) * 100 \\
        
        \midrule
        \textbf{Reliability} & \textbf{The issues that make the code behave differently as the developer was} \\
        & \textbf{intended.} \\
        
        \rule{0pt}{4ex} 
        \quad Number of bugs & \quad Total number of bugs in a project. A bug is defined as an issue that represents \\
        & \quad something wrong in the code. \\
        
        \midrule
        \textbf{Maintainability} & \textbf{The issues that make the code more difficult to update than it should.} \\
        
        \rule{0pt}{4ex} 
        \quad Number of code smells & \quad Total number of code smells in a project. A code smell is a violation of design \\
        & \quad patterns that may negatively impact software quality. \citep{rasool2015review}\\
        
        \rule{0pt}{4ex} 
        \quad Technical debt time & \quad The estimated time required to fix all the code smells. If the values are written \\
        & \quad in days, an 8-hour day is assumed. \\
        
        \midrule
        \textbf{Security} & \textbf{The issues that make potential weaknesses in terms of security. These} \\
        & \textbf{issues might be benefited by hackers.} \\
        
        \rule{0pt}{4ex} 
        \quad Number of vulnerabilities & \quad Total number of vulnerabilities in a project. A vulnebarity is a piece of code that \\
        & \quad could be exploit by a hacker. \\
        
        \rule{0pt}{4ex} 
        \quad Number of security hotspots & \quad Total number of security hotspots in a project. A security hotspot is the pieces of \\
        & \quad code that are security-sensitive. \\
        \midrule
        
 \end{tabular}
    }
    \label{tab:sonar_definition}
\end{table*}

\subsubsection{\textbf{Approach}}

We use SonarQube\footnote{\url{https://www.sonarqube.org/}}, a static code analysis tool that supports a large number of programming languages, including Python, Java, JavaScript, and Go, to statically measure the quality of the AIOps projects and the baselines, as AIOps projects are developed by different languages (RQ1). Many recent studies have utilized or evaluated SonarQube for code quality measurement~\citep{businge2019studying, saarimaki2019accuracy, lomio2021fault}.

\noindent \textbf{Code quality metrics.}
To understand the code quality of AIOps projects, we measure the metrics of each studied project along four dimensions: size, reliability, maintainability, and security. The detailed list of the measured metrics include: Lines of Code (LOC), comment lines, density of comments, number of bugs, number of code smells, number of vulnerabilities, number of security hotspots, and technical debt time. We define these metrics in Table~\ref{tab:sonar_definition}.

SonarQube also assigns a severity level to each of the issues. It categorizes the severities into four groups; Minor, Major, Critical, and Blocker, from the lowest to the highest severity\footnote{\url{https://docs.sonarqube.org/latest/user-guide/rules/}}.

\noindent \textbf{Finding the most violated rules and rule categories.} To provide insights into the primary code quality issues among the AIOps projects, we count each issue’s assigned \textit{rules} and \textit{rule categories} (i.e., \textit{tags}) and report the most repeated ones. SonarQube evaluates the source code against its set of rules to detect specific issues. Also, rule categories are a way to categorize trivial rules into higher-level concepts. SonarQube rules include code smells, bugs, vulnerabilities, and security hotspots. Each rule is related to a specific defined issue, and different rules can be part of one rule category.

To find the most violated issues (i.e., rules and rule categories), it is essential to consider two aspects: the occurrence of an issue in each project and the percentage of projects that have that issue. Hence, we calculate the weight of violated issues (``W'' in Tables~\ref{tab:sonarqube_rules} and~\ref{tab:sonarqube_tags}) to find the most violated issues.
Each project can have multiple violated issues. We calculate the weight of issue \textit{i} in project \textit{j} using the following formula,
\[w_{ij} = \frac{n_{ij}}{n_{pj}}\]
where $n_{ij}$ is the frequency of issue \textit{i} in project \textit{j} and $n_{pj}$ is the total number of issues in project \textit{j}. Then, to calculate the weight of issue \textit{i} in all the projects, we calculate its average in all the projects.
\[w_{i} = \frac{\sum_{j=1}^{n}w_{ij}}{n}\]




\noindent\textbf{Preparing project data and running SonarQube.} We execute the following three steps to analyze the quality of each of the AIOps and baseline projects.


\textbf{Step 1. Clone the projects from GitHub.}
We clone the GitHub repositories on the local machine so that we can analyze their source code.

\textbf{Step 2. Preprocess the project data.} For two of the popular languages among our projects (i.e., Python and Java), we have to perform a preprocessing phase before performing the source code analysis using SonarQube. We do not need to perform this phase for other languages.

   
    \textit{Python code.} SonarQube does not support the \textit{.ipynb} format which is the file format of Jupyter notebooks.
    To overcome this limitation, we first convert the \textit{.ipynb} files in each project to the \textit{.py} format using the \textit{nbconvert} library~\citep{nbconvert} in Python.
   
    We then use SonarQube to analyze the resulting \textit{.py} files.
    
   
    \textit{Java code.} To analyze the Java code, SonarQube requires the compiled files (\textit{.class} files). However, in most cases, developers do not upload the \textit{.class} files in their GitHub repositories and only put the \textit{.java} files. 
    To address this issue, we compile the \textit{.java} files into \textit{.class} files.
   
    We primarily leverage the build automation tool (i.e., Gradle, Apache Maven, and Apache ANT) of the project to compile the project files. For example, for Maven projects, we use \textit{mvn compile}. If we could not find a build automation tool, we use \textit{javac} to compile the \textit{.java} files.

   

\textbf{Step 3. Execute SonarQube.} We write a script that
sends the source code of each project to a SonarQube web server that is installed locally. 
After SonarQube finishes the analysis, we extract the results.


We are able to analyze the source code of 6,101 out of the 7,777 projects (78\%) in all three groups of projects (AIOps projects and the two baselines). The success ratio is 95\%, 85\%, and 72\% for AIOps, ML, and General projects, respectively.
The main reasons of failures
include removed GitHub repositories
and failures in compiling the Java projects. Analyzing the source code of projects at this scale is rarely reported in the literature, and performing the SonarQube analysis experiments took several weeks.






\subsubsection{\textbf{Results}}
\label{sec:rq3_results}
\noindent\textbf{AIOps projects have poorer quality than the ML and General baselines.}
Table~\ref{tab:sonarqube_metrics} summarizes the code quality metrics extracted from SonarQube for the AIOps and the baseline projects. We report the mean and median number of each of the quality metrics. Due to the skewed data and similar to Section~\ref{rq1_results}, we mainly compare the metrics using their median values.

For bugs, code smells, vulnerabilities, security hotspots, and technical debt, we further report the values normalized by the LOC of each project. We report these normalized values to ensure that our findings are robust and not biased by the size of the projects.

\begin{table*}[h!]
        \caption{The code quality metrics of the AIOps projects and the two baselines extracted from SonarQube. The mean and median are represented by $\mu$ and $\widetilde{\chi}$, respectively. The ``raw'' values are the exact numbers extracted from SonarQube, and the ``norm'' indicates normalized values by LOC of each project. }
    \resizebox{1\textwidth}{!}
    {
    \centering
    \small

    \begin{tabular}{l|cccc|cccc|cccc|cccc}
        \toprule
         & \multicolumn{4}{c|}{\textbf{Bugs}} 
         & \multicolumn{4}{c|}{\textbf{Code smells}} 
         & \multicolumn{4}{c|}{\textbf{Vulnerabilities}} 
         & \multicolumn{4}{c}{\textbf{Security hotspots}} \\
         & \multicolumn{2}{c}{raw} & \multicolumn{2}{c|}{norm} 
         & \multicolumn{2}{c}{raw} & \multicolumn{2}{c|}{norm}
         & \multicolumn{2}{c}{raw} & \multicolumn{2}{c|}{norm}
         & \multicolumn{2}{c}{raw} & \multicolumn{2}{c}{norm} \\
         & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ \\
        \midrule
        \textbf{AIOps} & 109.8 & 3.0 & 4.7 & 1.1 & 641.3 & 57.0 & 54.5 & 34.8 & 0.2 & 0.0 & 0.1 & 0.0 & 28.5 & 1.5 & 2.2 & 0.5 \\
        \textbf{ML} & 55.6 & 1.0 & 6.8 & 0.5 & 164.0 & 27.0 & 44.2 & 25.2 & 0.1 & 0.0 & 0.0 & 0.0 & 7.8 & 0.0 & 2.2 & 0.0 \\
        \textbf{General} & 58.3 & 1.0 & 8.6 & 0.4 & 241.0 & 13.0 & 35.5 & 17.3 & 0.2 & 0.0 & 0.1 & 0.0 & 16.0 & 1.0 & 3.7 & 0.3 \\
    
        \midrule
        \rule{0pt}{2ex}
        & \multicolumn{4}{c|}{\textbf{Technical debt (hour)}}
        & \multicolumn{4}{c|}{\textbf{LOC (k)}}
        & \multicolumn{4}{c|}{\textbf{Comment lines (k)}}
        & \multicolumn{4}{c}{\textbf{Comment density (\%)}} \\
        & \multicolumn{2}{c}{raw} & \multicolumn{2}{c|}{norm} 
        & \multicolumn{4}{c|}{} & \multicolumn{4}{c|}{}\\
        & $\mu$ & $\widetilde{\chi}$ & $\mu$ & $\widetilde{\chi}$ & & $\mu$ & $\widetilde{\chi}$ & & & $\mu$ & $\widetilde{\chi}$ & & & $\mu$ & $\widetilde{\chi}$ \\
        \midrule
        \textbf{AIOps} & 118.6 & 6.6 & 5.6 & 4.6 & & 78.1 & 1.4 & & & 20.3 & 0.4 & & & 20.4 & 19.6 \\
        \textbf{ML} & 29.1 & 3.2 & 5.0 & 3.2 & & 43.6 & 0.9 & & & 2.7 & 0.2 & & & 21.8 & 20.2 \\
        \textbf{General} & 52.6 & 1.7 & 5.3 & 2.4 & & 16.6 & 0.7 & & & 2.2 & 0.1 & & & 11.3 & 7.6 \\
        \midrule
    \end{tabular}
    }

    \label{tab:sonarqube_metrics}
    \vspace{-4mm}
\end{table*}


\begin{figure*}
\centering
\includegraphics[width=0.6\textwidth]{sonar_issues-cropped.pdf}
\caption{The distribution of the issues among the AIOps and baseline projects.}
\label{fig:sonarqube_issues}    
\end{figure*}

As shown in Table~\ref{tab:sonarqube_metrics}, the number of issues (i.e., bugs, code smells, vulnerabilities, and security hotspots) in AIOps projects are higher than that of baselines considering both raw and normalized values. This difference is more evident in the median number of bugs (3 in AIOps, 1 in ML and General) and code smells (57 in AIOps, 27 in ML, and 13 in General). This pattern is also visible in the normalized numbers; for example, the normalized median of code smells in AIOps projects is 34.8 per KLOC, much higher than for ML (25.2) and General (17.3) baselines.

Following the same pattern, the needed time to resolve these issues (i.e., technical debt) in AIOps projects is twice that of ML projects and nearly four times that of General projects (6.6, 3.2, and 1.7 in AIOps, ML, and General projects, respectively). These findings are in line with the high volume of issues observed in AIOps repositories, as shown in Table~\ref{tab:github_metrics}. 

\noindent\textbf{AIOps projects have more lines of code than baselines, and same amount of comments as the ML baseline.
}
As shown in Table \ref{tab:sonarqube_metrics}, the median LOC in AIOps projects is higher than the baselines (1.4k compared to 0.9k and 0.7k). This is also aligned with our observation about the size of the projects in Table \ref{tab:github_metrics}.
Regarding the comment density, we can observe a similar pattern among the AIOps and ML projects, where the use of comments is much higher than in the General baseline (19.6\% and 20.2\% compared to 7.6\%). This indicates that AIOps and ML projects document their code more adequately.




\noindent\textbf{AIOps projects suffer from code smells more than other types of issues.}
Figure \ref{fig:sonarqube_issues} illustrates the distribution of bugs, code smells, vulnerabilities, and security hotspots among the projects. The percentage of security hotspots and vulnerabilities is much smaller than bugs and code smells in all the projects. In AIOps projects, the main issue is related to code smells, where they account for 82\% of all issues, ten percent more than the ML baseline. 



\begin{table}[t]
\centering
\caption{The top-10 violated SonarQube rules for AIOps projects and the baselines. ``Sev'' indicates the severity of issues, ``W'' represents the weight of rules, and ``N'' is the percentage of projects with that rule. ``Mn'' stands for Minor, ``Mj'' stands for Major, and ``Cr'' stands for Critical.}
    \resizebox{1\textwidth}{!}
    {
    \small
    \begin{tabular}{llll|llll|llll}
        \toprule
        \multicolumn{4}{c|}{\textbf{AIOps}} & \multicolumn{4}{c|}{\textbf{ML}} &
        \multicolumn{4}{c}{\textbf{General}}\\
        Rule & Sev & W(\%) & N(\%) &
        Rule & Sev & W(\%) & N(\%) &
        Rule & Sev & W(\%) & N(\%) \\
        \midrule
        python:S117 & Mn & 17.9 & 58.2 & python:S117 & Mn & 17.7 & 59.3 & python:S117 & Mn & 3.8 & 15.0 \\
        python:S125 & Mj & 15.5 & 71.4 & python:S125 & Mj & 13.7 & 64.4 & javascript:S1117 & Mj & 3.4 & 19.4 \\
        python:S1192 & Cr & 7.6 & 64.8 & python:S1481 & Mn & 6.9 & 54.9 & python:S125 & Mj & 3.0 & 17.0 \\
        python:S905 & Mj & 5.6 & 26.4 & python:S905 & Mj & 6.4 & 26.8 & Web:S5254 & Mj & 2.5 & 22.6 \\
        python:S1481 & Mn & 3.8 & 52.7 & python:S1192 & Cr & 6.1 & 48.0 & python:S1192 & Cr & 2.1 & 15.2 \\
        python:S3776 & Cr & 3.2 & 47.3 & python:S1542 & Mj & 4.1 & 39.9 & html:S1100 & Mn & 2.1 & 12.3 \\
        python:S1542 & Mj & 2.9 & 48.4 & python:S2320 & Mj & 3.9 & 14.1 & python:S1481 & Mn & 2.0 & 15.3 \\
        python:S2208 & Cr & 2.6 & 28.6 & python:S3776 & Cr & 3.6 & 41.3 & javascript:S125 & Mj & 1.9 & 16.8 \\
        python:S100 & Mn & 2.5 & 23.1 & python:S2208 & Cr & 2.5 & 25.2 & python:S2320 & Mj & 1.8 & 7.0 \\
        python:S116 & Mn & 1.5 & 20.9 & python:S5754 & Cr & 1.7 & 24.1 & css:S4666 & Mj & 1.7 & 14.7
    \end{tabular}
    }
        
    \label{tab:sonarqube_rules}
\end{table}


\begin{table}[t]
\centering
        \caption{The top-10 violated SonarQube tags (rule categories) for AIOps projects and the baselines. ``W'' represents the weight of tags, and ``N'' is the percentage of projects with that tag.
        }
    \resizebox{1\textwidth}{!}
    {
    \small

    \begin{tabular}{lll|lll|lll}
        \toprule
        \multicolumn{3}{c|}{\textbf{AIOps}} & 
        \multicolumn{3}{c|}{\textbf{ML}} &
        \multicolumn{3}{c}{\textbf{General}} \\
        Tag & W(\%) & N(\%) &
        Tag & W(\%) & N(\%) &
        Tag & W(\%) & N(\%) \\

        \midrule
        convention & 26.3 & 76.9 & unused & 24.7 & 82.7 & unused & 13.7 & 60.8 \\
        unused & 24.1 & 94.5 & convention & 23.1 & 68.9 & convention & 9.2 & 36.7 \\
        design & 8.8 & 76.9 & cwe & 6.4 & 53.0 & suspicious & 7.7 & 51.4 \\
        cwe & 7.7 & 63.7 & design & 5.6 & 52.6 & cwe & 6.3 & 48.6 \\
        brain-overload & 4.2 & 61.5 & suspicious & 4.1 & 48.4 & pitfall & 6.3 & 46.0 \\
        pitfall & 4.1 & 49.4 & brain-overload & 3.9 & 47.8 & accessibility & 5.1 & 29.2 \\
        suspicious & 4.1 & 64.8 & pitfall & 3.5 & 40.0 & design & 4.8 & 35.9 \\
        accessibility & 2.1 & 26.4 & obsolete & 2.9 & 19.5 & brain-overload	& 4.4 & 46.8 \\
        obsolete & 1.8 & 23.0 & python3 & 2.5 & 15.7 & wcag2-a & 3.2 & 28.1 \\
        bad-practice & 1.8 & 53.8 & accessibility & 1.9 & 13.5 & clumsy & 3.0 & 38.4
    \end{tabular}
    }

    \label{tab:sonarqube_tags}
\end{table}

\begin{table}[h!]
\centering
        \caption{The most violated SonarQube rules and their definitions in AIOps projects.
        }
    \resizebox{1\textwidth}{!}
    {
    \small

    \begin{tabular}{|ll|}
        \toprule
        \multicolumn{2}{|c|}{\textbf{Rules}} \\
        Name & Definition \\

        \midrule
        python:S117 & Local variable and function parameter names should \\
        & comply with a naming convention. \\
        python:S125 & Sections of code should not be commented out. \\
        python:S1192 & String literals should not be duplicated \\
        python:S905 & Non-empty statements should change control flow or have \\
        & at least one side-effect. \\
        python:S1481 & Unused local variables should be removed. \\
        python:S3776 & Cognitive Complexity of functions should not be too high. \\
        python:S1542 & Function names should comply with a naming convention. \\
        python:S2208 & Wildcard imports should not be used. \\
        python:S100 & Method names should comply with a naming convention. \\
        python:S116 & Field names should comply with a naming convention. \\
        \midrule
    \end{tabular}
    }

    \label{tab:rules_definition}
\end{table}


\begin{table}[h!]
\centering
        \caption{The most violated SonarQube tags (rule categories) and their definitions in AIOps projects.}
        
    \resizebox{1\textwidth}{!}
    {
    \small

    \begin{tabular}{|ll|}
        \toprule
        \multicolumn{2}{|c|}{\textbf{Tags}} \\
        Name & Definition \\

        \midrule
        convention & Coding convention - typically formatting, naming, whitespace.  \\
        unused & Unused code; e.g., a private variable that is never used. \\
        design & There is something questionable about the design of the code. \\
        cwe & Relates to a rule in the Common Weakness Enumeration. \\
        & For more information, visit `https://cwe.mitre.org/'. \\
        brain-overload & There is too much to keep in your head at one time. \\
        pitfall & Nothing is wrong yet, but something could go wrong in the \\
        & future; a trap has been set for the next person, and they'll \\
        & probably fall into it and screw up the code. \\
        suspicious & It's not guaranteed that this is a bug, but it looks suspiciously \\
        & like one. At the very least, the code should be re-examined \\
        & and likely refactored for clarity. \\
        accessibility & Rules to make content more accessible, clear, and simple. \\
        obsolete & Using deprecated elements and attributes. \\
        bad-practice & The code likely works as designed, but the way it was designed \\
        & is widely recognized as being a bad idea. 


 \\
        \midrule
    \end{tabular}
    }

    \label{tab:tags_definition}
\end{table}


\noindent\textbf{AIOps and ML projects have similar types of issues in terms of violated SonarQube rules and rule categories.}
We report the most violated rules and rule categories in Tables \ref{tab:sonarqube_rules} and \ref{tab:sonarqube_tags}. We also define these violated rules and rule categories in AIOps field (based on SonarQube website\footnote{\url{https://rules.sonarsource.com/}}\footnote{\url{https://docs.sonarqube.org/latest/user-guide/built-in-rule-tags/}}) in Tables~\ref{tab:rules_definition} and~\ref{tab:tags_definition}. As shown in the tables, AIOps and ML projects have similar rule and rule category violations. In terms of violated rules, 8 out of the top-10 violated rules are common between AIOps and ML projects. Furthermore, the first two most violated rules (i.e., python:S117 and python:S125) are the same in AIOps and ML projects, with similar weights (17.9\% and 15.5\% for AIOps, and 17.7\% and 13.7\% for ML projects). Also, in terms of most violated rule categories, 9 out of the top-10 violated rule categories are common between AIOps and ML projects.

Comparing issues between the AIOps projects and the General baseline, 4 of the top-10 violated rules and 8 of the top-10 violated rule tags are common. However, the weight and percentage of projects having these issues in AIOps projects are much higher. As an example, Python:S117 is the most violated rule in both AIOps and General projects. However, the weight and percentage of projects having this violated rule in AIOps are 17.9\% and 58.2\% but in the General baseline are 3.8\% and 15.0\%, respectively.

We provide the complete list of violated rules and rule categories in AIOps projects and other baselines in our replication package. 










\noindent\textbf{Naming convention, commented-out code, duplicated string literals, high complexity of functions, and wildcard imports are among the top violated rules in AIOps projects.}
Regarding the most violated rules in AIOps, the first three are \textit{python:S117}, \textit{python:S125}, and \textit{python:S1192}. The first one has a minor severity and is about the non compliance of naming convention. Shared naming conventions are vital and allow teams to collaborate effectively. The second one has a major severity and is about commenting out the unused sections of code which reduces readability. Instead, unused code should be deleted. The third one has a critical severity and is about using duplicated string literals. It makes the process of refactoring error-prone because the programmer must be sure to update all occurrences of the string.

In addition to these violated rules, two more rules with critical severity exist in the top 10 violated rules of AIOps projects; \textit{python:S3776} and \textit{python:S2208}. \textit{Python:S3776} is about the cognitive complexity of code. It measures how hard the control flow of a function is to understand. Functions with high cognitive complexity will be difficult to maintain. \textit{Python:S2208} is about using wildcard imports (i.e., from module import *). Importing all public names from a module has multiple disadvantages. It can lead to conflicts between local names and imported ones, or same names between two different packages. It also reduces code readability and may cause confusion about which classes are imported and used.

\noindent\textbf{Naming convention, unused or commented-out code, and poor design are among the top violated rule categories.} Regarding the most violated rule categories in AIOps repositories, \textit{convention}, \textit{unused}, and \textit{design} are the worst issues. \textit{Convention} category is about fulfilling coding conventions, including naming functions and variables, complying white-spaces and indentations. The second most violated rule category is \textit{unused}. This category is about unused code that decreases the performance of the system. Some examples of this category are unused assignments, unused private classes, and empty test cases. The third most ignored rule category, \textit{design}, is about the bad design of the software. Some examples of this group are unstable tests, duplicated string literals, and using randomized data in test cases. 

Having rule category issues such as naming conventions, unused code, and bad design indicate that developers tend to write AIOps code in an \textit{ad hoc} manner, which hinders their project’s reusability and maintainability. Also, looking at the most violated rules and rule categories, we realize paying more attention to details can increase the quality
of AIOps solutions. For example, 3 out of the top-10 violated rules are related to naming conventions. Only following the naming conventions can reduce the number of issues heavily. Furthermore, since most of the rules (8 of top-10) and rule categories (9 of top-10) are common in AIOps and ML projects, AIOps projects can benefit from the quality assessment and quality assurance tools and techniques that have been developed for ML systems. The above points are discussed in more detail in Section~\ref{sec:discussion}.



\newenvironment{mybox3}[1]{%
    \begin{tcolorbox}[title={Summary of RQ3}]%
    }{
    \end{tcolorbox}
}
\begin{mybox3}{}
Although AIOps projects have an adequate amount of comments compared to the ML and General baselines, AIOps projects exhibit poorer quality than the two baselines in terms of a variety of quality metrics (e.g., bugs, code smells, and technical debt). In particular, code smells are the dominant type of issue in AIOps projects. We also identify the most common issues in AIOps projects: naming convention, unused code, and bad design are the top-3 violated rule categories. Moreover, we observe the similarity of violated rules between AIOps and ML projects. Future efforts are needed to reduce the issues identified in this work and improve the quality of AIOps projects, e.g., by designing tools for fixing bugs/smells or deriving coding guidelines. We also encourage future AIOps projects to reduce the \textit{ad-hoc}ness of their code to improve reusability and maintainability.
\end{mybox3}


\section{Discussion}
\label{sec:discussion}

\textbf{Natural Language Processing (NLP) techniques can receive more attention in AIOps in the coming years.}
Our results in Section~\ref{sec:rq2} indicate that only 2\% of the studied projects use NLP techniques. On the other hand, approximately one-third of the projects use logs as their input data. Prior research has shown that logs can approximately be represented as natural language text since the logs are generated by logging statements in the source code written by humans~\citep{hindle2016naturalness}. Other works also illustrate that software systems are even more predictable and repetitive than human languages, such as English~\citep{tu2014localness, allamanis2014mining}. Hence, we suggest that AIOps projects can benefit from leveraging the advances in NLP techniques to analyze and model the input data such as logs. 

\bigskip 
\noindent\textbf{More attention should be paid to increasing automation and reducing human interventions in AIOps solutions.}
Regarding the goal of the studied projects, most projects aim to detect and predict anomalies, monitor, and analyze the root causes of failures. All these systems need human interventions when the goal is reached. For example, when an anomaly is detected, an operator should decide the next required action to handle the situation. As demonstrated in Figure~\ref{fig:techniques}, only 1\% of projects have self-healing, meaning that the system can make necessary changes when needed without human interventions. We believe AIOps solutions should move toward becoming more automated, detecting/predicting the incidents and resolving them autonomously.

\bigskip 
\noindent\textbf{Paying more attention to simple details can increase the quality of AIOps solutions.}
We identify the most common issues in AIOps projects in Section~\ref{sec:rq3_results}. When looking at the most violated rules, some of them are challenging and time-consuming to fix and need structural changes in code, such as \textit{python:S3776} (cognitive complexity of functions) and \textit{python:S1192} (duplication of string literals). On the other hand, some are easy to handle. 3 of the 10 most violated rules are related to naming conventions. Only following the naming conventions can reduce the number of issues heavily. Furthermore, rules such as \textit{python:S2208} (using wildcard imports) and \textit{python:S1481} (removing unused variables) are quick to determine and easy to fix. Identifying the most common issues help practitioners and researchers in the field to become aware of them and take measurements to reduce them. 

\bigskip 
\noindent\textbf{Most of the tools and techniques used for the quality assurance of machine learning systems can also be used for AIOps solutions.}
Our results indicate that a high proportion of issues in ML and AIOps techniques are the same (80\% of most violated rules and 90\% of most violated rule categories are shared). Besides, the quality assurance of ML systems has received special attention in the previous years, and different approaches have been built to preserve the quality of ML-based systems~\citep{nakajima2018quality, poth2020quality,Nikanjam21,NikanjamDRL,Braiek22,TAMBON2023107129}. Hence, we believe most of the developed tools and techniques for quality assurance in ML systems can be utilized in AIOps techniques. Applying these tools and techniques may reduce the quality issues mentioned in Section~\ref{sec:rq3_results}.

\section{Threats to Validity} 
\label{sec:threats}
This section discusses threats to the validity of our results.

\noindent\textbf{External validity.}
In this work, we identified and studied a set of AIOps projects on GitHub. These projects may not cover all AIOps projects on GitHub, those hosted on other platforms, or private projects. To maximize our coverage of AIOps projects and avoid false positives, we follow a process that combines automated search (two rounds), keywords expansion, manual verification, and filtering. Two authors of the paper carefully examined each of the candidate projects to select the ones for our study. The third author stepping in to resolve any disagreement. 
Nevertheless, future work can leverage our replication package and extend our study by analysing more AIOps projects.

We also collect the ML baseline based on two keywords of ``machine learning'' and ``deep learning''. However, having these two keywords may not include all machine learning repositories on GitHub. Future work can expand our results by analyzing more projects extracted from more diverse keywords.


\noindent\textbf{Internal validity.} We study the code quality metrics of the AIOps projects using a set of metrics (e.g., code smells). Nevertheless, these metrics may not accurately represent the quality of the projects. To reduce the effect of this threat, we leveraged a variety of metrics that represent different aspects of each project. These metrics have also been used in other articles to measure code quality~\citep{tan2018towards, businge2019studying, lenarduzzi2019diffuseness}.

\noindent\textbf{Construct validity.} To expand our set of AIOps projects, we perform pattern mining and choose 4 out of 194 pairs of keywords. We choose these 4 pairs of keywords after a systematic process and based on the discussions between the authors, in order to reduce the bias.
All of the keywords are among the most frequently used keywords.

To answer RQ2, we use qualitative analysis to categorize the input data, analysis techniques, and goals of each AIOps project. Our results may be biased by personal deductions like any other qualitative study. In order to mitigate this threat, two authors of the paper performed the qualitative analysis carefully and followed a 5-step process to deduce the categories. We achieve a Cohen's kappa value of 0.81 which shows a strong and reliable agreement. 

We rely on the code issues detected by SonarQube to answer our RQ3. We are aware that SonarQube might have false positives or false negatives. However, SonarQube claim to have zero false-positives for code smells and bugs\footnote{\url{https://docs.sonarqube.org/latest/user-guide/rules/}}. Furthermore, we choose SonarQube since it is one of the most used tools for analyzing code quality~\citep{lenarduzzi2017analyzing, lenarduzzi2018survey}.

