


\section{Introduction}
\label{sec:introduction}
\input{introduction}

\section{Goals and Method's Overview}
\label{sec:rqs}
\input{methodology}

\section{\textbf{RQ$_1$} -- Flaky Tests: Nature, Origin, and Fixing}
\label{sec:rq1}
\input{rq1}

\section{\textbf{RQ$_2$} -- Flaky Tests: Relevance}
\label{sec:rq2}
\input{rq2}

\section{\textbf{RQ$_3$} -- Flaky Tests: Challenges}
\label{sec:rq3}
\input{rq3}

\section{Discussion and Implications}
\label{sec:discussion}
\input{discussion}

\section{Threats to Validity}
\label{sec:threats}
\input{threats}

\section{Related Work}
\label{sec:related}
\input{related}

\section{Conclusion}
\label{sec:conclusion}
\input{conclusion}

\begin{acks}
	The authors would like to thank all the \textsc{Mozilla}\xspace developers who participated in the creation of the dataset, as well as the respondents to our survey: You provided us and our research community with very precious data.
	Bacchelli and Palomba gratefully acknowledge the support of the Swiss National Science Foundation through the SNF Project No. PP00P2\_170529.
\end{acks}
\balance

\newpage


\bibliographystyle{ACM-Reference-Format}

\subsection{Research Questions}
We start our research by asking professional developers to classify (in terms of nature, origin, and fixing effort) flaky tests they previously fixed. Differently from Luo et al.\xspace \cite{Luo:fse2014}, we do not classify the underlying causes of flakiness ourselves, rather the original developers do and add information on the origin and fixing effort. Moreover, we complement the developers' analysis with information on the fixing strategies. Our goal is to triangulate the taxonomy proposed by Luo et al.\xspace \cite{Luo:fse2014} from a different data source and with a different methodology.
Hence, our first research question:

\begin{center}	
	\begin{examplebox}
		\textbf{RQ$_1$.} \emph{How do professional developers categorize flaky tests, in terms of nature, origin, and fixing effort? How do they fix them?\xspace}
	\end{examplebox}	 
\end{center}

Subsequently, we turn to a broader audience of respondents. We conduct a survey targeting software developers on their experience with flaky tests.
We investigate the frequency of flaky tests in practice and how problematic flaky tests are from the developer's perspective. 
We ask:

\begin{center}	
	\begin{examplebox}
		\textbf{RQ$_2$.} \emph{How prominent is test flakiness and how problematic is it as perceived by developers?\xspace}
	\end{examplebox}	 
\end{center}

Finally, we investigate the challenges that developers perceive as most critical when they have to deal with flaky tests. We ask:

\begin{center}	
	\begin{examplebox}
		\textbf{RQ$_3$.} \emph{What are the main challenges that developers face when dealing with flaky tests?}
	\end{examplebox}	 
\end{center}


\subsection{Overview Of The Research Method}
Our study features a mixed-methods approach \cite{johnson2004mixed} where quantitative and qualitative research are run in parallel with the goal of converging toward an empirical understanding of test code flakiness. We design and conduct a study to obtain data from two main sources: A novel annotated dataset (\textbf{\textsc{ds}}) of flaky tests (with 200\xspace instances annotated by 21\xspace professional developers) and the responses to an online survey (\textbf{\textsc{os}}) (with 121\xspace valid responses from developers).




\subsection{Developers' Analysis Of Flaky Tests}

In the following, we describe the methodological steps conducted to address our first research question.\smallskip

\textbf{Subject Flaky Tests.}
We first need to collect a dataset of flaky tests, which we then present to developers for their analysis.
To this aim, we mine the \textsc{Bugzilla} issue tracker of \textsc{Mozilla} (i.e.,\xspace a large free software organization counting more than 40,000 active contributors spread around the world and developing a variety of software, ranging from layout engines to operating systems). \textsc{Mozilla} has a database of flaky tests, verified and fixed by developers. According to such a database, the organization has 100 to 150 new flaky tests being detected every week: this makes it particularly suitable for our study because developers are very frequently in charge of diagnosing and fixing flaky tests, being therefore able to provide authoritative information on the nature of test flakiness, its origin, and the fixing effort.

We extract the data available after a flaky test has been reported. Thus, we gather the log files produced by the continuous integration system and the source code associated with the fixed flaky test (if available). 
In a first step, we identify all (869) fixed flaky tests the \textsc{Mozilla}\xspace's bug tracking system tagged \texttt{resolved} and \texttt{fixed} between May 2017 and April 2018. We consider this time window since the company only stores the log files for twelve months after their creation. 
From the set of retrieved flaky tests, we remove all instances for which no patch has been applied or other type of attachment is available (e.g.,\xspace some flaky tests stop occurring after a while and are closed as fixed), as they were never fixed or were false alarms.
This step filters out 295 tests; resulting in a set of 574 flaky tests, which we use in the next steps.

\smallskip
\textbf{Recruiting \textsc{Mozilla}\xspace Developers.}
The remaining 574 actual flaky tests are taken into further consideration and sorted according to the developer who is listed as the assignee or patch creator. From this list, to ensure we select people with a concrete knowledge on flaky tests, we exclude 177 programmers who have only fixed less than five  flaky tests. We end up with a list of 29 developers for a total of 304 fixed flaky tests. We contact these programmers through email and ask them to analyze the flaky tests as detailed in the following subsection.

\smallskip
\textbf{Developers' Analysis.}
We create a spreadsheet for each developer that contains two columns: the first reports the link to the bug report related to each flaky test fixed by a certain developer; the second is used by the programmer to classify the nature of each flaky test in the list. The participating developers are allowed to use the taxonomy defined by Luo et al.\xspace~\cite{Luo:fse2014} as a starting point for the classification process, as it (1) eases their task and (2) standardizes their answers---this is commonly done in both confirmatory and exploratory surveys \cite{flanigan2008conducting}. 
However, they are also allowed to create additional categories if none of the existing ones fit; at the same time, they can indicate more than one type to specify the reasons behind the flakiness of a test (it may be flaky because of both a race condition and a network problem).
We also ask the developers to specify the development effort they spent to fix the flaky tests, based on a Likert scale \cite{Oppenheim:1992} between 1 (very low) to 5 (very high), as well as the origin (test or production code) of the flakiness.
Once each developers completes the analysis, they send back their spreadsheets with the annotated classification. 
We received a total of 248 responses from 21\xspace professional \textsc{Mozilla}\xspace developers. These developers have 7.5 years (median) of experience with \textsc{Mozilla}\xspace, 958 bug reports (median) assigned to, and---over the last year---14 flaky tests fixed (median). Each developer provided us with 8.5 (median) answers (min = 4, max = 48). Of those responses, we excluded 48: (i) 24 cases turned into permanent failures, i.e.,\xspace they were labeled as intermittent by a \textsc{Mozilla}\xspace sheriff~\cite{mozillaSheriff}, but the fixing developer found them to be permanent failures; (ii) in the other 24 cases, the type of flakiness was unknown even to the developers, meaning that the fixing developers worked around the flakiness rewriting the whole test and code-under-test, but they could not ascertain the cause of the flakiness. By removing these cases, we intend to rely on precise information, only. Nevertheless, the excluded cases are available to our online appendix~\cite{eck:appendix}.
\smallskip

\textbf{Analysis Of The Developers' Responses.}
We use the 200\xspace classified flaky tests to define a taxonomy on the nature of flaky tests. In 72 cases (31\%) the assigned categories do not correspond to those available in the taxonomy by Luo et al.\xspace~\cite{Luo:fse2014} and, therefore, they are new and need to be defined. To this aim, we conduct an \emph{iterative content analysis} \cite{white2006content} to assign a common name to these new categories. The process involves three software engineering researchers, all authors of this paper, (one graduate student, one research associate, and one faculty member) and consists of the two iterative sessions:

\begin{description}[leftmargin=0.3cm]
	\item[Iteration 1:] The first author of this paper goes over the classifications made by developers. If the categories assigned belong to the taxonomy by Luo et al.\xspace~\cite{Luo:fse2014}, he leaves them as they are; if not, he assigns a temporary label to them. As an output, this step provides a draft categorization of the types of flakiness. 
	
	\smallskip
	\item[Iteration 2:] The three researchers open a discussion on the draft taxonomy, with the aim of reaching a consensus on the assigned categories. Afterwards, the first author re-categorizes flaky tests according to the decisions taken during the discussion. Finally, the second author of the paper double-checks the classifications to reduce the risk of errors (he found no wrong classifications, thus reaching a total agreement).
	
\end{description}

The resulting extended taxonomy is then used to address \textbf{RQ$_1$}. Besides reporting the list of such categories, we also characterize them with additional information on \begin{inparaenum}[(i)]
	\item their frequency, computed on the basis of the distribution of the categories within the dataset, 

	and 
	\item the fixing strategies.
\end{inparaenum}
As for the latter point, to characterize the fixing strategies adopted by developers when dealing with flaky tests, two authors of this paper conduct a new iterative content analysis that---similarly to the previous one---consists of two iterations. In the first iteration, the first author analyzes the patches associated to the resolution of the considered flaky tests to label them with a name and a short description of the fixing action performed by developers. In the second iteration, the two authors open a discussion on the initial labels assigned to reach a consensus. Then, the first author applies the changes according to the discussion while the second re-checks the final classifications to reduce threats in the interpretation of the results. 


\subsection{Survey: Research Method}\label{sec:survey_method}

To answer \textbf{RQ$_2$} and \textbf{RQ$_3$}, we design and deploy an online survey, and analyze the collected answers.

\smallskip
\textbf{Overall Survey Design.} Following the guidelines provided by Flanigan et al.\xspace~\cite{flanigan2008conducting}, we limit common issues possibly arising in survey studies and affecting the response rate: we keep the survey short, respecting the anonymity of participants and preventing our influence in the answers. We create an anonymous online survey using a professional tool (i.e.,\xspace \textsc{Surveygizmo}~\cite{surveygizmo}).

The survey first describes to the respondents the concept of flaky tests, following the definition accepted by the research community: \emph{``[flaky tests are] intermittent tests that sometimes pass, sometimes fail, even though there are no changes in the code they test''}. Then, the survey 
asks for demographic information about the participants, including programming/testing experience as well as company and team size/domain. Subsequently, the survey contains other two main sections to collect data about the relevance of flakiness (\textbf{RQ$_2$}) and its challenges (\textbf{RQ$_3$}).\smallskip

\textbf{\textbf{RQ$_2$} -- Collecting Information About Relevance.} We gather data on \begin{inparaenum}[(i)]
	\item how many times flaky tests occur,
	\item how problematic they are,
	and
	\item what are the top 3 to 5 problems caused by test flakiness.
\end{inparaenum}
Participants can answer the first two questions by using a 5-point Likert scale indicating the extent to which the problem exists and how much it is serious, while they are free to write down the problems (if any) in a text box.\smallskip

\textbf{\textbf{RQ$_3$} -- Collecting Information About Challenges.}

We first conduct a multivocal literature review (MLR)~\cite{garousi2016need}. This allows us to let emerge eight pieces of information that may be linked to challenges in the process of fixing flaky tests.
Afterwards, we ask the survey respondents to rate how important is each of those information pieces from `Not at all important' to `Extremely important' (on a 4-point Likert scale) and the difficulty in obtaining it from `Very easy' to `Very difficult' (on a 5-point Likert scale). We also let the respondents indicate any information needs and/or challenges they face that are not included in the list (respondents indicated a total of another eleven needs/challenges).

For our MLR, we analyze (as described in the following protocol) both previously published papers on the topic (i.e.,\xspace white literature) and online sources (e.g.,\xspace blog posts, websites, and documents written by practitioners) related to flaky tests (i.e.,\xspace gray literature).

\begin{itemize}[leftmargin=0.3cm]
	\item \emph{Identifying relevant white literature:} We perform a keyword search on \textsc{Google Scholar}, \textsc{IEEE Explore}, and \textsc{ACM Digital Library} to create an initial selection of articles. As keywords we use: `flaky tests', `flaky test resolution practices', `intermittent failures'. This step results in an initial set of 15 papers.
	We also perform forward and backward snowballing \cite{wohlin2014guidelines}, inspecting sources referred or that refer to those belonging to the initial set of primary studies. 
	At the same time, we consider the proceedings of all the relevant, top-tier conference venues in Software Engineering such as ICSE \cite{icse}, ISSTA \cite{issta}, ICST \cite{icst}, ICSME \cite{icsme}, ESEC/FSE \cite{fse} and journals such as IEEE TSE \cite{tse}, ACM TOSEM \cite{tosem}, and Springer's EMSE \cite{emse}. This step results in the inclusion of additional four papers.
	Once having this set of resources, we filter out the papers that do not report information on what are the practices used by developers to diagnose and/or fixing flaky tests: This is achieved by reading study setting, methodology, and conclusions made in all the considered papers. The filtering process is jointly discussed among all the authors of the paper and, in the end, we agree on a list of three papers. 
	
	\smallskip
	\item \emph{Identifying relevant gray literature:} We conduct the gray literature survey following the guidelines by Garousi et al.\xspace \cite{garousi2016need}. We query \textsc{Google} by using `flaky tests', `flaky test resolution practices', `intermittent failures' as keywords. We perform our search on the results appearing in all the relevant \textsc{Google} pages until saturation is reached, i.e.,\xspace until the search results contain references to blogs, documents, or websites reporting information on flaky tests. We take great care to ensure that considered sources originate from reputable companies, organizations, and authors. We also take steps from preventing our personal search bias from playing a role in the search results, hence we search on \textsc{Google} using the incognito mode. Overall, we find 56 relevant sources to further consider. Whenever possible, we also follow the web links contained in the initial set of sources as they might refer to other documents. This step results in the addition of three sources. The high number of relevant sources identified in the gray literature, as opposed to that of the white one, hints at the fact that the problem of understanding flaky tests has been only partially addressed by the research community. This situation calls for further empirical studies to create a stronger scientific knowledge around test flakiness.
	
	\smallskip
	\item \emph{Data Extraction:} We summarize each of the 62 identified relevant sources and create an enumeration of the aspects that are mentioned and may impact on the flaky tests fixing process. Then, we conduct another iterative content analysis sessions to group those aspects into the higher level themes that are related to the important information developers need to have when diagnosing and fixing test flakiness. This is a two-step process similar to the one conducted in the context of \textbf{RQ$_1$}: the second author of this paper performs a first iteration and then the draft themes are refined with the help of the last author. This iterative process results in eight relevant information types, which we list in the survey as previously mentioned.
	
\end{itemize}

\textbf{Analysis Of The Open Answers.}
For the open answers in the survey, we conduct a two-stage process comprising Descriptive and Pattern coding analysis to name them \cite{corbin1990grounded}. Specifically, in a first step two authors shortly summarize the topic of each passage from the open answers; then, they identify explanatory codes to create themes that are finally discussed among all the authors. For example, in case of the challenges, with this process we end up with nine additional challenges grouped into two categories that indicate the point in time in which they occur, i.e.,\xspace (1) \emph{Test case design} and (2) \emph{Flaky test fixing}.

\smallskip
\textbf{Attracting Participants.} We advertised the online survey using the personal social network accounts of the authors (i.e.,\xspace \textsc{Facebook}, \textsc{Twitter}, and \textsc{LinkedIn}), through private contacts, and also published on practitioners blogs (e.g.,\xspace \textsc{Reddit}). To stimulate the participation, we allow the participants to indicate a non-profit organization of their choice to which we donate 2 USD. 

\smallskip
\textbf{Respondents.} As a result, we collect 188 answers: We filter out 68 partial answers, thus, we only consider 121\xspace full responses. Among these respondents, 75\% have more than 6 years of programming experience (median 10) and two years of industrial programming as well as testing experience (median 5). Moreover, 45\% respondents come from large companies having more than 250 employees. Our sample is composed of developers who are experienced with (industrial) programming and tests, thus we consider the collected responses as valuable for our research questions.





