\section{Introduction}\label{intro}

Many  researchers comment that healthy OSS projects are characterized by much developer activity~\cite{wahyudin2007monitoring,jansen2014measuring,manikas2013reviewing,link2018assessing,wynn2007assessing,crowston2006assessing}. 
For example, Han et al. note that popular open-source software (OSS) projects tend to be more active~\cite{han2019characterization}.

For  
OSS projects  to be active  and popular, they need the support of a large  
 community
of developers.
This support can may take many forms
including:
\begin{itemize}
\item
Programmer time (to fix bugs or write documentation or build APIs);
\item or
financial donations (to fund infrastructure);
\item or public endorsements
(e.g. when        companies join the board
of directors of OSS foundations).
\end{itemize}
 Attracting such support is  
easier when projects   show  healthy trends in their   development. 
 For this reason, in a
survey of  hundreds of decision makers from
dozens of open source projects~\cite{xia2022predicting}, it was    found that
79\% to 93\% of decision makers  wanted methods to predict
{\em open source health indicators} of Table~\ref{healthind}
such as 
 {\em commits};  {\em closed pull requests}; and  {\em number of contributors}.




 
Making such predictions    a {\em small data problem}; i.e. it 
may be necessary to make conclusions from just a few dozen rows
described in just a dozen column. For example,
Xia et al.~\cite{xia2022predicting} explored   what project health information could
be consistently collected across 1000s of Github projects.
They found they could reliably access   around data with around 15 columns
(see Table~\ref{healthind}) and 
and 60 rows\footnote{Why 60?
   Most of their projects have a lifespan of 3 to 7 years~\cite{xia2022predicting,shrikanth2021early}
   which meant that, for each project, they could reason about 15 columns and 36 to 84 rows (i.e. one row collected monthly for 3 to 7 years).}.
Such small data problems can suffer from
 {\em large prediction errors}~\cite{ngo2011data} since,
when training data from such tiny
data sources,  learners can make mistakes   as they struggle to fit a small amount of data to a model.
For example, when we apply state-of-the-art data mining tuning algorithms
(OPTUNA~\cite{akiba2019OPTUNA}) to   learning open source health predictors, even their best
tunings exhibited very large error rates. Specifically,  across 20 runs we found the mean magnitude  of  errors (MRE) was 61\% to 149\%  for predictions about 
commits;   closed  pull requests; and number of contributors. MRE is defined as follows:
\begin{equation}\label{mre}
\mathit{MRE}=
 \mathit{abs}(\mathit{actual}- \mathit{prediction})/\mathit{actual}
\end{equation}
The goal of this paper is to produce better predictions for project health with   lower  
error rates. Sarro et al.~\cite{Sarro16} assert that for this kind of project estimation,
acceptable MRE values are 30\% to 40\%. They say that:
\begin{quote}
``..the reason for choosing these two thresholds derives from evidence that industrial practice based on human judgment hopes/claims
to produce predictions within these tolerances. The evidence for these thresholds comes from a survey of current industry practices by Molokken and J{\o}rgensen~\cite{Molokken}''.
\end{quote}



\begin{table}[!t]
\scriptsize
\begin{center}
\begin{tabular}{r | l}
Feature &Description\\\hline
dates &The end date of monthly data collection\\
monthly commits& Total number of commits created in last month\\
monthly commit comments &Total number of commit comments created in last month\\
monthly contributors &Total number of contributors that at least have one commit in last month\\
monthly open PRs &Total number of pull requests opened in last month\\
monthly closed PRs &Total number of pull requests closed in last month\\
monthly merged PRs& Total number of pull requests merged in last month\\
monthly PR mergers &Total number of pull requests mergers in last month\\
monthly PR comments& Total number of pull request comments created in last month\\
monthly open issues &Total number of issues opened in last month\\
monthly closed issues &Total number of issues closed in last month\\
monthly issue comments &Total number of issue comments created in last month\\
monthly stargazer& Total number of new stars acquired in last month\\
monthly forks &Total number of forks occurred in last month\\
monthly watchers& Total number of new watchers acquired in last month
\end{tabular}
\end{center}
\caption{Project health indicators. From Xia et al. ~\cite{xia2020predicting}.}\label{healthind}
\end{table}

To satisfy Sarro et al.,  we 
explored
{\em hyperparameter optimization}.
A common result in software analytics~\cite{agrawal2018better,agrawal2019dodge,agrawal2020better, Fu2016Tuning, fu2016differential,xia2020sequential,tu2021frugal,debtfree,9226105} (and other domains outside of SE~\cite{bergstra2011algorithms,storn1997differential,nair2018finding,chen2018sampling,akiba2019OPTUNA,9463120}) is that automatic hyperparameter optimization  algorithms can find
good  configuration settings.
But for small data sets, 
these methods are highly error prone.
For example, 
Xia et al.~\cite{xia2022predicting}'s
hyperparameter studies
  with our small data, 
  found models with
error rates twice as bad  as what Sarro et al. would consider acceptable.
Hence, in this work, we attempy to out-perform Xia et al.


We show here that a new   optimizer called {\sffamily{SNEAK}}, based on 
{\em  landscape analytics},  can dramatically reduce prediction errors.  
{\em Landscape analytics}~\cite{Malan21,bosman2020visualising,mersmann2011exploratory,ochoa2008study,belaidouni1999landscapes} 
maps out the space of data being explored  in order to find the most information-rich part of the data.
Once that is known, then we can leap to the more informative parts of the problem space.  Most landscape analytics methods require a pre-enumeration of most of the   space~\cite{Malan21}. But in the case of predicting for open source project health, this is impractical.
For example,  Table~\ref{tab:paramGrids}
lists   960,000 different ways a learner might be configured for predicting open source project health.   Assuming five seconds per learner (which might be an underestimate), then a study of all 960,000 options,  repeated 20 times (to check for
external validity) would terminate after
 \mbox{$960,000 * 5 /(3600*24)*20 = 1,111$ days}
(and note  that if Table~\ref{tab:paramGrids}  explored more options
or more learners, or slower methods such as deep learning, then those runtimes would get even longer.) 

\begin{wraptable}{r}{3.6in} 
\caption{Hyperparameters explored here (for ensembles of regression trees).}
\footnotesize
\begin{tabular}{r|rrl|r}
\textbf{Parameter}               & \textbf{Min}         & \textbf{Max}         & \textbf{Step}       & \#options \\ \hline
\textbf{n\_estimators}           & 10                   & 200                  & 10                  & 20 \\
\textbf{min\_sample\_leaves}     & 1                    & 20                   & 1                   & 20\\
\textbf{min\_impurity\_decrease} & 0                    & 10                   & 0.25                & 40\\
\textbf{max\_depth}              & 1                    & 20                   & 1                   & 20\\ 
\textbf{criterion}               & \multicolumn{3}{c|} {one of: [squared, absolute, poisson]}         & 3\\\cline{2-5}
\multicolumn{3}{c}{~} & Total: & 960,000
\end{tabular}
\label{tab:paramGrids}
 
\end{wraptable} Here again, by applying landscape
analytics,  
we can reduce the cost of that search For example,
{\sffamily{SNEAK}} can find   configurations that result in low errors after looking at around 100 options, while other tools needed
to explore 3500, and 5000 (for OPTUNA~\cite{akiba2019OPTUNA}, HYPEROPT~\cite{bergstra2011algorithms}). Note only that, 
{\sffamily{SNEAK}}'s
predictions have errors rates within the Sarro et al. limits. In fact, when
predicting for one particular project health indicator (number of closed
issues), we found a median error rate of 0\%\footnote{To be complete, we must
add that 30\% of the time, error rates up to 30\% were observed.}.
We conjecture that those
other optimizers performed worse
since they used a somewhat uninformed
search based on  random mutations.
On the other hand, 
{\sffamily{SNEAK}} works well
 because  it carefully reflects on  the shape
of the data before deciding where to go next.

The above results strongly recommend   landscape analytics
like {\sffamily{SNEAK}} for ``data starved'' SE domains
such as predicting open source project health.
To date, we have   applied these optimizers based on  landscape analytics to only this domain. But we see nothing in principle that prevents the  
application of this technique to a wider range of problems.

The rest of this paper is structured according
to guidelines from Wohlin \& Runeson et al.~\cite{wohlin2012experimentation} on how to structure experiments in software engineering. We present our methods in \S\ref{algo}, define and explain our case study and experiments in \S\ref{methods}, followed by our analysis at \S\ref{results}.
 To assist other researchers in repeating, improving, or even refuting our results, all our scripts
and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak.

 








 
 













Before beginning, we digress to  address the following threat to validity. 
Any study that learns configurations from local data could be guilty of over-fitting, or worse, using the target data as part of the tuning work. To address that concern, we assert here that in all our studies, we (a)~sort the data by time stamps; then (b)~tune using past data to (c)~test on future data. 
That is, {\em our test data is never used as part of the tuning process}.
 



 
\section{Background}\label{background}

\subsection{Open source project adoption}\label{adoption}

 
Hauge et al.~\cite{hauge2010risks} and by Kenett et al.~\cite{kenett2014adoption} argue that open source software (OSS) projects:
\noindent\begin{enumerate}
\item Reduce in overall development costs
\item Increase independent from proprietary software
\item Simplify license management
\item Offer other benefits related to  high availability
\noindent\end{enumerate}
But there are risks associated with achieving these benefits.
 Hauge et al., report that OSS projects often have hidden costs, from the time used to evaluate the project, to needs for user training and configuration. 
Also in some cases many organizations   require premium professional support
(which might be lacking in the community). 
Apart from that, although there are many OSS projects available, there may still exist a lack of projects that have a specific needed functionality, and as such, the necessity for a high level of customization of the OSS project to fit them for the context in which they are expected to be used.
Finally there is always the risk that the pertinent license made available for an OSS project could impact in the ability of a company to integrate it to their proprietary software. 

To say this another way, before an organization
uses opens source software, they need to assess
the health of that project. But how?  Our reading of the   Hauge et al.~\cite{hauge2010risks} report
is that much of their advice was      too general to identify specify healthy projects. 

A similar pattern is seen in other reports.
Kenett et al.~\cite{kenett2014adoption} established one of the earliest models to assess OSS project health. They did so through establishing Key Risk Indicators (KRI) from metrics obtained through mining OSS repositories. These metrics in their system worked as indicators to establish a level of measurable risk in adopting any given OSS project. 
Although fruitful research, 
we comment that the Kenett et al. only comments on the {\em present} state of a project. When deciding
about what OSS to use, a more insightful measure might also include a prediction on the {\em future}
state of a project.

Hence, in this paper, we turn to the problem
of predicting the {\em future} health of an  OSS project.



\subsection{From Effort to Health Estimation}\label{sapm}



The traditional method for assessing software projects
is to predict how much funds are required for completion. In that kind of analysis,
risk factors that predict for project failure manifest as large {\em effort multipliers} and, hence, greater completion costs.
For example, the COCOMO model from Boehm et al.~\cite{chulani1999calibrating}   generated estimates of total project effort (measured in terms of 150-hour work-months) that were within 30\% of actual, 71\% of the time.  Robbles et al.~\cite{robles2022development} report that for a large class of OSS projects,  developers will spend much of their time working full time on a single project. 
Hence, for those projects, it is possible to estimate the total effort that will be spent on a project, early in its development.
In the Robbles et al. approach, project features were extracted from online version control systems (e.g. GitHub).

COCOMO was designed for waterfall-style projects. Those  waterfall assumptions have to be updated in this age of agile. Whereas waterfall project assume some finish date, agile projects continue to grow and evolve based on feedback from users and developers.  

COCOMO's waterfall assumptions may mean it cannot be deployed in current software developments.
In 2020 and 2021, Xia et al. tried applying the Boehm and Robbles methods to OSS teams~\cite{xia2020predicting}; specifically the APACHE and LINUX kernel development teams.
Those teams expressed little interest in estimates of ``time to finish'', since no one in these teams expected to ever ``finish'' their projects (those developers are funded in perpetuity to maintain and extend their tools). 

Sice COCOMO failed,  Xia et al.  then conducted extensive interviews and email surveys to determine what kind of decision support dashboards were useful to those projects. 
They report that, for those teams,   {\em project reputation} is  more important than effort estimations:
\begin{itemize}
\item
Understanding and managing project reputation  is vitally important in the OSS community. 
\item
Many commercial companies use open-source packages  in the products they sell to customers.  
Those companies use projects that they   trust will remain viable for the foreseeable future.
\item
Projects with an unhealthy reputation lose this kind of community support.
\end{itemize}
Not only that, but
OSS projects grow when they attract developers:
\begin{itemize}
\item
Large OSS projects (e.g. the LINUX or APACHE foundation) have board members representing organizations that allocate considerable resources to maintaining and extending   software. 
\item Attracting such investments is far easier when  projects can show potential investors   healthy trends in their project development.
\item
For example, in the Apache ecosystem, there are 14 real-time time operating systems. 
Measured in terms of code commits per month, Figure~\ref{zep} shows that most of these are stagnant (e.g. LiteOS and myNewt),
while the remaining two (Zephyr and mbed~OS) show a strong increase in the number of commits per month. 
\end{itemize}
\begin{figure}[!t]
\begin{center}
\includegraphics[width=.7\textwidth]{Diagrams/zep.png}
\end{center}
\caption{Trends in number of contributors across 14 real-time OS projects since
2018. Zephyr (purple) has out-paced other similar software projects.}\label{zep}
\end{figure}
  \begin{wraptable}{r}{3in}
\centering
\caption{Importance of of indicators to core OSS developers.
From Xia et al.~\cite{xia2020predicting}. }
\label{tbl:survey}
\begin{adjustbox}{max width=0.98\textwidth}
\footnotesize
\begin{tabular}{lc|c|c}\cline{2-4}
& \multicolumn{3}{|c|}{Important?}\\
\rowcolor{gray}
 &\textcolor{white}{ Very}   & \textcolor{white}{ Somewhat }  & \textcolor{white}{ Not }  \\ \hline
 
  closed pull-requests   & 46\% & 34\% & 20\% \\

   closed issues         & 40\% & 53\% & 7\% \\

   contributors          & 33\% & 46\% & 21\% \\

  opened issues         & 33\% & 59\% & 8\% \\

   opened pull-requests & 31\% & 56\% & 13\% \\
  stars.                & 12\% & 36\% & 52\%
\end{tabular}
\end{adjustbox}
\end{wraptable}
Table~\ref{tbl:survey} shows results of Xia et al.'s survey of dozens of OSS projects. 
That survey sought the opinion of ``core developers''; i.e. those who make pivotal decisions about software.  
Xia et al. found 112 core contributors from 68 projects and asked them ``what product health indicators would prompt them to take action within their project?''.  
Some indicators were not so important (e.g. number of stars assigned to a project in GitHub) while others were seen to be far more valuable (e.g. closed issues). 
 
The rest of this paper explores methods for predicting future trends for the  top-ranked items in Table~\ref{tbl:survey}; i.e.
(a)~number of commits; (b)~number of closed issues, and (c)~number of closed pull requests.  

Before doing that, we note that many other researchers
have explored   project health indicators~\cite{bao2019large,jarczyk2018surgical,kikas2016using,han2019characterization,qi2017software,chen2014predicting,wahyudin2007monitoring,jansen2014measuring,manikas2013reviewing,link2018assessing,wynn2007assessing,crowston2006assessing}.  For example, Bao et al.~\cite{bao2019large} used a set of methods (Naive Bayes, SVR, Decision Tree, KNN and Random Forest) on 917 projects from GHTorrent to predict long term contributors (which they determine as the time interval between their first and last commit in the project is larger than a threshold.). They benchmarked their results and found that random forest achieves the best performance.
Kikas et al.~\cite{kikas2016using} built random forest models to predict the issue close time on GitHub projects, with multiple static, dynamic and contextual features. They report that the dynamic and contextual features are critical in such prediction tasks.
Jarczyk et al.~\cite{jarczyk2018surgical} used generalized linear models for predictions of issue closure rate. Based on multiple features (stars, commits, issues closed by the team, etc.), they found that larger teams with more project members have lower issue closure rates than smaller teams, while increased work centralization improved issue closure rates. 
Other developing related feature predictions also include the information of commits, which is used by Qi et al.~\cite{qi2017software} in their software effort estimation research of OSS projects, where they treat the number of commits as an indicator of human effort.
Chen et al.~\cite{chen2014predicting} used linear regression models on 1,000 GitHub projects to predict the number of forks, they concluded that this prediction could help GitHub recommend popular projects, and guide developers to find projects which are likely to succeed and worthy of their contribution.

There are several important differences between our work and the above.
For the most part, the above   explored a single measure of project health (and not the many measures explored here). Also, their data collection methods were somewhat opportunistic in that they seem to have found a data source, then taken steps to exploit that data. We say this since when
we went looking for their kinds of   data, we often found that the available information
was somewhat different to that used in the above. Like Xia et al., we found that, for many projects,  we could reliable collect 
information about the Table~\ref{healthind} items. Hence, we base our predictors on those attributes. 

Another difference of our work to the above is that,
with the exception of Bao et al.~\cite{bao2019large},
most of the prior work used data miners    {\em  without} hyperparameter optimization. The importance of
hyperparameter optimization is discussed in the next section.

\subsection{Hyperparameter Optimization}\label{hpola}



When a learner executes, it uses many decisions about what things to explore and what things to ignore. 
Those decisions are controlled by the {\em hyperparameters} of the learner.
Example hyperparameters include (a)~how maeny neighbors to use in nearest neighbor classification or (b)~how many trees to include in a random forest or (c)~how to configure the architecture of a deep learner.




\newcommand{\cellcolor{gray!15}}{\cellcolor{gray!15}}



Hyperparameter optimization is a very active field of research. 
We list below some of the algorithms commonly seen in the literature.  
Our reading of the literature is that, of the following, HYPEROPT and OPTUNA represent the prior state-of-the-art. 
One complaint we have against the Xia et al. analysis is that they made scant use of those state-of-the-art tools (their paper makes only one small study with HYPEROPT, on limited data).




\subsubsection{Grid Search (GS)~\cite{bergstra2011algorithms}}
As described by Bergstra et al.~\cite{bergstra2011algorithms}, GS defines all hyperparameter options in a set of vectors. 
GS then executes over the cross product of those options. Note that GS will execute all combinations, and as such for limiting CPU usage one must define a smaller search space.  

\subsubsection{Random Search (RS)~\cite{bergstra2011algorithms}}

Random search starts with the same option vectors as grid search.  
But instead of evaluating all of the models in the grid, RS evaluates a random subset of these models selecting the best configuration from this subset. 
Both GS and RS have been previously used towards the optimization of models for defect prediction~\cite{nevendra2022empirical}


\subsubsection{Differential Evolution (DE)~\cite{storn1997differential}}
Differential Evolution is an evolutionary algorithm that executes in several {\em generations}
of {\em mutate-crossover-select}. 
DE starts like random search and creates a small initial population of randomly selected configuration options (say, 10 vectors of options per feature being configured). 
For each generation, each option is  compared to a newly created option that is a mixture of three other options in the population. 
The new option replaces the old, if it is better.
After generation\#1, the invariant of DE is that each option in  a  population in generation $g$ is superior to  at least $g-1$ other option. 
Hence, as the generations mature, DE mixes together new items  built from  items of increasing  superior options.
DE was previously used, for defect prediction, by (e.g.) Fu et al.~\cite{Fu2016Tuning} on data from open-source JAVA systems.



\subsubsection{FLASH~\cite{nair2018finding}}
FLASH is a sequential model-based optimization (SMBO) method, introduced by Nair et al.~\cite{nair2018finding}, that outperformed the then-known state-of-the-art.  SMBO methods use a surrogate model built from options-evaluated-to-date to make guesses about all the as-yet unevaluated options.
Then (a)~the ``most interesting'' option is then evaluated; and (b)~the  surrogate is then updated so that we can make better guesses in the future.  
When used in practice, FLASH's surrogate model might make 100,000s of very fast guesses in order the select the best $10^1$ to $10^2$ configurations that actually deserve evaluation.
Nair et al. used FLASH to find the best possible configurations for 7 different software systems.

\subsubsection{{\sffamily{SWAY}}~\cite{chen2018sampling}}\label{sway0}
The {\sffamily{SWAY}} oversampling approach  recursively
clusters 10,000 randomly generated configurations (perhaps generated via theorem prover from domain constraints).
{\sffamily{SWAY}} is a greedy-search  algorithm that always commits to the top-most partition  (without checking if any other partition is better).
Later in this paper, we will discuss (\S\ref{zitzler}) the Zitzler domination predicate
and other details present in {\sffamily{SWAY}}.
Previously, in software engineering,
{\sffamily{SWAY}}~\cite{chen2018sampling} was applied towards the optimization of software process models and software product lines.



\subsubsection{HYPEROPT~\cite{bergstra2015hyperopt}}
HYPEROPT is a hyperparameter optimization framework introduced in 2015 as a python package.
At the time of this writing, the two papers that lead to HYPEROPT~\cite{bergstra2012random,bergstra2011algorithms} have been cited 7668 and 3543 times, making this the most mentioned algorithm in this survey. 
At its core, it uses either RS or the Tree-Parzen Estimator (TPE) as the optimization algorithm. TPE is an SMBO algorithm introduced by Bergstra et al.~\cite{bergstra2011algorithms}, that will model the sequential model optimization via the use of non-parametric statistical densities. 
Yedida et al.~\cite{agrawal2021simpler, yedida2021value} have used HYPEROPT for software defect prediction.   
Classic HYPEROPT does not allow for multi-objective optimization and so, when we use it, we will just tune for minimizing MRE values.

\subsubsection{OPTUNA~\cite{akiba2019OPTUNA}}
OPTUNA is a more recent hyperparameter optimization framework than HYPEROPT that uses
MOTPE~\cite{ozaki2020multiobjective}, a more advanced version of TPE. 
MOTPE allows for the optimization of multiple performance metrics at once by analyzing the results of the current models against the currently observed frontier of solutions via using a function that calculates the expected hypervolume improvement at each step of its optimization.
To the best of our knowledge, OPTUNA has not been used extensively for
software engineering problems (but it has been used in domains such as disease diagnosis~\cite{zhou2022predicting}, equipment performance forecast~\cite{mashlakov2019hyper} and others). 
We assert that it is important to baseline our new methods against OPTUNA (and MOTPE) since this is a clear extension and improvement of HYPEROPT.



\subsection{Landscape Analytics}



The experiments of this paper show that, at least for predicting future trends in OSS projects, we can out-perform {\em all} of the above hyperparameter optimization
methods through {\em landscape analytics}.  
This section describes that kind of analytics.

To understand landscape analytics, and its application to hyperparameter optimization, consider a function:
\begin{equation}\label{one}Y=f(X)\end{equation}
where $X$ contains one of more {\em independent features} and $Y$ contains one of more {\em dependent features}. 
If we cluster together with the $X$ points or the $Y$ points, we can draw out the shape of the data. 
We call this shape the ``landscape''.  

The process of data mining can be characterized as a search across landscapes.
Given many examples of  $(X_1,Y_1),(X_2,Y_2),..\mathit{etc}$ then a  learner $L$ seeks some model $f$ that knows where parts of the $X$ landscape connect to particular parts of the $Y$ landscape.
Also, a learner may contain hyperparameters $W$ about (e.g.) how many neighbors to use in the nearest neighbor classifier or (e.g.) how many times to divide numerics. 
Different hyperparameters will result in different models, so we must rewrite Equation~\ref{one} as:
\begin{equation}
Y=L(W)(X)
\end{equation}
To further guide that search of the hyperparameters, we note that in most domains, there is some preference function $P$ that tells us which parts of the dependent landscape are better than others. 
E.g. if $Y$ contains some measure of false alarms, then {\em lower} false alarms might be scored {\em higher}.  
Within this framework, the goal of hyperparameter optimization is to find the hyperparameters $W$ that lead to dependent variable settings that maximize the score:
\newcommand{\argmax}{\mathop{\mathrm{argmax}}}  
\begin{equation}
\argmax_{w_i  \subseteq W} \left(P(L(w_i)(X))\right)
\end{equation}
The list of all $W$ configurations also forms a landscape (separate to the $X,Y$ landscapes).
{\em Landscape Analytics} is the process of understanding, then exploiting the shape of these landscapes. Not all landscapes are ``cliffs'' where
the data changes suddenly and steeply. 
When changing many options has similar effects,
the landscape can be quite smooth and easily
mapped out with just a few samples. 
Hence, many methods~\cite{liu2016comparative,van2020survey,ros2017machine,mit06,h2022}  reason by cluster the data,  then explore just a few representatives from
each cluster. 
To say that another way, when the landscape is smooth,   it is not necessary to search everywhere. 
Rather, it can be (much) faster to:
\noindent\begin{enumerate}
\item  Look  around that space with just a few samples;
\item Then    leap   over to regions of denser information.
\noindent\end{enumerate}
While it is not widely acknowledged, in our opinion, landscape analytics generalizes a range of algorithms from different fields:

\begin{table}[t!]
\caption{Assumptions made in landscape analytics methods (those surveyed by Malan~\cite{Malan21}).}
\footnotesize
\begin{tabular}{l|l|c|l} 
\rowcolor{black!50}  \textcolor{white}{\textbf{Assumption}} &\textcolor{white}{Notes} & \textcolor{white}{\%papers} & \textcolor{white}{Example} \\ \hline
    \textcolor{ao}{\bf Binary space  }
    & All features have arity=2 and are often used in a theorem prover.                          & 33\%  & HDBL~\cite{belaidouni1999landscapes}\\
\rowcolor{gray!15}
    \textcolor{ao}{\bf Fully enumerable space }  & The entire search space can be pre-enumerated and cached.                          & 35\%  & LON~\cite{ochoa2008study}\\
 \textcolor{ao}{\bf Continuous space  }                        &  Useful  for (e.g.) gradient descent. Discrete features not allowed.              & 15\%  & ELA~\cite{mersmann2011exploratory}\\
\rowcolor{gray!15}
 \textcolor{ao}{\bf Knowledge of optima  }   
    & The value of the achievable best solution already known.            & 30\%  & Static-$\phi$~\cite{whitley1995hyperplane} \\
      \textcolor{ao}{\bf Evalaution is fast/cheap  }              & Optimizers can call the evaluation function 1,000,000s of times.  & 33\%  & LGC~\cite{bosman2020visualising} 
\end{tabular}
\label{tab:landscapesummary}
\end{table}



\noindent\begin{enumerate}
    \item When used on the $X$ shape, landscape analytics  might be called ``clustering''.
    \item When used on the  $X,Y$ shape,     limited sampling might be called ``semi-supervised learning'';
    \item Similarly, in a joint analysis of the $X,Y$ shape, if we bias our ``leaps'' towards regions that (in the past) had good $Y$ scores, landscape analytics  might be called ``reinforcement learning''.
    \item And if we use landscape analytics to jointly explore the $W,Y$ shapes, then this could be called ``hyperparameter optimization''.
\noindent\end{enumerate}
In a recent survey of landscape analytics methods, Malan~\cite{Malan21} notes
that the term ``landscape''
is a generalization of Sewell's 1932
concept of ``fitness landscape''~\cite{wr32}  which was first defined for
evolutionary biology.
Malan reports no less than 33 ``families'' of landscape analytics methods. Most of these methods
tend to enumerate the  whole landscape, before exploring it.
This approach is seen in many parts of the Table~\ref{tab:landscapesummary} summary of
the Malan survey; e.g.:
\noindent\begin{enumerate}
    \item Methods that assume that    
    \textcolor{ao}{\bf evaluation is fast/cheap} usually exploit that property to make many samples across the landscape.
    \item Methods that assume
     \textcolor{ao}{\bf knowledge of optima}
      also assume that the search space has been pre-explored (to find that best point).
    \item Methods that require a
       \textcolor{ao}{\bf  fully enumerable space}, by definition, must generate the whole space before reasoning can begin.
\noindent\end{enumerate}
For hyperparameter optimization, it is problematic to explore and cache a large part of the search space. 
This is especially true when conducting large experiments and/or exploring slower data mining algorithms (e.g. deep learners).
Quite apart from CPU issues, there are other reasons we need to look beyond standard landscape analytics tools. 
As shown in Table~\ref{tab:landscapesummary}:
\begin{itemize}
    \item Many current landscape analytics methods require
    \textcolor{ao}{\bf   knowledge of optima}.
    For many data mining applications, the achievable maximum is not recall=1, false alarm=0 but some point struggling to achieve (but not reaching) that zenith.
    \item Many current landscape analytics methods cannot handle features of mixed type since they rely on (e.g.) gradient descent over a 
       \textcolor{ao}{\bf  continuous space}.
       or (e.g.) theorem proving executed on a {\em binary  space}.   
    This is problematic for hyperparameter optimization since the policies that control data miners are often a combination of numeric and discrete choices.
\end{itemize}
For these reasons, we built our own landscape analytics tool, differing from existing ones   with the following properties:
\begin{itemize}
    \item It can explore heterogeneous data (discrete and symbolic data) through the use of simple discretization algorithms.
    \item It only enumerates a subset of the $W,X$ hyperparameters and independent features.
    \item It only enumerates a small subset of the $Y$ independent features.
\end{itemize}
That tool is called {\sffamily{SNEAK}} and is described in the next section.



 

\section{About {\sffamily{SNEAK}} }\label{algo}

{\sffamily{SNEAK}} is based on Chen et al.'s {\sffamily{SWAY}} algorithm~\cite{chen2018sampling}. It takes a binary dataset comprising of multiple configurations as an input and outputs (a) a binary tree of clustered configurations, from which it reports (b) the best configuration.
While an interesting prototype, {\sffamily{SWAY}} suffers from some serious drawbacks.
Both SWAY and {\sffamily{SNEAK}} recursively
partition the data, but {\sffamily{SWAY}}'s greedy search always commits
to the top-most partition  (without checking if any other partition is better) whereas
{\sffamily{SNEAK}}  takes a global approach.
Specifically, after all the data is clustered, {\sffamily{SNEAK}} reflects over all the nodes to find which sub-clusters ``best'' split the data
(and ``best'' means ``splits the data such that some attribute range most separates the two splits'').

As such we include {\sffamily{SWAY}} as a baseline algorithm when performing this hyperparameter optimization task.

\subsection{Preliminaries}
To understand our algorithms, we must first
know how to 
``evaluate'' hyperparameters to find ``better'' options. 

 


\subsubsection{``Evaluation''}
\begin{table}[!t]
\begin{tabular}{|p{.95\linewidth}|}\hline
We assess our project health predictors using   widely-used criteria from the literature~\cite{reddy2010software, 40sarro2016, shepperd2012evaluating}.

\begin{itemize}
\item
The calculation of ``magnitude of relative error'' (MRE) was shown in Equation~\ref{mre}.
\item
Pred40 assesses the Sarro limit    and is defined as 
   $ \mathit{Pred40} =  \mathit{Count(MREs,} \leq 40)/||\mathit{MREs}||$
 \item
Standardized accuracy (SA) baselines the  observed error against some reasonable, fast, but unsophisticated measurement. 
    SA is based on the Mean Absolute Error (MAE)
    i.e. $\mathit{MAE} = \frac{1}{N}\sum_{i=1}^{n} |\mathit{PREDICT}_i - \mathit{ACTUAL}_i|$ where $N$ is the size of the test set used for evaluating performance. 
    SA then is defined as the   ratio 
 $\mathit{SA} = (1 - (\mathit{MAE}/\mathit{MAE}_\mathit{guess})) \times 100$
    
    \item
    $\mathit{MAE_{guess}}$  (used in  {\em SA}) is the $\mathit{MAE}$ of a set of guesses. 
    For this paper, we follow the standards defined by Xia et al.~\cite{xia2018hyperparameter} so  $\mathit{MAE_{guess}}$  is the median of previous months values as the guessing values.
\end{itemize}
With the above, we seek to
  {\em maximize} Pred40 and SA and 
  {\em minimize} MRE and D2H.
  
  D2H 
(distance to heaven)  
  assumes some  omniscient oracle has can rank all samples.  Given a vector $Z$ containing all   options ranked from best to worst, the D2H of a   configuration $s$ seen at   index $i$ is
$ \mathit{D2H}_{s} = i / |Z|$
 Note that, the closer we get to ``heaven'', the smaller D2H and   more we are succeeding on all       goals. To rank the vectors,  we use the Zitzler indicator discussed elsewhere in this paper.\\\hline
 \end{tabular}
 \caption{``Better'' = \{MRE, Pred40, SA, D2H\}.}\label{evals}
\end{table}
~{\em Evaluation}
means configuring a learner with a one set of hyperparamters, running it on some training data,
then applying the resulting model to   test data.




\subsubsection{``Better''}
The result of such an {\em evaluation} is a set of  performance measures associated with a working learner. As shown in   Table~\ref{evals}, we have four such measures (Pred40, SA, MRE, D2H). When dealing with multiple goals,
a 
{\em domination} predicate is required
        to decide if one set  of performance measures
        $A=\{\mathit{Pred40}_i, \mathit{SA}_i, \mathit{MRE}_i, \mathit{D2H}_i\}$
        is ``better''   than $B=\{\mathit{Pred40}_j, \mathit{SA}_j, \mathit{MRE}_j, \mathit{D2H}_j\}$. 
Two such predicates are {\em boolean domination} and the 
the {\em Zitzler multi-objective indicator}~\cite{zit02}.   Boolean domination   says one thing is better than another if it is no worse on any single goal and better on at least one goal. 
We prefer Zitzler to boolean domination since we have a  four-goal optimization problem, and it is known that boolean domination often  fails for two or more goals~\cite{Wagner:2007, Sayyad:2013}. 
Zitzler favors
$B$ over $A$  model if jumping to $A$ ``loses'' most:
\begin{itemize}
\item Let$ \textit{worse}(A,B)  = \textit{loss}(A,B) > \textit{loss}(B,A)$
\item Let
$ \textit{loss}(A,B) = \sum_{j=1}^n -e^{\Delta(j,A,B,n)}/n$
\item Let 
       $ \Delta(j,A,B,n)  =  w_j(o_{j,A}  - o_{j,B})/n$
       \end{itemize}
where ``$n$'' is the number of objectives (for us, $n=4$) and $w_j\in \{-1,1\}$ depending on whether we seek to maximize goal $x_j$ and $o_{j,A}, o_{j,B}$ are the scores seen for objective $o_j$ for $A,B$, respectively.


 One nuanced point to be made here is that some of the performance measures of Table~\ref{evals} require extra evaluations. 
 For example, as discussed below, {\sffamily{SWAY}} only needs to evaluate $E=2\sqrt{N}$ options to identify some  good options. But  to check if {\sffamily{SWAY}} finds options better than (e.g.) {\sffamily{SNEAK}},
we may   have to look for any other good options missed by {\sffamily{SWAY}}. This means that for certification purposes, we will also have to evaluate   other 
options, looking for anything overlooked by {\sffamily{SWAY}}

 Hence we distinguish two evaluation counters:  $E$ and $E^+$. Here,  $E$ is the number of evaluations made by the inference procedure and $E^+$ are all the extra evaluations used to test if one algorithm is better than another (e.g. when applying the D2H measure of  Table~\ref{evals}). 
The important point about $E,E^+$ is that  suppose researchers (like us) use $E^+=N$  of some data   $D_i$ to certify that {\sffamily{SNEAK}} is a useful approach. After that,
when practitioner use this algorithm on their own data  $D_{j\not=i}$, they will only need to make $E \ll E+$ evaluations.

Having taken care to  distinguished
these two evaluation counters
 for the rest of this paper, when we say ``number of evaluations'', we are  reporting $E$  (and not $E+$).




\subsection{SWAY}
 

Chen et al.'s {\sffamily{SWAY}}~\cite{chen2018sampling} algorithm was discovered by accident
while  porting a JAVA version of a genetic algorithm  optimizer to PYTHON. Genetic algorithms use a mutator to explore different options.  Chen accidentally broke that mutator, then fixed it. Puzzlingly, after the fix,  the    resulting optimizations did not improve    (where ``improvement'' was measured by
the  Zitzler indicator).

On investigation Chen found that, at least for the 
SE data sets he was exploring,   what  seemed to matter more than the mutator was the size of the initial set of randomly-generated candidates. Standard genetic algorithms mutate
a population of 100 individuals for 100 generations~\cite{holland92}. Chen found he could optimize just as well using one generation of 10,000 randomly generated individuals (with no subsequent mutation or genetic cross-over).

The key to this approach was a simplistic greedy landscape analysis that   recursively clustered the candidates, using extreme examples. Informally, we   say that {\sffamily{SWAY}}
  draws a ``rope'' between two distant ``peaks'',  measures the height of the peaks,
then focuses on all the land nearer the higher ``peak''. More precisely, 
in  an approach inspired by 
Faloutsos et al.~\cite{faloutsos1995fastmap}, {\sffamily{SWAY}} evaluated two distant examples, then pruned   50\% of the data closest to the worst point 
\footnote{This can be done in linear time, as follows.
Pick any option $Z$ at random
then find the options $A,B$ most different to $Z,A$
(respectively).  Let $c={\mathit dist}(A,B)$ and let every   point have distances $a_i,b_i$ to $A,B$ respectively. Project
every point   onto a line between $A,B$ at a distance $x=(a_i^2 +c^2-b_i^2)/(2c)$ from $A$. Divided the data in two using the   median $x$ value. } (where ``worst'' was defined by the Zitzler indicator described above).
When repeated recursively,
{\sffamily{SWAY}} finds   {\em best}=$\sqrt{N}$ best  options within $N$ examples after just  $2\log_2({N})$ evaluations
(i.e. one evaluation for each of the two distance examples found at each level of the recursion).


For more details on {\sffamily{SWAY}}, see
the pseudo code of Figure~\ref{sway}.

\subsection{{\sffamily{SNEAK}}}

Conceptually, 
{\sffamily{SWAY}} builds a cluster tree of options but is hardwired to always select the split at the top of the current tree.
The idea behind {\sffamily{SNEAK}} is that, as shown by our results section below, better results can be  generated  by  looking around a little before committing to a particular jump.

    
\newcommand{\color{blue}{FUNCTION}}{\color{blue}{FUNCTION}}
\newcommand{\color{blue}{INPUT}}{\color{blue}{INPUT}}
\newcommand{\color{blue}{OUTPUT}}{\color{blue}{OUTPUT}}
\newcommand{\CODE}[1]{\bfseries{\sffamily{#1}}}
\newcommand{{\CODE IF}}{{\CODE if}}
\newcommand{{\CODE THEN}}{{\CODE then}}
\newcommand{{\CODE ELSE}}{{\CODE else}}
\newcommand{{\CODE AND}}{{\CODE and}}
\newcommand{{\CODE FOR}}{{\CODE for}}
\newcommand{{\CODE IN}}{{\CODE in}}
\newcommand{{\CODE DO}}{{\CODE do}}
\newcommand{{\CODE END}}{{\CODE end}}
\newcommand{{\CODE RETURN}}{{\CODE return}}
\newcommand{{\CODE WHILE}}{{\CODE while}}


\begin{figure}
\begin{minipage}{2.2in}
{\fontsize{6}{7}
\begin{alltt}
{\color{blue}{FUNCTION}} {\color{red}{SWAY}}(optionSpace, evaluationFun,
              stop = sqrt(length(rows)))
              
{\color{blue}{INPUT}}:  stop         : when to stop searching
        optionSpace  : dataset of possible options
        evaluationFun: run  model, get Table~\ref{evals} scores      
{\color{blue}{OUTPUT}}: selected node

selectionFun = all {\color{aoc} // See Table \ref{select}.}
half = {\color{red}HALF}(rows)
{{\CODE IF}} count(half.lefts, half.rights) > stop
   {{\CODE IF}} {\color{red}BEST\_LEFT}(half, evaluationFun) {{\CODE THEN}}
      return {\color{red}SWAY}(half.left, evaluationFun, stop)
   {{\CODE ELSE}} 
      return {\color{red}SWAY}(half.left, evaluationFun, stop)
{{\CODE ELSE}}
    survivors = join(half.lefts, half.rights)
    {{\CODE RETURN}} selectionFun(survivors)

\hrule 
{\color{blue}{FUNCTION}} {\color{red}{BEST\_LEFT}}(node, evaluationFun)

{\color{blue}{INPUT}}:  node          :  node being evaluated
         evaluationFun :  method to evaluate an option                    
{\color{blue}{OUTPUT}}:  true if the left subtree is better than the right 

leftScore = evaluationFun(node.left)
rightScore = evaluationFun(node.right)
{{\CODE RETURN}} dominates(leftScore, rightScore)  
\end{alltt}}
\end{minipage}\begin{minipage}{.7in}~\end{minipage}~\begin{minipage}{3in}
{\fontsize{6}{7}
\begin{alltt} \texttt
{\color{blue}{FUNCTION}} {\color{red}{HALF}}(rows,
              stop=sqrt(length(rows)),
              furthest=.95)

{\color{blue}{INPUT}}: rows     : N rows, each with x independent 
                  and y dependent columns
        stop    : stopping criteria
        furthest: when looking for distant points, 
                  avoid outliers, don’t go
                  all the way
{\color{blue}{OUTPUT}}:  the rows, divided into two

tmp = any(rows)   {\color{aoc}// picked at random}
left = furthest row from tmp 
right = furthest row from left
{{\CODE FOR}} row {{\CODE IN}} rows {{\CODE DO}} 
   a = dist(row, left)
   b = dist(row, right) 
   c = dist(left, right)
   row.x = (a^2 + c^2 - b^2) / (2c)  {{\CODE END}}
lefts,rights = [], []
{{\CODE FOR}} i,row {{\CODE IN}} enumerate(sort(rows, key=x)) {{\CODE DO}}
   {{\CODE IF}}    i< length(rows)/2 
   {{\CODE THEN}} push(lefts, row) 
   {{\CODE ELSE}}  push(rights, row) 
   {{\CODE END}} 
{{\CODE END}}
{{\CODE RETURN}} \{left=left, right=right, lefts=lefts, rights=rights\}

\end{alltt}}
\end{minipage} 
\caption{{\sffamily{SWAY}}, pseudo-code.}\label{sway}
\end{figure}









{\sffamily{SNEAK}}'s tree construction algorithm
is the $\mathit{TREE}$ algorithm of 
Figure~\ref{tree}. Unlike the {\sffamily{SWAY}} code of Figure~\ref{sway}, we see that $\mathit{TREE}$     returns the entire tree of splits.


{\sffamily{SNEAK}}  reflects on every sub-tree that divides $n_i$ examples
into two sub-trees of size $n_j,n_k$ (where $n_i=n_j+n_k$),
After all   numerics are discretized into  equal sized bins,   the entropy of a set of rows   is the the sum of the entropy of the columns found in those rows\footnote{The entropy of a set of symbols is analogous to the standard deviation of  set of numbers 
(in that both are  a measure of how much we expect that distribution lies {\em away}
from the central tendency). Entropy can be thought of as the effort require to ``round up'' $1..i..n$ signals of probability $p_i$.
A binary chop can isolate a signal with effort $\log_2{p_i}$. For distribution $i$, the odds that we will need to expend that effort
is $p_i$. Summing over the all the distributions, we say that entropy is     $-\sum_i^np_i\log_2{p_i}$.}.
 The  `best sub-tree'' with 
  the ``best seperation'' is the one with greatest difference in the entropy between the root of the sub-tree and its two children. To avoid trivial cases (near the leaves), this entropy value is weighted with the number of rows $n_i$ found in the root $i$ and   two sub-trees $j,k$:
  \[ \begin{array}{rcl}
  w_1&=& n_i\times \mathit{entropy}_i \\
    w_2&=& n_j\times \mathit{entropy}_j + 
               n_k\times \mathit{entropy}_k \\
   \mathit{best}&=& w_1-w_2
    \end{array} \]


\begin{wrapfigure}{r}{2.5in}
\fontsize{6}{7}
\begin{alltt} 
 
{\color{blue}{FUNCTION}} {\color{red}{TREE}}(rows, stop=sqrt(N))

{\color{blue}{INPUT}}:  rows : N rows, each with x independent 
               and y dependent columns    
        stop : stopping criteria
    
{\color{blue}{OUTPUT}}: balanced binary tree 

here= \{ rows=rows \}
{{\CODE IF}} |rows| < stop {{\CODE THEN}} {{\CODE RETURN}} here {{\CODE END}}
here.left, here.right, lefts, rights = {\color{red}HALF}(rows, top)
here.lefts = {\color{red}TREE}(lefts, stop)
here.right = {\color{red}TREE}(rights, stop)
{{\CODE RETURN}} here
\end{alltt}
\caption{{\sffamily{SNEAK}} builds a tree of all options (using {\color{red}{HALF}}, from Figure~\ref{sway}).}\label{tree}
\end{wrapfigure}

Once a    `best sub-tree'' with 
  the ``best seperation''  is found, 
  our sub-routine   
  ``xPASS''  evaluates two extreme points in that split and prunes the sub-tree containing
the worst options. To avoid asking too many questions, xPASS
  values querying sub-trees that
  \begin{itemize}
  \item Have not been queried before;
  \item Have different values in the 
$n_j$ and $n_k$ rows
\item Have high variability; i.e. has the largest entropies.
\end{itemize}
Once the best sub-tree is decided, xPASS finds and deletes the rows of the "worst" sub-tree. 
After that, xPASS loops again looking for other sub-trees to probe
(stopping when the tree is smaller than $\sqrt{N}$). 

In summary, {\sffamily{SNEAK}} uses the nested functions of  Equation~\ref{sneak}:


 
 \begin{equation}\label{sneak}
\mathit{SNEAK}(\mathit{rows}))=
 \underbrace{\mathit{SELECT}}_{\mathit{Table~\ref{select}}}(\;
  \mathit{yPASS}(
    \underbrace{\mathit{xPASS}}_{\mathit{see \;this section}}(
        \mathit{TREE}(
           \mathit{rows})))
       \end{equation}

The ``yPASS'' sub-routine is where we configure, then execute a pair  of  machine 
learning models (in this case here, random forests)
and only the best half of each pair is collected for SELECT.
Given all the work done on the right-hand-side of yPASS,  we need to run far fewer evaluations than,
other approaches  (for exact statistics on that, see Table~\ref{tab:globmedian}).  

Given how, similarly to {\sffamily{SWAY}}, {\sffamily{SNEAK}} outputs around $\sqrt{N}$ possible configurations, a {\em SELECT} function must be applied in order to select the best among these final configurations. This paper experiments with four such final selection methods (see Table~\ref{select})
As far as we know, {\sffamily{SNEAK}} represents the first time that this style of reasoning has been applied to SE hyperparameter optimization.



\begin{table}[!t]
\caption{Four different  {\bf selection} policies for
reporting final items to report to user.}\label{select}
\footnotesize
\begin{tabular}{cp{3.3in}l}
\rowcolor{black!50}& & \color{white}{\#evaluations ($E$)}\\\hline
{\bf ANY} & Evaluate and return any  one  of the {\em best} items.  & $2\log_2({N})+1$\\ 
 
\rowcolor{gray!15}{\bf SANY} & Pass {\em best} to a second round of    {\sffamily{SWAY}} to return $\sqrt{\sqrt{N}}$ items.
Return any item from this set.
&
  $2\log_2({N})+2\log_2({\sqrt{N}})$ \\
  
{\bf ALL} & Evaluate  {\em best};  return the one 
with smallest  D2H (see Table \ref{evals}).
&   $2\log_2({N}) + \sqrt{N}$.\\
 
\rowcolor{gray!15}{\bf SALL}&   Pass {\em best} to a second round of {\sffamily{SWAY}};
apply {\bf ALL} to the   $\sqrt{\sqrt{N}}$ items found in this way. & 
$2\log_2({N})+2\log_2({\sqrt{N}}) +
\sqrt{\sqrt{N}}$  .\\\hline
\end{tabular}

\end{table}



 

\section{Experimental Methods}\label{methods}

 
This paper uses software project health data from Xia et al. (obtained from their GitHub repository) to comparatively assess the hyperparameter optimizers of \S\ref{hpola} with the landscape analytics of {\sffamily{SNEAK}}.
Several of our algorithms have some stochastic components. 
Hence  following the advice of Arcuri and Briand~\cite{arcuri2011practical}, our experimental rig repeats the following procedure 20 times, each time using different random number seeds.

At MSR'22, Majumder et al.~\cite{Majumder22} reported twelve clusters of projects within the projects explored by Xia et al. 
This paper applies our methods to the projects nearest to those twelve clusters.
Table~\ref{tab:features} shows statistics for a dozen features collected from those 12 projects.
Data was originally collected from these 12 projects in one-month increments. These projects exist for different lengths of time, specifically 32 to 63 months.
This meant our 12 projects had between 32 to 63 rows.
We generated  predictions 12 months into the future.
To do so, we used all rows from 12 months prior to predicting for the 12 months.
For example, if a project had  40 months of data, then we
predicted for months 29,30,31...40 using
just the data found in the first 17,18,19,...28 rows
of data (respectively). 
For target   of this learning, we used Table~\ref{tbl:survey} to select for targets   that would most prompt core developers to take action within their projects; i.e.:
\noindent\begin{enumerate}
    \item {\em Indicator1}= number of commits;
    \item {\em Indicator2}= number of closed issues;
    \item {\em Indicator3}= number of closed pull requests.
\noindent\end{enumerate}
Then, for each indicator, we built predictors using  that indicator as to the $Y$ features, and every other feature as independent $X$ features.  
This means that within our 20 repeats, we:
\begin{itemize}
    \item Looped three  times (once for each indicator)
    \item Looped 12 times (once for each data set) 
        \begin{itemize}
            \item The seven hyperparameter optimizers of \S\ref{hpola} 
            \item Plus {\sffamily{SNEAK}} used on all of four configurations (see  {\sffamily{SNEAK}}+all,
            {\sffamily{SNEAK}}+Sall, {\sffamily{SNEAK}}+Sany and {\sffamily{SNEAK}}+any from  policies for {\sffamily{SNEAK}} from Table~\ref{select}).
            \item Plus one   call  with   off-the-shelf {\em default} parameters.
        \end{itemize}
\end{itemize}
For the above $20*3*12*12=8,640$ runs (except for the {\em default} runs), we tuned the hyperparameters of a random forest. 
This learner was selected since a wide range of
software analytics researchers have reported
that random forests are very useful for this kind of estimation~\cite{weber2014makes, kikas2016using, mustapha2019investigating, banimustafa2018predicting, xia2018hyperparameter}. 

Table~\ref{tab:paramGrid} shows the space of hyperparameters we explored. 
Four of these hyperparameters were chosen since they were also used by Xia et al. while the fifth (number of estimators) is important for random forests (since it sets the size of the forest). 

Different optimizers treated the ranges of
Table~\ref{tab:paramGrid} in different ways. For example,
DE extrapolating any value at all between the min and max values. As to {\sffamily{SNEAK}}, that algorithm selected at random from   used 20 steps between min and max.
Also, for continuous parameters (min impurity), 
{\sffamily{SNEAK}} did a more fine-graned approach and used 40 steps. 
The cross product of Table~\ref{tab:paramGrid} returns 984,000 possible model configurations.  
From this space, we randomly selected 10,000 models, ran and cached the results. 
Each tuning algorithm was then scored by how few of those evaluation results they used in their reasoning. Without that caching,
our experiments would take weeks to terminate but
with caching, on a 16-core machine~\footnote{Windows 10 machine with an Intel i7-11700K, 64GB of DDR4 Ram and an Nvidia RTX 3080}, our experiments terminated in under six hours.





\begin{table}[t!]
\caption{Summary of project health datasets used in this study.
Median is the 50th percentile and IQR (short for ``inter-quartile range'') is the (75-25)th percentile.}
\footnotesize
\begin{center}
\begin{tabular}{l|cccc}
                        & \textbf{Min} & \textbf{Max} & \textbf{Median} & \textbf{IQR} \\ \hline
\textbf{Commits}        & 0            & 919          & 22              & 69           \\
\textbf{Contributors}   & 0            & 29           & 2               & 6            \\
\textbf{Open PRs}       & 0            & 82           & 1               & 5            \\
\textbf{Closed PRs}     & 0            & 17           & 0               & 1            \\   
\textbf{Merged PRs}     & 0            & 77           & 0               & 3            \\
\textbf{PR comments}    & 0            & 1098         & 0               & 7            \\
\textbf{Open issues}    & 0            & 137          & 1               & 4            \\
\textbf{Closed issues}  & 0            & 385          & 8               & 31           \\
\textbf{Issue comments} & 0            & 1553         & 17              & 96           \\
\textbf{Stars}          & 0            & 2543         & 25              & 54           \\
\textbf{Forks}          & 0            & 379          & 10              & 12           \\
\textbf{Watchers}       & 0            & 15           & 0               & 1        
\end{tabular}
\end{center}
\label{tab:features}
\vspace{-10pt}
\end{table}



\begin{table}[!t]
\caption{Parameter Grid for generating {\sffamily{SNEAK}}'s option space. (Note that criterion is a discrete parameter)}
\footnotesize
\begin{tabular}{l|lll}
\textbf{Parameter}               & \textbf{Min}         & \textbf{Max}         & \textbf{Step}        \\ \hline
\textbf{n\_estimators}           & 10                   & 200                  & 10                   \\
\textbf{min\_sample\_leaves}     & 1                    & 20                   & 1                    \\
\textbf{min\_impurity\_decrease} & 0                    & 10                   & 0.25                 \\
\textbf{max\_depth}              & 1                    & 20                   & 1                    \\ 
\textbf{criterion}               & \multicolumn{3}{c} {one of: [squared, absolute, poisson]}
\end{tabular}
\label{tab:paramGrid}
\vspace{-10pt}
\end{table}




\subsection{Algorithms}

The {\sffamily{SNEAK}} system is implemented in Python 3.8.
That code is made available available on-line at \url{https://github.com/zxcv123456qwe/niSneak}.
In order to run most of our baseline algorithms, we have used the NUE framework~\footnote{\url{https://github.com/lyonva/Nue}}.
The NUE framework contains the implementation of most of the state-of-the-art hyperparameter optimization algorithms as an extension to scikit-learn~\cite{sklearn_api}. 
Through this framework, we have executed our runs for Random Search, {\sffamily{SWAY}}, Differential Evolution, Grid Search and Flash.

In order to use HYPEROPT and OPTUNA, we have made use of the implementations made publicly available through pip. 
Which can be installed via ``pip3 install hyperopt'' and ``pip3 install optuna''.
To run RF with its default parameters we have simply trained our learner on all target features using standard scikit-learn settings.
 


\subsection{Statistical Methods}\label{stats}

Our results in Table~\ref{tab:medianResults} reports the median (50th percentile) and inter-quartile range (IQR$=75\mathit{th}-25\mathit{th}$ percentile) of our methods' performance across all datasets. 
In order to rank these methods, and decide if any method is better we use a combination of the Friedman test with the Nemenyi Post-Hoc test. 
This methodology follows from suggestions from Dem{\v{s}}ar et al.~\cite{demvsar2006statistical}, Herbold et al~\cite{herbold2017comments, herbold2018correction} and Xia et al.~\cite{xia2018hyperparameter}.

The Friedman test is a non-parametric test that determines whether or not there is a statistically significant difference between populations~\cite{friedman1940comparison}. 
As such when comparing results for our treatments on a particular metric predicting a particular indicator we first check for the presence of a statistical significant difference. 
For our case, we run this test, and if the corresponding p-value is larger than $0.05$ we consider that there is no significant difference between the methods for this indicator and metric.
If the p-value is, however, lower than $0.05$, we then run the Post-Hoc test~\cite{nemenyi1963distribution} to determine the groupings of treatments that are significantly different. 
This test uses a measure of Critical Distances between average ranks to define significantly different populations. 
As a threshold for separating these groups, we also set the p-value to $0.05$.



\section{Results}\label{results}












Our results aim to answer the following research question: 
\begin{RQ}
 How does {\sffamily{SNEAK}} fare against the HPO algorithms previously used in a software engineering context?
\end{RQ}
Table~\ref{tab:medianResults} reports the median (50th percentile) and IQR (75th - 25th percentile) over 20 runs. 
On those populations, we have also run the statistical analysis described in \S\ref{stats}. 

In Table~\ref{tab:medianResults}, darker cells
show the better results (where ``better'' is determined by our statistical analysis). Some of these better results are very good indeed, particularly for predicting the number of closed pull requests:
\noindent\begin{enumerate}
\item Zero median and IQR values ((75-25)th value) for MRE
\item For some of the niSNEAK variants, 93\% of the results have errors inside the Sarro limit (see the Pred40 results). 
\noindent\end{enumerate}
Table~\ref{tab:medianResults} splits the algorithms into two groups, traditional hyperparameter optimizers, and landscape analytics-based optimizers.
In that split, we see that with   rare exceptions:
\noindent\begin{enumerate}
\item Traditional hyperparameter optimizers are seldom in the best group of methods statistically speaking.
\item While using the ``all'' and ``Sall'' approach, {\sffamily{SNEAK}} is consistently at the best group for all features on all metrics.
\noindent\end{enumerate}
Before running, we conjectured,  that the ``Sany'' and ``any'' budgets were possibly too naive to be considered good candidates for HPO.
Nevertheless the low cost of these budgets made them a valid case study. And through it we showcase the need for at least the process described by the ``Sany'' budget.
Interestingly enough {\sffamily{SWAY}} performed unexpectedly well on it's own, with results comparable to other traditional HPO techniques, usually staying in the same statistical ranking as them.

One unexpected result was from OPTUNA, which was unable to reach good optimization results. 
Recall from the above that OPTUNA uses
use MOTPE
(the  optimization of multiple performance metrics at once by analyzing the results of the current models against the currently observed frontier of solutions via using a function that calculates the expected hypervolume improvement at each step of its optimization).
We conjecture   that the hypervolume calculation used in    MOTPE is not suited for such small datasets as the ones we are working with here. (i.e. 20 to 40 data points).

  




\begin{table}[t!]
    \caption{  Median and IRQ results (over 20 runs).    Median = 50th  percentile.     IQR =(75-25)th percentile. {\em Darker} cells, on the topmost tables, show {\em better} results. Here ``better'' is computed relative to other results in the same column, using the statistical methods of \S\ref{stats}.  In this table. {\em lower} values for MRE and D2H are {\em better}
    while {\em higher} values for SA and Pred40 are {\em better}. The negative values
    seen in  median SA are not errors-- that measure can indeed fall below zero. 
    }
      \tiny   \begin{tabular}[t]{lccccccccccccccc}
\multicolumn{1}{l}{\underline{Algorithm}} &  \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  && 
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  &&  
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  &&  
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  \\                                                
\multicolumn{1}{l}{{\sffamily{SNEAK}}+all} &{\DARK}\textcolor{white}{38} & {\DARK}\textcolor{white}{0}    & {\DARK}\textcolor{white}{26}  && 
                             {\DARK}\textcolor{white}{52}    & {\DARK}\textcolor{white}{93}   & {\DARK}\textcolor{white}{66}  &&
                             {\DARK}\textcolor{white}{36}    & {\DARK}\textcolor{white}{46}   & {\DARK}\textcolor{white}{27}  &&
                             {\cellcolor[HTML]{CCCCCC}}54\%                      & {\DARK}\textcolor{white}{18\%} & {\cellcolor[HTML]{CCCCCC}}27\%                    \\
\multicolumn{1}{l}{{\sffamily{SNEAK}}+Sall}   &{\cellcolor[HTML]{CCCCCC}}47                  & {\DARK}\textcolor{white}{0}    & {\cellcolor[HTML]{CCCCCC}}33                      &&
                             {\DARK}\textcolor{white}{43}    & {\DARK}\textcolor{white}{93}   & {\DARK}\textcolor{white}{60}  &&
                             {\DARK}\textcolor{white}{33}    & {\DARK}\textcolor{white}{47}   & {\DARK}\textcolor{white}{19}  &&
                             {\DARK}\textcolor{white}{5\%}   & {\DARK}\textcolor{white}{5\%}  & {\DARK}\textcolor{white}{5\%} \\
\multicolumn{1}{l}{{\sffamily{SWAY}}}    &{\cellcolor[HTML]{CCCCCC}}52                    & {\cellcolor[HTML]{CCCCCC}}52                       & {\cellcolor[HTML]{EFEFEF}}52                    &&
                             {\cellcolor[HTML]{CCCCCC}}48                        & {\cellcolor[HTML]{CCCCCC}}42                       & {\cellcolor[HTML]{CCCCCC}}39                      &&
                             {\cellcolor[HTML]{CCCCCC}}26                        & {\cellcolor[HTML]{CCCCCC}}34                       & {\cellcolor[HTML]{CCCCCC}}23                      &&
                             {\cellcolor[HTML]{CCCCCC}}45\%                      & {\cellcolor[HTML]{CCCCCC}}45\%                     & {\cellcolor[HTML]{CCCCCC}}45\%                    \\    
\multicolumn{1}{l}{{\sffamily{SNEAK}}+Sany} &{\cellcolor[HTML]{EFEFEF}}120                 & {\cellcolor[HTML]{EFEFEF}}117                    & {\cellcolor[HTML]{EFEFEF}}70                    &&
                             {\cellcolor[HTML]{EFEFEF}}16                      & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{CCCCCC}}24                      &&
                             {\cellcolor[HTML]{EFEFEF}}8                       & {\cellcolor[HTML]{CCCCCC}}-5                       & {\cellcolor[HTML]{CCCCCC}}6                       &&
                             {\cellcolor[HTML]{EFEFEF}}63\%                    & {\cellcolor[HTML]{EFEFEF}}77\%                   & {\cellcolor[HTML]{CCCCCC}}41\%                    \\
\multicolumn{1}{l}{{\sffamily{SNEAK}}+any} &{\cellcolor[HTML]{EFEFEF}}120                  & {\cellcolor[HTML]{EFEFEF}} 120                   & {\cellcolor[HTML]{EFEFEF}}72                    &&
                             {\cellcolor[HTML]{EFEFEF}}8                       & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{CCCCCC}}24                      &&
                             {\cellcolor[HTML]{EFEFEF}}2                       & {\cellcolor[HTML]{CCCCCC}}-5                       & {\cellcolor[HTML]{EFEFEF}}6                     &&
                             {\cellcolor[HTML]{EFEFEF}}72\%                    & {\cellcolor[HTML]{EFEFEF}}81\%                   & {\cellcolor[HTML]{CCCCCC}}54\%                    \\
  
                             \hline                
\multicolumn{1}{l}{Default} &{\cellcolor[HTML]{CCCCCC}}70                       & {\cellcolor[HTML]{CCCCCC}}100                      & {\cellcolor[HTML]{EFEFEF}}66                    &&
                             {\cellcolor[HTML]{CCCCCC}}23                        & {\cellcolor[HTML]{CCCCCC}}14                       & {\cellcolor[HTML]{CCCCCC}}32                      &&
                             {\cellcolor[HTML]{EFEFEF}}14                      & {\cellcolor[HTML]{CCCCCC}}-6                       & {\cellcolor[HTML]{EFEFEF}}-5                    &&
                             {\cellcolor[HTML]{CCCCCC}}45\%                      & {\cellcolor[HTML]{CCCCCC}}50\%                     & {\cellcolor[HTML]{CCCCCC}}59\%                    \\\hline
\multicolumn{1}{l}{RS}      &{\cellcolor[HTML]{CCCCCC}}51                       & {\cellcolor[HTML]{CCCCCC}}29                       & {\cellcolor[HTML]{EFEFEF}}43                    &&
                             {\cellcolor[HTML]{CCCCCC}}48                        & {\cellcolor[HTML]{CCCCCC}}60                       & {\cellcolor[HTML]{CCCCCC}}42                      &&
                             {\cellcolor[HTML]{CCCCCC}}26                        & {\cellcolor[HTML]{CCCCCC}}36                       & {\cellcolor[HTML]{CCCCCC}}23                      &&
                             {\cellcolor[HTML]{CCCCCC}}27\%                      & {\DARK}\textcolor{white}{18\%} & {\cellcolor[HTML]{CCCCCC}}27\%                    \\
\multicolumn{1}{l}{DE}      &{\cellcolor[HTML]{CCCCCC}}78                       & {\cellcolor[HTML]{CCCCCC}}80                       & {\cellcolor[HTML]{EFEFEF}}50                    &&
                             {\cellcolor[HTML]{CCCCCC}}23                        & {\cellcolor[HTML]{EFEFEF}}7                      & {\cellcolor[HTML]{CCCCCC}}36                      &&
                             {\cellcolor[HTML]{CCCCCC}}15                        & {\cellcolor[HTML]{CCCCCC}}-2                       & {\cellcolor[HTML]{CCCCCC}}6                       &&
                             {\cellcolor[HTML]{CCCCCC}}32\%                      & {\cellcolor[HTML]{CCCCCC}}54\%                     & {\cellcolor[HTML]{CCCCCC}}50\%                    \\
\multicolumn{1}{l}{GS}      &{\cellcolor[HTML]{CCCCCC}}40                       & {\cellcolor[HTML]{CCCCCC}}34                       & {\cellcolor[HTML]{EFEFEF}}57                    &&
                             {\cellcolor[HTML]{CCCCCC}}53                        & {\cellcolor[HTML]{CCCCCC}}59                       & {\cellcolor[HTML]{CCCCCC}}32                      &&
                             {\DARK}\textcolor{white}{32}    & {\cellcolor[HTML]{CCCCCC}}33                       & {\cellcolor[HTML]{CCCCCC}}16                      &&
                             {\cellcolor[HTML]{CCCCCC}}32\%                      & {\cellcolor[HTML]{CCCCCC}}50\%                     & {\cellcolor[HTML]{CCCCCC}}36\%                    \\
\multicolumn{1}{l}{FLASH}   &{\cellcolor[HTML]{CCCCCC}}77                       & {\cellcolor[HTML]{EFEFEF}}94                     & {\cellcolor[HTML]{EFEFEF}}66                    &&
                             {\cellcolor[HTML]{CCCCCC}}17                        & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{CCCCCC}}28                      &&
                             {\cellcolor[HTML]{CCCCCC}}19                        & {\cellcolor[HTML]{EFEFEF}}-10                    & {\cellcolor[HTML]{CCCCCC}}15                      &&
                             {\cellcolor[HTML]{CCCCCC}}36\%                      & {\cellcolor[HTML]{EFEFEF}}72\%                   & {\cellcolor[HTML]{CCCCCC}}54\%                    \\
\multicolumn{1}{l}{HYPEROPT}&{\cellcolor[HTML]{CCCCCC}}55                       & {\cellcolor[HTML]{CCCCCC}}33                       & {\cellcolor[HTML]{CCCCCC}}35                      &&
                             {\cellcolor[HTML]{EFEFEF}}5                       & {\cellcolor[HTML]{EFEFEF}} 7                     & {\cellcolor[HTML]{EFEFEF}}7                     &&
                             {\cellcolor[HTML]{EFEFEF}}-1                      & {\cellcolor[HTML]{CCCCCC}} 0                       & {\cellcolor[HTML]{CCCCCC}}0                       &&
                             {\cellcolor[HTML]{EFEFEF}}77\%                    & {\cellcolor[HTML]{CCCCCC}}45\%                     & {\cellcolor[HTML]{EFEFEF}}86\%                  \\
\multicolumn{1}{l}{OPTUNA}  &{\cellcolor[HTML]{EFEFEF}}149                    & {\cellcolor[HTML]{EFEFEF}} 119                   & {\cellcolor[HTML]{EFEFEF}}61                    &&
                             {\cellcolor[HTML]{EFEFEF}}2                       & {\cellcolor[HTML]{EFEFEF}} 0                     & {\cellcolor[HTML]{EFEFEF}}4                     &&
                             {\cellcolor[HTML]{EFEFEF}}0                       & {\cellcolor[HTML]{CCCCCC}} 3                       & {\cellcolor[HTML]{CCCCCC}}0                       &&
                             {\cellcolor[HTML]{EFEFEF}}99\%                    & {\cellcolor[HTML]{EFEFEF}}77\%                   & {\cellcolor[HTML]{EFEFEF}}86\%                  \\  
& \multicolumn{3}{c} {(a) Median MRE  }    && \multicolumn{3}{c} {(b) Median Pred40  }   && \multicolumn{3}{c} {(c)  Median SA  }   && \multicolumn{3}{c} {(d) Median D2H  }\\ 
~\\
\multicolumn{16}{c}{\small{{\bf Table \ref{tab:medianResults}.a: Median results (over 20 runs).}}}\\
  &  \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  && 
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  &&  
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  &&  
                        \rotatebox{60}{commits}              & \rotatebox{60}{closedPRs}      & \rotatebox{60}{closedIssues}  \\  
                        
& \multicolumn{3}{c} {   (lower values are better)}    && \multicolumn{3}{c} {   (higher values are  better)}   && \multicolumn{3}{c} {  (higher values are  better)}   && \multicolumn{3}{c} {   (lower values are  better)}\\   


\multicolumn{1}{l}{{\sffamily{SNEAK}}+all} &{\cellcolor[HTML]{EFEFEF}}59                   & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{EFEFEF}}71                    && 
                             {\cellcolor[HTML]{EFEFEF}}51                      & {\cellcolor[HTML]{EFEFEF}}45                     & {\cellcolor[HTML]{EFEFEF}}48                    &&
                             {\cellcolor[HTML]{EFEFEF}}42                      & {\cellcolor[HTML]{EFEFEF}}69                     & {\cellcolor[HTML]{EFEFEF}}30                    &&
                             {\cellcolor[HTML]{EFEFEF}}47\%                    & {\cellcolor[HTML]{EFEFEF}}5\%                    & {\cellcolor[HTML]{EFEFEF}}25\%                  \\
\multicolumn{1}{l}{{\sffamily{SNEAK}}+Sall}   &{\cellcolor[HTML]{EFEFEF}}85                & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{EFEFEF}}69                    &&
                             {\cellcolor[HTML]{EFEFEF}}56                      & {\cellcolor[HTML]{EFEFEF}}45                     & {\cellcolor[HTML]{EFEFEF}}60                    &&
                             {\cellcolor[HTML]{EFEFEF}}52                      & {\cellcolor[HTML]{EFEFEF}}73                     & {\cellcolor[HTML]{EFEFEF}}30                    &&
                             {\cellcolor[HTML]{EFEFEF}}41\%                    & {\cellcolor[HTML]{EFEFEF}}9\%                    & {\cellcolor[HTML]{EFEFEF}}11\%                  \\
\multicolumn{1}{l}{{\sffamily{SWAY}}}    &{\cellcolor[HTML]{EFEFEF}}72                  & {\cellcolor[HTML]{EFEFEF}}100                    & {\cellcolor[HTML]{EFEFEF}}50                    &&
                             {\cellcolor[HTML]{EFEFEF}}55                      & {\cellcolor[HTML]{EFEFEF}}68                     & {\cellcolor[HTML]{EFEFEF}}44                    &&
                             {\cellcolor[HTML]{EFEFEF}}17                      & {\cellcolor[HTML]{EFEFEF}}85                     & {\cellcolor[HTML]{EFEFEF}}8                     &&
                             {\cellcolor[HTML]{EFEFEF}}20\%                    & {\cellcolor[HTML]{EFEFEF}}20\%                   & {\cellcolor[HTML]{EFEFEF}}41\%                  \\ 
\multicolumn{1}{l}{{\sffamily{SNEAK}}+Sany} &{\cellcolor[HTML]{EFEFEF}}127                 & {\cellcolor[HTML]{EFEFEF}}24                     & {\cellcolor[HTML]{EFEFEF}}88                    &&
                             {\cellcolor[HTML]{EFEFEF}}28                      & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{EFEFEF}}43                    &&
                             {\cellcolor[HTML]{EFEFEF}}24                      & {\cellcolor[HTML]{EFEFEF}}11                     & {\cellcolor[HTML]{EFEFEF}}19                    &&
                             {\cellcolor[HTML]{EFEFEF}}16\%                    & {\cellcolor[HTML]{EFEFEF}}20\%                   & {\cellcolor[HTML]{EFEFEF}}54\%                  \\
\multicolumn{1}{l}{{\sffamily{SNEAK}}+any} &{\cellcolor[HTML]{EFEFEF}}131                  & {\cellcolor[HTML]{EFEFEF}}33                     & {\cellcolor[HTML]{EFEFEF}}91                    &&
                             {\cellcolor[HTML]{EFEFEF}}28                      & {\cellcolor[HTML]{EFEFEF}}8                      & {\cellcolor[HTML]{EFEFEF}}52                    &&
                             {\cellcolor[HTML]{EFEFEF}}24                      & {\cellcolor[HTML]{EFEFEF}}15                     & {\cellcolor[HTML]{EFEFEF}}18                    &&
                             {\cellcolor[HTML]{EFEFEF}}16\%                    & {\cellcolor[HTML]{EFEFEF}}14\%                   & {\cellcolor[HTML]{EFEFEF}}72\%                  \\ \hline
\multicolumn{1}{l}{Default} &{\cellcolor[HTML]{EFEFEF}}82                     & {\cellcolor[HTML]{EFEFEF}}0                      & {\cellcolor[HTML]{EFEFEF}}98                    &&
                             {\cellcolor[HTML]{EFEFEF}}41                      & {\cellcolor[HTML]{EFEFEF}}31                     & {\cellcolor[HTML]{EFEFEF}}49                    &&
                             {\cellcolor[HTML]{EFEFEF}}81                      & {\cellcolor[HTML]{EFEFEF}}15                     & {\cellcolor[HTML]{EFEFEF}}52                    &&
                             {\cellcolor[HTML]{EFEFEF}}36\%                    & {\cellcolor[HTML]{EFEFEF}}83\%                   & {\cellcolor[HTML]{EFEFEF}}47\%                  \\ \hline
\multicolumn{1}{l}{RS}      &{\cellcolor[HTML]{EFEFEF}}69                     & {\cellcolor[HTML]{EFEFEF}}99                     & {\cellcolor[HTML]{EFEFEF}}50                    &&
                             {\cellcolor[HTML]{EFEFEF}}41                      & {\cellcolor[HTML]{EFEFEF}}60                     & {\cellcolor[HTML]{EFEFEF}}44                    &&
                             {\cellcolor[HTML]{EFEFEF}}15                      & {\cellcolor[HTML]{EFEFEF}}66                     & {\cellcolor[HTML]{EFEFEF}}22                    &&
                             {\cellcolor[HTML]{EFEFEF}}29\%                    & {\cellcolor[HTML]{EFEFEF}}20\%                   & {\cellcolor[HTML]{EFEFEF}}47\%                  \\
\multicolumn{1}{l}{DE}      &{\cellcolor[HTML]{EFEFEF}}104                    & {\cellcolor[HTML]{EFEFEF}}50                     & {\cellcolor[HTML]{EFEFEF}}112                   &&
                             {\cellcolor[HTML]{EFEFEF}}37                      & {\cellcolor[HTML]{EFEFEF}}27                     & {\cellcolor[HTML]{EFEFEF}}40                    &&
                             {\cellcolor[HTML]{EFEFEF}}68                      & {\cellcolor[HTML]{EFEFEF}}16                     & {\cellcolor[HTML]{EFEFEF}}45                    &&
                             {\cellcolor[HTML]{EFEFEF}}50\%                    & {\cellcolor[HTML]{EFEFEF}}20\%                   & {\cellcolor[HTML]{EFEFEF}}36\%                  \\
\multicolumn{1}{l}{GS}      &{\cellcolor[HTML]{EFEFEF}}49                     & {\cellcolor[HTML]{EFEFEF}}100                    & {\cellcolor[HTML]{EFEFEF}}48                    &&
                             {\cellcolor[HTML]{EFEFEF}}43                      & {\cellcolor[HTML]{EFEFEF}}62                     & {\cellcolor[HTML]{EFEFEF}}53                    &&
                             {\cellcolor[HTML]{EFEFEF}}15                      & {\cellcolor[HTML]{EFEFEF}}84                     & {\cellcolor[HTML]{EFEFEF}}24                    &&
                             {\cellcolor[HTML]{EFEFEF}}32\%                    & {\cellcolor[HTML]{EFEFEF}}54\%                   & {\cellcolor[HTML]{EFEFEF}}29\%                  \\
\multicolumn{1}{l}{FLASH}   &{\cellcolor[HTML]{EFEFEF}}68                     & {\cellcolor[HTML]{EFEFEF}}41                     & {\cellcolor[HTML]{EFEFEF}}87                    &&
                             {\cellcolor[HTML]{EFEFEF}}38                      & {\cellcolor[HTML]{EFEFEF}}15                     & {\cellcolor[HTML]{EFEFEF}}37                    &&
                             {\cellcolor[HTML]{EFEFEF}}62                      & {\cellcolor[HTML]{EFEFEF}}15                     & {\cellcolor[HTML]{EFEFEF}}45                    &&
                             {\cellcolor[HTML]{EFEFEF}}63\%                    & {\cellcolor[HTML]{EFEFEF}}14\%                   & {\cellcolor[HTML]{EFEFEF}}50\%                  \\
\multicolumn{1}{l}{HYPEROPT}&{\cellcolor[HTML]{EFEFEF}}131                    & {\cellcolor[HTML]{EFEFEF}}76                     & {\cellcolor[HTML]{EFEFEF}}76                    &&
                             {\cellcolor[HTML]{EFEFEF}}4                       & {\cellcolor[HTML]{EFEFEF}}6                      & {\cellcolor[HTML]{EFEFEF}}6                     &&
                             {\cellcolor[HTML]{EFEFEF}}7                       & {\cellcolor[HTML]{EFEFEF}}18                     & {\cellcolor[HTML]{EFEFEF}}15                    &&
                             {\cellcolor[HTML]{EFEFEF}}36\%                    & {\cellcolor[HTML]{EFEFEF}}50\%                   & {\cellcolor[HTML]{EFEFEF}}20\%                  \\
\multicolumn{1}{l}{OPTUNA}  &{\cellcolor[HTML]{EFEFEF}}157                    & {\cellcolor[HTML]{EFEFEF}}52                     & {\cellcolor[HTML]{EFEFEF}}76                    &&
                             {\cellcolor[HTML]{EFEFEF}}4                       & {\cellcolor[HTML]{EFEFEF}}3                      & {\cellcolor[HTML]{EFEFEF}}5                     &&
                             {\cellcolor[HTML]{EFEFEF}}15                      & {\cellcolor[HTML]{EFEFEF}}24                     & {\cellcolor[HTML]{EFEFEF}}30                    &&
                             {\cellcolor[HTML]{EFEFEF}}14\%                    & {\cellcolor[HTML]{EFEFEF}}38\%                   & {\cellcolor[HTML]{EFEFEF}}36\%                  \\ 
& \multicolumn{3}{c} {(e) MRE IQR}    && \multicolumn{3}{c} {(f) Pred40 IQR}   &&\multicolumn{3}{c} {(g) SA IQR}      && \multicolumn{3}{c} {(h) D2H IQR} \\
~\\
\multicolumn{16}{c}{\small{{\bf Table \ref{tab:medianResults}.b: IQR results (over 20 runs).}}}\\

        \end{tabular}
        \label{tab:medianResults}
\end{table}


Not only was {\sffamily{SNEAK}} better than traditional hyperparameter optimization across the board, we were able to achieve these results while using fewer evaluations.  
In Table~\ref{tab:globmedian}, we see in the last column a Rank measurement, this column was obtained via using the Ziztler's predicate to generate scores for each of the techniques on all of our metrics. 
That top ranking holds all the ``Sall'' and ``all'' budgets perform by far better than the other algorithms. 
When comparing {\sffamily{SNEAK}} to the other methods we can also see a notable increase in performance looking individually at all metrics: specifically, low MRE and average Pred40s of over 60\%.
Better yet, Table~\ref{tab:globmedian} shows us that {\sffamily{SNEAK}} is orders of magnitude faster than traditional optimizers. 
{\sffamily{SNEAK}}+Sall finds good configurations after looking at around 100 options, while other tools (OPTUNA, HYPEROPT) needed to explore 3500, and 5000 configurations. 

\begin{table}
\caption{Runtime and performance comparison. (Medians on all datasets and all features)}
\small
\begin{center}
\begin{tabular}{l@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c@{~}|c}
\textbf{Method}                 & \textbf{\#models}      & \textbf{MRE}               & \textbf{PRED40}            & \textbf{SA}                & \textbf{Runtime (s)}     & \textbf{D2H}                 & \textbf{Rank}   \\ \hline
\rowcolor[HTML]{EFEFEF}{\sffamily{SNEAK}}+Sall           & 115       & 33         & 60         & 33       & 10     & 5.00\%       & 1      \\
{\sffamily{SNEAK}}+all                                   & 6068      & 33         & 66         & 36       & 487    & 27.00\%      & 1      \\
RS                                          & 120       & 43         & 26         & 26       & 4      & 27.00\%      & 2      \\
{\sffamily{SWAY}}                                     & 44        & 52         & 26         & 26       & 94     & 45.00\%      & 2      \\
DE                                          & 132       & 132        & 6          & 6        & 4      & 50.00\%      & 3      \\
GS                                          & 2560      & 40         & 32         & 32       & 395    & 36.00\%      & 3      \\
FLASH                                       & 120       & 77         & 15         & 15       & 6      & 54.00\%      & 3      \\
{\sffamily{SNEAK}}+Sany                                  & 66        & 118        & 6          & 6        & 7      & 63.00\%      & 4      \\
{\sffamily{SNEAK}}+any                                   & 52        & 120        & 2          & 2        & 5      & 72.00\%      & 4      \\  
Default                                     & 1         & 70         & -5         & -5       & 1      & 50.00\%      & 4      \\
HYPEROPT                                    & 5000      & 35         & 7          & 0        & 237    & 77.00\%      & 4      \\
OPTUNA                                      & 3500      & 119        & 2          & 0        & 318    & 86.00\%      & 4  
\end{tabular}
\end{center}
\label{tab:globmedian}
\vspace{-14pt}
\end{table}

Given that for most HPO techniques evaluating configurations by calling a learner and collecting performance scores are the slowest part of this kind of reasoning, it comes with no surprise that {\sffamily{SNEAK}} on the ``Sall'' budget also runs at a much smaller runtime, being 20 to 30 times faster than traditional techniques. And while this difference seems significant here, we remind the reader that the training and evaluation of a random forest in such small datasets is significantly cheap, taking around $0.03s$ per evaluation. As such  {\sffamily{SNEAK}}'s reasoning results in a general overhead of 4 seconds on a dataset of 10.000 initial configurations, while the pass of {\sffamily{SWAY}} adds another 1 to 2 seconds of overhead. Meanwhile HYPEROPT added around $90s$ of overhead, and OPTUNA added around $213s$ of overhead, which are very significant for datasets of this size. However, if we were to simulate a slightly slower model, say one that took $1s$ to train and evaluate, {\sffamily{SNEAK}} theoretically would take around 2 minutes and HYPEROPT and OPTUNA taking between 1 hour and 1 hour and 20 minutes. Which means for larger models these overheads should not even be considered for runtime.

{\sffamily{SNEAK}} also showed that this style of reasoning using landscape analytics is very scalable, by demonstrating results on much larger datasets, both in number of features and number of samples. In summary this evidence makes us conclude that:
\begin{quote} {{\bf  {\sffamily{SNEAK}} is a more efficient approach than traditional SE hyperparameter optimization techniques} (for predicting future trends in OSS software).}
\end{quote}


\section{Threats to Validity}\label{threats}
As with any empirical study, biases can affect the final results. 
Therefore, any conclusions made from this work must be considered with the following issues in mind:

\textbf{Sampling Bias} - Any given empirical study using datasets suffers from sampling biases. 
In this work we have applied {\sffamily{SNEAK}} to 12 different datasets containing OSS project health indicators, and to one learning algorithm (Random Forest). 
The behaviour of {\sffamily{SNEAK}} upon different domains and different learners still needs to be evaluated and should be explored in future work.

\textbf{Parameter bias} - {\sffamily{SNEAK}} contains many modular subprocesses within itself, containing different  hyperparameters. 
While this study shows that the current configuration of {\sffamily{SNEAK}} is capable of producing results comparable to state-of-the-art algorithms, different configurations of the {\sffamily{SNEAK}} system might prove valuable. 
That said, in defense of our current setup, we were able to find solutions that were significantly better than those found by state-of-the-art HPO techniques.

\textbf{Algorithm bias} - We have selected seven commonly applied and state-of-the-art HPO techniques to baseline {\sffamily{SNEAK}} against. 
There might exist other HPO algorithms that were not explored in this study that may prove to be competitive to {\sffamily{SNEAK}} in this domain, however the point of these baselines was to showcase the performance of {\sffamily{SNEAK}} against other commonly applied techniques that are proven to obtain good results.

\section{Conclusion}

Using hyperparameter optimization
can be like
walking around a broken car and randomly
hitting it till it works. Our approach, called {\sffamily{SNEAK}}, adopts a landscape analytics approach that looks at the shape of
the data before deciding to move one way or another.
 
We have applied {\sffamily{SNEAK}} to the open-source software projects health domain, to aid in the prediction of indicators of project health one year into the future. 
Previous work in this arena has obtained good results~\cite{xia2018hyperparameter} but suffered
from high prediction errors. 
{\sffamily{SNEAK}} can significantly reduce those errors such that they satisfy the Sarro limits )discussed 
in our introduction).

More generally,
when compare to  seven other state-of-the-art HPO techniques,
{\sffamily{SNEAK}}  outperforms all  seven techniques by a statistically
significantly amount.
Further, we have achieved these significantly better results after evaluating a fraction of the models needed by several hyperparameter optimization.
When compared to a current state-of-the-art optimizer
(HYPEROPT and  OPTUNA):
\noindent\begin{enumerate}
\item {\sffamily{SNEAK}} is orders of magnitude faster
\item {\sffamily{SNEAK}}  only evaluates around 2\% 
(115/5000) of the models needed by other methods.
\noindent\end{enumerate}
This suggests that landscape analytics might be   applicable for more expensive evaluation functions in the future (i.e. deep learning).


Given these results, our conclusion is that {\sffamily{SNEAK}} is a useful approach for  hyperparameter optimization
in the software engineering data domain, and should be considered by future work when there is a need for tuning learners
(especially when the evaluation function of these learners is time consuming). 





\section*{Acknowledgments}

This work was partially funded under an NSF
grant \#1908762.

{\footnotesize \bibliographystyle{ACM-Reference-Format}
