\section{Introduction}
Recommender-system development has always been a challenge. Particularly, identifying the best recommendation algorithm and parameters for a given scenario is difficult. `Intuition', even of experienced data scientists, is often not good enough to identify the ideal algorithm and parameters \cite{gomez2015netflix}. Minor variations in implementations and parameters may lead to significantly different performances in different scenarios \cite{beel2016towards}.

The machine learning community faces similar challenges and tackled these quite successfully with so-called Automated Machine Learning (AutoML). AutoML eases the configuration of machine learning pipelines, particularly the algorithm selection and configuration process. AutoML applies hyperparameter optimization techniques beyond standard grid or random search, not only to hyperparameters but also to algorithm selection \cite{hutter2019automated}. Typical AutoML methods include Bayesian optimization \cite{snoek2012practical}, Sequential model-based optimization \cite{hutter2011sequential} or hierarchical planning \cite{mohr2018ml}. Sometimes, metalearning is used to ‘warm-start’ the process, i.e. to predict a set of algorithms and parameters that are promising for a given task \cite{hutter2019automated}. AutoML is easily accessible for machine-learning engineers through AutoML software libraries including H2O \cite{candel2016deep}, TPOT \cite{olson2019tpot}, AutoWEKA \cite{thornton2013auto}, AutoSklearn \cite{feurer2015efficient}, AutoKeras \cite{jin2019auto}, and MLPlan \cite{mohr2018ml}. 

The recommender-system community has fallen behind the advances in the (automated) machine-learning community. While there are many recommender-system libraries such as Mahout \cite{lyubimov2016apache}, LibRec \cite{guo2015librec}, Surprise \cite{hug2017surprise}, CaseRec \cite{da2018case}, and Lenskit \cite{ekstrand2011rethinking}, there is – to the best of our knowledge – only one Automated Recommender System library, namely Librec-Auto \cite{mansoury2019algorithm}. Librec-Auto extends the LibRec recommender-system library with some automated algorithm selection and configuration functionality, though this functionality is limited. LibRec-Auto iterates over parameter spaces in one scripted experiment, whereas the user still must define the parameter spaces and write the script. This system is useful for experienced data scientists who wish to experiment with different configurations and analyze the models. However, LibRec-Auto is not as advanced as the typical AutoML tools, and a user with no prior experience may have difficulties in setting up such a solution.

We introduce Auto-Surprise\footnotemark[1], the first automated recommender system library (AutoRecSys) with fully automated algorithm selection and configuration, comparable to state of-the-art AutoML libraries.

\section{Auto-Surprise}

Auto-Surprise is built as a wrapper around the Python Surprise \cite{hug2017surprise} library. Auto-Surprise uses a sequential model-based optimization approach for the algorithm selection and configuration, is open-source and brings the advances of AutoML to the recommender-system community. Auto-Surprise offers all 11 algorithms (see Table \ref{tab:results}) that Surprise has implemented. To use Auto-Surprise, a user needs to import the auto-surprise package and pass data to the trainer method. Auto-Surprise then automatically identifies the best performing algorithm and hyperparameters out of the 11 algorithms. As such, almost no prior knowledge is needed. 

\begin{figure}[h]
  \centering
  \includegraphics[width=.7\linewidth]{images/overview.png}
  \caption{Simple overview of the working of Auto-Surprise}
  \Description{A simple overview of the working of Auto-Surprise}
\end{figure}

The overall optimization strategy of Auto-Surprise is similar to AutoWEKA \cite{thornton2013auto}. Auto-Surprise first evaluate a baseline score for the given dataset using random predictor. This sets the minimum loss that each algorithm must achieve. Each algorithm is then optimized in parallel until a user defined time limit or a maximum evaluations limit is reached. If any of the algorithms perform worse than the baseline after a number of evaluations, it is not optimized any further. Once this process is completed, the best performing algorithm with optimized hyperparameters is returned along with a dictionary of the performance of all the algorithm.

For all algorithms (except for those which do not require any hyperparameters), we defined a hyperparameter space which is used to identify optimal hyperparameters in the given range. Auto-Surprise can use three hyperparameter optimization methods as implemented by Hyperopt \cite{bergstra2013hyperopt} - Tree of Parzens Estimator (TPE) \cite{bergstra2011algorithms}, Adaptive TPE (ATPE) \cite{hypermaxAtpe} and Random Search. The user sets a target metric such as RMSE or MAE which is to be minimized. All of this is done in just one line of code.
 
\section{Evaluation}

We compared Auto-Surprise against all eleven algorithms in Surprise with a) the algorithms' default parameters and b) the algorithms' being optimized with Grid search as implemented by Surprise with concurrency enabled. We used the Movielens 100k dataset \cite{harper2015movielens}, the de-facto gold-standard dataset in the recommender system community \cite{beel2019data}. Jester-2 \cite{goldberg2001eigentaste} and Book Crossing \cite{ziegler2005improving} datasets were also used, though only using a 100k sample of them to reduce resource requirements. Separate configurations of Auto-Surprise were evaluated using Adaptive TPE and TPE as the hyperparameter optimization algorithm. The target metric to minimize was RMSE and a maximum evaluation time of 2 hours was set for Auto-Surprise.

\begin{table*}
\caption{Comparison of Auto-Surprise with other Surprise algorithms and Grid Search. Results in bold is for the overall best performing algorithm in its default configuration}
\label{tab:results}
\centering
\begin{tabular}{lccccccccc}
\hline
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Algorithm}}} & \multicolumn{3}{c}{\textbf{MovieLens 100k}}                      & \multicolumn{3}{c}{\textbf{Jester 2}}                          & \multicolumn{3}{c}{\textbf{Book Crossing}}                      \\ \cline{2-10} 
\multicolumn{1}{c}{}                                    & \textbf{RMSE}   & \textbf{MAE}    & \textbf{Time}                & \textbf{RMSE}  & \textbf{MAE}   & \textbf{Time}                & \textbf{RMSE}   & \textbf{MAE}   & \textbf{Time}                \\ \hline
Normal Predictor                                        & 1.5195          & 1.2200          & 00:00:01                     & 7.277          & 5.886          & 00:00:01                     & 4.8960          & 3.866          & 00:00:01                     \\
SVD                                                     & 0.9364          & 0.7385          & 00:00:23                     & 4.905          & 3.97           & 00:00:13                     & 3.5586          & 3.013          & 00:00:11                     \\
SVD++                                                   & 0.9196          & 0.7216          & 00:14:23                     & 5.102          & 4.055          & 00:00:29                     & 3.5842          & 2.991          & 00:01:48                     \\
NMF                                                     & 0.9651          & 0.7592          & 00:00:25                     & --             & --             & --                           & --              & --             & --                           \\
Slope One                                               & 0.9450          & 0.7425          & 00:00:15                     & 5.189          & 3.945          & 00:00:02                     & --              & --             & --                           \\
KNN Basic                                               & 0.9791          & 0.7738          & 00:00:18                     & 5.078          & 4.034          & 00:02:14                     & 3.9108          & 3.562          & 00:00:38                     \\
KNN with Means                                          & 0.9510          & 0.7490          & 00:00:19                     & 5.124          & 3.955          & 00:02:16                     & 3.8574          & 3.301          & 00:00:35                     \\
KNN with   Z-score                                      & 0.9517          & 0.7470          & 00:00:21                     & 5.219          & 3.955          & 00:02:20                     & 3.8526          & 3.292          & 00:00:37                     \\
\textbf{KNN Baseline}                                   & \textbf{0.9299} & \textbf{0.7329} & \textbf{00:00:22}            & \textbf{4.898} & \textbf{3.896} & \textbf{00:02:14}            & \textbf{3.6181} & \textbf{3.101} & \textbf{00:00:36}            \\
Co-clustering                                           & 0.9678          & 0.7581          & 00:00:08                     & 5.153          & 3.917          & 00:00:12                     & 4.0168          & 3.409          & 00:00:19                     \\
Baseline Only                                           & 0.9433          & 0.7479          & 00:00:01                     & 4.849          & 3.934          & 00:00:01                     & 3.5760          & 3.095          & 00:00:02                     \\ \hline
GridSearch                                              & 0.9139          & 0.7167          & \multicolumn{1}{l}{27:02:48} & 4.7409         & 3.8147         & \multicolumn{1}{l}{80:52:35} & 3.5467          & 2.9554         & \multicolumn{1}{l}{48:29:46} \\
Auto-Surprise (TPE)                                     & 0.9136          & 0.7280          & 02:00:01                     & 4.6489         & 3.6837         & 02:00:10                     & 3.5221          & 2.8871         & 02:00:58                     \\
Auto-Surprise (ATPE)                                    & 0.9116          & 0.7244          & 02:00:02                     & 4.6555         & 3.6906         & 02:00:01                     & 3.5190          & 2.8739         & 02:00:06                     \\ \hline
\end{tabular}
\end{table*}


The best default algorithm in Surprise for the MovieLens dataset was SVD++ with an RMSE of 0.9196. Auto-Surprise was able to perform best with adaptive TPE with an RMSE of 0.9116. This is a small - but statistically significant difference (2 tailed p value < 0.05) in RMSE of 0.86\%. We see a similar result for the Book Crossing dataset which is optimized from an RMSE of 3.5586 with SVD to 3.5190 with Auto-Surprise, a 1.11\% difference. However, we see a more pronounced difference for the Jester dataset from Baseline Only algorithm with an RMSE of 4.8490 and Auto-Surprise with an RMSE of 4.6489, a difference of 4.12\%. It is also important to note that the best performing algorithm by default may not be the algorithm selected after optimization. For Movielens, Auto-Surprise found that NMF performed best after optimization even though the best performing default algorithm was SVD++. Similarly with Jester, KNN Baseline was selected and for Book Crossing, SVD was selected. While GridSearch can result in decent results, the time taken is far longer than Auto-Surprise.

\section{Conclusion}
We found that when compared to the default configuration of Surprise algorithms, Auto-Surprise performs anywhere from 0.8\% to 4\% better in terms of RMSE in our tests. Though the actual run time of the combined default Surprise algorithms is still lower than Auto-Surprise, it still widely outperforms Gridsearch in that respect. It is also worth noting that the selected algorithm may have a much lower runtime compared to the default algorithm as shown for the Movielens dataset where the selected algorithm NMF only has a runtime of 25 seconds compared to the ~15 minutes runtime for SVD++, the best performing default algorithm. And, of course, Auto-Surprise eases the entire process of algorithm selection and hyperparameter optimization by automating it in a single line of code.


\bibliographystyle{ACM-Reference-Format}
