\section{Introduction}

Many classification problems across different application domains can be naturally modeled hierarchically (\autoref{fig:classifiers} and Supplementary Figures \ref{fig:music}-\ref{fig:phylogeny}), typically in the form of trees or directed acyclic graphs \citep{silla2011survey}. Examples of hierarchical classification problems are vast, ranging from musical genre classification \citep{ariyaratne2012novel, iloga2018sequential} to text categorization \citep{javed2021hierarchical, ma2022hybrid}, taxonomic classification of viral sequences in metagenomic data \citep{shang2021cheer} and identification of COVID-19 in chest X-ray images \citep{pereira2020covid}.

Nonetheless, many classifiers proposed in the literature are designed to completely ignore the existing hierarchy between classes by usually predicting only leaf nodes in a methodology known as flat approach. Although easy to implement, the flat approach is incapable of dealing with problems where making a prediction for leaf nodes is not mandatory. Furthermore, since they consider the hierarchy during training, hierarchical models generally produce better results when compared with flat approaches, consistently achieving improvements in predictive performance \citep{silla2011survey}. Hence, in this manuscript we introduce \texttt{HiClass}, a Python library that implements the most common patterns for local hierarchical classifiers (\autoref{fig:classifiers} and Supplementary Material for more details), which can be employed in different application domains where the data can be hierarchically structured in the shape of trees or directed acyclic graphs. While the full hierarchical structure is ideally known, classification is also possible with only partially known structures and missing values in the outermost levels of the hierarchy.

\begin{figure}[!htb]
\centerline{\includegraphics[width=\linewidth]{images/classifiers.pdf}}
\caption{Depiction of the flat approach and the local hierarchical classification models implemented in \texttt{HiClass}, for a two-level hierarchy. a) Flat approach, where a single multi-class classifier is trained to predict only the leaf nodes in hierarchical data, completely ignoring previous levels of the hierarchy. b) Local classifier per node, where several binary classifiers are trained for each existing node in the hierarchy, excluding the root node. c) Local classifier per parent node, where various multi-class classifiers are trained for each parent node in the hierarchy, with the goal of predicting the children nodes. d) Local classifier per level, where a multi-class classifier is trained for each level in the hierarchy.}\label{fig:classifiers}
\end{figure}

\section{Overview and Design}

The API of \texttt{HiClass} is designed to be compatible with the \texttt{scikit-learn} API \citep{pedregosa2011scikit}, offering users a familiar API to train hierarchical models via \textit{fit()}, store trained models with \texttt{pickle}, predict labels with the \textit{predict()} method, and create machine learning pipelines. Moreover, each local classifier in \texttt{HiClass} is, by default, \texttt{scikit-learn}'s \textit{BaseEstimator} class, enabling users to employ any of the classifiers already implemented in \texttt{scikit-learn}'s library. However, classes other than the BaseEstimator can also be used, as long as they implement the methods \textit{fit()}, \textit{predict()} and \textit{predict\_proba()}, thus enabling users to code their own local classifiers using third-party libraries, e.g., \texttt{TensorFlow} \citep{abadi2016tensorflow} or \texttt{PyTorch} \citep{paszke2019pytorch}.

To allow a better integration with \texttt{scikit-learn} and faster training, \texttt{HiClass} uses for both features and labels \texttt{NumPy}'s structured arrays, whose data type is a composition of simpler data types \citep{harris2020array}. While features are exactly the same shape as expected for training flat models in \texttt{scikit-learn}, hierarchical training labels are represented as an array of shape \textit{n\_samples} $\times$ \textit{n\_levels}, where each column must contain either a label for the respective level in the hierarchy or an empty string that indicates missing labels in the leaf nodes.

The directed acyclic graph (DAG) implemented in the library \texttt{NetworkX} \citep{hagberg2008exploring} was the data structure chosen to make the models as generic as possible, since hierarchical data in the form of trees can also be represented with a DAG. Training can be performed in parallel for each local classifier by leveraging either the library \texttt{Ray}$^1$\footnotetext{1. \url{https://www.ray.io/}} or \texttt{Joblib}$^1$\footnotetext{2. \url{https://joblib.readthedocs.io/en/latest/parallel.html}}, while prediction, which is not a bottleneck, is performed from top to bottom, following the hierarchical structure to keep consistency among the several hierarchical levels.

According to \citet{silla2011survey}, the use of flat classification metrics might not be adequate to give enough insight on which algorithm is better at classifying hierarchical data. Hence, in \texttt{HiClass} we implemented the metrics of hierarchical precision (hP), hierarchical recall (hR) and hierarchical F-score (hF), which are extensions of the renowned metrics of precision, recall and F-score, but tailored to the hierarchical classification scenario. These hierarchical counterparts were initially proposed by \citet{kiritchenko2006learning}, and are defined as follows:

\vspace{0.2cm}

\centerline{
$\displaystyle{hP = \frac{\sum_i|\alpha_i\cap\beta_i|}{\sum_i|\alpha_i|}}$, $\displaystyle{hR = \frac{\sum_i|\alpha_i\cap\beta_i|}{\sum_i|\beta_i|}}$, $\displaystyle{hF = \frac{2 \times hP \times hR}{hP + hR}}$
}

\noindent where $\alpha_i$ is the set consisting of the most specific classes predicted for test example $i$ and all their ancestor classes, while $\beta_i$ is the set containing the true most specific classes of test example $i$ and all their ancestors, with summations computed over all test examples.

\section{Code Quality Assurance}

To ensure high code quality, \textit{inheritance} is applied whenever possible to keep the code concise and easier to maintain, and all implementations adhere to the \textit{PEP 8} code style \citep{van2001pep} enforced by \texttt{flake8} and the uncompromising code formatter \texttt{black}. API documentation is provided through \textit{docstrings} \citep{goodger2010docstring}, and the implementations are accompanied by \textit{unit tests} that cover \textit{98\%} of our code and are automatically executed by our \textit{continuous integration workflow} upon commits.

\section{Installation and Usage}

\texttt{HiClass} is hosted on GitHub$^2$\footnotetext{2. \url{https://github.com/mirand863/hiclass}}, while tutorials and API documentation are available on the platform Read the Docs$^3$\footnotetext{3. \url{https://hiclass.readthedocs.io/}}. Packages for Python 3.7-3.9 are available for Linux, macOS and Windows and can be obtained with \texttt{pip install hiclass} or \texttt{conda install -c conda-forge hiclass}. Code \ref{code} shows a basic example of fitting and evaluating a local hierarchical model with \texttt{HiClass}. More elaborate examples can be found in the tutorials.

\begin{lstlisting}[language=Python, caption=Example on how to use \texttt{HiClass} to train and evaluate a hierarchical classifier.]
	from hiclass import LocalClassifierPerNode
	from hiclass.metrics import f1
	from sklearn.ensemble import RandomForestClassifier
	
	# define mock data
	X_train = X_test = [[1, 2], [3, 4]]
	Y_train = Y_test = [
	["Animal", "Mammal", "Cat"],
	["Animal", "Reptile", "Turtle"],
	]
	
	# Use random forest classifiers for every node
	rf = RandomForestClassifier()
	lcpn = LocalClassifierPerNode(local_classifier=rf)
	lcpn.fit(X_train, Y_train)  # Train model
	predictions = lcpn.predict(X_test)  # Predict test data
	
	# Print hierarchical F-score
	print(f"f1: {f1(y_true=Y_test, y_pred=predictions)}")
\end{lstlisting}

\section{Comparison with Flat Classifiers}

While \texttt{HiClass} focuses on ease of use and is fully written in a high level language, care has
been taken to maximize computational efficiency. In \autoref{fig:results}, we compare the hierarchical F-score, computational resources (measured with the command \textit{time}) and disk usage. This comparison was performed between two flat classifiers from the library \texttt{scikit-learn} and Microsoft's LightGBM \citep{ke2017lightgbm} versus the local hierarchical classifiers implemented in \texttt{HiClass}. In order to avoid bias, cross-validation and hyperparameter tuning were performed on the local hierarchical classifiers and flat classifiers. For comparison purposes, we used a snapshot from 02/11/2022 of the consumer complaints dataset provided by the Consumer Financial Protection Bureau of the United States \citep{bureau2022consumer}, which after preprocessing contained 727,495 instances for cross-validation and hyperparameter tuning as well as training and 311,784 more for validation. Additional descriptions about this dataset, preprocessing, feature extraction, and experimental setup are available in the Supplementary Material. The benchmark was computed on multiple cluster nodes running GNU/Linux with 512 GB physical memory and 128 cores provided by two AMD EPYC\textsuperscript{\texttrademark} 7742 processors. A reproducible Snakemake pipeline \citep{koster2012snakemake} is available in our public repository$^4$\footnotetext{4. \url{https://github.com/mirand863/hiclass/tree/main/benchmarks/consumer_complaints}}.

\begin{figure}[!htb]
\centerline{\includegraphics[width=0.87\linewidth]{images/results.pdf}}
\caption{Comparison between the flat approach and the local hierarchical classifiers implemented in \texttt{HiClass}, using the consumer complaints dataset and Microsoft's LightGBM \citep{ke2017lightgbm}, Logistic Regression and Random Forest as the base classifiers. For this benchmark, the metrics used were hierarchical F-score, memory usage in gigabyte, disk usage in megabyte or gigabyte, and training time in seconds. Overall, the hierarchical classifiers improved the F-score when compared with the flat approaches, while in some occasions the local hierarchical classifiers further reduced memory consumption, disk usage, and training time.}\label{fig:results}
\end{figure}

Our results reveal that the hierarchical F-score was enhanced by all local hierarchical approaches when compared with the flat classifiers, achieving a maximum improvement of $\approx113\%$ when comparing the local classifier per node (LCPN) with the flat LightGBM approach. Regarding training time, the LCPN and local classifier per parent node (LCPPN) decreased computational time by $\approx93\%$ and $\approx44\%$, respectively, when compared with the flat Logistic Regression classifier. When compared with the flat Random Forest, the LCPN and LCPPN reduced memory usage by $\approx66\%$ and $\approx70\%$ and disk usage by $\approx67\%$ and $\approx70\%$, respectively.

\section{Conclusion}

 \texttt{HiClass} is a Python package that provides implementations of popular machine learning models and evaluation metrics for local hierarchical classification. Thanks to its compatibility with the \texttt{scikit-learn} API, users can choose among existing classifiers to create powerful hierarchical classifiers for hierarchical data.

In future releases, we plan to implement multi-label hierarchical classification to expand the range of problems the library can solve. Additionally, we also plan to implement the global approach, which trains a single model that learns the entire hierarchical structure. Lastly, we also intend to add support for incremental learning in order to enable training of larger than memory datasets.


\acks{BYR gratefully acknowledges support by the BMBF-funded Computational Life Science initiative (project DeepPath, 031L0208, to B.Y.R.). The authors gratefully acknowledge all users for requesting useful new features and reporting bugs.}




\iffalse

