\section{Introduction}


The usefulness of feature engineering for applications of deep learning has long been considered a settled question in the negative, as neural networks are on their own universal function approximators \cite{GoodBengCour16}. However, even in the context of deep learning, tabular features are often treated with some form of encoding for preprocessing. Automunge \cite{anonymous_github} is a platform for encoding dataframes developed by the authors. This python library was originally built for a simple use case of basic encoding conventions for numeric and categoric features, like z-score normalization and one-hot encodings. Along the iterative development journey we began to flesh out a full library of encoding options, including a series of options for numeric and categoric features that now include scenarios for normalization, binarization, hashing, and missing data infill under automation. Although it was expected that these range of encoding options would be superfluous for deep learning, that does not rule out their utility in other paradigms which could range from simple regression, support vector machines, decisions trees, or as will be the focus of this paper, gradient boosting.

The purpose of this work is to present the results of a benchmarking study between alternate encoding strategies for numeric and categoric features for gradient boosted tabular learning. We were particularly interested in validating the library’s default encoding strategies, and found that in both primary performance metrics of tuning duration time and model performance the current defaults under automation of categoric binarization and numeric z-score normalization demonstrated merit to serve as default encodings for the Automunge library. We also found that in addition to our default binarization, even a frequency sorted variant of ordinal encoding on average outperformed one hot encoding.


\section{Gradient Boosting}

Gradient boosting \cite{friedman2000greedy} refers to a paradigm of decision tree learning \cite{quinlan:induction} similar to random forests \cite{10.1023/A:1010933404324} but in which the optimization is boosted by recursively training an iteration’s model objective to correct the performance of the preceding iteration’s model. It is commonly implemented in practice by the XGBoost library \cite{Chen_2016} for GPU acceleration, although there are architecture variations available for different fortes, like LightGPM \cite{NIPS2017_6449f44a} which may train faster on CPU’s than XGBoost (with a possible performance tradeoff).

Gradient boosting has traditionally been found as a winning solution for tabular modality competitions on the Kaggle platform, and its competitive efficacy has even been demonstrated for more sophisticated applications like time series sequential learning when used for window based regression \cite{https://doi.org/10.48550/arxiv.2101.02118}. Recent tabular benchmarking papers have found that gradient boosting may still mostly outperform sophisticated neural architectures like transformers \cite{gorishniy2021revisiting}, although even a vanilla multi layer perceptron neural network could have capacity to outperform gradient boosting with comprehensively tuned regularizers \cite{kadra2021welltuned}. Gradient boosting can also be expected to have higher latency inference than neural networks \cite{https://doi.org/10.48550/arxiv.2110.01889}.

Conventional wisdom is that one can expect gradient boosting models to have capacity for better performance than random forests for tabular applications but with a tradeoff of increased probability of overfitting without hyperparameter tuning \cite{Howard:20}. With both more sensitivity to tuning parameters and a much higher number of parameters in play than random forest, gradient boosting usually requires more sophistication than a simple grid or random search for tuning. One compromise method available is for a sequential grid search through different subsets of parameters \cite{procedural_tuning}, although more automated and even parallelized methods are available by way of black box optimization libraries like Optuna \cite{https://doi.org/10.48550/arxiv.1907.10902}. There will likely be more improvements to come both in libraries and tuning conventions, this is an active channel of industry research.

\section{Feature Encodings}

Feature encoding refers to feature set transformations that serve to prepare the data for machine learning. Common forms of feature encoding preparations include normalizations for numeric sets and one hot encodings for categoric, although some learning libraries may accept categoric features in string representations for internal encodings. Before the advent of deep learning, it was common to supplement features with alternate representations of extracted information or to combine features in some fashion. Such practices of feature engineering are sometimes still applied in gradient boosted learning, and it was one of the purposes of these benchmarks to evaluate benefits of the practice in comparison to directly training on the data.

An important distinction of feature encodings can be considered as those that can be applied independent of an esoteric domain profile verses those that rely on external structure. An example could be the difference between supplementing a feature with bins derived based on the distribution of populated numeric values verses extracting bins based on an external database lookup. In the case of Automunge, the internal library of encodings follows almost exclusively the former, that is most encodings are based on inherent numeric or string properties and do not consider adjacent properties that could be inferred based on relevant application domains. (An exception is made for date-time formatted features which under automation automatically extract bins for weekdays, business hours, holidays, and redundantly encodes entries based on cyclic periods of different time scales \cite{time_encoding}.) The library includes a simple template for integrating custom univariate transformations \cite{anonymous_customtransforms} if a user would like to integrate into a pipeline alternate conventions.

\subsection{Numeric}

Numeric normalizations in practice are most commonly applied similar to our default of z-score `nmbr' (subtract mean and divide by standard deviation) or min-max scaling `mnmx' (converting to range between 0–1). Other variations that may be found in practice include mean scaling `mean'(subtract mean and divide by min max delta), and max scaling `mxab' (divide by feature set absolute max). More sophisticated conventions may convert a distribution shape in addition to the scale, such as the box-cox power law transformation `bxcx' \cite{https://doi.org/10.1111/j.2517-6161.1964.tb00553.x} or Scikit-Learn’s \cite{JMLR:v12:pedregosa11a} quantile transformer `qttf', which both may serve the purpose of converting a feature set to closer resemble a Gaussian distribution. In general, numeric normalizations are more commonly applied for learning paradigms other than those based on decision trees, where for example in neural networks they serve the purpose of normalizing gradient updates across features. We did find that the type of normalizations applied to numeric features appeared to impact performance, and we will present these findings below.

\subsection{Categoric}

Categoric encodings are most commonly derived in practice as a one hot encoding, where each unique entry in a received feature is translated to boolean integer activations in a dedicated column among a returned set thereof. The practice of one hot encoding has shortcomings in the high cardinality case (where a categoric feature has an excessive number of unique entries), which in the context of gradient boosting may be particularly impactful as an inflated column count impairs latency performance of a training operation — or when the feature is targeted as a classification label may even cause training to exceed memory overhead constraints. The Automunge library attempts to circumvent this high cardinality edge case in two fashions, first by defaulting to a binarization encoding instead of one hot, and second by distinguishing highest cardinality sets for a hashed encoding \cite{NIPS1988_82161242} \cite{10.1145/1553374.1553516} \cite{anonymous_customtransforms} which may stochastically consolidate multiple unique entries into a shared ordinal representation for a reduced number of unique entries.

The library default of categoric binarization `1010' refers to translating each unique entry in a received feature to a unique set of zero, one, or more boolean integer activations in a returned set of boolean integer columns. Where one hot encoding may return a set of n columns for n unique entries, binarization will instead return a smaller count of log2(n) rounded up to nearest integer. We have previously seen the practice discussed in the blogging literature, such as \cite{onehot_blog}, although without validation as offered herein.

A third common variation on categoric representations includes ordinal encodings, which simply refers to returning a single column encoding of a feature with a distinct integer representation for each unique entry. Variations on ordinal encodings in the library may sort the integer representations by frequency of the unique entry `ord3' or based on alphabetic sorting `ordl'.

Another convention for categoric sets unique to the Automunge library we refer to as parsed categoric encodings `or19' \cite{anonymous_Strings}. Parsed encodings search through tiers of string character subsets of unique entries to identify shared grammatical structure for supplementing encodings with structure derived from a training set basis. Parsed encodings are supplemented with extracted numeric portions of unique entries for additional information retention in the form received by training.

\section{Benchmarking}

The benchmarking sought to evaluate a range of numeric and categoric encoding scenarios by way of two key performance metrics, training time and model performance. Training was performed over the course of ~1.5 weeks on a Lambda workstation with AMD 3970X processor, 128Gb RAM, and two Nvidia 3080 GPUs. Training was performed by way of XGBoost tuned by Optuna with 5-fold fast cross-validation \cite{NIPS2013_f33ba15e} and early stopping criteria of 50 tuning iterations without improvement. Performance was evaluated against a partitioned 25\% validation set based on a f1 score performance metric, which we understand is a good default for balanced evaluation of bias and variance performance of classification tasks \cite{stevens2020learning}. This loop was repeated and averaged across 5 iterations and then repeated and averaged across 31 tabular classification data sets sourced from the OpenML benchmarking repository \cite{Vanschoren_2014}. Rephrasing for clarity, the reported metrics are averages of 5 repetitions of 31 data sets for each encoding type as applied to all numeric or categoric features for training. The distribution bands shown in the figures are across the five repetitions. The data sets were selected for diverse tabular classification applications with in-memory scale training data and tractable label cardinality.

We found that these benchmarks gave us comfort in the Automunge library's defaults of numeric z-score normalization and categoric binarization. An interesting result was the outperformance of categoric binarization in comparison to one-hot encoding, as the latter is commonly used in mainstream practice as a default. Further dialogue for the interpretation of the results presented in Figures \ref{fig1:test} and \ref{fig2:test} are provided in the Appendix.



\begin{figure}
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{numeric_time_031422.png}
  \caption{Numeric tuning time comparison}
  \label{fig1:sub1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{numeric_performance_031422.png}
  \caption{Numeric model performance comparison}
  \label{fig1:sub2}
\end{subfigure}
\caption{Numeric Results}
\label{fig1:test}
\end{figure}



\begin{figure}
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{categoric_time_031422.png}
  \caption{Categoric tuning time comparison}
  \label{fig2:sub1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{categoric_performance_031422.png}
  \caption{Categoric model performance comparison}
  \label{fig2:sub2}
\end{subfigure}
\caption{Categoric Results}
\label{fig2:test}
\end{figure}



\section{Conclusion}

We hope that these benchmarks may have provided some level of user comfort by validating the default encodings applied under automation by the Automunge library of z-score normalization and categoric binarization, both from a training time and model performance standpoint. If you would like to try out the library we recommend the tutorials folder found on GitHub \cite{anonymous_github} as a starting point.



































\newpage



