\section{Introduction}
\label{sec:intro}
The discovery of gene-disease associations is made difficult by incomplete knowledge of biological and physiological processes. When approaching complex, multi-gene diseases and traits, it is hard to disentangle each gene's contribution and computational biological approaches for predicting Gene-Disease Associations (GDAs) \cite{Opap2017, Piro2012} can support and address experimental approaches (e.g., Genome-Wide Association Studies -GWAS- or linkage studies, among others) which are expensive and time demanding. 

The fuzzy background of \emph{yet unknown} or \emph{truly unassociated} GDAs contributes to make the computational identification of disease genes difficult to carry out with accuracy.
In machine learning, this setting translates into the ability to identify new positive instances among a set of positive and unlabelled instances. This is known as Positive-Unlabelled (PU) learning~\cite{liu2003,Bekker2020}.
This task can be addressed through semi-supervised learning algorithms, trained using two approaches: in the first, the set of unlabelled instances is assumed to be a contaminated set of negative instances and the contamination is considered during the modelling process by weighting the data points or adding penalties on misclassification \cite{elkan2008, mordelet2014, claesen2015, ke2018}. In the specific case of gene discovery, this contamination is given by the possibility of the negative instances to contain not yet discovered positive genes. The second approach, called two-step technique, aims at relabelling the instances and then to train a supervised learning algorithm \cite{liu2003, Yang2012, Yang2014}. For instance, Yang \textit{et al.}~\cite{Yang2012} introduced a multi-class labelling procedure considering five different item labels, namely Positive ($P$), Likely Positive ($LP$), Weakly Negative ($WN$), Likely Negative ($LN$) and Reliable Negative ($RN$), based on a Markov process with restart~\cite{can2005}, widely applied in disease genes identification ~\cite{kohler2008, li2010, li2010genome}. Then, a supervised learning algorithm is trained on the relabelled data. 

In the present work we consider the multi-class labelling approach since it allows to identify a set of originally unlabelled items, namely the $LP$ set, whose features are close to that of Positive items. This translates into the identification of a small set of genes which is more likely to contain true positive instances, providing therefore a set of new candidates disease genes.

We here propose several significant modifications of the method in  \cite{Yang2012} which are related to the distance matrix defining the Markov process and to the selection of the different classes. Some of these modifications were needed in order to apply the method to general PU data sets, while others were proposed in order to make the process of class formation more rigorous and, at the same time, flexible.
The approach considered, being a two-step technique, is based on the separability and smoothness assumptions \cite{Bekker2020}, which require that the features should be able to distinguish between positive and negative instances, and that instances which have similar features are more likely to have the same label. Therefore, as further contribution, we propose the use of specific features, one of them introduced for the first time in this work, based on protein-protein interaction (PPI) data, which provide a ranking of genes with respect to disease seed genes. The introduction of these new features is fundamental because the use of classical topological network measures provides a ranking of genes with respect to the entire network, without considering information carried by seed genes. The network-informed adaptive positive-unlabeled (\textsc{NIAPU}\xspace) framework is therefore formed by two components: the network diffusion and biology-informed topological (NeDBIT) features, and the adaptive positive-unlabelled (\textsc{APU}\xspace) label propagation strategy.

\section{Materials and methods}
\label{sec:method}

\subsection{Data sources and preprocessing}
The methodology here proposed exploits two types of data, i.e., reliable PPIs and known GDAs data. 
PPI data provide valuable biological knowledge for the identification of undiscovered disease genes \cite{Piro2012,Doncheva2012,Silverman,TIERI2019805}. Human PPI data were gathered from the BioGRID~\cite{stark2006biogrid} dataset version 4.4.206 (the latest available at the time of the analysis, Jan. 2022) and were used to build the human PPI network, a.k.a. the human interactome. BioGRID is among the most complete protein interaction repositories with manually curated freely available data. It has been preferred over other PPI datasets such as the Human Reference Interactome (HuRI) \cite{Huri} and the STRING database \cite{String} because of its balance between completeness and reliability. Data cleaning on the interactome were performed by choosing \emph{Homo sapiens} genes (organism ID 9606), which yielded a network consisting of 19764 genes and 678932 non redundant, undirected interactions, from which only the largest connected component has been selected (19761 genes and same number of links -- \texttt{Supplementary file 1}). 

GDAs were derived from DisGeNET \cite{pinero2016disgenet, pinero2020disgenet}, a discovery platform containing one of the largest publicly available collections of genes and variants associated to human diseases. Five pathologies were considered, namely: malignant neoplasm of breast (disease ID C0006142, 1074 genes), schizophrenia (C0036341, 883 genes), liver cirrhosis (C0023893, 774 genes), colorectal carcinoma (C0009402, 702 genes) and malignant neoplasm of prostate (C0376358, 616 genes). Criteria for selection of these diseases were the highest cardinality of GDAs in the DisGeNET curated dataset\footnote{\url{https://www.disgenet.org/browser/0/0/0/0/source\_\_CURATED/\_b./}, downloaded in Feb. 2022}, so to assure coherent dataset dimensions for the ML tasks. After performing additional cleaning steps described in \texttt{Supplementary file 2}, we ended up having a set of seed genes, denoted by $\Sigma$, with a vector of scores for each disease. In particular, we have 1025 genes for disease C0006142, 832 for C0036341, 747 for C0023893, 672 for C0009402 and 606 for C0376358. 

\subsection{Multi-class labelling: Adaptive PU (\textsc{APU}\xspace) label propagation and classification}\label{sec:APU}

The \textsc{APU}\xspace algorithm consists of a multi-class labelling procedure that relies on the labels introduced in~\cite{Yang2012}: $P$, $LP$, $WN$, $LN$, $RN$. Positive ($P$) instances mirror known disease genes, Reliable Negative ($RN$) instances represent the genes whose features are the furthest from the average features in the $P$ sample, while the remaining labels are assigned through a Markov process with restart \cite{can2005}.

The hallmarks of the method proposed here are the construction of a new Markov diffusion process leading to a more effective labelling procedure. Starting from the symmetrical matrix representing the distance between the features of the elements, the Markov process is implemented so as to ensure the conservation of the total weights (GDA score) present in the $P$ and $RN$ initial distribution. Moreover, the class selection has been made flexible by using an adaptable quantile separation instead of fixed thresholds. These characteristics have been implemented in order to make the process of class formation more rigorous and, at the same time, more flexible, so as to be easily adapted to different settings, datasets and needs.

For what concerns the labelling process, let $V$ be a set whose generic $i^{th}$ element $v_{i=1,...,n}$ is characterised by the couple $\left(\boldsymbol{x}_{i},y_{i}\right)$ where $\boldsymbol{x}_{i}\in \left[0,1\right]^{d}$ represent the features vector, and $y_{i}\in \left\{0,1\right\}$ the initial label. 
The \textsc{APU}\xspace algorithm is defined by the following steps:\\
{\it \textbf{Step 1}:}
compute the matrix $\boldsymbol{W}$, whose elements $w_{ij}$, with values between 0 and 1, represents the similarity score between elements $i$ and $j$, defined as follows
\begin{equation}
w_{ij} = \begin{cases}1- \frac{e_{ij}-m}{M-m} &\mbox{if } i\neq j\\ 1 & \mbox{otherwise} 
\end{cases}\nonumber
\end{equation}
where $e_{ij}$ is the euclidean distance between the features of elements $i$ and $j$, 
$e_{ij}=\sum_k \left (x^k_i-x^k_j\right)^2$, $m=\mbox{min}_{ij}\left\{e_{ij}\right\}$ and $M=\mbox{max}_{ij}\left\{e_{ij}\right\}$. The present definition of $\boldsymbol{W}$ preserve the symmetry of the similarity score between the elements $i$ and $j$;\\
{\it \textbf{Step 2}:}
Compute the reduced matrix $\boldsymbol{W}_{r}$ as follows 
\begin{equation}
w_{r, ij}=\begin{cases}w_{ij} &\mbox{if } w_{ij}>s\\ 0 & \mbox{otherwise} 
\end{cases}.\nonumber
\end{equation}
where all the elements of $\boldsymbol{W}$ below the threshold score $s$ are set to zero. The threshold $s$ is computed as a given quantile of the distribution of the elements in the matrix $\boldsymbol{W}$ (for example the quantile $0.05$ can be used as threshold), in order to exclude from the label propagation process all the links between poorly related elements. To obtain a proper Markov process with restart that will be used for the label propagation process, the matrix $\boldsymbol{W}_{r}$ must be normalized as follows
\begin{equation}
\boldsymbol{W}_{n}= \boldsymbol{D}^{-1}\boldsymbol{W}_{r}\nonumber
\end{equation} 
where $\boldsymbol{D}$ is the diagonal matrix with elements $D_{ii}=\sum_{j}w_{r, ij}$.\\
{\it \textbf{Step 3}:}
Recalling that $P$ is the set of seed genes and $|P|$ its cardinality, the reliable negative $RN$ elements are defined as those genes whose features are the furthest from the average features $\boldsymbol{\hat x}$ of the $P$ set, namely ${\hat x}^k=1/|P|\sum_{i\in P}x^k_i$. The $RN$ genes are thus extracted using the ranking of genes with the most distant features (using euclidean distance) from the average feature of seed genes. In order to create a balanced sample, exactly the $|P|$ most distant genes from $\boldsymbol{\hat x}$ were selected. We initialized the label propagation process with the vector $G_{0}$, that contains the initial weights, which are set as 1 for elements in $P$, $-1$ for elements in $RN$, and 0 for the remaining elements. When needed, a larger number of $RN$ genes can be selected. In this case, the initial value of the $RN$ genes in the vector $G_{0}$ must be set to $-\vert P\vert/\vert RN\vert$, so that the two distributions of positive and negative values (associated to $P$ and $RN$ sets, respectively) are balanced in ${G}_{0}$, with the sum of its elements equal to zero.\\
{\it \textbf{Step 4}:}
Label propagation. Starting from the vector $G_0$, containing the initial weights, a Markov process with restart is introduced in order to obtain the stationary distribution of the process of propagating labels:
\begin{equation}
G_{r}=\left(1-\alpha\right)\boldsymbol{W^t}_{n}G_{r-1}+\alpha G_{0}\nonumber
\end{equation}
where the parameter $\alpha$ is usually set to $0.8$ \cite{Yang2012,li2010genome}. Such a process guarantees the conservation of the sum of the elements of $G_{r}$ for each $r$, and that the positive and negative values diffuse to their neighbors (with probability $(1-\alpha)$) and restart from the initial distribution (with probability $\alpha$) till a stationary distribution is considered reached ($\vert G_{r}-G_{r-1}\vert < 10^{-6}$). The asymptotic vector is called $G_{\infty}$.  \\
{\it \textbf{Step 5}:}
The ranking induced by $G_{\infty}$ vector is used to assign the remaining labels. That is, by selecting only those elements that belong neither to $P$ nor to $RN$ (i.e., whose $G_0$ values is 0), the values of the asymptotic distribution of those elements are sorted and the ranking of the corresponding elements is used to form the remaining classes, $LP$, $WN$ and $LN$. A simple rule is to divide the ranking into three equal parts and identify $LP$s with the first third (quantiles from $0.0$ to $0.333\Bar{3}$ of the distribution), $WN$s with the second third (quantile from $0.333\Bar{3}$ to $0.666\Bar{6}$ of the distribution) and $LN$s with the third third (quantile from $0.666\Bar{6}$ to $1.0$ of the distribution). However, depending on the type of analysis and the problem addressed, any division of the ranking can be considered acceptable. This type of classification is very flexible and can be easily adapted to different dataset and needs, not suffering from the problems encountered when using a single rigid threshold as in \cite{Yang2012}, e.g. empty classes or highly unbalanced classes.\\
{\it \textbf{Step 6}:} Classification. A ML classifier is trained over the data set containing the new labelling. Three different ML algorithms have been used: Random Forest (RF) \cite{breiman2001}, Support Vector Machine (SVM) \cite{cortes1995,drucker1997} and Multilayer Perceptron (MLP) \cite{hastie2001} (details in \texttt{Supplementary file 2}).

\subsection{Network diffusion and biology-informed topological (NeDBIT) features}\label{sec:features}
The \textsc{APU}\xspace algorithm can be used with any set of features, but its proper functioning is based on separability and smoothness assumptions, i.e., features should be able to distinguish between positive and negative instances, and instances with similar features must be, with good probability, in the same class. The NeDBIT features here introduced include two network diffusion-based features, namely heat diffusion and balanced diffusion, and two biology-informed topological metrics, namely NetShort and NetRing. Network diffusion methods are widely used in disease gene discovery processes \cite{Lancour,Picart,Janyasupab}. Here we coupled network diffusion methods and innovative topological-based features in order to make the most of the combined predictive power of both approaches. Moreover, for all the selected features we use the GDA score, denoted by $s_{i}$, of the seed genes for the particular disease under investigation. In this way, both the network diffusion methods and the topological measures provide features that are certainly more relevant to the disease under investigation than considering all seed genes with the same weight.\\

\noindent
{\bf Heat diffusion features}\\
The first feature is obtained by using a heat diffusion process over networks, that is among the most used process for disease gene prioritisation and prediction (see \cite{diffusioncyto} and references therein). Starting with a distribution of weights, with positive values only on the seed genes, their evolution is determined by using the diffusion equation on graph \cite{pmid20840752}
\begin{equation}
{\mathbf s}'(t)+{\mathbf L} {\mathbf s}(t)=0\,,
\label{eqheatdiff}
\end{equation}
where ${\mathbf L}$ is the Graph Laplacian matrix, ${\mathbf L}={\mathbf D} - {\mathbf A}$, where ${\mathbf D}_{ij} = k_i\delta_{ij}$ is the diagonal matrix with the degree of nodes on the diagonal and ${\mathbf A}$ is the adjacency matrix of the PPI. The weights at time $t$ are given by the formal solution of Eq.~(\ref{eqheatdiff})
\begin{equation}
{\mathbf s}(t)=\exp\left ( -{\mathbf L} t \right){\mathbf s}(0)
\label{solheatdiff}
\end{equation}
where $\exp$ is the exponential of the matrix. Regarding the initial distribution of weights, we assign ${\mathbf s}_i(0) = s_i$ for seed genes in $\Sigma$ and 0 otherwise, where $s_i$ is the GDA score.\\ 

\noindent
{\bf Balanced diffusion features}\\
The balanced diffusion features are obtained by using another version for the Graph Laplacian matrix, i.e., ${\mathbf L_b}={\mathbf I} - {\mathbf D}^{-1}{\mathbf A}.$ This form of the graph diffusion operator differs from the heat diffusion in the fact that the operator ${\mathbf L}$ diffuses the same amount of score for each link, whereas ${\mathbf L_b}$ diffuses the same amount of score for each node. This implies a different short time behaviour of the diffusion process on the graph. In the first case, the diffusion process proceeds faster from well-connected nodes with high weight, in fact each link carries the same amount of weight and therefore those nodes are drained quickly, whereas in
the second case, the same amount of weight diffuses from each node and therefore there is no great difference between well-connected and poorly connected nodes with regard to the outflow of weight.

Similarly to the previous dynamics, the evolution of the weight is determined by using the diffusion equation on graph
\begin{equation}
{\mathbf s}'(t)+\hat L_b {\mathbf s}(t)=0\,.
\label{eqheatdiffb}
\end{equation}

Also in this case the weights at time $t$ are given by the formal  solution of Eq.~\eqref{eqheatdiffb}
\begin{equation}
{\mathbf s}(t)=\exp\left ( -\hat L_B t \right){\mathbf s}(0)
\label{solheatdiffb},\nonumber
\end{equation}
and the initial weights ${\mathbf s}(0)$ are the GDA scores for seed genes in $\Sigma$ and zero otherwise, as defined for the heat diffusion. \\

\noindent
{\bf NetShort}\\
The NetShort measure \cite{white2003algorithms} is based on the idea that a generic node is topologically important for a disease if a large number of seed nodes must be traversed to reach it.
Therefore, the measure is implemented computing the weighted average shortest paths reaching the node under consideration, where each link is weighted favouring links connecting seed nodes and penalising links connecting non-seed nodes.
The links are weighted using the GDA score through the following formula 
\begin{equation}
w_{ij}={\mathbf A}_{ij} \frac{2}{\tilde s_i+\tilde s_j}, \quad \mbox{where} \quad \tilde s_i =\left \{
	\begin{matrix} 
		{\displaystyle \frac{s_i}{\max{s}}} \qquad\mbox{if}\qquad i\in \Sigma \vspace{0.2truecm}\\
		\alpha {\displaystyle \frac{\min{s}}{\max{s}}} \qquad\mbox{if}\qquad i\notin \Sigma
        \end{matrix} \right \},
\label{renormscore}
\end{equation}
and $\min{s}$ and $\max{s}$ are the minimum and the maximum of the scores of seed nodes, and $\alpha$ is the penalization parameter given to non-seed nodes. We use $\alpha=0.5$ so that all non-seed nodes have normalised score
$\tilde s_i =0.5\frac{\min s}{\max s}$ while seed nodes have normalised score $\frac{\min s}{\max s}\leq \tilde s_i \leq 1$.

Finally, the NetShort measure of non-seed node $i$ is defined as
\begin{equation}
NS_{i}=\sum_{j\neq i} \frac{1}{d_{ij}},
\label{netshort}\nonumber
\end{equation}
where $d_{ij}$ is the length of the weighted shortest path from $i$ to $j$.\\

\noindent
{\bf NetRing}\\
The NetRing measure, introduced for the first time in this work, is based on the concept of ring structure~\cite{baronchelli2006ring} generalized to a set of seed nodes. Starting from seed nodes, a partition of the graph in sub-graphs, or rings, is introduced with the following property
\begin{equation}
R(l)\equiv \left \{j\in V \,\,\,|\,\,\,\min_{i\in \Sigma}l_{ij}=l\right \}.
\label{ring}
\end{equation}
where $l_{ij}$ is the (unweighted) length of the shortest path from $i$ to $j$. $R(l)$ contains all the non-seed nodes with a minimal distance $l$ from, at least, one seed node. From the definition follows that $R(0)\equiv\Sigma$, $R(l_1)\cap R(l_2)={\emptyset}$ if $l_1\neq l_2$ and $V=\cup_{l=0}^{L} R(l)$, where $L$ is the highest value of the minimal distance from non-seed nodes to seed nodes. Additional important considerations about the NetRing measure can be found in \texttt{Supplementary file 2}.

This measure has been implemented via a two-steps procedure, i) an initial rank for each seed node is defined by using the GDA score: $\hat r_i=1 - \frac{s_i}{\max{s}}$, ii) the rank of the seed nodes is then computed by a convex combination of two terms, the initial rank $\hat r_i$, and the average of the initial rank of the neighbors of the node
\begin{equation}
r_i(0) = \alpha \hat r_i + 
         (1-\alpha) \frac{1}{k_i}\sum_{j|A_{ij}\neq 0}\hat r_j\,.
\label{netring0}
\end{equation}
In this way the seed nodes that have many other seed nodes as neighbors have the highest ranks.  

Then, the rank of non seed-nodes in the ring $l$, is defined by the following formula:
\begin{equation}
r_i(l) = l + \frac{1}{k_i}\left( \sum_{j\in R_i(l-1)} \left(r_j(l-1) - (l-1)\right) +
\sum_{j\in O_i} \hat r_j \right) \,,
\label{netringl}
\end{equation}
namely, it is obtained by summing two terms: 1) the level of the ring, $l$, 2) the average rank of the neighbors of node $i$ separated in two terms: the first is related to nodes in the previous ring $l-1$ (that we collected in the set $R_i(l-1)=\left \{j\in R(l-1) | A_{ij}\neq 0\right\}$) and the second related to the other neighbors of the node $i$, that can be in the same ring $l$ or in the next ring $l+1$ (we collected them in the set $O_i=\left \{j\notin R(l-1) | A_{ij}\neq 0\right\}$).  The quantity $\left(r_j(l-1) - (l-1)\right)$ is the rank of node $j$ inside the ring $l - 1$, in order to make a fair ranking with neighbors in higher rings with initial rank $\hat r_j$ (equal to 1). This measure rewards non-seed nodes that are close to high-ranking nodes in the previous ring and that are linked with a few nodes of the same or higher ring.


\section{Results}
\label{sec:results}

We tested the performances of \textsc{NIAPU}\xspace on the five aforementioned selected disease datasets, testing the effectiveness of both the NeDBIT features and the \textsc{APU}\xspace algorithm in several settings. A visual overview of our workflow can be grasped in Figure~\ref{fig:NIAPU-workflow}.




 

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{images/NIAPU-workflow.png}
    \caption{\small{The complete \textsc{NIAPU}\xspace pipeline. PPI and GDAs are used to obtain a disease-related network. Features are extracted (Section~\ref{sec:features}) and \textsc{APU}\xspace is applied (Section~\ref{sec:APU}) to assign new labels to train ML algorithms for the final gene classification. The new labels can be used for disease gene-discovery purposes (Section~\ref{sec:gene-discovery}). Network image taken from \url{http://snap.stanford.edu/deepnetbio-ismb/ipynb/diseasome.png}}}
    \label{fig:NIAPU-workflow}
\end{figure}

\subsection{NeDBIT vs TFO classification performances}
\label{subsec:5foldCV}

We tested the classification performances of three ML algorithms with the multi-class labelling provided by \textsc{APU}\xspace. Moreover, we compared the effectiveness of the NeDBIT features against features based on topological, functional and ontological (TFO) information (see \texttt{Supplementary files 2, and 3}), considering binary classification and multi-class \textsc{APU}\xspace classification. 

To test the ML classification the five GDA datasets were split into training set (70\%) and test set (30\%), keeping class balance.

Figure~\ref{fig:binary-class} and~\ref{fig:multi-class} show the confusion matrices over the test set in the cases of respectively binary (positive vs unlabelled instances) and multi-class classification exploiting both TFO and the newly introduced NeDBIT features, for malignant neoplasm of breast. Analogous results can be found in \texttt{Supplementary file 2} for the rest of the diseases, along with the results of the five-fold cross-validation over the training set. 

\begin{figure}
        \vspace{-15mm}
            \centering
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__MLP_normalized_binary_central_GO.png}
                 \subcaption{\footnotesize{MLP + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                
                 \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__MLP_normalized_binary.png}
                 \subcaption{\footnotesize{MLP + NeDBIT features}}
            \end{minipage}
            
            
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__RF_normalized_binary_central_GO.png}
                 \subcaption{\footnotesize{RF + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__RF_normalized_binary.png}
                 \subcaption{\footnotesize{RF + NeDBIT features}}
            \end{minipage} \vfill
            
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__SVM_normalized_binary_central_GO.png}
                \subcaption{\footnotesize{SVM + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__SVM_normalized_binary.png}
                \subcaption{\footnotesize{SVM + NeDBIT features}}
            \end{minipage}
            \caption{\footnotesize{CMs for binary classification on malignant neoplasm of breast. The newly defined NeDBIT features properly separate the positive $P$ class from the unlabelled $U$ genes, while TFO features completely fail in such task.}}
            \label{fig:binary-class}
\end{figure}


\begin{figure}
        \vspace{-15mm}
            \centering
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__MLP_normalized_multiclass_central_GO.png}
                 \subcaption{\footnotesize{MLP + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                
                 \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__MLP_normalized_multiclass.png}
                 \subcaption{\footnotesize{MLP + NeDBIT features}}
            \end{minipage}
            
    \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__RF_normalized_multiclass_central_GO.png}
                 \subcaption{\footnotesize{RF + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__RF_normalized_multiclass.png}
                 \subcaption{\footnotesize{RF + NeDBIT features}}
            \end{minipage} \vfill
            
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__SVM_normalized_multiclass_central_GO.png}
                \subcaption{\footnotesize{SVM + TFO features}}
            \end{minipage}\hfill
            \begin{minipage}{0.40\textwidth}
                \centering
                \includegraphics[width=\textwidth]{images/C0006142_Malignant_neoplasm_of_breast__SVM_normalized_multiclass.png}
                \subcaption{\footnotesize{SVM + NeDBIT features}}
            \end{minipage}
            \caption{\footnotesize{CMs for multi-class classification on malignant neoplasm of breast. The \textsc{APU}\xspace labelling along with the newly defined NeDBIT features allows for a better and clear distinction of both the $P$ class and the pseudo-classes.}}
            \label{fig:multi-class}
\end{figure}

\subsection{\textsc{NIAPU}\xspace performances in disease gene identification}
\label{subsec:LP}

We tested the ability of \textsc{NIAPU}\xspace to identify new candidate genes. We performed a validation by excluding the 20\% of seed genes, setting them as unlabelled, in the NeDBIT features computation and in the label propagation process. We repeated the procedure for five times with, such that all the seed genes were set as unlabelled once and only once. We wanted to inspect whether \textsc{NIAPU}\xspace was able to properly classify the removed positive genes as $LP$. For brevity, the results for malignant neoplasm of breast only are reported in Table~\ref{tab:lp-APU} (other diseases in \texttt{Supplementary file 2}). 

\begin{table}[!ht]
\centering
\caption{\small{Labelling of 205 unlabelled seed genes by \textsc{NIAPU}\xspace for malignant neoplasm of breast. Results are intended as average with standard deviation over the five runs.}}
\label{tab:lp-APU}
\begin{small}
\begin{tabular}{@{}lllllll@{}}
\toprule
label & \% genes & \# genes     & GDA mean        & GDA median      & \multicolumn{2}{l}{GDA mode}       \\ \midrule
$LP$    & 43.122 $\pm$ 0.436 & 88.4 $\pm$ 0.894 & 0.383 $\pm$ 0.019 & 0.343 $\pm$ 0.025 & \multicolumn{2}{l}{0.32 $\pm$ 0.045} \\
$WN$    & 27.805 $\pm$ 2.365 & 57.0 $\pm$ 4.848 & 0.347 $\pm$ 0.011 & 0.32 $\pm$ 0.01   & \multicolumn{2}{l}{0.3 $\pm$ 0.0}    \\
$LN$    & 16.683 $\pm$ 4.784 & 34.2 $\pm$ 9.808 & 0.326 $\pm$ 0.009 & 0.304 $\pm$ 0.005 & \multicolumn{2}{l}{0.3 $\pm$ 0.0}    \\
$RN$   & 12.39 $\pm$ 3.191 & 25.4 $\pm$ 6.542 & 0.324 $\pm$ 0.015 & 0.3 $\pm$ 0.0     & \multicolumn{2}{l}{0.3 $\pm$ 0.0}    \\ \bottomrule
\end{tabular}
\end{small}
\end{table}

By applying ML methods in this setting, with a reduced seed genes set, we also tested whether their performances were consistent with the ones shown in section \ref{subsec:5foldCV}. 
The results for malignant neoplasm of breast can be appreciated in Table~\ref{tab:ML-class-kfold-bin} and~\ref{tab:ML-class-kfold-multi}.

\begin{table}[!ht]
\centering
\caption{\small{Classification scores as average values and standard deviations over five runs for the binary setting.}}
\label{tab:ML-class-kfold-bin}
\begin{small}
\noindent\makebox[\textwidth]{
\begin{tabular}{@{}llll@{}}
\toprule
                 & precision       & recall      & f1-score        \\ \midrule
\textbf{MLP}                                      \\ 
$P$                     & 0.999 $\pm$ 0.003 & 1.0 $\pm$ 0.0 & 0.999 $\pm$ 0.002 \\
$U$                     & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\

macro avg             & 0.999 $\pm$ 0.002 & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.001   \\
weighted avg          & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\ \midrule
accuracy              & 1.0 $\pm$ 0.0                 \\\midrule
\textbf{RF}                                       \\ 
$P$                     & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\
$U$                     & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\

macro avg             & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\
weighted avg          & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\ \midrule
accuracy              & 1.0 $\pm$ 0.0                \\\midrule
\textbf{SVM}                                       \\ 
$P$ & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\
$U$ & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\

macro avg             & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\
weighted avg          & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0 & 1.0 $\pm$ 0.0     \\  \midrule
accuracy              & 1.0 $\pm$ 0.0                \\\bottomrule
\end{tabular}
}
\end{small}
\end{table}

\begin{table}[!ht]
\centering
\caption{\small{Classification scores as average values and standard deviation over five runs for the multi-class setting.}}
\label{tab:ML-class-kfold-multi}

\begin{small}
\noindent\makebox[\textwidth]{
\begin{tabular}{@{}llll@{}}
\toprule
             & precision            & recall          & f1-score        \\ \midrule
\textbf{MLP} &                      &                 &                 \\
$P$            & 0.993 $\pm$ 0.005      & 1.0 $\pm$ 0.0     & 0.997 $\pm$ 0.002 \\
$LP$           & 0.991 $\pm$ 0.002      & 0.995 $\pm$ 0.002 & 0.993 $\pm$ 0.001 \\
$WN$           & 0.993 $\pm$ 0.002      & 0.987 $\pm$ 0.002 & 0.99 $\pm$ 0.001  \\
$LN$           & 0.987 $\pm$ 0.006      & 0.719 $\pm$ 0.042 & 0.831 $\pm$ 0.03  \\
$RN$           & 0.782 $\pm$ 0.026      & 0.994 $\pm$ 0.004 & 0.875 $\pm$ 0.017 \\
macro avg    & 0.949 $\pm$ 0.006      & 0.939 $\pm$ 0.009 & 0.937 $\pm$ 0.009 \\
weighted avg & 0.94 $\pm$ 0.007       & 0.926 $\pm$ 0.011 & 0.924 $\pm$ 0.012 \\ \midrule
accuracy     & 0.926 $\pm$ 0.011      &                 &                 \\ \midrule
\textbf{RF}  & \multicolumn{1}{c}{} &                 &                 \\
$P$            & 1.0 $\pm$ 0.0          & 1.0 $\pm$ 0.0     & 1.0 $\pm$ 0.0     \\
$LP$           & 0.993 $\pm$ 0.001      & 0.992 $\pm$ 0.004 & 0.993 $\pm$ 0.001 \\
$WN$           & 0.989 $\pm$ 0.003      & 0.987 $\pm$ 0.003 & 0.988 $\pm$ 0.002 \\
$LN$           & 0.989 $\pm$ 0.002      & 0.991 $\pm$ 0.001 & 0.99 $\pm$ 0.001  \\
$RN$           & 0.995 $\pm$ 0.001      & 0.995 $\pm$ 0.003 & 0.995 $\pm$ 0.001 \\
macro avg    & 0.993 $\pm$ 0.0        & 0.993 $\pm$ 0.0   & 0.993 $\pm$ 0.0   \\
weighted avg & 0.992 $\pm$ 0.001      & 0.992 $\pm$ 0.001 & 0.992 $\pm$ 0.001 \\ \midrule
accuracy     & 0.992 $\pm$ 0.001      &                 &                 \\ \midrule
SVM          &                      &                 &                 \\
$P$            & 0.997 $\pm$ 0.004      & 1.0 $\pm$ 0.0     & 0.999 $\pm$ 0.002 \\
$LP$           & 0.866 $\pm$ 0.009      & 0.614 $\pm$ 0.034 & 0.718 $\pm$ 0.022 \\
$WN$           & 0.467 $\pm$ 0.01       & 0.905 $\pm$ 0.01  & 0.616 $\pm$ 0.008 \\
$LN$           & 0.99 $\pm$ 0.022       & 0.027 $\pm$ 0.032 & 0.051 $\pm$ 0.06  \\
$RN$           & 0.756 $\pm$ 0.027      & 1.0 $\pm$ 0.0     & 0.861 $\pm$ 0.017 \\
macro avg    & 0.815 $\pm$ 0.004      & 0.709 $\pm$ 0.004 & 0.649 $\pm$ 0.01  \\
weighted avg & 0.775 $\pm$ 0.004      & 0.645 $\pm$ 0.004 & 0.572 $\pm$ 0.013 \\ \midrule
accuracy     & 0.645 $\pm$ 0.004      &                 &                 \\ \bottomrule
\end{tabular}
}
\end{small}
\end{table}

\subsection{\textsc{NIAPU}\xspace vs other disease genes identification tools}
\label{sec:gene-discovery}

\textsc{NIAPU}\xspace can be employed for disease gene discovery, in which candidate disease genes are those that are classified in the $LP$ class. We considered the same validation aforementioned (20\% of seed genes stripped of the $P$ label). In such setting, the algorithm should be able to label as $LP$ many of the unlabelled seed genes. We considered the ranking retrieved by \textsc{NIAPU}\xspace at different quantile thresholds and we compared the performances of \textsc{NIAPU}\xspace against known gene discovery algorithms (Table~\ref{tab:gene-discovery-res}), namely DIAMOnD~\cite{pmid25853560}, Markov clustering (MCL)~\cite{pmid11917018, sun2011prediction} and random walk with restart (RWR)~\cite{kohler2008, valdeolivas2019random}. 

\begingroup
\setlength{\LTleft}{-20cm plus -1fill}
\setlength{\LTright}{\LTleft}
\begin{scriptsize}
\begin{longtable}{
>{\columncolor[HTML]{F2F2F2}}l 
>{\columncolor[HTML]{F2F2F2}}l ccccc}
\caption{\small{Gene discovery comparison results over five repetitions as average and standard deviation. We report the disease IDs:  C0006142 (malignant neoplasm of breast), C0009402 (colorectal carcinoma), C0023893 (liver cirrhosis), C0376358 (malignant neoplasm of prostate) and C0036341 (schizophrenia). Results are stratified by the top K retrieved genes, which correspond to the first K genes found by any method. N corresponds to the number of known associated genes and it is different for each disease (see Section~\ref{sec:intro}}).}
\label{tab:gene-discovery-res} \\

\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}}                                      & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   25}}                                                                                                                                                                                                                               \\
\multicolumn{2}{l}{\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{}}}           & \cellcolor[HTML]{F2F2F2}\textbf{C0006142} & \cellcolor[HTML]{F2F2F2}\textbf{C0009402} & \cellcolor[HTML]{F2F2F2}\textbf{C0023893} & \cellcolor[HTML]{F2F2F2}\textbf{C0376358} & \cellcolor[HTML]{F2F2F2}\textbf{C0036341} \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.024 $\pm$ 0.048                                                 & 0.016 $\pm$ 0.02                                          & 0.016 $\pm$ 0.02                                     & 0.056 $\pm$ 0.032                                                   & 0.04 $\pm$ 0.044                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & 0.003 $\pm$ 0.006                                                 & 0.003 $\pm$ 0.004                                         & 0.003 $\pm$ 0.003                                    & 0.012 $\pm$ 0.007                                                   & 0.006 $\pm$ 0.007                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.176 $\pm$ 0.078}                                                 & \textbf{0.08 $\pm$ 0.051}                                          & \textbf{0.032 $\pm$ 0.03}                                     & \textbf{0.144 $\pm$ 0.074}                                                   & \textbf{0.088 $\pm$ 0.059}                                 \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & \textbf{0.021 $\pm$ 0.01}                                                  & \textbf{0.015 $\pm$ 0.009}                                         & \textbf{0.005 $\pm$ 0.005 }                                   & \textbf{0.03 $\pm$ 0.015}                                                   & \textbf{0.013 $\pm$ 0.009}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.032 $\pm$ 0.03                                                  & 0.016 $\pm$ 0.02                                          & 0.016 $\pm$ 0.02                                     & 0.048 $\pm$ 0.059                                                   & 0.048 $\pm$ 0.03                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.004 $\pm$ 0.004                                                 & 0.003 $\pm$ 0.004                                         & 0.003 $\pm$ 0.003                                    & 0.01 $\pm$ 0.012                                                    & 0.007 $\pm$ 0.005                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.04 $\pm$ 0.025                                                  & 0.016 $\pm$ 0.032                                         & 0.024 $\pm$ 0.032                                    & 0.032 $\pm$ 0.03                                                    & 0.056 $\pm$ 0.02                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.005 $\pm$ 0.003                                                 & 0.003 $\pm$ 0.006                                         & 0.004 $\pm$ 0.005                                    & 0.007 $\pm$ 0.006                                                   & 0.008 $\pm$ 0.003                                  \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   50}}                                                                                                                                                                                                                               \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.024 $\pm$ 0.023                                                 & 0.036 $\pm$ 0.029                                         & 0.02 $\pm$ 0.018                                     & 0.056 $\pm$ 0.015                                                   & 0.052 $\pm$ 0.02                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & 0.006 $\pm$ 0.006                                                 & 0.013 $\pm$ 0.011                                         & 0.007 $\pm$ 0.006                                    & 0.023 $\pm$ 0.006                                                   & 0.016 $\pm$ 0.006                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} &\textbf{ 0.108 $\pm$ 0.048}                                                 & \textbf{0.088 $\pm$ 0.041 }                                        & \textbf{0.016 $\pm$ 0.015}                                    & \textbf{0.112 $\pm$ 0.055}                                                   & \textbf{0.06 $\pm$ 0.031}                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & \textbf{0.026 $\pm$ 0.012}                                                 & \textbf{0.033 $\pm$ 0.015   }                                      & \textbf{0.005 $\pm$ 0.005}                                    & \textbf{0.046 $\pm$ 0.022}                                                   & \textbf{0.018 $\pm$ 0.009}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.024 $\pm$ 0.015                                                 & 0.012 $\pm$ 0.016                                         & 0.008 $\pm$ 0.01                                     & 0.036 $\pm$ 0.034                                                   & 0.028 $\pm$ 0.016                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.006 $\pm$ 0.004                                                 & 0.004 $\pm$ 0.006                                         & 0.003 $\pm$ 0.003                                    & 0.015 $\pm$ 0.014                                                   & 0.008 $\pm$ 0.005                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.028 $\pm$ 0.027                                                 & 0.012 $\pm$ 0.016                                         & 0.016 $\pm$ 0.023                                   & 0.024 $\pm$ 0.023                                                   & 0.032 $\pm$ 0.016                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.007 $\pm$ 0.007                                                 & 0.004 $\pm$ 0.006                                         & 0.005 $\pm$ 0.008                                    & 0.01 $\pm$ 0.01                                                     & 0.01 $\pm$ 0.005                                   \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   N/10}}                                                                                                                                                                                                                             \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.035 $\pm$ 0.018                                                 & 0.036 $\pm$ 0.032                                         & \textbf{0.019 $\pm$ 0.011}                                    & 0.072 $\pm$ 0.025                                                   & \textbf{0.051 $\pm$ 0.016}                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & 0.018 $\pm$ 0.009                                                 & 0.018 $\pm$ 0.016                                         & \textbf{0.009 $\pm$ 0.005}                                    & 0.036 $\pm$ 0.012                                                   & \textbf{0.025 $\pm$ 0.008}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.08 $\pm$ 0.036}                                                  & \textbf{0.078 $\pm$ 0.038}                                         & 0.016 $\pm$ 0.005                                    & \textbf{0.1 $\pm$ 0.045 }                                                    & 0.048 $\pm$ 0.013                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & \textbf{0.04 $\pm$ 0.018}                                                  & \textbf{0.039 $\pm$ 0.019}                                         & 0.008 $\pm$ 0.003                                    & \textbf{0.049 $\pm$ 0.022}                                                   & 0.024 $\pm$ 0.007                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.022 $\pm$ 0.011                                                 & 0.021 $\pm$ 0.012                                         & 0.005 $\pm$ 0.007                                    & 0.037 $\pm$ 0.029                                                   & 0.031 $\pm$ 0.021                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.011 $\pm$ 0.006                                                 & 0.01 $\pm$ 0.006                                          & 0.003 $\pm$ 0.003                                    & 0.018 $\pm$ 0.014                                                   & 0.016 $\pm$ 0.011                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.027 $\pm$ 0.017                                                 & 0.015 $\pm$ 0.016                                         & 0.014 $\pm$ 0.021                                    & 0.02 $\pm$ 0.019                                                    & 0.024 $\pm$ 0.011                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.014 $\pm$ 0.008                                                 & 0.007 $\pm$ 0.008                                         & 0.007 $\pm$ 0.01                                     & 0.01 $\pm$ 0.01                                                     & 0.012 $\pm$ 0.005                                  \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   N/5}}                                                                                                                                                                                                                              \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.044 $\pm$ 0.014                                                 & 0.028 $\pm$ 0.019                                         & \textbf{0.027 $\pm$ 0.011}                                    & 0.051 $\pm$ 0.016                                                   & 0.042 $\pm$ 0.01                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & 0.044 $\pm$ 0.014                                                 & 0.028 $\pm$ 0.019                                         & \textbf{0.027 $\pm$ 0.011}                                    & 0.051 $\pm$ 0.016                                                   & 0.042 $\pm$ 0.01                                   \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.051 $\pm$ 0.017}                                                 & \textbf{0.048 $\pm$ 0.017}                                         & 0.024 $\pm$ 0.005                                    & \textbf{0.069 $\pm$ 0.033}                                                   & \textbf{0.046 $\pm$ 0.014}                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & \textbf{0.051 $\pm$ 0.017}                                                 & \textbf{0.048 $\pm$ 0.017}                                         & 0.024 $\pm$ 0.005                                    & \textbf{0.069 $\pm$ 0.033}                                                   & \textbf{0.046 $\pm$ 0.014}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.024 $\pm$ 0.007                                                 & 0.01 $\pm$ 0.006                                          & 0.007 $\pm$ 0                                        & 0.026 $\pm$ 0.01                                                    & 0.033 $\pm$ 0.012                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.024 $\pm$ 0.007                                                 & 0.01 $\pm$ 0.006                                          & 0.007 $\pm$ 0                                        & 0.026 $\pm$ 0.01                                                    & 0.032 $\pm$ 0.012                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.028 $\pm$ 0.01                                                  & 0.015 $\pm$ 0.011                                         & 0.013 $\pm$ 0.013                                    & 0.013 $\pm$ 0.012                                                   & 0.018 $\pm$ 0.008                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.028 $\pm$ 0.01                                                  & 0.015 $\pm$ 0.01                                          & 0.013 $\pm$ 0.013                                    & 0.013 $\pm$ 0.012                                                   & 0.018 $\pm$ 0.008                  \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   N/4}}                                                                                                                                                                                                                              \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.045 $\pm$ 0.01}                                                  & 0.03 $\pm$ 0.016                                          & 0.027 $\pm$ 0.012                                    & 0.043 $\pm$ 0.015                                                   & 0.04 $\pm$ 0.01                                    \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & \textbf{0.056 $\pm$ 0.013}                                                 & 0.037 $\pm$ 0.019                                         & 0.034 $\pm$ 0.015                                    & 0.054 $\pm$ 0.019                                                   & 0.05 $\pm$ 0.012                                   \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.045 $\pm$ 0.014                                                 & \textbf{0.049 $\pm$ 0.013}                                         & \textbf{0.028 $\pm$ 0.002}                                    & \textbf{0.058 $\pm$ 0.024}                                                   & \textbf{0.05 $\pm$ 0.01}                                    \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & 0.056 $\pm$ 0.018                                                 & \textbf{0.061 $\pm$ 0.016}                                         & \textbf{0.035 $\pm$ 0.003}                                    & \textbf{0.073 $\pm$ 0.03}                                                    & \textbf{0.062 $\pm$ 0.012}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.027 $\pm$ 0.007                                                 & 0.012 $\pm$ 0.007                                         & 0.005 $\pm$ 0                                        & 0.024 $\pm$ 0.009                                                   & 0.03 $\pm$ 0.012                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.034 $\pm$ 0.008                                                 & 0.015 $\pm$ 0.008                                         & 0.007 $\pm$ 0                                        & 0.03 $\pm$ 0.011                                                    & 0.035 $\pm$ 0.012                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.029 $\pm$ 0.011                                                 & 0.015 $\pm$ 0.006                                         & 0.014 $\pm$ 0.009                                    & 0.017 $\pm$ 0.013                                                   & 0.019 $\pm$ 0.008                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.036 $\pm$ 0.013                                                 & 0.019 $\pm$ 0.008                                         & 0.017 $\pm$ 0.012                                    & 0.021 $\pm$ 0.016                                                   & 0.024 $\pm$ 0.01                                   \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top   N/2}}                                                                                                                                                                                                                              \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.039 $\pm$ 0.003}                                                 & 0.024 $\pm$ 0.009                                         & \textbf{0.024 $\pm$ 0.006}                                    & 0.032 $\pm$ 0.009                                                   & 0.03 $\pm$ 0.004                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & \textbf{0.098 $\pm$ 0.007}                                                 & 0.061 $\pm$ 0.022                                         & \textbf{0.06 $\pm$ 0.014}                                    & 0.079 $\pm$ 0.023                                                   & 0.076 $\pm$ 0.01                                   \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.03 $\pm$ 0.009                                                  & \textbf{0.029 $\pm$ 0.007}                                         & 0.02 $\pm$ 0.003                                     & \textbf{0.034 $\pm$ 0.01}                                                    & \textbf{0.033 $\pm$ 0.006}                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & 0.075 $\pm$ 0.022                                                 & \textbf{0.071 $\pm$ 0.017}                                         & 0.05 $\pm$ 0.007                                     & \textbf{0.086 $\pm$ 0.024}                                                   & \textbf{0.083 $\pm$ 0.015}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.022 $\pm$ 0.003                                                 & 0.018 $\pm$ 0.004                                         & 0.007 $\pm$ 0.001                                    & 0.016 $\pm$ 0.005                                                   & 0.026 $\pm$ 0.013                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.056 $\pm$ 0.009                                                 & 0.046 $\pm$ 0.011                                         & 0.017 $\pm$ 0.003                                    & 0.04 $\pm$ 0.012                                                    & 0.041 $\pm$ 0.01                                   \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.027 $\pm$ 0.005                                                 & 0.013 $\pm$ 0.006                                         & 0.015 $\pm$ 0.004                                    & 0.013 $\pm$ 0.007                                                   & 0.018 $\pm$ 0.004                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.066 $\pm$ 0.013                                                 & 0.031 $\pm$ 0.014                                         & 0.038 $\pm$ 0.009                                    & 0.033 $\pm$ 0.016                                                   & 0.044 $\pm$ 0.01                                   \\
\multicolumn{2}{l}{\cellcolor[HTML]{F2F2F2}\textbf{}}                             & \multicolumn{5}{c}{\cellcolor[HTML]{F2F2F2}\textbf{Top N}}                                                                                                                                                                                                                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & \textbf{0.03 $\pm$ 0.003}                                                  & \textbf{0.018 $\pm$ 0.004}                                         & \textbf{0.019 $\pm$ 0.006}                                    & \textbf{0.025 $\pm$ 0.006}                                                   & \textbf{0.025 $\pm$ 0.003}                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{NIAPU}}       & \textbf{Recall}    & \textbf{0.152 $\pm$ 0.017}                                                 & \textbf{0.091 $\pm$ 0.021}                                         & \textbf{0.094 $\pm$ 0.029}                                    & \textbf{0.127 $\pm$ 0.032}                                                   & \textbf{0.124 $\pm$ 0.017}                                  \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.027 $\pm$ 0.007                                                 & 0.016 $\pm$ 0.004                                         & 0.013 $\pm$ 0.001                                    & 0.02 $\pm$ 0.006                                                    & 0.02 $\pm$ 0.004                                   \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{DIAMOnD}}   & \textbf{Recall}    & 0.133 $\pm$ 0.034                                                 & 0.08 $\pm$ 0.019                                          & 0.066 $\pm$ 0.006                                    & 0.102 $\pm$ 0.028                                                   & 0.1 $\pm$ 0.021                                    \\

\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.02 $\pm$ 0.003                                                  & 0.015 $\pm$ 0.002                                         & 0.007 $\pm$ 0.002                                    & 0.012 $\pm$ 0.004                                                   & 0.024 $\pm$ 0.014                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{MCL}}       & \textbf{Recall}    & 0.099 $\pm$ 0.014                                                 & 0.076 $\pm$ 0.008                                         & 0.033 $\pm$ 0.01                                     & 0.058 $\pm$ 0.018                                                   & 0.042 $\pm$ 0.009                                  \\
\cellcolor[HTML]{F2F2F2}                                     & \textbf{Precision} & 0.029 $\pm$ 0.002                                                 & 0.01 $\pm$ 0.001                                          & 0.013 $\pm$ 0.001                                    & 0.017 $\pm$ 0.006                                                   & 0.021 $\pm$ 0.003                                  \\
\multirow{-2}{*}{\cellcolor[HTML]{F2F2F2}\textbf{RWR}}       & \textbf{Recall}    & 0.145 $\pm$ 0.011                                                 & 0.049 $\pm$ 0.006                                         & 0.067 $\pm$ 0.007                                    & 0.087 $\pm$ 0.031                                                   & 0.105 $\pm$ 0.017                                 


\end{longtable}
\end{scriptsize}
\endgroup



\section{Discussion}
\label{sec:discussion}

Results suggest that NeDBIT features are far superior to the more common TFO approach. The joint usage of \textsc{APU}\xspace with the NeDBIT features (\textsc{NIAPU}\xspace) succeeded to discriminate the class $P$ from the rest of the genes and in better separating the new classes $LP$, $WN$, $LP$ and $RN$, as shown in Section~\ref{subsec:5foldCV}. In the binary case, NeDBIT features were able to properly characterize the positive class from the unlabelled samples $U$, while TFO features definitely failed. With regards to the new classes, they were correctly identified also using TFO features, even if with a drop in accuracy compared to NeDBIT, highlighting the effectiveness of the \textsc{APU}\xspace label assignment. RF and MLP delivered the best performances. Regarding SVM, $LN$ samples were sometimes misclassified as either $WN$ or $RN$, which is still an acceptable error, being in the negative spectrum.

With regards to the disease gene identification performances on the retained 20\% of seed genes (Section~\ref{subsec:LP}), results show that nearly 50\% of unlabelled seed genes fell in the $LP$ class and a decreasing trend of classification in the \emph{more negative} classes was noticed.
We also observed a clear association between the positive labelling and the GDA score: the higher GDA score a gene had, the more likely they were found in the positive classes spectrum. This underlines the influence of scores on the NeDBIT features. Analogous results can be found in the \texttt{Supplementay File 2} for the remaining diseases.
Similarly to the results obtained using the full set of seed genes, all the classes were properly identified by RF and MLP with high scores for all classes, while SVM delivered lower scores, particularly with regards to the $LN$ class. Therefore, \textsc{NIAPU}\xspace turned out to be robust also in more challenging settings, both for binary and multi-class classification.
    
Most of the times, our methodology outperformed or was at par with the state-of-the-art algorithms for disease gene identification (Table~\ref{tab:gene-discovery-res}), being often the best-performing method when looking for a large number of candidate genes and the runner up for lower ones. Moreover, \textsc{NIAPU}\xspace was among the methods that shown lower variance in the performances. DIAMOnD performs at its best when considering a low number (e.g., from 5 to 50) of predicted genes. On the contrary, \textsc{NIAPU}\xspace is resilient also to larger quantities of candidate genes, outperforming DIAMOnD is such scenarios. In fact, the latter becomes unreliable, as stated by the authors themselves, when exceeding 200 predicted genes~\cite{pmid25853560}. Furthermore, \textsc{NIAPU}\xspace is able to use, when available, additional information on the seed genes (represented here by the GDA score), improving the discovery process.

\section{Conclusions}
\label{sec:conclu}

In this paper we presented the \textsc{NIAPU}\xspace pipeline, that has been shown to be able to work in the typical context of the computational identification of new GDAs. The principal advantage of the proposed algorithm is that it allows a more accurate control of the set of $LP$ which, extracted from the set of unlabelled elements, contains, with the highest probability, positive non-labelled elements.

The effectiveness of \textsc{NIAPU}\xspace is threefold: firstly, it enables PU learning on graph-based data rendering effective the application of ML methods; secondly, the pseudo-label assignment renders the classification of the unlabelled instances meaningful by assigning decreasing levels of \emph{positiveness}; thirdly, the $LP$ can be exploited for the task of disease-gene discovery, delivering results at par or outperforming existing methodologies.

It is worth to note that by uncoupling the features from the construction of the Markov process, \textsc{APU}\xspace could be used also in settings where data points are already organized in a graph and the features are exploited to compute edge weights. Moreover, features coming from different sources, e.g. multi-omics data, can be used without leveraging on data aggregation/assimilation methods, using one subset of features to build the graph and the others to weight the edges.   

One of the possible limitations of \textsc{NIAPU}\xspace lies in the accuracy of the dataset. \textsc{NIAPU}\xspace methodology is clearly influenced by the reliability of seed genes, by the association score assigned to them, and by the background network (the PPI network and its reliability, in the present case). The use of datasets that are as error-free and reliable as possible (e.g., using an interactome validated in the specific pathological context, possibly with weighted PPIs) could potentially improve the classification performance of the method. In this regard, it is worth mentioning that \textsc{NIAPU}\xspace is being applied in different contexts (e.g. nephrology, gastroenterology, rare diseases)~\cite{11573_1631720}, paying particular attention to the selection of seed genes and reference interactomes. 


\bibliographystyle{unsrt_first.bst}
