




\section{Background}
\label{sec:intro}

Identifying structural brain changes related to cognitive impairments is an important research topic in Alzheimer's Disease (AD) study. 
Regression models have been extensively studied to predict cognitive outcomes using morphometric measures 
that are extracted from structural magnetic resonance imaging (MRI) scans \cite{wan2014, yan2015}.
These studies are able to advance our understanding on the neuroanatomical basis of cognitive impairments. 
However, they are not designed to have direct impacts on clinical practice. 
To bridge this gap, in this manuscript we develop a novel learning paradigm to rank cognitive assessments 
based on their relevance to AD using brain MRI data.

Cognitive assessments represent the most common clinical routine for AD diagnosis. 
Given a large number of cognitive assessment tools and time-limited office visits, 
it is important to determine a proper set of cognitive tests for the subjects. 
Most current studies 
create guidelines of cognitive test selection for a targeted population \cite{Cordell2013, Scott2018}, 
but they are not customized for each individual subject.
In this work, we develop a novel learning paradigm that incorporate the ideas of precision medicine 
and customizes the cognitive test selection process to the characteristics of each individual patient. 
Specifically, we conduct a novel application of a newly developed learning-to-rank approach, 
denoted as \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace \cite{he2018tcbb}, to the structural MRI and cognitive assessment data of 
the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort \cite{Weiner17}. 
Using structural MRI measures as the individual characteristics, we are able to not only identify 
individual-specific cognitive biomarkers but also prioritize them and their corresponding assessment 
tasks according to AD-specific abnormality. 
{We also extend \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace to \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace using hinge loss \cite{gentile1999linear} to} 
 {more effectively prioritize individual-specific cognitive biomarkers.}
The study presented in this manuscript is a substantial extension from our preliminary study~\cite{Peng2019}. 

Our study is unique and innovative from the following two perspectives. 
First, conventional regression-based studies for  
cognitive performance prediction using MRI data focus on identifying relevant imaging biomarkers at the population level. 
However, our proposed model aims to identify AD-relevant cognitive biomarkers customized to each individual patient. 
Second, the identified cognitive biomarkers and assessments are prioritized based on 
the individual's brain characteristics. Therefore, they can be used to guide the selection of 
cognitive assessments in a personalized manner in clinical practice; 
it has the potential to enable personalized diagnosis and disease subtyping.

\subsection{Literature Review}
\label{sec:literature}
\subsubsection{Learning to Rank}
\label{sec:literature:LR}
Learning-to-Rank (\mbox{$\mathop{\mathtt{LETOR}}\limits$}\xspace)~\cite{Liu2011} is a popular technique used in 
information retrieval~\cite{Li2011}, 
web search~\cite{agichtein2006} and recommender systems~\cite{Karatzoglou2013}.
Existing \mbox{$\mathop{\mathtt{LETOR}}\limits$}\xspace methods can be classified into three categories~\cite{Liu2011}.
The first category is point-wise methods~\cite{Cao2007}, in which a function is learned to score individual instance, 
and then instances are sorted/ranked based on their scores.
The second category is pair-wise methods~\cite{Burges2007}, which maximize the number of correctly ordered pairs 
in order to learn the optimal ranking structure among instances.
The last category is list-wise methods~\cite{Lebanon2002}, in which a ranking function is learned to explicitly 
model the entire ranking.
Generally, pairwise and listwise methods have superior performance over point-wise methods
due to their ability to leverage order structure among instances in learning~\cite{Liu2011}.
Recently, \mbox{$\mathop{\mathtt{LETOR}}\limits$}\xspace has also been applied in drug discovery and drug selection~\cite{Liu2017,Zhang2015,Liu2017a,Liu2017b}.
For example, Agarwal \mbox{\emph{et al.}}\xspace~\cite{Agarwal2010} developed a bipartite ranking method to prioritize drug-like compounds.
He \mbox{\emph{et al.}}\xspace~\cite{he2018tcbb} developed a joint push 
and learning-to-rank method to select cancer drugs for each individual patient.
These studies demonstrate the great potential of \mbox{$\mathop{\mathtt{LETOR}}\limits$}\xspace 
in computational biology and computational medicine, particularly for biomarker prioritization.

\subsubsection{Machine Learning for AD Biomarker Discovery}
\label{sec:literature:AD}

The importance of {using big data} to enhance AD biomarker study has been widely recognized \cite{Weiner17}. 
As a result, numerous data-driven machine learning models have been developed for early AD detection 
and  AD-relevant biomarker identification including cognitive measures. 
These models are often designed to accomplish tasks such as 
classification (e.g., \cite{wang2017ipmi}), 
regression (e.g., \cite{wan2014,yan2015,yan2019frontneurosci}) or 
both (e.g., \cite{wang2012bioinfo2,brand2018miccai}), 
where imaging and other biomarker data are used to predict diagnostic, cognitive and/or other outcome(s) of interest. 
A drawback of these methods is that, although outcome-relevant biomarkers can be identified, 
they are identified at the population level and not specific to any individual subject. To bridge this gap, 
we 
{adapt} 
the \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace method  for biomarker prioritization at the individual level, 
which has greater potential to directly impact personalized diagnosis. 

\section{Methods}
\label{sec:Med}

\subsection{Materials}
\label{sec:materials}

The imaging and cognitive data used in our study were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) 
database  \cite{Weiner17}. 
The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. 
The primary goal of ADNI has been to test whether serial MRI, PET, other biological markers, and clinical and 
neuropsychological assessment can be combined to measure the progression of mild cognitive impairment 
(MCI, a prodromal stage of AD) and early AD. For up-to-date information, Please refer to~\cite{adni} for more detailed, 
up-to-date information. 


Participants include 819 ADNI-1 subjects with 229 healthy control (HC), 397 MCI and 193 AD participants. 
We consider both MCI and AD subjects as patients, and thus we have 590 cases and 229 controls.
We downloaded the 1.5T baseline MRI scans and cognitive assessment data from the ADNI website~\cite{adni}. 
We processed the MRI scans using Freesurfer version 5.1~\cite{Risacher2013}, 
where volumetric and cortical thickness measures of 101 regions relevant to AD were extracted to characterize 
brain morphometry. 

We focus our analysis on 151 scores assessed in 15 neuropsychological tests. For convenience, 
we denote these measures as {\em cognitive features} and these tests as {\em cognitive tasks}. 
The 15 studied tasks include Alzheimer's Disease Assessment Scale (ADAS), Clinical Dementia Rating Scale (CDR),
Functional Assessment Questionnaire (FAQ), Geriatric Depression Scale (GDS), Mini-Mental State Exam (MMSE), 
Modified Hachinski Scale (MODHACH), Neuropsychiatric Inventory Questionnaire (NPIQ), Boston Naming Test (BNT), 
Clock Drawing Test (CDT), Digit Span Test (DSPAN), Digit Symbol Test (DSYM), Category Fluency Test (FLUENCY), 
Weschler's Logical Memory Scale (LOGMEM), Rey Auditory Verbal Learning Test (RAVLT) and Trail Making Test (TRAIL). 





\subsection{Joint Push and Learning-To-Rank using Scores -- \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace}
\label{sec:method:PLTR}

We use the joint push and learning-to-rank method that we developed in He \mbox{\emph{et al.}}\xspace~\cite{he2018tcbb},
denoted as \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, for personalized cognitive feature prioritization.
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace has also been successfully applied in our preliminary study~\cite{Peng2019}. 
We aim to prioritize cognitive features for each individual patient that are most relevant to
his/her disease diagnosis. We will 
{use} 
patients' brain morphometric measures that are extracted from their MRI scans for the 
cognitive feature prioritization. 
The cognitive features are in the form of scores or answers in the cognitive tasks that the patients take. 
The prioritization outcomes can potentially be used in clinical practice to suggest the most relevant 
cognitive features or tasks that can most effectively facilitate diagnosis of an individual subject. 

In order to prioritize MCI/AD cognitive features,  
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace learns and uses patient latent vector representations and their imaging features to
score each cognitive feature for each individual patient. Then, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace ranks the cognitive features based on their scores.  
Patients with similar imaging feature profiles will have similar latent vectors and thus similiar ranking of cognitive features {\cite{cf,wang2006unifying}}. 
During the learning, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace explicitly pushes the most relevant cognitive features on
top of the less relevant features for each patient, and therefore
optimizes the latent patient vectors and cognitive feature vectors in a way that they will reproduce the feature ranking
structures {\cite{Liu2011}}. 
In \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, these latent vectors are learned via solving the following optimization problem:
\begin{equation}
  \label{eqn:obj}
 
    \min_{U, V} {\mathcal{L}_s} = (1 - \alpha) {P^{\uparrow}_s} + \alpha {O^{\scalebox{0.6}{$+$}}_s}
    + \frac{\beta}{2} R_{uv} + \frac{\gamma}{2} R_{\text{csim}},
   
\end{equation}
where
$\alpha$, $\beta$ and $\gamma \in [0,1]$ are coefficients of $O^{\scalebox{0.6}{$+$}}_s$,
$R_{uv}$ and $R_{\text{csim}}$ terms, respectively;
\mbox{$U = [\mathbf{u}_1, \mathbf{u}_2, \cdots, \mathbf{u}_m]$}
and \mbox{$V=[\mathbf{v}_1, \mathbf{v}_2, \cdots, \mathbf{v}_n]$}
are the latent matrices for patients and features, respectively ($\mathbf{u}$ and $\mathbf{v}$ 
are column latent patient vector and feature vector, respectively); 
${\mathcal{L}_s}$ is the overall loss function. In Problem~\ref{eqn:obj}, 
${P^{\uparrow}_s}$ measures the average number of relevant cognitive features 
ranked below an irrelevant cognitive feature, defined as follows, 
\begin{equation}
  \label{eqn:push}
  {P^{\uparrow}_s} = \sum\limits_{p = 1}^m
  \frac{1}{n_p^{\scalebox{0.6}{$+$}}n_p^{\scalebox{0.6}{$-$}}}\sum\limits_{\scriptsize{\mbox{$\mathop{f}\limits$}\xspace^{\scalebox{0.6}{$-$}}_i\in \mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}}
  \sum\limits_{\scriptsize{\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}\in \mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p^{\scalebox{0.6}{$+$}}}}
  \mathbb{I} ( {s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}) \le {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}) ),
 
\end{equation}
where $m$ is the number of patients, $\mbox{$\mathop{f}\limits$}\xspace^{\scalebox{0.6}{$+$}}_j$ and $\mbox{$\mathop{f}\limits$}\xspace^{\scalebox{0.6}{$-$}}_i$ are the
relevant and irrelevant features of patient $\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p$, 
$n^{\scalebox{0.6}{$+$}}_p$ and $n^{\scalebox{0.6}{$-$}}_p$ are their respective numbers, and
$\mathbb{I}(x)$ is the indicator function 
($\mathbb{I}(x) = 1$ if $x$ is true, otherwise 0).
In Equation~(\ref{eqn:push}), $s_p(\mbox{$\mathop{f}\limits$}\xspace_i)$ is a scoring function defined as follows, 
\begin{ceqn}
  \label{eqn:pred}
  \begin{align}
   
    s_p(\mbox{$\mathop{f}\limits$}\xspace_i) = \mathbf{u}^{\mathsf{T}}_p \mathbf{v}_i, 
  \end{align}
 
\end{ceqn}
that is, it calculates the score of feature $\mbox{$\mathop{f}\limits$}\xspace_i$ on patient $\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p$ using their respective latent
vectors $\mathbf{u}_p$ and $\mathbf{v}_i${\cite{koren2009matrix}}. 
By minimizing $P^{\uparrow}_s$, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace learns to assign higher scores to relevant features than irrelevant features
so as to rank the relevant features at the top of the final ranking list.
Note that, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace learns different latent vectors and ranking lists for different subjects, and therefore enables
personalized feature prioritization.
In Problem~(\ref{eqn:obj}), ${O^{\scalebox{0.6}{$+$}}_s}$ measures the ratio of mis-ordered feature pairs over the relevant features among
all the subjects, defined as follows,
\begin{eqnarray}
  \begin{aligned}
  \label{eqn:order_rel}
  {O^{\scalebox{0.6}{$+$}}_s}  
     	 = \sum\limits_{p=1}^m \frac{1}{|\{\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}} \succ_{\scriptsize{\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}} \mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}\}|}
         \sum\limits_{\scriptsize{\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}} \succ_{\tiny{\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}} \mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}}}
        
         \mathbb{I}({s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}}) < {s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}})),
         \hspace{7pt} 
        
  \end{aligned}
  %

\end{eqnarray}
where $\mbox{$\mathop{f}\limits$}\xspace_i \succ_{\scriptsize{\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}} \mbox{$\mathop{f}\limits$}\xspace_j$ represents
that $\mbox{$\mathop{f}\limits$}\xspace_i$ is ranked higher than $\mbox{$\mathop{f}\limits$}\xspace_j$ for patient $\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p$.
By minimizing $O^{\uparrow}_s$, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace learns to push the most relevant features on top of the less 
relevant features. Thus, most relevant features are pushed to the very top of the ranking list.
In Problem~\eqref{eqn:obj}, $R_{uv}$ is a regularizer on $U$ and $V$ to prevent overfitting,
defined as,
\begin{equation}
  \label{eqn:reg}
    R_{uv} = \frac{1}{m}\|U\|^2_{{F}} + \frac{1}{n}\|V\|^2_{{F}}, 
\end{equation}
where $\|X\|_{{F}}$ is the Frobenius norm of matrix $X$.
$R_{\text{csim}}$ is a regularizer on patients
to constrain patient latent vectors, defined as
\begin{equation}
  \label{eqn:sim}
    R_{\text{csim}} = \frac{1}{m^2}\sum_{p=1}^m\sum_{q=1}^m w_{pq}\|\mathbf{u}_p-\mathbf{u}_q\|_2^2,    
\end{equation}
where $w_{pq}$ is the similarity between subject $\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p$ and $\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_q$ that is calculated
using the imaging features of the subjects. 
The assumption here is that patients who are similar in terms of imaging features 
could also be similar in terms of cognitive features.

\subsection{Joint Push and Learning-To-Rank with Marginalization -- \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace}
\label{sec:method:PLTRH}
The objective of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace is to score 
relevant features higher than less relevant features 
as shown in Equation~\ref{eqn:push} and Equation~\ref{eqn:order_rel}.
However, in some cases, the score of relevant features is expected to be higher than that of less relevant features 
by a large margin.
For example, patients can be very sensitive to a few cognitive tasks but less sensitive to many others.
In order to incorporate such information, we propose a {new}
hinge loss~\cite{gentile1999linear}
based \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, denoted as \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace.
In \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace, the overall loss function is very similar to Equation~\ref{eqn:obj}, defined as follows,
\begin{equation}
  \label{eqn:H:obj}
 
    \min_{U, V} {\mathcal{L}_h} = (1 - \alpha) {P^{\uparrow}_h} + \alpha {O^{\scalebox{0.6}{$+$}}_h}
    + \frac{\beta}{2} R_{uv} + \frac{\gamma}{2} R_{\text{csim}},
   
\end{equation}
where ${\mathcal{L}_h}$ is the overall loss function; 
$U$, $V$, $R_{uv}$ and $R_{\text{csim}}$ are identical as those in Equation~\ref{eqn:obj}. 
In \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace, ${P^{\uparrow}_h}$ measures the average loss between the relevant features
 and irrelevant features using hinge loss as follows,
 %
\begin{eqnarray}
  \begin{aligned}
  \label{eqn:H:push}
  {P^{\uparrow}_h} = \sum\limits_{\mathclap{p = 1}}^m
  \frac{1}{n_p^{\scalebox{0.6}{$+$}}n_p^{\scalebox{0.6}{$-$}}}\!\!\sum\limits_{{\scriptsize{\mbox{$\mathop{f}\limits$}\xspace^{\scalebox{0.6}{$-$}}_i\in \mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}}}
  \sum\limits_{\mathclap{\scriptsize{~~\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}\in \mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p^{\scalebox{0.6}{$+$}}}}}
  \text{max} (0, t_p\hspace{-0.3em} -\hspace{-0.3em} ( {s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}})\hspace{-0.3em} -\hspace{-0.3em} {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}) ) ),
  \hspace{7pt}
  \end{aligned}
\end{eqnarray}
where $\text{max} (0, t_p - ( {s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}) - {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}) ) )$ is the hinge loss
($\text{max}(0, x) = x \text{ if $x>0$}$, otherwise $0$)
between the relevant feature $\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}$ and the irrelevant feature $\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}$, 
and $t_p$ is the pre-defined margin. 
Specifically, only when ${s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}) - {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}) > t_p$ will 
not induce any loss during optimization. 
Otherwise, the hinge loss will be positive and increase as ${s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}) - {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$-$}}) $ gets smaller
than $t_p$.
Thus, the hinge loss forces the scores of relevant features higher than those of irrelevant features by at least $t_p$. 
By doing this,  the relevant features are ranked higher than irrelevant features in the ranking list.
Similarly, ${O^{\scalebox{0.6}{$+$}}_h}$ measures the average loss among the relevant features 
also using hinge loss as follows,
\begin{eqnarray}
  \begin{aligned}
  \label{eqn:H:order_rel}
  {O^{\scalebox{0.6}{$+$}}_h}  
         = \sum\limits_{p=1}^m \frac{1}{|\{\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}}\!\!\succ_{\mathclap{\scriptsize{\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}}} \mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}\}|}
         \sum\limits_{\mathclap{~~~\scriptsize{\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}} \succ_{\tiny{\mbox{$\mathop{\mathcal{P}}\limits$}\xspace_p}} \mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$+$}}}}}
	 \text{max}  (0, 
	 t_o \hspace{-0.3em}-\hspace{-0.3em} ( {s_p}(\mbox{$\mathop{f}\limits$}\xspace_i^{\scalebox{0.6}{$+$}}) \hspace{-0.3em} - \hspace{-0.3em}{s_p}(\mbox{$\mathop{f}\limits$}\xspace_j^{\scalebox{0.6}{$-$}}) ) ),
	 \hspace{7pt}
        
        
  \end{aligned}
  %

\end{eqnarray}
where
$t_o$ is also the pre-defined margin.

\subsection{Data Processing}
\label{sec:data}

\subsubsection{Data Normalization}
\label{sec:materials:norm}

Following the protocol in our preliminary study~\cite{Peng2019}, 
we selected all the MCI and AD patients from ADNI
and conducted the following data normalization for these patients.
We first performed a $t$-test on each cognitive feature between patients and controls,
and selected those features if there is a significant difference
between patients and controls on these features.
Then, we converted the selected features into $[0, 1]$ by shifting and scaling the feature
values.
We also converted all the normalized feature values according to the Cohen's $d$ of the features between patients and controls,
and thus, smaller values always indicate higher AD possibility. 
After that, we filtered out features with values $0$, $1$ or $0.5$ for more than 95\% patients. 
This is to discard features that are either not discriminative, or extremely dominated by patients or controls.
{After the filtering step, we have 112 cognitive features remained and used in experiments. Table S1 in the supplementary 
materials presents these 112 cognitive features. }
We conducted the same process as above on the imaging features. {Table S2 in the supplementary materials presents 
these imaging features used in experiments. }




\subsubsection{Patient Similarities from Imaging Features}
\label{sec:data:sim}

Through the normalization and filtering steps as in Section~\ref{sec:materials:norm},
we have 86 normalized imaging features remained. We represent each patient using a vector of these features, 
denoted as $\mathbf{r}_p = [r_{p1}, r_{p2}, \cdots, r_{p86}]$, in which $r_{pi}$ ($i = 1, \cdots, 86$)
is an imaging feature for patient $p$. 
We calculate the patient similarity from imaging features using the radial basis function (RBF) kernel, that is, $w_{pq} = \exp( -\frac{\|\mathbf{r}_p - \mathbf{r}_q\|^2}{2\sigma^2})$,
where $w_{pq}$ is the patient similarity used in $R_{\text{csim}}$. 

\section{Results}
\label{sec:results}



\subsection{Baseline Methods}
\label{sec:method:baselines}
We compare \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace with two baseline methods:
the Bayesian Multi-Task Multi-Kernel Learning (\mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace) method~\cite{costello2014community}
and the Kernelized Rank Learning (\mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace) method~\cite{he2018kernelized}.

\subsubsection{Bayesian Multi-Task Multi-Kernel Learning (\mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace)}
\label{sec:method:baselines:BMTBKL}
\mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace is a {state-of-the-art} baseline for biomarker prioritization.
It was originally proposed to rank cell lines for drugs and won the DREAM 7
challenge~\cite{dream}.
In our study, \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace uses the multi-task and multi-kernel learning within kernelized regression to predict cognitive feature values
and learns parameters by conducting Bayesian inference.
We use the patient similarity matrix calculated from FreeSurfer features as the kernels in \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace.


\subsubsection{Kernelized Rank Learning (\mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace)}
\label{sec:method:baselines:KRL}
KRL represents another {state-of-the-art} baseline for biomarker prioritization.
In our study, \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace uses kernelized regression with a ranking loss to learn the ranking structure of patients
and to predict the cognitive feature values.
The objective of \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace is to maximize the hits among the top k of the ranking list.
We use the patient similarity matrix calculated from FreeSurfer features as the kernels in \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace.


\subsection{Training-Testing Data Splits}
\label{sec:exp:split}


Following the protocol in our preliminary study~\cite{Peng2019}, 
we test our methods in two different settings: cross validation (\mbox{$\mathop{\mathtt{CV}}\limits$}\xspace) and leave-out validation (\mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace).
In \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace, we randomly split each patient's cognitive tasks into
5 folds: all the features of a cognitive task will be either split into training or testing set.
We use 4 folds for training and the rest fold for testing, and do such experiments
5 times, each with one of the 5 folds as the testing set. The overall performance of the methods
is averaged over the 5 testing sets. This setting corresponds to the goal to prioritize additional cognitive tasks
that a patient should complete.
In \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace, 
we split patients (not patient tasks) into training and testing sets, and a certain patient and all his/her
cognitive features will be either in the training set or in the testing set.
This corresponds to the use scenario to identify the most relevant cognitive tasks that
a new patient needs to take, based on the existing imaging information of the patient, when the patient
has not completed any cognitive tasks. 
Figure~\ref{fig:cv} and Figure~\ref{fig:5fold} demonstrate the \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace and \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace data split processes, respectively.

Please note that as presented in Section~\ref{sec:materials:norm}, for 
normalized cognitive features, smaller values always 
indicate more AD possibility. Thus, in both settings,
we use the ranking list of normalized cognitive
features of each patient as ground truth for training and testing.


\begin{figure}[t!]
  \centering
 
 
    \begin{center}
     
     
	\input{random_loo_legend.tex}
        \scalebox{0.8}{
        \input{random_5_fold.tex}
      }
     
 
      \caption{Data split for cross validation (\mbox{$\mathop{\mathtt{CV}}\limits$}\xspace)}
      \label{fig:cv}
      \end{center}
     
\end{figure}
 
  %
 
\begin{figure}[!t]
  \centering
 
  \begin{center}
   
   
   
      \input{random_loo_legend.tex}
      \scalebox{0.8}{
      \input{random_loo.tex}
    }
    

    \caption{Data split for leave-out validation (\mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace)}
    \label{fig:5fold}
  \end{center}
 
  %
 
 
 
\end{figure}


\subsection{Parameters}
\label{sec:method:parameters}


We conduct  grid search 
to identify the best parameters {on each evaluation metric}
for each model. 
We use 0.3 and 0.1 as the value of $t_p$ and $t_o$, respectively.
In the experimental results, we report the combinations of parameters 
that achieve the best performance on evaluation metrics.
{We implement {\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace} using Python 3.7.3
and Numpy 1.16.2, and run the experiments on Xeon
E5-2680 v4 with 128G memory.}


\subsection{Evaluation Metrics}
\label{sec:method:metrics}

\input{cv.tex}

\subsubsection{Metrics on Cognitive Feature Level}
\label{sec:methods:metrics:question}

We use a metric named average feature hit at $k$ (QH@$k$) as in our preliminary study~\cite{Peng2019} to evaluate the ranking performance, 
\begin{equation}
  \label{eqn:qh}
  \text{QH}@k(\mathbf{\tau}^q, \tilde{\mathbf{\tau}}^q) = \sum\nolimits_{i = 1}^k \mathbb{I}(\tilde{{\tau}}^q_i \in {\mathbf{\tau}^q(1:k)}),

\end{equation}
where $\mathbf{\tau}^q$ is the ground-truth ranking list of all the features in all the tasks,
$\mathbf{\tau}^q(1:k)$ is the top $k$ features in the list, 
$\tilde{\mathbf{\tau}}^q$ is the predicted ranking list of all the features, and 
$\tilde{\mathbf{\tau}}^q_i$ is the $i$-th ranked features in $\tilde{\mathbf{\tau}}^q$. 
That is, QH@$k$ calculates the number of features among top $k$ in the predicted feature lists that are also in
the ground truth (i.e., hits).
Higher QH@$k$ values indicate better prioritization performance.  

We use a second evaluation metric weighted average feature hit at $k$ (WQH@$k$) as follows:
\begin{equation}
  \label{eqn:wqh}
 
  \text{WQH}@k(\mathbf{\tau}^q, \tilde{\mathbf{\tau}}^q) = {\sum\nolimits_{j=1}^k  QH@j(\mathbf{\tau}^q, \tilde{\mathbf{\tau}}^q)}/k,

\end{equation}
that is, $\text{WQH}@k$ is a weighted version of $\text{QH}@k$ that calculates the average of $\text{QH}@j$ ($j = 1, \cdots, k$)
over top $k$.
Higher $\text{WQH}@k$ indicates more feature hits and those hits are ranked on top in the ranking list.

\subsubsection{Metrics on Cognitive Task Level}
\label{sec:methods:metrics:questionnaire}



In in Peng \mbox{\emph{et al.}}\xspace~\cite{Peng2019}, 
we use the mean of the top-$g$ normalized ground-truth scores/predicted scores on the features of each
cognitive task for a patient as the score of that task for that patient.
For each patient, we rank the tasks using their ground-truth scores and use the ranking
as the ground-truth ranking of these tasks.
Thus, these scores measure how much relevant to AD the task indicates for the patients.
We use the predicted scores to rank cognitive tasks into the predicted ranking of the tasks.
We define a third evaluation metric task hit at $k$ (NH$_g$@$k$) as follows  
to evaluate the ranking performance in terms of tasks, 
\begin{equation}
  \label{eqn:nh}
  \text{NH}_g@k(\mathbf{\tau}_g^n, \tilde{\mathbf{\tau}}_g^n) = \sum\nolimits_{i = 1}^k \mathbb{I}(\tilde{{\tau}}^n_{gi} \in {\mathbf{\tau}_g^n(1:k)}),

\end{equation}
where $\mathbf{\tau}_g^n$/$\tilde{\mathbf{\tau}}_g^n$ is the ground-truth/predicted
ranking list of all the tasks using top-$g$ question scores. 


\section{Experimental Results}
\label{sec:results}


\subsection{Overall Performance on \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace}
\label{sec:results:overall:cv}

Table~\ref{table:cv} presents the performance of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace and two baseline methods in the \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace setting.
{Note that overall, {\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace} have similar standard deviations; {\mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace}
have higher standard deviations compared to {\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace}. This indicates that {\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace}
are more robust than {\mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace} and {\mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace} for the prioritization tasks. }


\subsubsection{Comparison on cognitive feature level}
\label{sec:results:overall:cv:feature}
For cognitive features from all tasks,
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace is able to identify on average 2.665{{$\pm$}0.07} out of the top-5 most relevant ground-truth cognitive features
 among its top-5 predictions (i.e., QH@5=2.665{$\pm$0.07}).
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace achieves similar performance as \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, 
and identifies on average 2.599{$\pm$0.09} most relevant ground-truth cognitive features on its top-5 predictions (i.e., QH@5=2.599{$\pm$0.09}).
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace significantly
outperform the baseline methods in terms of all the evaluation metrics on cognitive feature level (i.e., QH@5 and WQH@5).
Specifically, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace outperforms the best baseline method \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace at 9.1{$\pm$3.7}\% and 22.1{$\pm$9.5}\% 
on QH@5 and WQH@5, respectively.
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace also outperforms \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace at 6.4{$\pm$4.3}\% and 19.2{$\pm$10.1}\% on QH@5 and WQH@5, respectively.
These experimental results demonstrate that among the top 5 features in the ranking list, 
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to rank more relevant features on top than 
the two {state-of-the-art}
 baseline methods and the positions of those hits are also higher than those in the baseline methods.

\subsubsection{Comparison on cognitive task level}
\label{sec:results:overall:cv:task}
For the scenario to prioritize cognitive tasks that each patient
should take, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to identify the top-1 most relevant task for 72.5{$\pm$6.0}\% and 74.3{$\pm$4.0}\% of all the
patients when using 3 features to score cognitive tasks, respectively (i.e., NH$_3$=0.725{$\pm$0.06} for \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and NH$_3$=0.743{$\pm$0.04} for 
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace). 
This indicates the strong power of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace in prioritizing cognitive features
and in recommending relevant cognition tasks for real clinical applications. 
We also find that \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to outperform baseline methods on most of the metrics 
on cognitive task level (i.e., NH$_g@1$).
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace outperforms the best baseline method at 11.6{$\pm$5.6}\%, 16.7{$\pm$6.1}\% and 14.2{$\pm$6.6}\% on NH$_1@1$, 
NH$_2@1$ and NH$_3@1$, respectively.
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace performs even better than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on NH$_1@1$ and NH$_3@1$, in addition to that 
it outperforms the best performance of baseline methods at 13.7{$\pm$5.3}\%, 14.7{$\pm$4.8}\% and 17.0{$\pm$8.8}\% on NH$_1@1$, 
NH$_2@1$ and NH$_3@1$, respectively.
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace perform slightly worse than baseline methods on NH$_5@1$ and NH$_\text{all}@1$ 
(0.760{$\pm$0.05} vs 0.784{$\pm$0.05} on NH$_5@1$ and 0.707{$\pm$0.03} vs 0.760{$\pm$0.06} on NH$_\text{all}@1$).
These experimental results indicate that \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to push the most 
relevant task to the top of the ranking list than baseline methods when using a small number 
of features to score cognitive tasks.
Note that in \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace, each patient has only a few cognitive tasks in the testing set. Therefore, we only
consider the evaluation at the top task in the predicted task rankings (i.e.,
only NH$_g@1$ in Table~\ref{table:cv}).

Table~\ref{table:cv} also shows that \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace
on most of the metrics on cognitive task level (i.e., NH$_g@1$).
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace at 1.9{$\pm$0.5}\%, 2.5{$\pm$1.2}\%, 1.5{$\pm$0.3}\% and 2.6{$\pm$0.9}\%
on NH$_1$@1, NH$_3$@1, NH$_5$@1 and NH$_\text{all}$@1, respectively.
This indicates that generally \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace is better than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on ranking cognitive tasks in \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace setting.
The reason could be that the hinge-based loss functions with pre-defined margins can enable significant
difference between the scores of
relevant features and irrelevant features, and 
thus effectively push relevant features upon irrelevant features.

\subsection{Overall Performance on \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace}
\label{sec:results:overall:lov}

\input{loo.tex}
\input{loo_52.tex}
Table~\ref{table:lov:26} and Table~\ref{table:lov:52} present the performance of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace, \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace 
and two baseline methods in the \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace setting.
{Due to space limit, we did not present the standard deviations in the tables, but they have similar trends
as those in Table~{\ref{table:cv}}. }
We first hold out 26 (Table~\ref{table:lov:26}) and 52 (Table~\ref{table:lov:52}) 
AD patients as testing patients, respectively. 
We determine these hold-out AD patients as the ones that have more than 10 similar
AD patients in the training set with corresponding patient similarities higher than 0.67 and 0.62, respectively.


\subsubsection{Comparison on cognitive feature level}
\label{sec:results:overall:lov:feature}
Table~\ref{table:lov:26} and Table~\ref{table:lov:52} show that \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace significantly outperform the baseline methods 
in terms of all the evaluation metrics on cognitive feature level (i.e., QH@5 and WQH@5), 
which is consistent with the experimental results in \mbox{$\mathop{\mathtt{CV}}\limits$}\xspace setting.
When 26 patients are hold out for testing,
with parameters $\alpha = 0.5$, $\beta = 1.5$, $\gamma = 1.0$ and $\text{d} = 30$, 
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace outperforms the best baseline method \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace at 13.4\% and 1.3\% on QH@5 and WQH@5, respectively.
The performance of \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace is very comparable with that of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace" 
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace at 13.4\% and 0.5\% on QH@5 and WQH@5, respectively.
When 52 patients are hold out for testing,
with parameters $\alpha = 0.5$, $\beta = 0.5$, $\gamma = 1.0$ and $\text{d} = 50$,
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace outperforms the best baseline method \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace at 18.1\% and 7.8\% on QH@5 and WQH@5, respectively.
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace even performs better than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace in this setting. In addition, 
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace at 19.7\% and 9.5\% 
on QH@5 and WQH@5, respectively.
These experimental results demonstrate that for new patients, 
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to rank more relevant features 
to the top of the ranking list than the two baseline methods. 
They also indicate that for new patients, ranking based methods (e.g., \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace) are more effective 
than regression based methods (e.g., \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace and \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace) for biomarker prioritization. 

\subsubsection{Comparison on cognitive task level}
\label{sec:results:overall:lov:task}
Table~\ref{table:lov:26} also shows that 
when 26 patients are hold out for testing, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are both able to identify the top most relevant questionnaire for
84.6\% of the testing patients (i.e., 22 patients) under NH$_1@1$.
Table~\ref{table:lov:52} shows that
when 52 patients are hold out for testing, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are both able to identify for 80.8\% of the testing patients
(i.e., 42 patients) under NH$_1@1$.
Note that the hold-out testing patients in \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace do not have any cognitive features. Therefore,
the performance of \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace as above demonstrates their strong capability in identifying most
AD related cognitive features based on imaging features only. 
We also find that \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace are able to achieve similar or even better results compared to baseline methods
in terms of the evaluation metrics on cognitive task level (i.e., NH$_g$@1 and NH$_g$@5).
When 26 patients are hold out for testing,
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperform the baseline methods in terms of NH$_g$@1 (i.e., $g = 1, 2 \ldots 5$).
They are only slightly worse than \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace on ranking relevant tasks 
on their top-5 of predictions when $g = 1$ or $g = 5$ (3.308 vs 3.423 on NH$_1$@5 and 3.808 vs 3.962 on NH$_5$@5).
When 52 patients are hold out for testing,
\mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace also achieve the best performance on most of the evaluation metrics.
They are only slightly worse than \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace on NH$_2$@1, NH$_5$@5 (0.423 vs 0.481 on NH$_2$@1 and 3.712 vs 3.808 on NH$_5$@5).
These experimental results demonstrate that among top 5 tasks in the ranking list, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace 
rank more relevant task on top than \mbox{$\mathop{\mathtt{KRL}}\limits$}\xspace. 

It's notable that in Table~\ref{table:lov:26} and Table~\ref{table:lov:52}, as the number of features used to score cognitive tasks (i.e.,
$g$ in NH$_g@k$) increases, the performance of all the methods in NH$_g@1$ first declines and then increases.
This may indicate that as $g$ increases, irrelevant features which happen to have relatively high scores will be included 
in scoring tasks, and thus degrade the model performance on NH$_g@1$. 
However, generally, the scores of irrelevant features are considerably lower than those of relevant ones. 
Thus, as more features are included, the scores for tasks are more dominated by the scores of relevant features and thus the performance increases.

We also find that \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace performs poorly on NH$_3@1$ in both Table~\ref{table:lov:26} and Table~\ref{table:lov:52}. 
This indicates that \mbox{$\mathop{\mathtt{BMTMKL}}\limits$}\xspace, a regression-based method, could not well rank relevant features and irrelevant features. 
It's also notable that generally the best performance for the 26 testing patients is better than that for 52 testing patients.
This may be due to that the similarities between the 26 testing patients and their top 10 similar training patients are 
higher than those for the 52 testing patients.
The high similarities enable accurate latent vectors for testing patients. 

Table~\ref{table:lov:26} and Table~\ref{table:lov:52} also show that
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace is better than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on ranking cognitive tasks in \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace setting.
When 26 patients are hold out for testing,
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on NH$_1$@5, NH$_5$@5 and NH$_\text{all}$@5 and achieves
very comparable performance on the rest metrics.
When 52 patients are hold out for testing,
\mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace is able to achieve better performance than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on QH@5, WQH@5,
NH$_3$@1, NH$_5$@1, NH$_5$@5 and NH$_\text{all}$@5 and also achieves very comparable performance on
the rest metrics.
Generally, \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace outperforms \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace in terms of  metrics on cognitive task level. 
This demonstrates the effectiveness of hinge loss-based methods in separating relevant and irrelevant 
features during modeling. 

\section{Discussion}
\label{sec:results:case}

Our experimental results show that 
when NH$_1@1$ achieves its best performance of 0.846
for the 26 testing patients in the \mbox{$\mathop{\mathtt{LOV}}\limits$}\xspace setting (i.e.,
the first row block in Table~\ref{table:lov:26}), 
the task that is most commonly prioritized for the testing patients is
Rey Auditory Verbal Learning Test (RAVLT), including the following cognitive features:
1) trial 1 total number of words recalled;
2) trial 2 total number of words recalled; 
3) trial 3 total number of words recalled; 
4) trial 4 total number of words recalled; 
5) trial 5 total number of words recalled; 
6) total Score; 
7) trial 6 total number of words recalled; 
8) list B total number of words recalled; 
9) 30 minute delay total; and 
10) 30 minute delay recognition score. 
RAVLT is also the most relevant task in the ground truth if tasks
are scored correspondingly. 
RAVLT assesses learning and memory, and has shown promising
performance in early detection of AD~\cite{Moradi2017}. A number of studies have reported high correlations
between various RAVLT scores with different brain regions~\cite{Balthazar2010}.
For instance, RAVLT recall is associated with medial prefrontal cortex and hippocampus;
RAVLT recognition is highly correlated with thalamic and caudate nuclei.
In addition, genetic analysis of $APOE$ $\varepsilon$4 allele, the most common variant of AD, 
reported its association with RAVLT score in an early-MCI (EMCI) study \cite{Risacher2013}.
The fact that RAVLT is prioritized demonstrates that \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace is powerful in prioritizing
cognitive features to assist AD diagnosis. 

Similarly, we find the top-5 most frequent cognitive tasks corresponding to the performance at
NH$_3@5$=3.731 for the 26 hold-out testing patients. They are: Functional Assessment Questionnaire (FAQ), 
Clock Drawing Test (CDT), Weschler's Logical Memory Scale (LOGMEM), 
Rey Auditory Verbal Learning Test (RAVLT), and Neuropsychiatric Inventory Questionnaire (NPIQ). 
In addition to RAVLT discussed above, other top prioritized cognitive tasks 
have also been reported to be associated with AD or its progression. In an MCI to AD conversion study, FAQ, NPIQ and RAVLT 
showed significant difference between MCI-converter and MCI-stable groups~\cite{Risacher2009}.
We also notice that for some testing subjects, \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace is able to  
very well reconstruct their ranking structures. 
For example, when NH$_3@5$ achieves its optimal performance 3.731, for a certain testing subject, 
her top-5 predicted cognitive tasks RAVLT, LOGMEM, FAQ, NPIQ and CDT are exactly the top-5 
cognitive tasks in the ground truth.
These evidences further demonstrate the diagnostic power of our method. 

\section{Conclusions}
\label{sec:disc}

We have proposed a novel machine learning paradigm to prioritize cognitive assessments based on their relevance to 
AD at the individual patient level. The paradigm tailors the cognitive biomarker discovery and cognitive assessment selection 
process to the brain morphometric characteristics of each individual patient. It has been implemented using newly developed
 learning-to-rank method \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace and \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace.
Our empirical study on the ADNI data has produced promising results to identify and prioritize individual-specific cognitive biomarkers 
as well as cognitive assessment tasks based on the individual's structural MRI data.
In addition, \mbox{$\mathop{\mathtt{PLTR_h}}\limits$}\xspace shows better performance than \mbox{$\mathop{\mathtt{PLTR}}\limits$}\xspace on ranking cognitive assessment tasks.
The resulting top ranked cognitive biomarkers and assessment tasks have the potential to aid personalized diagnosis and 
disease subtyping, and to make progress towards enabling precision medicine in AD. 



\section{Abbreviations}
\label{sec:abbr}
\textbf{AD}: Alzheimer's Disease

\noindent
\textbf{MRI}: Magnetic Resonance Imaging

\noindent
\textbf{ADNI}: Alzheimer's Disease Neuroimaging Initiative

\noindent
\textbf{LETOR}: Learning-to-Rank

\noindent
\textbf{PET}: Positron Emission Tomography

\noindent
\textbf{MCI}: Mild Cognitive Impairment

\noindent
\textbf{HC}: Healthy Control

\noindent
\textbf{ADAS}: Alzheimer's Disease Assessment Scale

\noindent
\textbf{CDR}: Clinical Dementia Rating Scale

\noindent
\textbf{FAQ}: Functional Assessment Questionnaire 

\noindent
\textbf{GDS}: Geriatric Depression Scale

\noindent
\textbf{MMSE}: Mini-Mental State Exam

\noindent
\textbf{MODHACH}: Modified Hachinski Scale

\noindent
\textbf{NPIQ}: Neuropsychiatric Inventory Questionnaire

\noindent
\textbf{BNT}: Boston Naming Test

\noindent
\textbf{CDT}: Clock Drawing Test

\noindent
\textbf{DSPAN}: Digit Span Test

\noindent
\textbf{DSYM}: Digit Symbol Test

\noindent
\textbf{FLUENCY}: Category Fluency Test

\noindent
\textbf{LOGMEM}: Weschler's Logical Memory Scale

\noindent
\textbf{RAVLT}: Rey Auditory Verbal Learning Test

\noindent
\textbf{TRAIL}: Trail Making Test

\noindent
\textbf{RBF}: Radial Basis Function

\noindent
\textbf{PLTR}: Joint Push and Learning-to-Rank Method

\noindent
\textbf{PLTR$_\textbf{h}$}: Joint Push and Learning-to-Rank Method using Hinge Loss

\noindent
\textbf{BMTMKL}: Bayesian Multi-Task Multi-Kernel Learning

\noindent
\textbf{KRL}: Kernelized Rank Learning

\noindent
\textbf{CV}: Cross Validation

\noindent
\textbf{LOV}: Leave-Out Validation

\noindent
\textbf{QH@$k$}: Average Feature Hit at $k$

\noindent
\textbf{WQH@$k$}: Weighted Average Feature Hit at $k$

\noindent
\textbf{$\text{NH}_g$@$k$}: Task Hit at $k$

\noindent
\textbf{APOE}: Apolipoprotein E

\noindent
\textbf{EMCI}: Early-MCI
\section{Declaration}
\label{sec:dec}

\subsection{Ethics approval and consent to participate}

The dataset supporting the conclusions of this article is available in the 
Alzheimer’s Disease Neuroimaging Initiative (ADNI)~\cite{adni}.
ADNI data can be requested by all interested investigators; they can request it via the ADNI website 
and must agree to acknowledge ADNI and its funders in the papers that use the data. 
There are also some other reporting requirements; the PI must give an annual report of what the data have been used for, 
and any publications arising.
More details are available at http://adni.loni.usc.edu/data-samples/access-data/. 


\subsection{Consent for publication}
Not applicable

\subsection{Availability of data and materials}
\label{sec:dec:data}
The dataset supporting the conclusions of this article is available in the 
Alzheimer’s Disease Neuroimaging Initiative (ADNI)~\cite{adni}.

\subsection{Conflicts of interest}
\label{sec:dec:cof}
The authors declare that they have no competing interests. 

\subsection{Funding}
\label{sec:dec:fund}
This work was supported in part by NIH R01 EB022574,  
R01 AG019771, and P30 AG010133; NSF 
IIS 1837964 and 1855501.
Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do
not necessarily reflect the views of the funding agencies.


\subsection{Authors' contributions}
\label{sec:dec:contributions}

XN and LS designed the research study. BP and XY contributed to the
conduct of the study: XY extracted and processed the data from ADNI; BP conduced 
the model development and data analysis. 
The results were analyzed, interpreted and
discussed by BP, XY, SL, AJ, LS and XN. BP and XN drafted the manuscript and all
co-authors revised and approved the final version of the manuscript.

\subsection{Acknowledgements}
\label{sec:dec:ack}


Data collection and sharing for this project was funded by the Alzheimer's Disease Neuroimaging Initiative (ADNI) 
(National Institutes of Health Grant U01 AG024904) and 
DOD ADNI (Department of Defense award number W81XWH-12-2-0012). 
ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, 
and through generous contributions from the following: AbbVie, Alzheimer's Association; Alzheimer's Drug Discovery Foundation; 
Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Eisai Inc.; Elan Pharmaceuticals, Inc.; 
Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; 
GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research \& Development, LLC.; Johnson \& Johnson Pharmaceutical Research
 \& Development LLC.; Lumosity; Lundbeck; Merck \& Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; 
 Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. 
 The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions 
 are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California 
 Institute for Research and Education, and the study is coordinated by the Alzheimer's Disease Cooperative Study at the University of California, 
 San Diego. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.

\bibliographystyle{bmc-mathphys}




\section{Features used in experiments}
\label{sec:features}

Table~\ref{table:cog_features} presents the cognitive features that are used in our experiments after data processing. 
Table~\ref{table:img_features} presents the image features that are used in our experiments after data processing. 

\setcounter{table}{0}
\renewcommand{\thetable}{S\arabic{table}}

\onecolumn
\input{cog_features.tex}

\onecolumn
\input{image_features_xy.tex}


\end{frontmatter}

\bibliographystyle{bmc-mathphys}
