




\subsection{Subject-based Normal vs. Abnormal Comparison}
\vspace{-2pt}
In this section, we aim to compare the normal and simulated-impaired performances of the same subject and show that this analysis is insensitive to the choice of recording device and robust to the viewpoint or distance from the camera. Note that in our dataset, for each subject, we have four sets of recordings. Two of these recordings capture the normal performance of the test, while in the other two, the subject is asked to perform abnormally. In addition, two pairs of normal/abnormal recordings are captured by an iPhone (P) and an iPad (T). Let $N_P$/$N_T$ and $A_P$/$A_T$ denote the normal and abnormal recordings captured by iPhone/iPad.

For each feature and subject, we define A-A/N-N as the intra-class distance between the features derived from the abnormal/normal recordings of the subject captured on iPhone and iPad devices. In other words, A-A is the distance between features computed for $A_T$ and $A_P$ recordings, while N-N marks the difference between the features of $N_T$ and $N_P$ videos. 
For N-A, we consider the distance between $A_T$-$N_P$ and $N_T$-$A_P$ pairs and report the average.
We normalize the A-A, N-N, and N-A distances by the maximum of N-A distances.

Fig.~\ref{fig:diff_dist_als} illustrates the distribution of A-A, N-N, and N-A distances across $20$ different subjects for a subset of features of FTF test. While the intra-class values are concentrated near zero, the inter-class distances are spread out over a wider range. In addition, the mean A-A and N-N distances are strictly lower than the N-A distances. The higher concentration of A-A and N-N distances around zero shows that our feature set is robust to some minor changes in the viewpoint and is not affected by the recording device. Furthermore, it can be seen as a proof-of-concept, demonstrating the ability to compare the subject's performance across different time points. This validates the potential of neurological disease progression the effects of treatment tracking using our DNE system.

\begin{figure}[t!]
    \centering
    \resizebox{.92\columnwidth}{!}{
        \includegraphics[]{resources/features_diff_dist.pdf}}
    \caption{The inter-class and intra-class distances between some features of normal \textit{(N)} and abnormal \textit{(A)} FTF recordings.
    \ding{169} denotes the mean value.} 
    \label{fig:diff_dist_als}
\end{figure}

\vspace{-10pt}
\subsection{Abnormality Detection}
\vspace{-2pt}
\input{fig_tex/plot_pca}

\input{fig_tex/feature_density}

\noindent \textbf{Principal component analysis (PCA):} The feature set describing normal and abnormal recordings constitutes a high-dimensional vector. For a visual comparison of normal and abnormal recordings in terms of their derived features, we perform dimensionality reduction through PCA. For this purpose, for each test, we concatenate the set of features listed in Table~\ref{tab:dne-features} and normalize them before passing to PCA. Fig.~\ref{fig:pca-analysis} showcases the results for different tests. It is observed that the normal and abnormal recordings are separated in dimension reduced feature space. 
This implies that our defined features are descriptive and well differentiate normal from abnormal.

\noindent \textbf{Abnormal Class Distribution:} In Fig.~\ref{fig:features_density} we compare the distribution of normal versus abnormal features for FT, FTF, FR, and SAW tests. These plots clearly indicate the difference in distribution between two classes. Normal features are more concentrated in a specific range, however the abnormal features are often less regular and have a higher STD.    

\noindent \textbf{Abnormality Detection:} We assess the normal and abnormal classification performance using our proposed set of features. Therefore, we utilize several machine learning (ML) models that are grouped into: 1) tree-based methods such as Random Forest (RF), Gradient-Boosting Machine (GBM) \cite{friedman2001greedy}, XGBoost \cite{Chen:2016:XST:2939672.2939785} and 2) parametric models trained using gradient-descent updates, including Logistic Regression (LR),  Support Vector Machine with radial basis function (RBF) kernel (RSVM) and Multi-layer Perceptron (MLP) with rectified linear unit (ReLU) activation. 

We also benchmarked our ML classification performance against two deep learning (DL) baselines. Both DL models predict normal versus abnormal based on major keypoint pose sequence, unlike the ML based models which perform classification on the extracted spatio-temporal/kinematic features. In the first DL baseline, we adopt a long-short term memory (LSTM)~\cite{hochreiter1997long} based sequential model while in the second DL approach, similar to~\cite{kidzinski_automatic_2019}, we use convolutional neural networks (CNN). Details of ML and DL based classification models, data processing and hyper parameters are provided in the supplementary section II and Table S1. We evaluate different models via metrics such as accuracy, average precision, F1 score, and area under the ROC curve (AUC).

We have two splitting schemes to separate the train from test sets in our experiments. In \textit{video-based} splitting, videos from all subjects are divided independently based on a $80 \%$/$20 \%$ splitting ratio for train/test sets. In addition, to evaluate the performance of the models on unseen patients, the \textit{subject-based} division scheme splits a portion of the subjects into the train set while keeping the rest in the test set. Thus, videos belonging to the subjects in the train set are not used in the test set and vice versa. In subject-based splitting, we have $16$/$4$ subjects in train/test sets.

We performed $5$-fold cross validation and summarized the average classification performance of all ML and DL models
in Table \ref{tab:classification}. While all models perform well for various tests, among ML models RSVM and GBM/XGBoost tend to perform better on most metrics. However, the gap between the performance of all ML models is not significant across different tests. This suggests that the extracted set of features well-distinguish normal from abnormal samples.

Furthermore, comparing ML and DL models, we notice that: 1) While DL models perform well on FT, FR and FTF tasks (especially for video-based splitting), they are lagging behind ML models for SAW. We attribute this to the fact that SAW involves more complex motion patterns. Therefore, DL models require larger datasets to be able to learn the classification task from the pose data. 2) DL features extracted from the pose data lack clinical interpretability. 3) For subject-based splitting, ML models operating on the spatio-temporal/kinematic features outperform DL models on most metrics. This indicates better generalization capability of our features on unseen subjects compared to DL models operating on pose data.



\input{fig_tex/tab_classifier}
\vspace{-10pt}
\subsection{Feature Importance Analysis}
\vspace{-2pt}
One benefit of tree-based models is in the tractable decision-making process. Therefore, we investigate the importance of each feature, contributing to the decision process by analyzing our RF models. This analysis gives us the weight of all features, sorted in descending order in Supplementary Fig. S4. %

We notice that symmetry between specific R/L features for FT, FTF, and SAW tests is considered the most important, i.e., with the largest weight. For the SAW test, the most important feature is the similarity between the knee angle time series across different cycles ({${S}_{\text{saw}}^{\text{knee angle}}$}) while
for FT (Supplementary Fig. S4-(a)) and FTF (Supplementary Fig. S4-(c)), the features with the largest weights are frequency asymmetry
and horizontal ({${S}_{\text{ftf}}^{\text{finger-x}}$}) symmetry, respectively. Although this can be attributed to the nature of the simulated impairments in our dataset, it is consistent with the clinical practice, where the left and right asymmetry is a common biomarker~\cite{Babrak2019, Nagasaki2004AsymmetricVA, sawyer_asymmetry_1993} of different neurological disorders.

Furthermore, temporal and spatial features that characterize the periodic behavior of the movement are important metrics that the decision tree classification models rely on. Examples of these features are amplitude and period for FT, FTF, and FR tests, step length, width, and step time for SAW.
We also notice that for a subset of features, having large variations (i.e. STD) across different cycles is another indicator of abnormal performance in our dataset. This is captured in the large weight associated with STD values of some features for various tests. This result also affirms our observations in Fig.~\ref{fig:features_density}.

\vspace{-10pt}
\section{Discussion \& Challenges}
\vspace{-5pt}
In this section, we discuss various aspects of DNE including feature design, robustness, clinical relevance and application as well as the current challenges and our proposed solutions.

\vspace{-10pt}
\subsection{Discussion}
\vspace{-2pt}
\textbf{Feature Design:} The main goal of our DNE system is to provide an objective tool for quantifying and documentation of  recordings of neurological tests. Thus, it is critical to design a set of clinical interpretable features that explain the performance of a subject on various motor tasks. In addition, having powerful digital biomarkers reduces the workload of normal versus abnormal classification models and improves their generalization, especially when large training datasets are not available. Furthermore, unlike black-box DL models, the explainability of our diverse set of features allows clinicians to better understand and track patients' status over time.   

\textbf{Robustness:} DNE is resilient to changes in slight deviations from the camera view, distance to the camera, subject clothing, and mild pixel intensity changes due to intermediate data standardization and robust pose estimation steps (section \ref{ssec:pose-estimation}-\ref{sec:prep}). This is experimentally shown by the low intra-class feature distances in Fig. \ref{fig:diff_dist_als}. Data normalization and filtering in the pre-processing step also helps in eliminating noise and propagated errors from the pose estimation module.

In FT, FR and SAW tests, the abnormality in the motion is imposed by wearing equipment which are visible in the recordings. The pose estimation models we have used (OP and VP3D) are robust to the appearance of the equipment and can accurately predict the joint locations regardless of the presence of the equipment. The features
incorporated in the classification tasks are derived from the pose data. Therefore, the quantified features and the classification performance is not affected by the visual cues from the equipment.

\textbf{Clinical Relevance:} In our dataset, the abnormalities in the movements of the subjects were simulated. The simulated impairment in the FR test is the closest to what is observed in clinics for patients with neurological disorders. In the simulated impairment for FR, the arm with no moulage satellites around the weighted wrist, causing a decrease in the orbit frequency (Fig.~\ref{fig:fr_example}). This is coherent with the clinical observations of patients with neurological impairments. 

In the FTF test, the simulated abnormality would be more realistic, if the tremor or inaccuracy of movement increased as the finger got closer to its target (i.e. when the two fingers approach). In our current dataset, the subjects often simulated the tremor throughout their movements which is only seen in severe cases. In addition, for the FT test, often the abnormality is a combination of decreased amplitude and rate (Fig.~\ref{fig:ft_features}) and in Parkinson’s decrements of both. In our DNE dataset, some subjects simulated more of one or the other.

In SAW, the abnormality in real patients appear as a combination of slow time-to-stand, decreased step length, increased step-time, and asymmetry of gait features. In our dataset, the abnormality was imposed by wearing a knee brace. Alongside asymmetry between the R/L knee angles, we observed decreased step-length for the subjects wearing the knee brace (Fig. ~\ref{fig:features_density}-SAW). These are in-line with clinical observations from real patients.

Overall, features that clinicians observe were disrupted from normal findings to various degrees, although the pattern of disruption of features may have not been exact for a specific condition. We showed that DNE was able to define clinically interpretable features and detect differences between normal and simulated impaired recordings.
As future work, to expand its clinical impact, we will focus our analysis on real patients with various neurological impairment severity levels, 
and with other neurological tests, such as eye movement~\cite{Pretegiani2017}, facial activation~\cite{Jin2020,Rafayet2021}, or phonation \cite{Fabbri2017}.

\textbf{Clinical Application:} The initial clinical application of DNE is measuring and documenting features of various neurological exams. This would allow for improved communication of objective exam quantification and the ability to assess for changes over time. As future work, with clinicians' supervision, we will examine and report the performance of DNE on real patients. A longer term goal is to assist clinicians with classification of recordings and provide a platform for longitudinal monitoring of patients.

\vspace{-10pt}
\subsection{Challenges}
\vspace{-2pt}
\textbf{Depth Ambiguity: }Analyzing human motion from 2D RGB data requires dealing with uncertainties associated with lacking depth information. Furthermore, depth ambiguity becomes a more prominent challenge for the SAW test with frontal view recordings rather than sagittal view. It also avoids defining the spatial features in their absolute units. Currently, to mitigate the issues corresponding to these depth uncertainties, for upper limb tests, the subjects are asked to perform the tests while facing the camera and (roughly) in parallel to its image plane. In our processing steps, we also perform pose normalizations to compensate for scale variations due to variable distance from the camera. To further address this issue, we believe incorporating LiDAR depth maps captured by recent iOS devices, in the pose estimation step can prove helpful.

\textbf{Self-baselining:} Natural motion properties differ across various subjects. For example, one subject can be inherently slower or have less strength in performing some tests. In our dataset, we witnessed while some subjects had a slower inherent speed in their normal performance, they were mistakenly classified as abnormal. This highlights the importance of taking into account the history of a subject and self-baselining. 
In our experiments, we showcased an example of self-comparisons of normal and abnormal performance of the same subject (Fig.~\ref{fig:diff_dist_als}). The purpose of this study was to show the ability of our designed features to discriminate between the varying status of the subject at different test times. This result validates the potential of our DNE pipeline as a personalized medical assessment system that helps clinicians better monitor the disease progression and the effect of medical treatments.

\textbf{Real-time DNE:} Our current DNE system and the extracted kinematic/spatio temporal features rely on tracking the human pose from the video recordings in an offline step using off-the-shelf pose estimation modules. Currently, the pose estimation step is the most computationally expensive step, hindering real time processing and feature extraction. To address this challenge, on-device lighter pose estimation models (with small sacrifice on the accuracy), that focus on extracting major keypoints rather than the whole body pose are necessary.




\subsection{Pose estimation}
\vspace{-2pt}
\label{ssec:pose-estimation}
For upper body tests (FT, FTF, and FR), we use OpenPose (OP)~\cite{Cao2019_openpose} to estimate the 2D pose. On the other hand, for SAW tests, we compute the 3D pose using the VideoPose3D (VP3D) package~\cite{pavllo2019_videopose3d}. 
OP is designed for multi-person real time 2D hand \cite{Simon2017} and body ~\cite{Cao2019_openpose} pose estimation. Given an RGB image, OP first detects all visible body parts in the image and the corresponding part affinity fields. Associating body parts to each individual in an image boils down to a graph matching problem which can be solved in a greedy fashion. 
Meanwhile, VP3D adopts dilated temporal convolution to estimates 3D pose from sequence of 2D keypoints extracted from the video. 


For upper body tests, if the subject and the moving limb is located parallel to the camera plane, then the motion is well approximated in a plane, i.e. in two dimensions. That is why 2D pose is chosen for upper body tests. However, this might not hold for the SAW test (especially depending on the camera view), hence urging us to use 3D pose for this analysis. 

We use OP and VP3D to extract 2D/3D pose of the recordings in the DNE dataset. We measure the inference time of pre-trained OP models on Google Colab with one Tesla T4 GPU as $6$ FPS. In VP3D, the 2D pose data are extracted for each frame via Detectron2~\cite{wu2019detectron2}, which achieves the  runtime of $12.5$ FPS on a GeForce RTX 2080 GPU. Additionally, it takes roughly $0.2$ seconds for uplifting 2D pose to 3D per video.
\vspace{-10pt}
\subsection{Pre-processing}
\vspace{-2pt}
\label{sec:prep}
We truncate a recording to only include the sequence of frames that are related to the subject performing the test. 
To account for variable distance of the subjects from the camera, we normalize the estimated pose by a reference length. For FT, FTF and FR tests, the reference is the length of the forearm. For SAW, the reference is the distance between the pelvis and neck joints. We compute the reference lengths as the median of the value across all the frames. 
In addition, as the estimated pose can be erroneous at some frames we use median and Savitzky-Golay filtering~\cite{Savitzky1964}. In our dataset, we have excluded $27$ recordings due to unreliable and noisy estimated pose. 
Therefore, we only analyzed $348$ videos in total. %



\vspace{-10pt}
\subsection{Notations}
\vspace{-2pt}
    


\begin{figure}
    \centering
    \includegraphics[width=.8 \linewidth]{resources/keypoints.pdf}
    \vspace{-5pt}
    \caption{Skeleton tree for (a) body ${{B}}$, and (b) hand ${{H}}$. Examples of human pose estimation (c) in 2D (${B}_2, {H}_2$) using OpenPose~\cite{Cao2019_openpose} and (d) in 3D (${B}_3$) using VideoPose3D~\cite{pavllo2019_videopose3d}.} 
    \label{fig:pose_estimation}
\end{figure}
Given the pose sequence estimated from the RGB video, we extract a set of quantified features. Below, we first express our notations and then introduce the features we defined for each test. Let $\boldsymbol{v} = [v_1, ..., v_N]$ denote the set of $N$ frames ordered chronologically in video $\boldsymbol{v}$. There is a one-to-one correspondence between the time associated with each frame and the frame index,  where $\boldsymbol{t} = [t_1, ..., t_N]$ and $t_i = i/fps$, $fps$ denoting the frame per second rate of the video. Given $\boldsymbol{v}$ and the pose estimation module (such as OP or VP3D), we extract the location of $K$ keypoints in each frame. For convenience, we use the same indexing of the body joints for both 2D and 3D pose. However, to differentiate between the 2D and 3D pose, we denote each by ${B}_2$ and ${B}_3$, respectively. Furthermore, we use ${H}_2$ to represent the 2D hand keypoints. An illustration of the hand and body skeleton trees alongside our indexing notations are provided in Fig.~\ref{fig:pose_estimation}. Note that, for the sake of brevity, we have only indexed a subset of the keypoints that we are using in our analysis.





We reserve $\boldsymbol{s}_{k, *}[i]$ for the location of the $k$-th keypoint at frame $i$, corresponding to skeleton tree $* \in \{{H}_2, {B}_2, {B}_3\}$. For $* \in \{{H}_2, {B}_2\}$, $\boldsymbol{s}_{k, *}[i] \in \mathbb{R}^2$ and for $*={B}_3$, $\boldsymbol{s}_{k, *}[i] \in \mathbb{R}^3$. Furthermore, we add superscript $r$ and $l$ to point to right and left (R/L) body parts, respectively. For example, $\boldsymbol{s}^r_{3, {H}_2}[i]$ locates the tip of the right thumb at frame $i$.

To extract kinematic features that quantify the performance of a subject in a test, we track the location of various major keypoints and define a set of features accordingly. Major keypoints vary based on the test. For instance, the major keypoints in FT include the tip of the index and thumb fingers of two hands while in FR, we closely track the wrist joints. 

In different tests, the subjects are asked to move certain limbs repeatedly. 
Thus, it is natural to compute features such as frequency, and amplitude for periodic pose patterns and report the mean and standard deviations (STD) across different cycles. In addition, for a test performed normally, the features 
corresponding to the R/L body parts should be close. Thus, to quantify the difference between the right ${f}^r$ and left ${f}^l$ features, we define an asymmetry metric as:
\begin{align}
   \textrm{Asym}({f}^r, {f}^l) = \frac{\vert {f}^r- {f}^l \vert}{{f}^r + {f}^l}. \label{eq:asym} 
\end{align}
Another useful metric in our analysis is Pearson correlation coefficient denoted by $\text{CC}$. For two 1D discrete time series $\boldsymbol{x}_1$ and $\boldsymbol{x}_2$, we define $\text{CC}$ as:
\vspace{-5pt}
\begin{align}
    \textrm{CC}(\boldsymbol{x}_1, \boldsymbol{x}_2) = \frac{(\boldsymbol{x}_1-\bar{\boldsymbol{x}}_1)^T (\boldsymbol{x}_2-\bar{\boldsymbol{x}}_2)}{\Vert \boldsymbol{x}_1-\bar{\boldsymbol{x}}_1 \Vert_2 \Vert \boldsymbol{x}_2-\bar{\boldsymbol{x}}_2 \Vert_2}.\label{eq:cc}
\end{align}
where $\bar{.}$ and $.^T$ denote the mean and transpose operators.
For highly correlated series, $\vert \textrm{CC} \vert$ is close to one.




\vspace{-8pt}
\subsection{Feature definition}
\vspace{-2pt}
We list the features defined for various tests in Table~\ref{tab:dne-features} and describe them in detail below.

\noindent \textbf{Finger Tapping (FT)}: For this test, the major keypoints are the tip of the R/L thumb and index fingers alongside R/L wrist and elbow joints. To extract properties of the periodic motion, we examine the distance between the tip of the index and thumb fingers across time defined as:

\begin{align}
    \boldsymbol{d}_{\text{ft}}^*[i] = \Vert \boldsymbol{s}^*_{3, {H}_2}[i] - \boldsymbol{s}^*_{6, {H}_2}[i] \Vert_2, \quad * \in \{r, l\}. \label{eq:dist_ft}
\end{align}
Examples of $\boldsymbol{d}_{\text{ft}}^r$ and $\boldsymbol{d}_{\text{ft}}^l$ for normal and abnormal executions of the FT test are provided in Fig.~\ref{fig:ft_features}. In our dataset, to simulate abnormality in FT the subjects are wearing a rubber band around index and thumb fingers of one hand. As also revealed in Fig.~\ref{fig:ft_features}, this limits the tapping amplitude of the hand wearing the band and slows down the tapping rate.
Given $\boldsymbol{d}_{\text{ft}}^*$, we compute the period for the $*$ hand, $T_{\textrm{ft}}^*$, as the time (in seconds) between two consecutive local minima (or maxima) of $\boldsymbol{d}_{\text{ft}}^{*}$. Frequency $F_{\textrm{ft}}^*$ is the reciprocal of $T_{\textrm{ft}}^*$. 
We also report the magnitude of finger-tapping $A^*_{\textrm{ft}}$ as the difference in consecutive minima and maxima of $\boldsymbol{d}_{\text{ft}}^*$. We also report the asymmetry of the periods ($\text{Asym}(T_{\text{ft}}^{r}, T_{\text{ft}}^{l})$), frequencies ($\text{Asym}(F_{\text{ft}}^{r}, F_{\text{ft}}^{l})$) 
and amplitudes ($\text{Asym}(A_{\text{ft}}^{r}, A_{\text{ft}}^{l})$) of R/L hands following~\eqref{eq:asym}.

\input{fig_tex/ft_features}
\input{resources/tft/ftf_features}

Furthermore, we define the instant tapping speed and acceleration for R/L hands as the first and second order derivatives of $\boldsymbol{d}_{\text{ft}}^r$ and $\boldsymbol{d}_{\text{ft}}^l$ with respect to time. We adopt mean and maximum of instant speed and acceleration across tapping cycles as features.
We also introduce average tapping rate as the average number of finger taps per second, obtained by dividing the total number of finger tappings by the FT duration.

Finally, to evaluate the stability of the hands and arms during the FT recording, we examine the wrist and elbow joints. For this purpose, we introduce the relative height between $(\boldsymbol{s}^r_{7, {B}_2},\boldsymbol{s}^l_{7, {B}_2})$ and $(\boldsymbol{s}^r_{6, {B}_2},\boldsymbol{s}^l_{6, {B}_2})$ across $N$ frames:
\vspace{-5pt}
\begin{align}
    C_{\textrm{ft}}^{\textrm{wrist}} &= \frac{1}{N} \sum\limits_{i=1}^N \frac{\Vert \boldsymbol{s}^r_{7, {B}_2}[i]-\boldsymbol{s}^l_{7, {B}_2}[i] \Vert_2}{\Vert \boldsymbol{s}^r_{7, \textrm{B}_2}[i] \Vert_2},
\end{align}
\vspace{-10pt}
\begin{align}
    C_{\textrm{ft}}^{\textrm{elbow}} &= \frac{1}{N} \sum\limits_{i=1}^N \frac{\Vert \boldsymbol{s}^r_{6, {B}_2}[i]-\boldsymbol{s}^l_{6, {B}_2}[i] \Vert_2}{\Vert \boldsymbol{s}^r_{6, {B}_2}[i] \Vert_2}. 
    \label{eq:wrist_stable}
\end{align}
\vspace{-10pt}

\input{fig_tex/tab_features}

\noindent \textbf{Finger to Finger (FTF)}: 
In our dataset, we observe that the estimated pose by OP for middle joints of the index finger, i.e. joint index $5$ in ${H}_2$, is more stable than the outer fingertip. Hence, we focus on this joint for FTF test.
In a normal FTF, the horizontal and vertical trajectories of the R/L hands are symmetric up to a mirroring (Fig.~\ref{fig:ftf-lr-asymmetry}- top row), while this does not necessarily hold for abnormal case (Fig.~\ref{fig:ftf-lr-asymmetry}- bottom row). Thus, in each cycle, we define the cross correlation of the R/L horizontal ($x$) and vertical ($y$) coordinates as the horizontal ($S_{\text{ftf}}^{\text{finger-x}}$) and vertical symmetries ($S_{\text{ftf}}^{\text{finger-y}}$):

\begin{align}
    \label{eq:ftf_finger}
   S_{\text{ftf}}^{\text{finger-x}} &= \text{CC}(\left[\boldsymbol{s}_{5, {H}_2}^l\right]_x, - \left[\boldsymbol{s}_{5, {H}_2}^r\right]_x), \\
   S_{\text{ftf}}^{\text{finger-y}} &= \text{CC}(\left[\boldsymbol{s}_{5, {H}_2}^l\right]_y, \left[\boldsymbol{s}_{5, {H}_2}^r\right]_y)
\end{align}
\noindent where $\left[\boldsymbol{s}_{5, H_2}^* \right]_\dagger = \{\boldsymbol{s}^*_{5, {H}_2}[i](\dagger)\}_{i=1}^{N}$, $\dagger \in \{x, y\}$ and $* \in \{r, l\}$, is the $x$ or $y$ coordinates of the pose series.
We also compute the period and average speed. We derive the average speed by dividing the traversed distance of R/L finger within half a cycle's period by half the cycle's period.


Patients with neurological impairments tend to have tremors while moving their fingers during FTF test \cite{krishna_quantitative_2019}. This leads to a deviation of the fingers' trajectory from a smooth curve. To characterize this deviation, we first fit a smooth curve to the fingers' trajectory, in the form of a second order polynomial in terms of the $x$ and $y$ coordinates. We observe that fitting a second order function to the trajectories, well matches the FTF trajectories of normal subjects. We consider the length of this smooth curve as a reference to compare against the length of the original fingers' trajectory. We then define the ratio of the length of the actual fingers' trajectory during each FTF cycle by the length of the fitted smooth curve as path smoothness metric (PS). We report PS for R/L hands. Examples of normal and abnormal finger trajectories alongside the smooth fitted curves are plotted in Fig.~\ref{fig:ftf-traj}. %



Another feature we found helpful in detecting abnormal function in FTF is instant velocity. We derive the instant velocity vector by the first derivative of the horizontal and vertical pose with respect to time.
We then examine the angle between the vertical and horizontal components of this vector on the R/L hands. At time instant $t$, the \textit{velocity angle} $\boldsymbol{\theta}$ is:
\begin{align}
    \boldsymbol{\theta}^{*}(t) = \textrm{atan2} \left(\frac{\frac{d \, \left[\boldsymbol{s}^*_{5, {H}_2}\right]_{y}}{d t}}{\frac{d \, \left[\boldsymbol{s}^*_{5, {H}_2}\right]_x}{d t}} \right), * \in \{r, l\}. \label{eq:vel_ang}
\end{align}
Next, for each hand, we compare $\boldsymbol{\theta}$ across different cycles using CC in~\eqref{eq:cc}. Given $N_C$ number of cycles, we have $N_C \choose 2$ CC values assessing the symmetry of the R/L velocity angles across different cycles, which we summarize by reporting the mean and STD. Examples of normal and abnormal aligned velocity angles across different cycles are provided in Fig.~\ref{fig:ftf-vel-angle}. Note that for abnormal FTF, large magnitude fluctuations, caused by tremors in moving the hands, visibly appear in $\boldsymbol{\theta}$. 






\noindent \textbf{Forearm Rolling (FR)}: We include the wrist and elbow joints as the major keypoints for this test. We specifically attend to the vertical coordinate of the wrist joints to compute period $T^*_\textrm{fr}$
and amplitudes $A^*_\textrm{fr}$ for $* \in \{r, l\}$. Fig.~\ref{fig:fr_example} illustrates the vertical position of the R/L wrists for a normal and abnormal example. Note that, due to wearing the device in the abnormal recording, the period of the forearm roll cycles for both R/L hands are larger compared to its normal counterpart. In addition, similar to FT, we include the asymmetry of the aforementioned metrics in the FR features.

We also include the maximum instant speed and acceleration derived from vertical coordinates of the wrist joints. Similar to FT, we define rolling speed and rate. Rolling speed is computed as the difference between the minimum and maximum of $y$ coordinate of the R/L hands divided by half the rolling period. Also, rolling rate is defined as the number of rolling cycles per second. Finally, we report the stability of the elbows $C_{\textrm{fr}}^{\textrm{elbow}}$ and define it analogous to~\eqref{eq:wrist_stable}. %

\input{fig_tex/fr_features}
\input{resources/saw/saw_features}

\noindent \textbf{Stand-up and Walk (SAW):} We use the side-view SAW recordings in our analysis of SAW test. For SAW pose estimation, we use VP3D~\cite{pavllo2019_videopose3d}. In VP3D, the joint locations are defined relative to the pelvis joint. As a result, estimated pose by VP3D misses the global position of subjects within a frame which is essential to detect different segments
of the SAW test, i.e. stand-up (SU), walk (W), and turn (TU). This urged us to track the 2D position of the pelvis $\boldsymbol{s}_{0,B_2}$ extracted by OP as a notion of subject's global position in a video frame. Analyzing this position through time enables us to split a SAW recording into multiple non-overlapping SU, W, and TU segments. Supplementary Fig. S3 visualizes these segments.

For the SU segment of SAW, we focus on the time to stand~\cite{Jawad2012}, measured by the total time taken from the first SU effort to a full standing on feet state.
We derive time to stand by thresholding the magnitude of the pelvis joint's velocity.
Note that, since our subjects are asked to walk back and forth a designated room multiple times, at some points, they have to change direction and turn around. We report time to turn around as another indicative feature for SAW test.


The first set of features derived for the walking segment are obtained based on the distance between the two feet stated as:
\vspace{-15pt}
\begin{align}
    \boldsymbol{d}_{\textrm{saw}}[i] = ||\boldsymbol{s}^r_{2,{B}_3}[i]-\boldsymbol{s}^l_{2,{B}_3}[i]||_2.
\end{align}
Note that, the periodic nature of a normal gait also reflects in $\boldsymbol{d}_{\textrm{saw}}$ (see Fig.~\ref{fig:saw-step-info}). Given $\boldsymbol{d}_{\textrm{saw}}$, we highlight different W and TU segments in Fig.~\ref{fig:saw-step-info}. For a gait pattern derived based on $\boldsymbol{d}_{\textrm{saw}}$, \textit{step time} is the time to complete one step and computed as the time difference between two consecutive local maxima of $\boldsymbol{d}_{\textrm{saw}}$. Meanwhile, \textit{step length} defined as linear distance between two successive placements of the same foot~\cite{Herran2014} manifests as the local maxima of $\boldsymbol{d}_{\textrm{saw}}$. The \textit{step width}, on the other hand, is interpreted as the local minima of $\boldsymbol{d}_{\textrm{saw}}$. The calculations of these features in turning segments are excluded.



As two global features for gait, we report mean and STD of \textit{cadence} and \textit{average speed} across all W segments. We compute cadence as the number of steps divided by the duration of a walking segment. Average speed is determined by the total traveled distance of the pelvis joint divided by the duration of a walking segment.


To evaluate the symmetry of the R/L gait, we introduce the cross correlation between the knee angle series of R/L legs, denoted by {${S}_{\textrm{saw}}^{\textrm{knee angle}}$}. We find this feature a good descriptive of gait abnormality, as in our recordings, gait abnormality is introduced through wearing a knee band which limits the knee motion (Fig.~\ref{fig:dne_dataset}). For each frame, we define the knee angle as the angle between $\boldsymbol{s}^r_{3, {B}_3}-\boldsymbol{s}^r_{4, {B}_3}$ and $\boldsymbol{s}^r_{3, {B}_3}-\boldsymbol{s}^r_{2, {B}_3}$ for the right leg and $\boldsymbol{s}^l_{3, {B}_3}-\boldsymbol{s}^l_{4, {B}_3}$ and $\boldsymbol{s}^l_{3, {B}_3}-\boldsymbol{s}^l_{2, {B}_3}$ for the left leg. As there is a lag between the R/L gait cycles, we align the knee angle series of the R/L legs within each cycle and then report CC of the aligned series. Examples of aligned normal and abnormal knee angles for R/L legs are shown in Fig.~\ref{fig:saw-knee-angle}. For normal gait, the R/L knee angles are highly correlated after alignment (Fig.~\ref{fig:saw-knee-angle}-top row), while this does not hold for abnormal gait (Fig.~\ref{fig:saw-knee-angle}-bottom row).  

In addition, we define step symmetry between the R/L feet movements by comparing the horizontal position of R/L feet at different gait cycles. We represent this metric by {${S}_{\textrm{saw}}^{\textrm{feet-x}}$}. To compute {${S}_{\textrm{saw}}^{\textrm{feet-x}}$}, similar to {${S}_{\textrm{saw}}^{\textrm{knee angle}}$}, we first align the R/L horizontal positions within each gait stride and report the CC of the aligned series. We report mean and STD for both {${S}_{\textrm{saw}}^{\textrm{feet-x}}$} and {${S}_{\textrm{saw}}^{\textrm{knee angle}}$} across different cycles. %




\section{Introduction}
\input{sections/intro}

\vspace{-10pt}
\section{Related Work}
\vspace{-5pt}
\input{sections/related_work}

\vspace{-10pt}
\section{System Design}
\vspace{-5pt}
\input{sections/system_design}

\vspace{-10pt}
\section{Dataset Collection}
\vspace{-5pt}
\input{sections/dataset_collection}

\vspace{-8pt}
\section{DNE Vision-Based Analysis}
\input{sections/vision_analysis}

\vspace{-10pt}
\section{Results and Discussion}
\input{sections/results}

\vspace{-10pt}
\section{Conclusion}
\vspace{-2pt}
\input{sections/conclusion}

\vspace{-8pt}
\section*{Acknowledgement}
\vspace{-3pt}
This work was supported by the Jump ARCHES endowment through the Health Care Engineering Systems Center, by the National Institute of Health (NIH) R01 AI139401, and by the Vingroup Innovation Foundation VINIF.2021.DA00128.

\vspace{-8pt}
\small{
\bibliographystyle{IEEEtran}



























\section{Introduction}
\input{sections/intro}

\vspace{-10pt}
\section{Related Work}
\vspace{-5pt}
\input{sections/related_work}

\vspace{-10pt}
\section{System Design}
\vspace{-5pt}
\input{sections/system_design}

\vspace{-10pt}
\section{Dataset Collection}
\vspace{-5pt}
\input{sections/dataset_collection}

\vspace{-8pt}
\section{DNE Vision-Based Analysis}
\input{sections/vision_analysis}

\vspace{-10pt}
\section{Results and Discussion}
\input{sections/results}

\vspace{-10pt}
\section{Conclusion}
\vspace{-2pt}
\input{sections/conclusion}

\vspace{-8pt}
\section*{Acknowledgement}
\vspace{-3pt}
This work was supported by the Jump ARCHES endowment through the Health Care Engineering Systems Center, by the National Institute of Health (NIH) R01 AI139401, and by the Vingroup Innovation Foundation VINIF.2021.DA00128.

\vspace{-8pt}
\small{
\bibliographystyle{IEEEtran}








\subsection{Subject-based Normal vs. Abnormal Comparison}
\vspace{-2pt}
In this section, we aim to compare the normal and simulated-impaired performances of the same subject and show that this analysis is insensitive to the choice of recording device and robust to the viewpoint or distance from the camera. Note that in our dataset, for each subject, we have four sets of recordings. Two of these recordings capture the normal performance of the test, while in the other two, the subject is asked to perform abnormally. In addition, two pairs of normal/abnormal recordings are captured by an iPhone (P) and an iPad (T). Let $N_P$/$N_T$ and $A_P$/$A_T$ denote the normal and abnormal recordings captured by iPhone/iPad.

For each feature and subject, we define A-A/N-N as the intra-class distance between the features derived from the abnormal/normal recordings of the subject captured on iPhone and iPad devices. In other words, A-A is the distance between features computed for $A_T$ and $A_P$ recordings, while N-N marks the difference between the features of $N_T$ and $N_P$ videos. 
For N-A, we consider the distance between $A_T$-$N_P$ and $N_T$-$A_P$ pairs and report the average.
We normalize the A-A, N-N, and N-A distances by the maximum of N-A distances.

Fig.~\ref{fig:diff_dist_als} illustrates the distribution of A-A, N-N, and N-A distances across $20$ different subjects for a subset of features of FTF test. While the intra-class values are concentrated near zero, the inter-class distances are spread out over a wider range. In addition, the mean A-A and N-N distances are strictly lower than the N-A distances. The higher concentration of A-A and N-N distances around zero shows that our feature set is robust to some minor changes in the viewpoint and is not affected by the recording device. Furthermore, it can be seen as a proof-of-concept, demonstrating the ability to compare the subject's performance across different time points. This validates the potential of neurological disease progression the effects of treatment tracking using our DNE system.

\begin{figure}[t!]
    \centering
    \resizebox{.92\columnwidth}{!}{
        \includegraphics[]{resources/features_diff_dist.pdf}}
    \caption{The inter-class and intra-class distances between some features of normal \textit{(N)} and abnormal \textit{(A)} FTF recordings.
    \ding{169} denotes the mean value.} 
    \label{fig:diff_dist_als}
\end{figure}

\vspace{-10pt}
\subsection{Abnormality Detection}
\vspace{-2pt}
\input{fig_tex/plot_pca}

\input{fig_tex/feature_density}

\noindent \textbf{Principal component analysis (PCA):} The feature set describing normal and abnormal recordings constitutes a high-dimensional vector. For a visual comparison of normal and abnormal recordings in terms of their derived features, we perform dimensionality reduction through PCA. For this purpose, for each test, we concatenate the set of features listed in Table~\ref{tab:dne-features} and normalize them before passing to PCA. Fig.~\ref{fig:pca-analysis} showcases the results for different tests. It is observed that the normal and abnormal recordings are separated in dimension reduced feature space. 
This implies that our defined features are descriptive and well differentiate normal from abnormal.

\noindent \textbf{Abnormal Class Distribution:} In Fig.~\ref{fig:features_density} we compare the distribution of normal versus abnormal features for FT, FTF, FR, and SAW tests. These plots clearly indicate the difference in distribution between two classes. Normal features are more concentrated in a specific range, however the abnormal features are often less regular and have a higher STD.    

\noindent \textbf{Abnormality Detection:} We assess the normal and abnormal classification performance using our proposed set of features. Therefore, we utilize several machine learning (ML) models that are grouped into: 1) tree-based methods such as Random Forest (RF), Gradient-Boosting Machine (GBM) \cite{friedman2001greedy}, XGBoost \cite{Chen:2016:XST:2939672.2939785} and 2) parametric models trained using gradient-descent updates, including Logistic Regression (LR),  Support Vector Machine with radial basis function (RBF) kernel (RSVM) and Multi-layer Perceptron (MLP) with rectified linear unit (ReLU) activation. 

We also benchmarked our ML classification performance against two deep learning (DL) baselines. Both DL models predict normal versus abnormal based on major keypoint pose sequence, unlike the ML based models which perform classification on the extracted spatio-temporal/kinematic features. In the first DL baseline, we adopt a long-short term memory (LSTM)~\cite{hochreiter1997long} based sequential model while in the second DL approach, similar to~\cite{kidzinski_automatic_2019}, we use convolutional neural networks (CNN). Details of ML and DL based classification models, data processing and hyper parameters are provided in the supplementary section II and Table S1. We evaluate different models via metrics such as accuracy, average precision, F1 score, and area under the ROC curve (AUC).

We have two splitting schemes to separate the train from test sets in our experiments. In \textit{video-based} splitting, videos from all subjects are divided independently based on a $80 \%$/$20 \%$ splitting ratio for train/test sets. In addition, to evaluate the performance of the models on unseen patients, the \textit{subject-based} division scheme splits a portion of the subjects into the train set while keeping the rest in the test set. Thus, videos belonging to the subjects in the train set are not used in the test set and vice versa. In subject-based splitting, we have $16$/$4$ subjects in train/test sets.

We performed $5$-fold cross validation and summarized the average classification performance of all ML and DL models
in Table \ref{tab:classification}. While all models perform well for various tests, among ML models RSVM and GBM/XGBoost tend to perform better on most metrics. However, the gap between the performance of all ML models is not significant across different tests. This suggests that the extracted set of features well-distinguish normal from abnormal samples.

Furthermore, comparing ML and DL models, we notice that: 1) While DL models perform well on FT, FR and FTF tasks (especially for video-based splitting), they are lagging behind ML models for SAW. We attribute this to the fact that SAW involves more complex motion patterns. Therefore, DL models require larger datasets to be able to learn the classification task from the pose data. 2) DL features extracted from the pose data lack clinical interpretability. 3) For subject-based splitting, ML models operating on the spatio-temporal/kinematic features outperform DL models on most metrics. This indicates better generalization capability of our features on unseen subjects compared to DL models operating on pose data.



\input{fig_tex/tab_classifier}
\vspace{-10pt}
\subsection{Feature Importance Analysis}
\vspace{-2pt}
One benefit of tree-based models is in the tractable decision-making process. Therefore, we investigate the importance of each feature, contributing to the decision process by analyzing our RF models. This analysis gives us the weight of all features, sorted in descending order in Supplementary Fig. S4. %

We notice that symmetry between specific R/L features for FT, FTF, and SAW tests is considered the most important, i.e., with the largest weight. For the SAW test, the most important feature is the similarity between the knee angle time series across different cycles ({${S}_{\text{saw}}^{\text{knee angle}}$}) while
for FT (Supplementary Fig. S4-(a)) and FTF (Supplementary Fig. S4-(c)), the features with the largest weights are frequency asymmetry
and horizontal ({${S}_{\text{ftf}}^{\text{finger-x}}$}) symmetry, respectively. Although this can be attributed to the nature of the simulated impairments in our dataset, it is consistent with the clinical practice, where the left and right asymmetry is a common biomarker~\cite{Babrak2019, Nagasaki2004AsymmetricVA, sawyer_asymmetry_1993} of different neurological disorders.

Furthermore, temporal and spatial features that characterize the periodic behavior of the movement are important metrics that the decision tree classification models rely on. Examples of these features are amplitude and period for FT, FTF, and FR tests, step length, width, and step time for SAW.
We also notice that for a subset of features, having large variations (i.e. STD) across different cycles is another indicator of abnormal performance in our dataset. This is captured in the large weight associated with STD values of some features for various tests. This result also affirms our observations in Fig.~\ref{fig:features_density}.

\vspace{-10pt}
\section{Discussion \& Challenges}
\vspace{-5pt}
In this section, we discuss various aspects of DNE including feature design, robustness, clinical relevance and application as well as the current challenges and our proposed solutions.

\vspace{-10pt}
\subsection{Discussion}
\vspace{-2pt}
\textbf{Feature Design:} The main goal of our DNE system is to provide an objective tool for quantifying and documentation of  recordings of neurological tests. Thus, it is critical to design a set of clinical interpretable features that explain the performance of a subject on various motor tasks. In addition, having powerful digital biomarkers reduces the workload of normal versus abnormal classification models and improves their generalization, especially when large training datasets are not available. Furthermore, unlike black-box DL models, the explainability of our diverse set of features allows clinicians to better understand and track patients' status over time.   

\textbf{Robustness:} DNE is resilient to changes in slight deviations from the camera view, distance to the camera, subject clothing, and mild pixel intensity changes due to intermediate data standardization and robust pose estimation steps (section \ref{ssec:pose-estimation}-\ref{sec:prep}). This is experimentally shown by the low intra-class feature distances in Fig. \ref{fig:diff_dist_als}. Data normalization and filtering in the pre-processing step also helps in eliminating noise and propagated errors from the pose estimation module.

In FT, FR and SAW tests, the abnormality in the motion is imposed by wearing equipment which are visible in the recordings. The pose estimation models we have used (OP and VP3D) are robust to the appearance of the equipment and can accurately predict the joint locations regardless of the presence of the equipment. The features
incorporated in the classification tasks are derived from the pose data. Therefore, the quantified features and the classification performance is not affected by the visual cues from the equipment.

\textbf{Clinical Relevance:} In our dataset, the abnormalities in the movements of the subjects were simulated. The simulated impairment in the FR test is the closest to what is observed in clinics for patients with neurological disorders. In the simulated impairment for FR, the arm with no moulage satellites around the weighted wrist, causing a decrease in the orbit frequency (Fig.~\ref{fig:fr_example}). This is coherent with the clinical observations of patients with neurological impairments. 

In the FTF test, the simulated abnormality would be more realistic, if the tremor or inaccuracy of movement increased as the finger got closer to its target (i.e. when the two fingers approach). In our current dataset, the subjects often simulated the tremor throughout their movements which is only seen in severe cases. In addition, for the FT test, often the abnormality is a combination of decreased amplitude and rate (Fig.~\ref{fig:ft_features}) and in Parkinson’s decrements of both. In our DNE dataset, some subjects simulated more of one or the other.

In SAW, the abnormality in real patients appear as a combination of slow time-to-stand, decreased step length, increased step-time, and asymmetry of gait features. In our dataset, the abnormality was imposed by wearing a knee brace. Alongside asymmetry between the R/L knee angles, we observed decreased step-length for the subjects wearing the knee brace (Fig. ~\ref{fig:features_density}-SAW). These are in-line with clinical observations from real patients.

Overall, features that clinicians observe were disrupted from normal findings to various degrees, although the pattern of disruption of features may have not been exact for a specific condition. We showed that DNE was able to define clinically interpretable features and detect differences between normal and simulated impaired recordings.
As future work, to expand its clinical impact, we will focus our analysis on real patients with various neurological impairment severity levels, 
and with other neurological tests, such as eye movement~\cite{Pretegiani2017}, facial activation~\cite{Jin2020,Rafayet2021}, or phonation \cite{Fabbri2017}.

\textbf{Clinical Application:} The initial clinical application of DNE is measuring and documenting features of various neurological exams. This would allow for improved communication of objective exam quantification and the ability to assess for changes over time. As future work, with clinicians' supervision, we will examine and report the performance of DNE on real patients. A longer term goal is to assist clinicians with classification of recordings and provide a platform for longitudinal monitoring of patients.

\vspace{-10pt}
\subsection{Challenges}
\vspace{-2pt}
\textbf{Depth Ambiguity: }Analyzing human motion from 2D RGB data requires dealing with uncertainties associated with lacking depth information. Furthermore, depth ambiguity becomes a more prominent challenge for the SAW test with frontal view recordings rather than sagittal view. It also avoids defining the spatial features in their absolute units. Currently, to mitigate the issues corresponding to these depth uncertainties, for upper limb tests, the subjects are asked to perform the tests while facing the camera and (roughly) in parallel to its image plane. In our processing steps, we also perform pose normalizations to compensate for scale variations due to variable distance from the camera. To further address this issue, we believe incorporating LiDAR depth maps captured by recent iOS devices, in the pose estimation step can prove helpful.

\textbf{Self-baselining:} Natural motion properties differ across various subjects. For example, one subject can be inherently slower or have less strength in performing some tests. In our dataset, we witnessed while some subjects had a slower inherent speed in their normal performance, they were mistakenly classified as abnormal. This highlights the importance of taking into account the history of a subject and self-baselining. 
In our experiments, we showcased an example of self-comparisons of normal and abnormal performance of the same subject (Fig.~\ref{fig:diff_dist_als}). The purpose of this study was to show the ability of our designed features to discriminate between the varying status of the subject at different test times. This result validates the potential of our DNE pipeline as a personalized medical assessment system that helps clinicians better monitor the disease progression and the effect of medical treatments.

\textbf{Real-time DNE:} Our current DNE system and the extracted kinematic/spatio temporal features rely on tracking the human pose from the video recordings in an offline step using off-the-shelf pose estimation modules. Currently, the pose estimation step is the most computationally expensive step, hindering real time processing and feature extraction. To address this challenge, on-device lighter pose estimation models (with small sacrifice on the accuracy), that focus on extracting major keypoints rather than the whole body pose are necessary.




\subsection{Pose estimation}
\vspace{-2pt}
\label{ssec:pose-estimation}
For upper body tests (FT, FTF, and FR), we use OpenPose (OP)~\cite{Cao2019_openpose} to estimate the 2D pose. On the other hand, for SAW tests, we compute the 3D pose using the VideoPose3D (VP3D) package~\cite{pavllo2019_videopose3d}. 
OP is designed for multi-person real time 2D hand \cite{Simon2017} and body ~\cite{Cao2019_openpose} pose estimation. Given an RGB image, OP first detects all visible body parts in the image and the corresponding part affinity fields. Associating body parts to each individual in an image boils down to a graph matching problem which can be solved in a greedy fashion. 
Meanwhile, VP3D adopts dilated temporal convolution to estimates 3D pose from sequence of 2D keypoints extracted from the video. 


For upper body tests, if the subject and the moving limb is located parallel to the camera plane, then the motion is well approximated in a plane, i.e. in two dimensions. That is why 2D pose is chosen for upper body tests. However, this might not hold for the SAW test (especially depending on the camera view), hence urging us to use 3D pose for this analysis. 

We use OP and VP3D to extract 2D/3D pose of the recordings in the DNE dataset. We measure the inference time of pre-trained OP models on Google Colab with one Tesla T4 GPU as $6$ FPS. In VP3D, the 2D pose data are extracted for each frame via Detectron2~\cite{wu2019detectron2}, which achieves the  runtime of $12.5$ FPS on a GeForce RTX 2080 GPU. Additionally, it takes roughly $0.2$ seconds for uplifting 2D pose to 3D per video.
\vspace{-10pt}
\subsection{Pre-processing}
\vspace{-2pt}
\label{sec:prep}
We truncate a recording to only include the sequence of frames that are related to the subject performing the test. 
To account for variable distance of the subjects from the camera, we normalize the estimated pose by a reference length. For FT, FTF and FR tests, the reference is the length of the forearm. For SAW, the reference is the distance between the pelvis and neck joints. We compute the reference lengths as the median of the value across all the frames. 
In addition, as the estimated pose can be erroneous at some frames we use median and Savitzky-Golay filtering~\cite{Savitzky1964}. In our dataset, we have excluded $27$ recordings due to unreliable and noisy estimated pose. 
Therefore, we only analyzed $348$ videos in total. %



\vspace{-10pt}
\subsection{Notations}
\vspace{-2pt}
    


\begin{figure}
    \centering
    \includegraphics[width=.8 \linewidth]{resources/keypoints.pdf}
    \vspace{-5pt}
    \caption{Skeleton tree for (a) body ${{B}}$, and (b) hand ${{H}}$. Examples of human pose estimation (c) in 2D (${B}_2, {H}_2$) using OpenPose~\cite{Cao2019_openpose} and (d) in 3D (${B}_3$) using VideoPose3D~\cite{pavllo2019_videopose3d}.} 
    \label{fig:pose_estimation}
\end{figure}
Given the pose sequence estimated from the RGB video, we extract a set of quantified features. Below, we first express our notations and then introduce the features we defined for each test. Let $\boldsymbol{v} = [v_1, ..., v_N]$ denote the set of $N$ frames ordered chronologically in video $\boldsymbol{v}$. There is a one-to-one correspondence between the time associated with each frame and the frame index,  where $\boldsymbol{t} = [t_1, ..., t_N]$ and $t_i = i/fps$, $fps$ denoting the frame per second rate of the video. Given $\boldsymbol{v}$ and the pose estimation module (such as OP or VP3D), we extract the location of $K$ keypoints in each frame. For convenience, we use the same indexing of the body joints for both 2D and 3D pose. However, to differentiate between the 2D and 3D pose, we denote each by ${B}_2$ and ${B}_3$, respectively. Furthermore, we use ${H}_2$ to represent the 2D hand keypoints. An illustration of the hand and body skeleton trees alongside our indexing notations are provided in Fig.~\ref{fig:pose_estimation}. Note that, for the sake of brevity, we have only indexed a subset of the keypoints that we are using in our analysis.





We reserve $\boldsymbol{s}_{k, *}[i]$ for the location of the $k$-th keypoint at frame $i$, corresponding to skeleton tree $* \in \{{H}_2, {B}_2, {B}_3\}$. For $* \in \{{H}_2, {B}_2\}$, $\boldsymbol{s}_{k, *}[i] \in \mathbb{R}^2$ and for $*={B}_3$, $\boldsymbol{s}_{k, *}[i] \in \mathbb{R}^3$. Furthermore, we add superscript $r$ and $l$ to point to right and left (R/L) body parts, respectively. For example, $\boldsymbol{s}^r_{3, {H}_2}[i]$ locates the tip of the right thumb at frame $i$.

To extract kinematic features that quantify the performance of a subject in a test, we track the location of various major keypoints and define a set of features accordingly. Major keypoints vary based on the test. For instance, the major keypoints in FT include the tip of the index and thumb fingers of two hands while in FR, we closely track the wrist joints. 

In different tests, the subjects are asked to move certain limbs repeatedly. 
Thus, it is natural to compute features such as frequency, and amplitude for periodic pose patterns and report the mean and standard deviations (STD) across different cycles. In addition, for a test performed normally, the features 
corresponding to the R/L body parts should be close. Thus, to quantify the difference between the right ${f}^r$ and left ${f}^l$ features, we define an asymmetry metric as:
\begin{align}
   \textrm{Asym}({f}^r, {f}^l) = \frac{\vert {f}^r- {f}^l \vert}{{f}^r + {f}^l}. \label{eq:asym} 
\end{align}
Another useful metric in our analysis is Pearson correlation coefficient denoted by $\text{CC}$. For two 1D discrete time series $\boldsymbol{x}_1$ and $\boldsymbol{x}_2$, we define $\text{CC}$ as:
\vspace{-5pt}
\begin{align}
    \textrm{CC}(\boldsymbol{x}_1, \boldsymbol{x}_2) = \frac{(\boldsymbol{x}_1-\bar{\boldsymbol{x}}_1)^T (\boldsymbol{x}_2-\bar{\boldsymbol{x}}_2)}{\Vert \boldsymbol{x}_1-\bar{\boldsymbol{x}}_1 \Vert_2 \Vert \boldsymbol{x}_2-\bar{\boldsymbol{x}}_2 \Vert_2}.\label{eq:cc}
\end{align}
where $\bar{.}$ and $.^T$ denote the mean and transpose operators.
For highly correlated series, $\vert \textrm{CC} \vert$ is close to one.




\vspace{-8pt}
\subsection{Feature definition}
\vspace{-2pt}
We list the features defined for various tests in Table~\ref{tab:dne-features} and describe them in detail below.

\noindent \textbf{Finger Tapping (FT)}: For this test, the major keypoints are the tip of the R/L thumb and index fingers alongside R/L wrist and elbow joints. To extract properties of the periodic motion, we examine the distance between the tip of the index and thumb fingers across time defined as:

\begin{align}
    \boldsymbol{d}_{\text{ft}}^*[i] = \Vert \boldsymbol{s}^*_{3, {H}_2}[i] - \boldsymbol{s}^*_{6, {H}_2}[i] \Vert_2, \quad * \in \{r, l\}. \label{eq:dist_ft}
\end{align}
Examples of $\boldsymbol{d}_{\text{ft}}^r$ and $\boldsymbol{d}_{\text{ft}}^l$ for normal and abnormal executions of the FT test are provided in Fig.~\ref{fig:ft_features}. In our dataset, to simulate abnormality in FT the subjects are wearing a rubber band around index and thumb fingers of one hand. As also revealed in Fig.~\ref{fig:ft_features}, this limits the tapping amplitude of the hand wearing the band and slows down the tapping rate.
Given $\boldsymbol{d}_{\text{ft}}^*$, we compute the period for the $*$ hand, $T_{\textrm{ft}}^*$, as the time (in seconds) between two consecutive local minima (or maxima) of $\boldsymbol{d}_{\text{ft}}^{*}$. Frequency $F_{\textrm{ft}}^*$ is the reciprocal of $T_{\textrm{ft}}^*$. 
We also report the magnitude of finger-tapping $A^*_{\textrm{ft}}$ as the difference in consecutive minima and maxima of $\boldsymbol{d}_{\text{ft}}^*$. We also report the asymmetry of the periods ($\text{Asym}(T_{\text{ft}}^{r}, T_{\text{ft}}^{l})$), frequencies ($\text{Asym}(F_{\text{ft}}^{r}, F_{\text{ft}}^{l})$) 
and amplitudes ($\text{Asym}(A_{\text{ft}}^{r}, A_{\text{ft}}^{l})$) of R/L hands following~\eqref{eq:asym}.

\input{fig_tex/ft_features}
\input{resources/tft/ftf_features}

Furthermore, we define the instant tapping speed and acceleration for R/L hands as the first and second order derivatives of $\boldsymbol{d}_{\text{ft}}^r$ and $\boldsymbol{d}_{\text{ft}}^l$ with respect to time. We adopt mean and maximum of instant speed and acceleration across tapping cycles as features.
We also introduce average tapping rate as the average number of finger taps per second, obtained by dividing the total number of finger tappings by the FT duration.

Finally, to evaluate the stability of the hands and arms during the FT recording, we examine the wrist and elbow joints. For this purpose, we introduce the relative height between $(\boldsymbol{s}^r_{7, {B}_2},\boldsymbol{s}^l_{7, {B}_2})$ and $(\boldsymbol{s}^r_{6, {B}_2},\boldsymbol{s}^l_{6, {B}_2})$ across $N$ frames:
\vspace{-5pt}
\begin{align}
    C_{\textrm{ft}}^{\textrm{wrist}} &= \frac{1}{N} \sum\limits_{i=1}^N \frac{\Vert \boldsymbol{s}^r_{7, {B}_2}[i]-\boldsymbol{s}^l_{7, {B}_2}[i] \Vert_2}{\Vert \boldsymbol{s}^r_{7, \textrm{B}_2}[i] \Vert_2},
\end{align}
\vspace{-10pt}
\begin{align}
    C_{\textrm{ft}}^{\textrm{elbow}} &= \frac{1}{N} \sum\limits_{i=1}^N \frac{\Vert \boldsymbol{s}^r_{6, {B}_2}[i]-\boldsymbol{s}^l_{6, {B}_2}[i] \Vert_2}{\Vert \boldsymbol{s}^r_{6, {B}_2}[i] \Vert_2}. 
    \label{eq:wrist_stable}
\end{align}
\vspace{-10pt}

\input{fig_tex/tab_features}

\noindent \textbf{Finger to Finger (FTF)}: 
In our dataset, we observe that the estimated pose by OP for middle joints of the index finger, i.e. joint index $5$ in ${H}_2$, is more stable than the outer fingertip. Hence, we focus on this joint for FTF test.
In a normal FTF, the horizontal and vertical trajectories of the R/L hands are symmetric up to a mirroring (Fig.~\ref{fig:ftf-lr-asymmetry}- top row), while this does not necessarily hold for abnormal case (Fig.~\ref{fig:ftf-lr-asymmetry}- bottom row). Thus, in each cycle, we define the cross correlation of the R/L horizontal ($x$) and vertical ($y$) coordinates as the horizontal ($S_{\text{ftf}}^{\text{finger-x}}$) and vertical symmetries ($S_{\text{ftf}}^{\text{finger-y}}$):

\begin{align}
    \label{eq:ftf_finger}
   S_{\text{ftf}}^{\text{finger-x}} &= \text{CC}(\left[\boldsymbol{s}_{5, {H}_2}^l\right]_x, - \left[\boldsymbol{s}_{5, {H}_2}^r\right]_x), \\
   S_{\text{ftf}}^{\text{finger-y}} &= \text{CC}(\left[\boldsymbol{s}_{5, {H}_2}^l\right]_y, \left[\boldsymbol{s}_{5, {H}_2}^r\right]_y)
\end{align}
\noindent where $\left[\boldsymbol{s}_{5, H_2}^* \right]_\dagger = \{\boldsymbol{s}^*_{5, {H}_2}[i](\dagger)\}_{i=1}^{N}$, $\dagger \in \{x, y\}$ and $* \in \{r, l\}$, is the $x$ or $y$ coordinates of the pose series.
We also compute the period and average speed. We derive the average speed by dividing the traversed distance of R/L finger within half a cycle's period by half the cycle's period.


Patients with neurological impairments tend to have tremors while moving their fingers during FTF test \cite{krishna_quantitative_2019}. This leads to a deviation of the fingers' trajectory from a smooth curve. To characterize this deviation, we first fit a smooth curve to the fingers' trajectory, in the form of a second order polynomial in terms of the $x$ and $y$ coordinates. We observe that fitting a second order function to the trajectories, well matches the FTF trajectories of normal subjects. We consider the length of this smooth curve as a reference to compare against the length of the original fingers' trajectory. We then define the ratio of the length of the actual fingers' trajectory during each FTF cycle by the length of the fitted smooth curve as path smoothness metric (PS). We report PS for R/L hands. Examples of normal and abnormal finger trajectories alongside the smooth fitted curves are plotted in Fig.~\ref{fig:ftf-traj}. %



Another feature we found helpful in detecting abnormal function in FTF is instant velocity. We derive the instant velocity vector by the first derivative of the horizontal and vertical pose with respect to time.
We then examine the angle between the vertical and horizontal components of this vector on the R/L hands. At time instant $t$, the \textit{velocity angle} $\boldsymbol{\theta}$ is:
\begin{align}
    \boldsymbol{\theta}^{*}(t) = \textrm{atan2} \left(\frac{\frac{d \, \left[\boldsymbol{s}^*_{5, {H}_2}\right]_{y}}{d t}}{\frac{d \, \left[\boldsymbol{s}^*_{5, {H}_2}\right]_x}{d t}} \right), * \in \{r, l\}. \label{eq:vel_ang}
\end{align}
Next, for each hand, we compare $\boldsymbol{\theta}$ across different cycles using CC in~\eqref{eq:cc}. Given $N_C$ number of cycles, we have $N_C \choose 2$ CC values assessing the symmetry of the R/L velocity angles across different cycles, which we summarize by reporting the mean and STD. Examples of normal and abnormal aligned velocity angles across different cycles are provided in Fig.~\ref{fig:ftf-vel-angle}. Note that for abnormal FTF, large magnitude fluctuations, caused by tremors in moving the hands, visibly appear in $\boldsymbol{\theta}$. 






\noindent \textbf{Forearm Rolling (FR)}: We include the wrist and elbow joints as the major keypoints for this test. We specifically attend to the vertical coordinate of the wrist joints to compute period $T^*_\textrm{fr}$
and amplitudes $A^*_\textrm{fr}$ for $* \in \{r, l\}$. Fig.~\ref{fig:fr_example} illustrates the vertical position of the R/L wrists for a normal and abnormal example. Note that, due to wearing the device in the abnormal recording, the period of the forearm roll cycles for both R/L hands are larger compared to its normal counterpart. In addition, similar to FT, we include the asymmetry of the aforementioned metrics in the FR features.

We also include the maximum instant speed and acceleration derived from vertical coordinates of the wrist joints. Similar to FT, we define rolling speed and rate. Rolling speed is computed as the difference between the minimum and maximum of $y$ coordinate of the R/L hands divided by half the rolling period. Also, rolling rate is defined as the number of rolling cycles per second. Finally, we report the stability of the elbows $C_{\textrm{fr}}^{\textrm{elbow}}$ and define it analogous to~\eqref{eq:wrist_stable}. %

\input{fig_tex/fr_features}
\input{resources/saw/saw_features}

\noindent \textbf{Stand-up and Walk (SAW):} We use the side-view SAW recordings in our analysis of SAW test. For SAW pose estimation, we use VP3D~\cite{pavllo2019_videopose3d}. In VP3D, the joint locations are defined relative to the pelvis joint. As a result, estimated pose by VP3D misses the global position of subjects within a frame which is essential to detect different segments
of the SAW test, i.e. stand-up (SU), walk (W), and turn (TU). This urged us to track the 2D position of the pelvis $\boldsymbol{s}_{0,B_2}$ extracted by OP as a notion of subject's global position in a video frame. Analyzing this position through time enables us to split a SAW recording into multiple non-overlapping SU, W, and TU segments. Supplementary Fig. S3 visualizes these segments.

For the SU segment of SAW, we focus on the time to stand~\cite{Jawad2012}, measured by the total time taken from the first SU effort to a full standing on feet state.
We derive time to stand by thresholding the magnitude of the pelvis joint's velocity.
Note that, since our subjects are asked to walk back and forth a designated room multiple times, at some points, they have to change direction and turn around. We report time to turn around as another indicative feature for SAW test.


The first set of features derived for the walking segment are obtained based on the distance between the two feet stated as:
\vspace{-15pt}
\begin{align}
    \boldsymbol{d}_{\textrm{saw}}[i] = ||\boldsymbol{s}^r_{2,{B}_3}[i]-\boldsymbol{s}^l_{2,{B}_3}[i]||_2.
\end{align}
Note that, the periodic nature of a normal gait also reflects in $\boldsymbol{d}_{\textrm{saw}}$ (see Fig.~\ref{fig:saw-step-info}). Given $\boldsymbol{d}_{\textrm{saw}}$, we highlight different W and TU segments in Fig.~\ref{fig:saw-step-info}. For a gait pattern derived based on $\boldsymbol{d}_{\textrm{saw}}$, \textit{step time} is the time to complete one step and computed as the time difference between two consecutive local maxima of $\boldsymbol{d}_{\textrm{saw}}$. Meanwhile, \textit{step length} defined as linear distance between two successive placements of the same foot~\cite{Herran2014} manifests as the local maxima of $\boldsymbol{d}_{\textrm{saw}}$. The \textit{step width}, on the other hand, is interpreted as the local minima of $\boldsymbol{d}_{\textrm{saw}}$. The calculations of these features in turning segments are excluded.



As two global features for gait, we report mean and STD of \textit{cadence} and \textit{average speed} across all W segments. We compute cadence as the number of steps divided by the duration of a walking segment. Average speed is determined by the total traveled distance of the pelvis joint divided by the duration of a walking segment.


To evaluate the symmetry of the R/L gait, we introduce the cross correlation between the knee angle series of R/L legs, denoted by {${S}_{\textrm{saw}}^{\textrm{knee angle}}$}. We find this feature a good descriptive of gait abnormality, as in our recordings, gait abnormality is introduced through wearing a knee band which limits the knee motion (Fig.~\ref{fig:dne_dataset}). For each frame, we define the knee angle as the angle between $\boldsymbol{s}^r_{3, {B}_3}-\boldsymbol{s}^r_{4, {B}_3}$ and $\boldsymbol{s}^r_{3, {B}_3}-\boldsymbol{s}^r_{2, {B}_3}$ for the right leg and $\boldsymbol{s}^l_{3, {B}_3}-\boldsymbol{s}^l_{4, {B}_3}$ and $\boldsymbol{s}^l_{3, {B}_3}-\boldsymbol{s}^l_{2, {B}_3}$ for the left leg. As there is a lag between the R/L gait cycles, we align the knee angle series of the R/L legs within each cycle and then report CC of the aligned series. Examples of aligned normal and abnormal knee angles for R/L legs are shown in Fig.~\ref{fig:saw-knee-angle}. For normal gait, the R/L knee angles are highly correlated after alignment (Fig.~\ref{fig:saw-knee-angle}-top row), while this does not hold for abnormal gait (Fig.~\ref{fig:saw-knee-angle}-bottom row).  

In addition, we define step symmetry between the R/L feet movements by comparing the horizontal position of R/L feet at different gait cycles. We represent this metric by {${S}_{\textrm{saw}}^{\textrm{feet-x}}$}. To compute {${S}_{\textrm{saw}}^{\textrm{feet-x}}$}, similar to {${S}_{\textrm{saw}}^{\textrm{knee angle}}$}, we first align the R/L horizontal positions within each gait stride and report the CC of the aligned series. We report mean and STD for both {${S}_{\textrm{saw}}^{\textrm{feet-x}}$} and {${S}_{\textrm{saw}}^{\textrm{knee angle}}$} across different cycles. %

