\section{Introduction}
Stroke is the leading cause of disability and the most common neurological disorder worldwide \cite{feigin2017global}. An early and extensive physical therapy session with task-oriented exercises is an effective intervention for post-stroke subjects to regain their functional ability \cite{o2019physical}. However, post-stroke survivors can receive only limited amount of those supervised sessions due to the costs and the shortage of therapists.

Recent advance in computing and artificial intelligence empowers a robot with various autonomous capabilities to understand and interact with the world \cite{hirschberg2015advances}. Researchers have explore the possibility of supplementing health services with advanced computing \cite{webster2014systematic} and socially assistive robotics (SAR) \cite{feil2005defining}. For instance, researchers envision that a SAR can be integrated into the rehabilitation process by automatically monitoring patient's exercise and providing motivation on patient's in-home rehabilitation until patient's next visits to a therapist \cite{mataric2007socially}. Prior work demonstrates that the physical embodiment of a robot is considered more effective and acceptable for elderly people than the virtual robot embodiment \cite{fasola2013socially}. Elderly or post-stroke subjects can successfully exercise and stay engaged with a robot over a single \cite{swift2015effects} or multiple sessions \cite{gorer2017autonomous}. 

In spite of this potential, prior work is limited to provide pre-defined corrective feedback on patient's exercise performance (e.g. angular difference with a motion template \cite{fasola2013socially,gorer2017autonomous}), which is fixed over all subjects. Generating tailored feedback for an individual still remain a challenge \cite{gorer2017autonomous}. 











In this paper, we presents an approach of motion analysis that can automatically identify the most salient features for assessment to predict the quality of motion and generate personalized corrective feedback. 
this paper presents an interactive assistive robot (Figure \ref{fig:flow-diagram}) that can automatically identify salient features for assessment and predict the quality of motion. Utilizing identified kinematic features and user's held-out normal motions, a robot can analyze which kinematic variables have been deviated from normal motions, and generate audio-based real-time feedback on a patient's exercise motion (Figure \ref{fig:web-interface}). In future, this interactive personalized assessment will be deployed to a socially assistive robot \cite{shamsuddin2012initial} and post-stroke survivors will be recruited to conduct a study whether the presented interactive assistive robot with personalized corrective feedback can enhance user's engagement to participate in rehabilitation exercises. 


\begin{comment}
\begin{figure*}[tp!]
\centering
\begin{subfigure}[t]{.4\textwidth}
\centering
  \includegraphics[width=1.0\columnwidth]{figures/flow.png}
  \caption{}
  \label{fig:flow-diagram}
\end{subfigure}\hfill
\begin{subfigure}[t]{.6\textwidth}
  \centering
  \includegraphics[height=0.3\columnwidth]{figures/setup-together.png}
  \caption{}
  \label{fig:web-interface}
\end{subfigure}
    \caption{(a) Flow diagram of a Socially Assistive Robot for Personalized Stroke Rehabilitation Therapy. (b) Figure on the left describes the setup of a system with a Kinect sensor, a tablet with the interface, the NAO robot. Figure on the right indicates an example of rehabilitation assessment with corrective feedback.}
\end{figure*}
\end{comment}

\begin{figure*}[tp!]
\centering
\begin{subfigure}[t]{.4\textwidth}
\centering
  \includegraphics[width=1.0\columnwidth]{figures/flow.png}
  \caption{}
  \label{fig:flow-diagram}
\end{subfigure}
\begin{subfigure}[t]{.2\textwidth}
  \centering
  \includegraphics[height=0.9\columnwidth]{figures/setup-robot.png}
  \caption{}
  \label{fig:web-interface}
\end{subfigure}
\begin{subfigure}[t]{.35\textwidth}
  \centering
  \includegraphics[width=1.0\columnwidth]{figures/setup-feedback.png}
  \caption{}
  \label{fig:web-interface}
\end{subfigure}
    \caption{(a) Flow diagram of an interactive approach for an assistive robot during physical therapy. (b) the setup of a system with a Kinect sensor, a tablet with the visualization interface, the NAO robot. (c) An example output of the visualization interface that presents assessment of patient's exercise performance with corrective feedback.}
\end{figure*}

\section{Interactive Assistive Robot for Personalized Rehabilitation Assessment} \label{sect:monitor-exercise}


It is questionable how to select kinematic variables for assessment and how well a robot can provide personalized assessment of a patient's rehabilitation exercise. 
If a robot just notifies the output of pre-defined corrective feedback, a patient might consider a robot as not useful and reject to interact with it more \cite{pain2007readiness}. Thus, it is critical to derive a personalized assessment model of a robot for physical therapy sessions. 


Even if prior work demonstrated the feasibility of automatically assessing rehabilitation exercises \cite{Lee2019LAQ}, it is challenging to replicate therapist's assessment. In addition, simply presenting the predicted assessment on patient's rehabilitation performance is not convincing to a user. This work aims to support a transparent interaction between a system and user with a sparse model by feature selection \cite{biran2017explanation}. Specifically this paper applies reinforcement learning to identify salient feature for assessment. Utilizing an identified kinematic features, this approach predict the quality of motion and generate personalized corrective feedback with held-out user's normal motions (Figure \ref{fig:web-interface}). 

For identifying salient features of assessment, this paper utilizes Reinforcement Learning with Double Q-learning \cite{van2016deep} to train an agent that sequentially decides whether another feature is necessary to assess an exercise while receiving a negative reward for requesting an additional feature. For Prediction Model (PM), Neural Network (NN) is trained for the prediction model while grid-searching various architectures (i.e. one to three layers with 32, 64, 128, 256, 512 hidden units) and learning rates (i.e. $0.0001, 0.005, 0.001, 0.01, 0.1$) using \textit{`PyTorch'} libraries \cite{paszke2017automatic}. \textit{`ReLu'} activation and \textit{`AdamOptimizer'} are applied and a model is trained until the tolerance of optimization is $0.0001$ or the maximum $200$ iterations. The neural network of Q-learning has the same architecture of a Neural Network for the Prediction Model (PM).

For the evaluation, this paper utilizes the dataset of three task-oriented, upper-limb stroke rehabilitation exercises recommended by therapists (\textit{`Bring a cup to the mouth', `Switch a light on'}, and \textit{`Move forward a cane'}) \cite{Lee2019LAQ}. Fifteen stroke patients with different level of functional abilities (37 $\pm$ 21 Fugl Meyer Scores) and 11 healthy subjects were participated for data collection with a Kinect v2 sensor  (Microsoft, Redmond, USA) to record the trajectory of body joints at 30 Hz. A patient performed 10 repetitions of each exercise with both affected and unaffected sides. A healthy subject performed 15 repetitions of each exercise with the subject's dominant side. A therapist annotated the entire dataset for implementation.

In the result so far, the prediction model achieves the good agreement level with therapist's evaluation: 0.8090 - 0.8806 F1-scores. In addition, this paper implements an interface with Kinect SDK and Accord.NET framework \cite{de2014accord}, which can present a patient's exercise motion with joint trajectories, assessment, and real-time audio-based corrective feedback (Figure \ref{fig:web-interface}). A communication channel between an assessment model with python and an interface is implemented with socket programming, which shares the predicted assessment and corrective feedback using identified kinematic variables and user-data.

\section{Conclusion and Future Work}
This paper describes an interactive assessment technique with feature selection that derives personalized motion analysis while generating transparent corrective feedback of motion analysis in terms of real-time audio feedback. The presented interactive assessment achieves good congruence with therapist's evaluation. In future, this interactive technique will be deployed on a socially assistive robot, \cite{shamsuddin2012initial}. In addition, following the study design of \cite{fasola2013socially}, the presented technique will be evaluated on the usefulness of personalized corrective feedback during a post-stroke survivor's rehabilitation session. 



\begin{comment}
\section{Acknowledgement}
Thank you to Daniel Siewiorek, Asim Smailagic, Alexandre Bernardino, Sergi Berm{\'u}dez i Badia, Nicola Palmarini, and Peri Tarr for fruitful discussion that helped expand my thinking and research, although they may not agree with all interpretations/conclusions of this paper. The work on monitoring daily activities was done during an internship at IBM Research. All opinion expressed are my own and do not represent any institutional endorsement.



It is questionable how to select kinematic variables for assessment and how well a robot can provide personalized assessment of a patient's rehabilitation exercise. 
If a robot just notifies the output of pre-defined corrective feedback, a patient might consider a robot as not useful and reject to interact with it more \cite{pain2007readiness}. Thus, it is critical to derive a personalized assessment model of a robot for physical therapy sessions. 




In this paper, we present an interactive approach of an assistive robot, which can automatically identify the most salient features to predict the quality of rehabilitation exercises and generate personalized corrective feedback (Figure \ref{fig:approach}). For the development, our approach utilizes the dataset of three upper-limb stroke rehabilitation exercises from 15 post-stroke and 11 healthy subjects that are annotated by therapists. With this dataset, our approach utilizes reinforcement learning \cite{van2016deep} to identify salient features of assessment and predict the quality of motion with a supervised learning algorithm (e.g. Neural Network). When a new patient performs the exercise with patient's unaffected and affected side, our approach dynamically selects user-specific features for assessment and predict the quality of motion with corrective feedback (Figure \ref{fig:interface}).

A new patient can first perform exercise with patient's unaffected side 

When a new patient performs the exercise with patient's unaffected and affected side, 


Once a new patient performs the exercise with patient's unaffected and affected side, the visualization interface of our approach (Figure \ref{fig:web-interface}) shows patient-specific analysis: the predicted quality of motion on three performance components (i.e. \textit{`Range of Motion'}, \

With the selected features and held-out user's unaffected

g to identify salient features of assessmentwith reinforcement learning and predict the quality of motion withsupervised learning algorithms (e.g. Neural Networks). Utilizingidentified features and held-out user data, testing patient’s unaf-fected motions, our approach generates patient-specific analysisthat empowers therapists to understan


dynamically select the most important features for an individual's motion. 



From this dataset, we apply reinforcement learning to identify the most important features for assessment and learn a machine learning model to predict the scores on the exercises of patients. Once a new patient performs the exercise with patient's unaffected and affected side, the visualization interface of our approach (Figure \ref{fig:web-interface}) shows patient-specific analysis: the predicted quality of motion on three performance components (i.e. \textit{`Range of Motion'}, \textit{`Smoothness'}, \textit{`Compensation'}) and the comparison between unaffected and affected sides on the most important kinematic features. Therapists can review this analysis to better understand patient's performance and provide feedback (e.g. feature relevance) to tune a model for personalized rehabilitation assessment.




for the assessment on rehabilitation exercises

of an assistive robot for personalized physical therapy.

that can automatically identify the most salient features for assessment to predict the quality of motion and generate personalized corrective feedback  (Figure \ref{fig:flow-diagram}). 

\end{comment}

\section{Introduction}
An early and extensive physical therapy session is an effective intervention for patients with neurological and musculoskeletal problems (e.g stroke)  to regain their functional ability.
However, patients can receive only a limited amount of sessions due to the costs and the shortage of therapists.

Researchers have explored the possibility of supplementing health services with advanced computing and a socially assistive robot \cite{mataric2007socially}. For instance, researchers envision that a socially assistive robot can be integrated into the rehabilitation process by automatically monitoring patient's exercises and providing motivational feedback until the patient's next visits to a therapist \cite{mataric2007socially}. Prior work on robotic exercise coaching systems demonstrates elderly or post-stroke subjects can successfully exercise and stay engaged with a robot over a single \cite{swift2015effects} or multiple sessions \cite{fasola2013socially,gorer2017autonomous}. 
However, in spite of this potential of a robot to monitor and guide exercises, prior work is limited to provide pre-defined corrective feedback on patient's exercise performance (e.g. angular difference with a motion template \cite{fasola2013socially,gorer2017autonomous}).
Generating personalized interaction and feedback for an individual patient still remain a challenge \cite{gorer2017autonomous}. 


To address this challenge, this paper presents an interactive approach of a socially assistive robot for personalized post-stroke therapy (Figure \ref{fig:approach})   \cite{ijcai2019-lee}. This approach utilizes reinforcement learning to dynamically select the most important kinematic features of stroke rehabilitation assessment for individual patient's exercise motions to predict the quality of motion \cite{lee2020iha}. Utilizing selected features and patient's held-out normal motions, a robotic system can analyze which features of an affected motion have been deviated from those of normal motions, and generate personalized corrective feedback on a patient's exercise motion (Figure \ref{fig:interface}) \cite{ijcai2019-lee}. 





\begin{figure*}[tp!]
\centering
\begin{subfigure}[t]{.4\textwidth}
\centering
  \includegraphics[width=1.0\columnwidth]{figures/flow.png}
  \caption{}
  \label{fig:flow-diagram}
\end{subfigure}
\begin{subfigure}[t]{.2\textwidth}
  \centering
  \includegraphics[height=0.91\columnwidth]{figures/setup-robot.png}
  \caption{}
  \label{fig:setup}
\end{subfigure}
\begin{subfigure}[t]{.38\textwidth}
  \centering
  \includegraphics[width=1.0\columnwidth]{figures/setup-feedback.png}
  \caption{}
  \label{fig:interface}
\end{subfigure}
    \caption{(a) Flow diagram of an interactive approach of an assistive robot for personalized physical therapy. (b) the setup of a system with a Kinect sensor, a tablet with the visualization interface, and the NAO robot. (c) An example output of the visualization interface that presents predicted assessment on patient's exercise performance with corrective feedback.}
\end{figure*} \label{fig:approach}
\section{Interactive Assistive Robot for Personalized Rehabilitation Assessment} \label{sect:monitor-exercise}
\section{Interactive Approach of a Socially Assistive Robot for Personalized Physical Therapy}
This work aims to support transparent and personalized interaction of a robot exercise coach that utilizes a sparse machine learning model with feature selection \cite{biran2017explanation} to predict the quality of motion and generate corrective feedback (Figure \ref{fig:interface}) with held-out patient's normal motions \cite{ijcai2019-lee}. 

We represent an exercise motion with sequential joint coordinates from a Kinect v2 sensor and extract various kinematic features similar to \cite{Lee2019LAQ}: joint angles (e.g. elbow flexion, shoulder flexion, elbow extension, shoulder abduction, the tilted angle of head, spine, and shoulder), speed related features (e.g. speed, acceleration, jerk, etc.) on wrist and elbow joints, and normalized relative trajectory (i.e. the Euclidean distance head and wrist and head and elbow). 

For feature selection, this paper utilizes reinforcement learning (RL) to dynamically identify salient features of assessment for individual patient's motions \cite{lee2020iha}. Specifically, we apply Double Q-learning \cite{van2016deep} to train an agent that sequentially decides whether another feature is necessary to assess an exercise while receiving a negative reward for requesting an additional feature or misclassification. Although the classical approaches of feature selection (e.g. filter, wrapper, embedded methods) select a fixed set of features with training data for all patients, our approach of feature selection with RL finds an optimal set of features for individual patient's motions \cite{lee2020iha}. Thus, we hypothesize and demonstrate that feature selection with RL is beneficial over classical feature selection approaches to generate personalized rehabilitation assessment and feedback.

For a machine learning (ML) model to predict assessment, we utilize a Neural Network (NN) while grid-searching various architectures (i.e. one to three layers with 32, 64, 128, 256, 512 hidden units) and learning rates (i.e. $0.0001, 0.005, 0.001, 0.01, 0.1$) using \textit{`PyTorch'} libraries \cite{paszke2017automatic}. \textit{`ReLu'} activation and \textit{`AdamOptimizer'} are applied, and a model is trained until the tolerance of optimization is $0.0001$ or the maximum $200$ iterations.

\section{Dataset of Three Upper-Limb Exercises}
For the evaluation, this paper utilizes the dataset of three task-oriented, upper-limb stroke rehabilitation exercises suggested by therapists: \textit{`Bring a cup to the mouth', `Switch a light on'}, and \textit{`Move forward a cane'} \cite{lee2020opportunities}. Fifteen stroke patients with different levels of functional abilities (37 $\pm$ 21 Fugl Meyer Scores) and 11 healthy subjects participated in data collection with a Kinect v2 sensor (Microsoft, Redmond, USA) that records the trajectory of body joints at 30 Hz. A patient performed 10 repetitions of each exercise with both patient's affected and unaffected sides. A healthy subject performed 15 repetitions of each exercise with the subject's dominant side. 
Two therapists annotated the dataset to implement our approach and compute therapist's agreement (TP in Table \ref{tab:results-avg}). 





\section{Results}
The machine learning (ML) model with reinforcement learning based feature selection (ML - RL) achieves the good agreement level with therapist's annotation: 0.7973 - 0.8331 average F1-scores on three exercises, which is comparable with therapist's agreement (TP in Table \ref{tab:results-avg}). In addition, our approach (ML - RL) achieves 0.11 higher average F1-score than the ML model with Recursive Feature Elimination, one of classical feature selection methods (ML - RFE). Thus, these results show that our approach can perform better to generate personalized assessment and corrective feedback.
For the interaction with patients, we implement the interface that presents the tracked joints of a patient's exercise motion, predicted assessment on patient's exercise performance, and real-time audio and visual corrective feedback (Figure \ref{fig:interface}). 

\begin{table}[htp]
\centering
\caption{Performance (F1-scores) of our approach (ML - RL), the baseline approach with Recursive Feature Selection (ML - RFE), and therapist's agreement (TP)}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{ccccc} \toprule
 & Exercise 1 (E1) & Exercise 2 (E2) & Exercise 3 (E3) & Overall \\ \midrule  \midrule 
  ML - RL & 0.8331$\pm$	0.0059 & 0.7973 $\pm$ 0.0867
 & 0.8053 $\pm$	0.0496 & 0.8119 $\pm$ 0.0526\\ \midrule
 ML - RFE & 0.6742 $\pm$ 0.0715 & 0.7628 $\pm$ 0.1708
 & 0.6415 $\pm$	0.0806 & 0.6928 $\pm$ 0.1147 \\ \midrule
 
 \begin{tabular}[c]{@{}c@{}}TP\end{tabular} & {0.7455 $\pm$ 0.2054} & {0.8147 $\pm$ 0.1522} & {0.7254 $\pm$ 0.1838} & {0.7619 $\pm$ 0.1626} \\ \bottomrule
\end{tabular} 
}
\label{tab:results-avg}
\end{table}











\section{Conclusion and Future Work}
This paper describes an interactive approach of a socially assistive robot that applies reinforcement learning for dynamic feature selection on individual patient's rehabilitation exercises to assess the quality of motion and generate personalized corrective feedback. 
The evaluation with the annotated dataset of three stroke rehabilitation exercises shows that our approach achieves good congruence with therapist's annotation, but also allows to generate transparent and personalized corrective feedback. In future, we will evaluate the usefulness of personalized corrective feedback from the NAO robot (Figure \ref{fig:setup}) for coaching post-stroke subject's rehabilitation exercises. 

\subsection{Equations}

The equations are an exception to the prescribed specifications of this template. You will need to determine whether or not your equation should be typed using either the Times New Roman or the Symbol font (please no other font). To create multileveled equations, it may be necessary to treat the equation as a graphic and insert it into the text after your paper is styled. Number equations consecutively. Equation numbers, within parentheses, are to position flush right, as in (1), using a right tab stop. To make your equations more compact, you may use the solidus ( / ), the exp function, or appropriate exponents. Italicize Roman symbols for quantities and variables, but not Greek symbols. Use a long dash rather than a hyphen for a minus sign. Punctuate equations with commas or periods when they are part of a sentence, as in

$$
\alpha + \beta = \chi \eqno{(1)}
$$

Note that the equation is centered using a center tab stop. Be sure that the symbols in your equation have been defined before or immediately following the equation. Use Ò(1)Ó, not ÒEq. (1)Ó or Òequation (1)Ó, except at the beginning of a sentence: ÒEquation (1) is . . .Ó

\subsection{Figures and Tables}

Positioning Figures and Tables: Place figures and tables at the top and bottom of columns. Avoid placing them in the middle of columns. Large figures and tables may span across both columns. Figure captions should be below the figures; table heads should appear above the tables. Insert figures and tables after they are cited in the text. Use the abbreviation ÒFig. 1Ó, even at the beginning of a sentence.

\begin{table}[h]
\caption{An Example of a Table}
\label{table_example}
\begin{center}
\begin{tabular}{|c||c|}
\hline
One & Two\\
\hline
Three & Four\\
\hline
\end{tabular}
\end{center}
\end{table}


   \begin{figure}[thpb]
      \centering
      \framebox{\parbox{3in}{We suggest that you use a text box to insert a graphic (which is ideally a 300 dpi TIFF or EPS file, with all fonts embedded) because, in an document, this method is somewhat more stable than directly inserting a picture.
}}
     
      \caption{Inductance of oscillation winding on amorphous
       magnetic core versus DC bias magnetic field}
      \label{figurelabel}
   \end{figure}
   

Figure Labels: Use 8 point Times New Roman for Figure labels. Use words rather than symbols or abbreviations when writing Figure axis labels to avoid confusing the reader. As an example, write the quantity ÒMagnetizationÓ, or ÒMagnetization, MÓ, not just ÒMÓ. If including units in the label, present them within parentheses. Do not label axes only with units. In the example, write ÒMagnetization (A/m)Ó or ÒMagnetization {A[m(1)]}Ó, not just ÒA/mÓ. Do not label axes with a ratio of quantities and units. For example, write ÒTemperature (K)Ó, not ÒTemperature/K.Ó

\end{comment}




\addtolength{\textheight}{-12cm}  
                                 
                                 
                                 
                                 
                                 







\begin{comment}
\section*{APPENDIX}
Appendixes should appear before the acknowledgment.

\section*{ACKNOWLEDGMENT}
The preferred spelling of the word ÒacknowledgmentÓ in America is without an ÒeÓ after the ÒgÓ. Avoid the stilted expression, ÒOne of us (R. B. G.) thanks . . .Ó  Instead, try ÒR. B. G. thanksÓ. Put sponsor acknowledgments in the unnumbered footnote on the first page.
\end{comment}




\bibliographystyle{IEEEtran}
