\section{INTRODUCTION}
Building robots capable of adapting to their environments solely using environmental interactions is one of the long-standing goals of embodied artificial intelligence. 
Such a capability entails learning on the fly as the agent interacts with the physical world, also known as \emph{real-time learning}. The reinforcement learning (RL) framework is well-suited to formulate real-time learning tasks since an RL agent can discover optimal behaviour through trial-and-error interactions with its environment \shortcite{sutton2018reinforcement}. 

While researchers have developed impressive deep RL methods to solve complex motor control problems \shortcite{haarnoja2018sac_apps, schulman2017proximal, abdolmaleki2018maximum}, these techniques do not easily extend to real-time learning systems that operate under time and resource constraints (e.g., quadrotors and mobile robot bases). While approaches including learning from demonstrations \shortcite{gupta2016learning, vasan2017learning}, sim-to-real \shortcite{peng2018sim, bousmalis2018using}, and offline RL \shortcite{levine2020offline} have been used to develop pre-trained agents, there has been relatively little interest in studying real-time learning agents in the physical world. 

The primary challenge of real-time learning is that the real world does not pause while the agent computes actions or makes learning updates. In the real-time learning setting, an agent obtains sensorimotor information from various devices and executes action commands at specific frequencies. Given these constraints, an RL agent must compute an action within a chosen action cycle time and perform learning updates without disrupting the periodic execution of actions. 

\begin{figure}[t]
\centering
\includegraphics[width=0.45\textwidth]{ROA_env_agent_interaction.pdf}
\caption{Remote-Onboard system: An overview of the agent-environment interaction in this system.}
\label{fig:ROA_env_agent_interaction}
\end{figure}

The problems inherent to real-time learning systems are particularly exacerbated when only resource-limited computers are available to a robot. 
Most onboard embedded computers (e.g., Jetson Nano and Raspberry Pi) cannot run a computationally intensive algorithm such as Soft Actor-Critic. 
The primary goal of our work is to develop a system that leverages state-of-the-art deep RL methods to support real-time learning on resource-limited computers. On the other hand, the system should introduce minimum overhead to RL algorithms so that it is also suitable for conventional RL research and powerful workstations.
We propose a way to decouple the agent-environment interaction loop from policy learning to make computationally intensive learning methods feasible for real-time learning. We exploit multi-threading and multiprocessing capabilities to handle the intensive computations of state-of-the-art RL algorithms effectively. 

The primary contribution of our paper is a task-agnostic \textit{remote-onboard} system that can support various model-free deep RL algorithms. 
We decouple agent-environment interaction and policy learning into two distinct elements, namely the \textit{performer} and \textit{learner} as shown in Figure \ref{fig:ROA_env_agent_interaction}. 
In the proposed system, all computationally intensive operations such as gradient updates and sampling from replay buffer are occur within the learner and run on a powerful remote workstation. In contrast, all elements essential for the agent to interact with the real world are a part of the performer and run on a resource-limited onboard computer.

We hypothesize that such a decentralized system enables resource-limited systems to take advantage of resource-demanding deep RL algorithms in real-time learning setting. To fully utilize the computational resources available to a learning agent, our system supports three modes: 1) {remote-only}, 2) {remote-onboard}, and 3) {onboard-only} (see section \ref{arch} for details). We evaluate the performance of the three modes on two visuomotor policy learning tasks using the UR5 industrial robot arm and iRobot Create2, a mobile robot. We use two different learning algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), to assess the effectiveness of our system. 

Our second hypothesis considers the setting where all computations happen on a single computer. In this scenario, we presume that the performance of our system will be on par with off-the-shelf implementations that require lots of computational resource requirements to learn real-time tasks. To assert this, we compare the performance of our system with \citeA{yuan2021asynchronous}'s implementation on the UR5-VisualReacher task on a single workstation, although the latter overall learns slightly faster.


\section{RELATED WORK}

SenseAct is a computational framework for physical robots that allows learning experiments to be reproducible in different locations and under diverse conditions \shortcite{mahmood2018setting}. Although SenseAct supports the systematic design of robotic tasks for RL, it does not address how to design real-time learning agents. It is unclear how best to order and distribute computations relevant to learning. We use the guiding principles of SenseAct to design the tasks employed in this paper to showcase the effectiveness of our approach. 


\citeA{krishnan2019air} introduced an open-source simulator and a gym environment for quadrotors. Given that these aerial robots need to accomplish their tasks with limited onboard energy, it is impossible to run computationally intensive RL methods on the onboard hardware. Since onboard computing is a scarce resource and updating RL policies is computationally intensive, they carefully designed policies considering the power and computational resources available onboard. They focused on sim-to-real techniques for learning instead of using experience generated in the real world, thus making their method unsuited to real-time learning.   

\citeA{nair2015massively} proposed a distributed learning architecture called GORILA framework that mainly focused on using multiple actors and learners to collect data in parallel and accelerate training in simulation using clusters of CPUs and GPUs. On the other hand, our work focuses primarily on how best to distribute performer and learner between resource-limited onboard computers and powerful remote workstations to enable real-time learning. In other words, GORILA framework is conceptually akin to the DistBelief \shortcite{distbelief} architecture and our system is designed to allow resource limited computers to take advantage of resource demanding RL algorithms in real-time learning tasks. In addition, the GORILA framework is customized to Deep Q-Networks (DQN) while our system can support multiple RL algorithms.

A recent paper by \citeA{bloesch2021towards} used a distributed version of Maximum aposteriori Policy Optimization (MPO) \cite{abdolmaleki2018maximum} to learn how to walk with Robotis \href{https://emanual.robotis.com/docs/en/platform/op3/introduction/}{OP3 bipedal robots}. The robots’ onboard computer samples actions and periodically synchronizes the policy’s neural network weights with a remote learner process at the start of each episode. They roll out one episode using the current policy, after which the onboard computer sends all data from that episode to a remote replay buffer implemented using the Reverb library \shortcite{cassirer2021reverb}. \citeA{haarnoja2018learning} also proposed a similar asynchronous learning system tailored to learn a stable gait using SAC and the minitaur robot \cite{kenneally2016design}. However, this approach does not solve any image-based control task. Unlike these two methods, our system is task-agnostic and compatible with multiple algorithms. In addition, our code is publicly available to make it easier for the research community to incorporate the system in their work. 


\section{Background}
Reinforcement learning is a setting where a learning agent can discover the optimal behaviors through trial-and-error interactions with its environment. The agent-environment interaction is modeled using a Markov Decision Process (MDP), where an agent interacts with its environment
at discrete time steps. At each step t, the agent obtains its state information $S_t \in \mathcal{S}$ and a scalar reward signal $R_t \in \mathcal{R}$ from the environment. The agent uses a stochastic policy $\pi$ with a probability distribution $\pi(a|s) \doteq \mathbb{P} (A_t = a|S_t = s)$ to sample an action $A_t \in \mathcal{A}$. After executing the action, the agent transitions to the next state $S_{t+1}$ and obtains a reward $R_{t+1}$ at the subsequent timestep ${t+1}$ according to a transition probability distribution $p(s', r | s, a) = \mathbb{P} (S_{t+1}=s, R_{t+1}=r | S_t = s, A_t = a)$. The state space $\mathcal{S}$ and the action space $\mathcal{A}$ are assumed to be continuous or discrete. The standard RL objective maximizes the expected sum of rewards $\mathbb{E}_{A_t \sim \pi} [\sum_t  R_{t+1}]$. 

\subsection{Soft Actor-Critic}
Soft Actor-Critic (SAC) is an off-policy RL algorithm where the policy is trained to maximize a trade-off between the expected return and the policy's entropy \shortcite{haarnoja2018soft}. Instead of the standard RL objective,
it uses the maximum entropy objective proposed by \shortcite{ziebart2010modeling}:
\begin{align*}
    \pi^{*} = \argmax_\pi  \mathbb{E}_{A_t \sim \pi} \left[ \sum_{t=0}^T R_{t+1} + \alpha \mathcal{H}\pi(\cdot |S_t)\right],
\end{align*}
where $\alpha$ is the temperature parameter determining the relative importance of entropy compared to reward.

SAC concurrently learns a policy $\pi_\phi(A_t|S_t)$, where $\phi$ denotes the policy parameters and a soft-Q function $Q_\theta (S_{t}, A_{t})$, where $\theta$ denotes the Q-network parameters. The policy is learned by maximizing the following objective:
\begin{align*}
    J_\pi (\phi) &= \mathbb{E}_{S_t \sim \mathcal{D}} \left[ \mathbb{E}_{A_t \sim \pi_\phi} [\alpha \log{\pi_\phi (A_{t} | S_{t})} - Q_\theta (S_{t}, A_{t})] \right].
\end{align*}
The soft-Q function incorporates an entropy bonus to the action-value estimate unlike a regular Q-function. The parameters of the soft Q-function are trained to minimize the soft Bellman residual:
\begin{align*}
    J_Q (\theta) &= \mathbb{E}_D \left[ \frac{1}{2} \left( R_{t+1} + \gamma V_{\Tilde{\theta}}(S_{t+1})-Q_\theta (S_t, A_t)  \right)^2 \right],
\end{align*}
where the soft value function $V_{\Tilde{\theta}}$ is defined as:
\begin{align*}
    V_{\Tilde{\theta}}(S_{t}) &= \mathbb{E}_{\pi} \left[ Q_\theta (S_{t}, A_{t}) - \alpha \log{\pi_\phi (A_{t} | S_{t})} \right].
\end{align*}
The temperature parameter $\alpha$ can be learned by approximating dual gradient descent \shortcite{haarnoja2018sac_apps}:
\begin{align*}
    J (\alpha) &= \mathbb{E}_{S_t \sim \mathcal{D}} \left[\mathbb{E}_{A_t \sim \pi_t} [-\alpha \log{\pi_\phi (A_t|S_t)} - \alpha \Tilde{\mathcal{H}} ]    \right],
\end{align*}
where $\Tilde{\mathcal{H}}$ is a desired minimum expected entropy.



 

\subsection{Proximal Policy Optimization (PPO)}
PPO \shortcite{schulman2017proximal} is an on-policy policy gradient algorithm that alternates between collecting data via interactions with the environment and optimizing a clipped surrogate loss function. It limits the magnitude of the policy gradient step to prevent overly large updates that can prove detrimental to learning. The following objective function is used to train the policy:

\begin{align*}
 L_{\theta}^\text{CLIP} = &\mathbb{E}_{S_t,A_t \sim \pi_{\theta_{old}}} [ \min (\rho_\theta(A_t|S_t) {H}_{\theta_\text{old}}(S_t, A_t), \\ 
 & \quad\text{clip}(\rho_\theta(A_t|S_t), 1 - \epsilon, 1 + \epsilon) {H}_{\theta_\text{old}}(S_t, A_t))],
\end{align*}
where $\rho_\theta(A_t|S_t) = \frac{\pi_\theta(A_t \vert S_t)}{\pi_{\theta_\text{old}}(A_t \vert S_t)}$ and $H_{\theta_\text{old}}(S_t, A_t)$ is an estimate of the advantage function. The function $\text{clip}(\rho_\theta(A_t|S_t), 1 - \epsilon, 1 + \epsilon)$ clips the ratio to be no more than $1 + \epsilon$ and no less than $1 - \epsilon$. The policy is optimized by running several iterations of stochastic gradient ascent at each update. We use an adaptation of the PPO-Clip implementation from \citeA{SpinningUp2018} for our experiments.

\subsection{Random Augmented Data}
\citeA{laskin2020reinforcement} showed the effectiveness of adding random augmentations to image input, such as random cropping, in improving sample efficiency of state-of-the-art learning algorithms in pixel-based control tasks during training. Their proposed approach RAD regularizes representations by learning an augmentation-invariant critic. We also used RAD concurrently with both SAC and PPO to achieve higher data efficiency and overall performance. We apply random cropping during the training phase only. The original image is used during action computation. 

\section{Robots}

\begin{figure}
\centering
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{ur-visual-reacher.png}
       
        \label{fig:ur_visual_reacher}
    \end{subfigure}
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{ur-reacher-cam-img.png}
       
        \label{fig:ur_reacher_cam}
    \end{subfigure}
    \caption{The setup of the environment (left), and the image from the mounted camera (right).}
    \label{fig:ur_setup}
\end{figure}


\textbf{UR5} is an industrial robotic arm manufactured by Universal Robotics. It has six joints, and each joint can be controlled individually. UR5 uses conventional TCP/IP protocol to communicate the state and control packets with the host computer for each 8 ms. A state packet includes all six joints' angles, velocities, target accelerations, and currents. UR5 can be configured to use either velocity control or angular control. In both modes, users are allowed to send low-level control commands directly to the six joints, making UR5 an ideal platform for reinforcement learning studies. In this paper we used the velocity control. Since UR5 does not come with a camera, we attached a Logitech RGB camera to the tip of the arm to enable image-based tasks. The configuration is shown in Figure \ref{fig:ur_setup}.

\begin{figure}
\centering
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{roomba_reacher_1.jpg}
       
        \label{fig:create_reacher_1}
    \end{subfigure}
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{roomba_reacher_2.jpg}
       
        \label{fig:create_reacher_2}
    \end{subfigure}
    \caption{Create Reacher environment.}
    \label{fig:create2_setup}
\end{figure}

\textbf{Create2} is a robotic development platform based on commericially available iRobot Roomba robot vacuum cleaner. Unlike Roomba, it has an open interface to allow custom development so that it is suitable for many applications, including reinforcement learning studies. Create2 has two controllable wheels and various onboard sensors. It has six wall sensors to measure the distances to walls, three infrared sensors to locate the charger, four cliff sensors to detect falling edges, two bump sensors to detect bumping into obstacles, and one charging sensor to detect successful docks. The iRobot Create2 Open Interface (OI) streams sensory data and internal states to the host computer every 15 ms. Users are allowed to send the desired speeds of the two wheels to Create2 via the open interface. In this paper, we attached a Depstech 4K camera and a Jetson Nano 4GB attached to it to allow onboard inference. The configuration is shown in Figure \ref{fig:create2_setup}.


\section{TASKS}
To demonstrate the benefits of our system, we develop two visuomotor tasks that are not easy to solve using simple RL approaches since they require tens of thousands of images to learn good representations and control policies. All environments were designed using the guidelines proposed by SenseAct to reduce computational overhead and take advantage of its asynchronous architecture for handling different transmission rates. 

\subsection{UR5-VisualReacher}

This environment is a modification of the environment proposed by \citeA{yuan2021asynchronous}. It is an image-based task whose objective is to move UR5's fingertip to a random target designated by a red circle displayed on a monitor (Fig. \ref{fig:ur_setup}). The movement of UR5 is restricted within a bounding box to prevent collisions and accidental damage. The action space is the desired angular velocities for the five joints between $[-0.7, 0.7]$ rad/s. The observation vector includes the latest joint angles, the latest joint angular velocities, three consecutive images of dimension  $160\!\times\!90\!\times\!3$ taken by the camera, and the previous action. Every 40 ms, the Senseact environment process actuates the five joints of UR5 by sending low-level control commands to them and receiving the next observation. The Senseact communicator process transmits the joint data every 8 ms and the image data every $33.\Bar{3}$ ms. Since this environment is inherently non-Markovian, we stack three images to provide the RL agent temporal history. Each episode lasts 4 seconds (i.e., 100 timesteps). The reward function is designed to encourage the following two behaviors: 1) moving the fingertip closer to the target while keeping the target centralized, and 2) avoiding abrupt twisting of the joints. Specifically, the reward function is as follows \shortcite{yuan2021asynchronous}: 
\[
    r_t=\alpha \frac{1}{hw}M \circ W-\beta \left(\bigg|\pi-\sum_{n=1}^{3}\omega_n\bigg|+\bigg|\sum_{n=4}^{5}\omega_n\bigg|\right),
\]
where $h$ is the height and $w$ is the width of the image, $M$ is a mask matrix whose element $M_{i,j}$ is 1 if the target color is detected at location $(i,j)$ and 0 otherwise, $W$ is a constant weight matrix whose elements range from 0 to 1 to encourage centralization of the target, and $\omega$ is the vector of the five joint angles. The coefficient $\alpha$ and $\beta$ weigh the importance of the two behaviors. In this paper, we set $\alpha=800$ and $\beta=1$. During a reset, a new random target is generated and displayed on the monitor for the next episode, and UR5 is set to a predefined posture. The environment runs on a Lenovo Ideapad Y700 Laptop.

\subsection{Create-Reacher}
In this image-based task, the objective is to move Create2 to a target designated by a green square on a wall as soon as possible. We assume Create2 reached the target if the target square occupies more than 12\% pixels of the current image. The size of the image is $160\!\times\!120\!\times\!3$. The area of Create2's movement is limited by an arena of size $110 \text{cm} \times 70 \text{cm}$ built by wood and cardboard (Fig. \ref{fig:create2_setup}). The action is the desired speeds of the two wheels, limited to $[-150, 150]$mm/s to prevent damage to it. The observation includes the latest values of the six wall sensors, three consecutive images taken by the camera, and the previous action. The wall sensors are normalized to convert raw values into approximate distances to walls. 
Every 45ms, the Senseact environment process actuates the two wheels of Create2 by sending low-level control commands to them and receiving the next observation. The Senseact communicator process transmits the sensory data every 15 ms and the image data every $33.\Bar{3}$ ms. 
Similar to UR5-VisualReacher, we stack three images to give the learning agent  temporal history. Each episode has a maximum length of 666 timesteps. The episode terminates earlier if the Create2 reaches the target. The reward is $-1$ per step to encourage the agent to reach the target as fast as possible.
The reset routine brings Create2 to a random location in the arena. During a reset, Create2 first moves backward and reorients itself in a random direction. Our reset routine also handles recharging when the battery life falls below a specified threshold. All environment interface processes run on a Jetson Nano 4GB.


\section{The Remote-Onboard System}
\label{arch}
\begin{figure*}
\centering
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{OnboardAgent.png}
        \caption{Onboard Agent}
        \label{fig:onboard_agent}
    \end{subfigure}\hfill%
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{RemoteAgent.png}
        \caption{Remote Agent}
        \label{fig:remote_agent}
    \end{subfigure}
\caption{Remote-Onboard System}
\label{fig:remote-onboard}
\end{figure*}

In our proposed system, the \textit{agent} consists of two distinct computational units:
\begin{itemize}
    \item \textit{Performer.} This contains computational items responsible for mapping observations to actions sent to the environment. It consists of all essential components for a system to perform in the real world, such as, the actor network and its inference.
    \item \textit{Learner.} This contains all computationally intensive items involved in learning such as replay buffer, random sampling from it, mini-batch gradient updates, and data augmentation. This process periodically synchronizes with the performer by sharing new policy parameters.  
\end{itemize}

The system supports three modes to handle different computational configurations:
\begin{enumerate}
  \item \textbf{remote-only:} Both performer and learner run on the remote workstation. The onboard computer only needs to relay actions to robots. This mode requires the minimal onboard resources, suitable for small onboard computers such as Arduino Uno.
  \item \textbf{remote-onboard:} The performer runs on the onboard computer to sample actions, and the learner runs on the remote workstation to update policies. Policies are synchronized between the onboard computer and the remote workstation periodically. This mode has potential advantages over remote-only  when using powerful onboard computers such as Jetson-Nano that can perform policy inference.
  \item \textbf{onboard-only:} Both performer and learner run on the onboard computer. This mode assumes the availability of considerable computational resources onboard. This mode is impractical for RL algorithms that require large replay buffers and intensive computations. However, powerful onboard computers may support this mode for relatively simpler algorithms such as PPO. 
\end{enumerate}

The system is designed to work with various RL algorithms. Users can easily add more RL algorithms to the system by implementing the agent interfaces defined by the system. In this paper, we implement two state-of-the-art RL algorithms: PPO and SAC. The resulting systems are called PPO-agent and SAC-agent, respectively. The structure of the SAC-agent configured to the remote-onboard mode is demonstrated in Fig. \ref{fig:remote-onboard}. In this example, the performer runs on a resource-limited onboard computer, while the learner runs asynchronously on a powerful remote workstation. 

When SAC-agent runs in the remote-onboard mode, the performer forks three processes that run in parallel on the onboard computer to mitigate the negative effects of network latency. \emph{The agent-environment interaction process} is responsible for sending actuation commands, receiving observations, and collecting state transition samples. \emph{The send process} sends the collected state transition samples to the replay buffer process in the remote workstation. \emph{The receive process} receives new policy weights from the remote workstation and updates the local policy parameters used to compute actions (Fig. \ref{fig:onboard_agent}). In the onboard-only and the remote-only mode, the send process and the receive process are not required. The agent-environment interaction process will also send transition samples and receive/update policies.

In the remote workstation, the learner forks three concurrent processes. \emph{The main process} receives samples and sends updated polices. \emph{The replay buffer process} stores state transition samples, samples mini-batches, converts mini-batches to CUDA tensors, and applies random augmentation to images. \emph{The update process} performs gradient updates and shares new policy parameters with the onboard computer (Fig. \ref{fig:remote_agent}). The PPO-agent does not have separate replay buffer and update processes. The policy updates are performed in the main process. 

The remote agent and the remote workstation are not needed in the onboard-only mode. All learning and performing operations will run on the onboard computer.


\section{EXPERIMENTS}
In this paper, we aim to demonstrate the advantages of our remote-onboard system on resource-limited computers. We run experiments using PPO and SAC. While SAC is computationally intensive, PPO has relatively low computational requirements. We used an adaptation of the asynchronous SAC system proposed by \citeA{yuan2021asynchronous} to reduce the overhead of policy updates. The PPO-agent was adapted from \citeA{SpinningUp2018}, and we extended it to support learning with images. 

The default hyper-parameter configuration provided by \citeA{yuan2021asynchronous} and \citeA{SpinningUp2018} is used for SAC and PPO, respectively. We used a Jetson Nano 4GB as the onboard computer for Create-Reacher. However, the Jetson Nano's CPU is not fast enough to run UR5-VisualReacher. We observed that the Jetson Nano was only able to support an average action cycle time of $100 \text{ms}$ for UR5-VisualReacher. Thus, we used a Lenovo Ideapad Y700 as the onboard computer for UR5-VisualReacher instead. The remote workstation \emph{franka} for Create-Reacher has an AMD Ryzen Threadripper $3970 \times 32$ core processor and an NVidia 3090 GPU. The remote workstation \emph{helper} for UR5-VisualReacher has an AMD Ryzen Threadripper $2950 \times 16$ core processor and an NVidia 2080Ti GPU.

We compared the performance of SAC and PPO in the three modes of our system. The Jetson Nano and the laptop cannot fit the default buffer size of SAC. We used a binary search strategy to determine the maximum size for the onboard replay buffer. We performed five independent runs with different seeds for each task and mode.

We also run the proposed system on the workstation helper in the onboard-only mode to compare its performance to the system proposed by \citeA{yuan2021asynchronous}. This is to verify if the performance of our implementation is comparable to the off-the-shelf implementation in this setting. The environment is UR5-VisualReacher, and five independent runs were performed for each system.

One complete run of Create-Reacher that lasts $150K$ timesteps requires roughly 5h of robot operation. This includes the real experience time, the time needed to recharge the batteries, and the reset time of the environment. Similarly, a complete run of UR5-VisualReacher that lasts $100K$ timesteps requires roughly 2.5h of robot operation.




\section{RESULTS}
\begin{figure}
\centering
    \begin{subfigure}{.95\columnwidth}
        \includegraphics[width=\columnwidth]{create_reacher_all_runs_sac.png}
        \caption{Create-Reacher using SAC}
        \label{fig:create_reacher_sac}
    \end{subfigure}\hfill%
    \begin{subfigure}{.95\columnwidth}
        \includegraphics[width=\columnwidth]{ur5_reacher_all_runs_sac.png}
        \caption{UR5-VisualReacher using SAC}
        \label{fig:ur_reacher_sac}
    \end{subfigure}
\caption{Comparison of learning performance across different modes of the remote-onboard system using Soft Actor Critic. The onboard-only mode was unable to learn a policy using SAC. The solid lines for each mode are averaged over five independent runs. }
\label{fig:sac-all-modes}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{ur5_reacher_all_runs_ppo.pdf}
    \caption{UR5-VisualReacher using PPO. The solid lines for each mode are averaged over five independent runs.}
    \label{fig:ur_ppo}
\end{figure}

We show the learning curves of the remote-onboard and the remote-only modes of operation of the SAC-agent in Fig. \ref{fig:sac-all-modes}. Note that the real experience time in our plots does not include additional components required to run an experiment, including resetting the environment and recharging the robot. Unfortunately, we were unable to run the onboard-only mode for SAC-agent regardless of the choice of buffer size since the onboard computers were unable to perform learning updates within a reasonable time frame. Each wide learning curve was the average of 5 independent runs of the same hyper-parameter configuration. The remote-onboard mode outperformed remote-only mode consistently and by a large margin on both tasks. In addition, the remote-only mode exhibited a much higher variance in overall learning performance. 

The learning curves of the PPO-agent on UR5-VisualReacher is shown in Fig. \ref{fig:ur_ppo}. Given the relatively lower computational requirements of PPO, we were able to assess the performance of all three modes of operation on the UR5-VisualReacher task. Each solid learning curve was the average of 5 independent runs of the same hyper-parameter configuration. The performance of the remote-onboard mode was on par with onboard-only mode here, while remote-only mode was slightly worse than both.

\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{yufeng_ours_comparison.pdf}
    \caption{Comparing onboard-only mode of our system with an off-the-shelf implementation of \protect\citeA{yuan2021asynchronous} on a powerful workstation.  The solid lines for each mode are averaged over five independent runs. Both methods use SAC to learn a control policy. }
    \label{fig:yufeng_ours}
\end{figure}

We also compared our implementation in onboard-only mode with an off-the-shelf implementation of \citeA{yuan2021asynchronous}'s system in Fig. \ref{fig:yufeng_ours} using a powerful workstation. Both methods achieved similar overall learning performance. 


\section{DISCUSSION \& CONCLUSIONS}
In this paper, we proposed a remote-onboard system to allow resource-limited computers to take advantage of resource-demanding RL algorithms to solve robotic control problems. In total, we conducted 45 independent runs in our experiments, which took nearly 160 hours of usage on robots. We achieved effective learning without tuning the hyper-parameters, using only default hyper-parameters proposed by the original authors of PPO and SAC, which indicates the reliability of the algorithms, our learning system, and our task setup. The subtleties and interesting properties of real-time learning revealed by our experiments are summarized in this section.

\textit{Effects of latency in communicating actions and observation.} We believe that the relatively poor performance of remote-only mode in all tasks is primarily due to the variable latency in communicating actions and receiving observations over WiFi. \citeA{mahmood2018setting} showed that system delays occurring in different computational stages could be detrimental to learning. The variable latency of WiFi in our remote-only setup negatively impacted our learning system.

\textit{Comparison of SAC and PPO in this setting.} While we could not run SAC on resource-limited onboard computers in the onboard-only mode, it was relatively straightforward to use PPO in this setting. However, PPO takes much longer than SAC to learn a good policy. SAC achieves twice the return attained by PPO in a shorter time frame (see Fig. \ref{fig:ur_reacher_sac} and Fig. \ref{fig:ur_ppo} ). While SAC is more sample efficient since it can effectively leverage frequent off-policy gradient updates, it requires considerably more computational resources than PPO. Our results indicate that there exists a trade-off between the availability of computational resources and overall learning performance.  

\textit{Failure of SAC in onboard-only mode}. Through binary search, we found that the buffer size can be set to 25000 on the Jetson Nano and 40000 on the laptop for the SAC-agent. But unfortunately, we were unable to learn a good policy within a reasonable time frame. The main problem is that each SAC update takes enormous time on both platforms (e.g., $\sim 87s$ on the Jetson Nano). Given a fixed learning period, the total number of learning updates performed by an agent is inversely correlated with the time required to make one SAC update. Considering that both tasks require tens of thousands of updates to achieve good performance, we determined it is impossible to finish the training within a reasonable time frame. Thus, we decided that the Jetson Nano and the laptop does not have enough resources to take advantage of SAC. While SAC is more sample efficient than PPO, its computational requirements prohibit us from successfully deploying it to  resource-limited onboard computers in this mode. However, as mentioned earlier, we could still deploy PPO and potentially achieve good learning performance, albeit at a much slower rate. 

Our results show that the proposed \textit{remote-onboard} system successfully supported SAC and PPO on resource-limited computers such as Jetson Nano and Lenovo Ideapad Y700 Laptop in solving two real-time tasks, Create-Reacher and UR5-VisualReacher. Our results indicate that the remote-only mode performs worse than the remote-onboard and onboard-only modes due to network latency issues, causing variable delays in collecting observations and sending actions. On the other hand, the remote-onboard mode seems to sidestep the latency issues, thus avoiding the pitfalls of remote-only learning. In addition, our results also show that when our system is configured to the onboard-only mode and deployed to a powerful workstation, its performance is on par with off-the-shelf systems that require lots of resources to enable real-time learning. Thus, our experiments support the claim that the remote-onboard system can help resource-limited computers leverage resource-demanding deep RL algorithms in an effective manner. 


\addtolength{\textheight}{-5cm}  
                                 
                                 
                                 
                                 
                                 











\bibliographystyle{apacite}


\section{RESULTS}
\begin{figure}
\centering
    \begin{subfigure}{.95\columnwidth}
        \includegraphics[width=\columnwidth]{create_reacher_all_runs_sac.png}
        \caption{Create-Reacher using SAC}
        \label{fig:create_reacher_sac}
    \end{subfigure}\hfill%
    \begin{subfigure}{.95\columnwidth}
        \includegraphics[width=\columnwidth]{ur5_reacher_all_runs_sac.png}
        \caption{UR5-VisualReacher using SAC}
        \label{fig:ur_reacher_sac}
    \end{subfigure}
\caption{Comparison of learning performance across different modes of the remote-onboard system using Soft Actor Critic. The onboard-only mode was unable to learn a policy using SAC. The solid lines for each mode are averaged over five independent runs. }
\label{fig:sac-all-modes}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{ur5_reacher_all_runs_ppo.pdf}
    \caption{UR5-VisualReacher using PPO. The solid lines for each mode are averaged over five independent runs.}
    \label{fig:ur_ppo}
\end{figure}

We show the learning curves of the three configurations of the SAC-agent in Fig. \ref{fig:sac-all-modes}. Note that the real experience time in our plots does not include additional components required to run an experiment, including resetting the environment and recharging the robot. Unfortunately, we were unable to run the local-only configuration for SAC-agent on Create-Reacher regardless of the choice of buffer size since Jetson Nano was unable to perform learning updates within a reasonable time frame. Each wide learning curve was the average of 5 independent runs of the same hyper-parameter configuration. The remote-local configuration outperformed the other two consistently and by a large margin on both tasks. In addition, the remote-only mode exhibited a much higher variance in overall learning performance. Surprisingly, the performance of the local-only configuration on the UR5-VisualReacher is very close to the remote-only though it uses much less resources.

The learning curves of the PPO-agent on UR5-VisualReacher is shown in Fig. \ref{fig:ur_ppo}. Each solid learning curve was the average of 5 independent runs of the same hyper-parameter configuration. The performance of the remote-local configuration was on par with local-only configuration here, while remote-only configuration was slightly worse than both.

\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{yufeng_ours_comparison.pdf}
    \caption{Comparing onboard-only mode of our system with an off-the-shelf implementation of \protect\citeA{yuan2021asynchronous} on a powerful workstation.  The solid lines for each mode are averaged over five independent runs. Both methods use SAC to learn a control policy. }
    \label{fig:yufeng_ours}
\end{figure}

We also compared our implementation in local-only mode with an off-the-shelf implementation of \citeA{yuan2021asynchronous}'s system in Fig. \ref{fig:yufeng_ours} using a powerful workstation. Both methods achieved similar overall learning performance. 


\section{DISCUSSION \& CONCLUSIONS}
In this paper, we used our proposed remote-local system to compare the performance of three different distributions of computations of a learning system between a resource-limited local computer and a powerful remote computer. In total, we conducted 45 independent runs in our experiments, which took nearly 160 hours of usage on robots. We achieved effective learning without tuning the hyper-parameters, using only default hyper-parameters proposed by the original authors of PPO and SAC, which indicates the reliability of the algorithms, our learning system, and our task setup. The subtleties and interesting properties of real-time learning revealed by our experiments are summarized in this section.

\textit{Effects of latency in communicating actions and observation.} We believe that the relatively poor performance of remote-only mode in all tasks is primarily due to the variable latency in communicating actions and receiving observations over WiFi. \citeA{mahmood2018setting} showed that system delays occurring in different computational stages could be detrimental to learning. The variable latency of WiFi in our remote-only setup negatively impacted our learning system.

\textit{Comparison of SAC and PPO on UR5-VisualReacher.} In all of the three configurations, PPO takes much longer than SAC to learn a good policy and SAC achieves twice the return attained by PPO (see Fig. \ref{fig:ur_reacher_sac} and Fig. \ref{fig:ur_ppo} ). While SAC is more sample efficient since it can effectively leverage frequent off-policy gradient updates, it requires considerably more computational resources than PPO. Our results indicate that there exists a trade-off between the availability of computational resources and overall learning performance.  

\textit{Failure of SAC on Jetson Nano.} Unfortunately, we were unable to learn a good policy within a reasonable time frame on Create-Reacher with a Jetson Nano. The main problem is that each SAC update takes enormous time on both platforms (e.g., $\sim 87s$ on the Jetson Nano). Given a fixed learning period, the total number of learning updates performed by an agent is inversely correlated with the time required to make one SAC update. Considering that both tasks require tens of thousands of updates to achieve good performance, we determined it is impossible to finish the training within a reasonable time frame. Thus, we decided that the Jetson Nano does not have enough resources to take advantage of SAC. While SAC is more sample efficient than PPO, its computational requirements may prohibit us from successfully deploying it to  resource-limited local computers in local-only configuration.

Our results indicate that the remote-only configuration significantly suffers from variable delays in collecting observations and sending actions. 
On the other hand, the remote-local configuration seems to sidestep the latency issues, thus avoiding the pitfalls of remote-only learning. 
Our results also demonstrate that the performance of state-of-the-art RL algorithms can be compromised in the local-only configuration due to resource limitation.
In addition, our results verify our second hypothesis that when our system is configured to the local-only mode and deployed to a powerful workstation, its performance is on par with off-the-shelf systems that require lots of resources to enable real-time learning. Thus, our experiments support the claim that the remote-onboard system can help resource-limited computers leverage resource-demanding deep RL algorithms in an effective manner. 


\addtolength{\textheight}{-5cm}  
                                 
                                 
                                 
                                 
                                 











\bibliographystyle{apacite}


\section{INTRODUCTION}

Building robotic agents capable of adapting to their environments based on environmental interactions is one of the long-standing goals of embodied artificial intelligence. 
Such a capability entails learning on the fly as the agent interacts with the physical world, also known as \textit{real-time learning}. 
When learning in real-time, the real world does not pause while the agent computes actions or makes learning updates (Mahmood et al.\ 2018a, Ramstedt \& Pal 2019).
Moreover, the agent obtains sensorimotor information from various onboard devices and executes action commands at a specific frequency. 
Given these constraints, a real-time learning agent must compute an action within a chosen action cycle time and perform learning updates without disrupting the periodic execution of actions (Yuan \& Mahmood 2022).

\begin{figure}[t]
\centering
\includegraphics[width=0.375\textwidth]{ROA_env_agent_interaction-cropped.pdf}
\caption{Our proposed system ReLoD distributes computations of a learning system between a local and a remote computer.}
\label{fig:ROA_env_agent_interaction}
\end{figure}

Reinforcement Learning (RL) is a natural way of formulating real-time control learning tasks.
Although many deep RL methods have been developed to solve complex motor control problems
(Schulman et al.\ 2017, Abdolmaleki et al.\ 2018, Haarnoja et al.\ 2018)
they do not easily extend to the real-time learning setting that operates under time and resource constraints, for example, in quadrotors and mobile robot bases.
While approaches including learning from demonstrations  (Gupta et al.\ 2016, Vasan \& Pilarski 2017), sim-to-real (Peng et al.\ 2018, Bousmalis et al.\ 2018), and offline RL (Levine et al.\ 2020) have been used to develop pre-trained agents, there has been relatively little interest in studying real-time learning in the real world.

State-of-the-art RL algorithms such as SAC are computationally intensive, and hence for real-time robotic control, they go together with a computationally powerful computer tethered to the robot (Yuan \& Mahmood 2022).
On the other hand, a robot agent deployed in the real world typically uses a tethered resource-limited computer and a wirelessly connected workstation (Haarnoja et al.\ 2019, Bloesch et al.\ 2021).
In this paper, we use \emph{local} to refer to the computer tethered to the robot and \emph{remote} to refer to the wirelessly connected computer.
Computations of a learning system using these two computers can be distributed in different ways (e.g., see Fig.\ \ref{fig:ROA_env_agent_interaction}).
However, it is unclear how much the performance of a learning system is impacted due to the wireless connection or resource limitations.
Moreover, prior works do not systematically study distributions of computations between local and remote computers nor suggest how to achieve an effective distribution.

In this paper, we develop two vision-based tasks using a robotic arm and a mobile robot, and propose a real-time RL system called the \emph{Re}mote-\emph{Lo}cal \emph{D}istributed (ReLoD) system. 
Similarly to Yuan and Mahmood's (2022) work, ReLoD  parallelizes computations of RL algorithms to maintain small action-cycle times and reduce the computational overhead of real-time learning. But unlike the prior work, it is designed to utilize both a local and a remote computer.
ReLoD supports three different modes of distribution: \emph{Remote-Only} that allocates all computations on the remote computer,  \emph{Local-Only} that allocates all computations on the local computer, and \emph{Remote-Local} that carefully distributes the computations between the two computers in a specific way.


Our results show that the performance of SAC on a tethered resource-limited computer drops substantially compared to its performance on a tethered powerful workstation.
Surprisingly, when all computations of SAC are deployed on a wirelessly connected powerful workstation, the performance does not improve notably, which contradicts our intuition since this mode fully utilizes the workstation.
On the other hand, SAC's Remote-Local mode consistently improves its performance by a large margin on both tasks, which indicates that a careful distribution of computations is essential to utilize a powerful remote workstation.
However, the Remote-Local mode only benefits computationally expensive and sample efficient methods like SAC since the relatively simpler learning algorithm PPO learns similar policies in all three modes.
We also notice that the highest average return attained by PPO is about one-third of the highest average return attained by SAC, which indicates that SAC is more effective in complex robotic control tasks.

Our system in the Local-Only mode can achieve a performance that is on par with a system well-tuned for a single computer (Yuan \& Mahmood 2022), though the latter overall learns slightly faster.
This property makes our system suitable for conventional RL studies as well.



\section{RELATED WORK}

A system comparable to ours is SenseAct, which provides a computational framework for robotic learning experiments to be reproducible in different locations and under diverse conditions (Mahmood et al.\ 2018b).
Although SenseAct enables the systematic design of robotic tasks for RL, it does not address how to distribute computations of a real-time learning agent between two computers, and the original work does not contain vision-based tasks.
We use the guiding principles of SenseAct to design the vision tasks and systematically study the effectiveness of different distributions of computations of a learning agent. 

Krishnan et al.\ (2019) introduced an open-source simulator and a gym environment for quadrotors. 
Given that these aerial robots need to accomplish their tasks with limited onboard energy, it is prohibitive to run current computationally intensive RL methods on the onboard hardware. 
Since onboard computing is scarce and updating RL policies with existing methods is computationally intensive, they carefully designed policies considering the power and computational resources available onboard. 
However, they focused on sim-to-real techniques for learning, making their approach unsuited for real-time learning.

Nair et al.\ (2015) proposed a distributed learning architecture called the GORILA framework that mainly focuses on using multiple actors and learners to collect data in parallel and accelerate training in simulation using clusters of CPUs and GPUs. 
GORILA is conceptually akin to the DistBelief (Dean et al.\ 2012) architecture. 
In contrast to the GORILA framework, our system focuses primarily on how best to distribute the computations of a learning system between a resource-limited local computer and a powerful remote computer to enable effective real-time learning.
In addition, the GORILA framework is customized to Deep Q-Networks (DQN), while our system supports two different policy gradient algorithms using a common agent interface.

Lambert et al.\ (2019) used a model-based reinforcement learning approach for high-frequency control of a small quadcopter. 
Their proposed system is similar to our Remote-Only mode. 
A recent paper by Smith et al.\ (2022) demonstrated real-time learning of walking gait from scratch on a Unitree A1 quadrupedal robot on various terrains. Their real-time synchronous training of SAC on a laptop is similar to our Local-Only mode. 
The effectiveness of both these approaches on vision-based tasks is untested.

Bloesch et al.\ (2021) used a distributed version of Maximum aposteriori Policy Optimization (MPO) (Abdolmaleki et al.\ 2018) to learn a vision-based control policy that can walk with Robotis \href{https://emanual.robotis.com/docs/en/platform/op3/introduction/}{OP3 bipedal robots}. 
The robot's onboard computer samples actions and periodically synchronizes the policy's neural network weights with a remote learning process at the start of each episode. Haarnoja et al.\ (2019) also proposed a similar asynchronous learning system tailored to learn a stable gait using SAC and the minitaur robot (Kenneally et al.\ 2016). 
These tasks do not use images.
Although their proposed systems are similar to our Remote-Local mode, these two papers aim at solving tasks instead of systematically comparing different distributions of computations of a learning agent between a resource-limited computer and a powerful computer. 
In addition, their systems are tailored to specific tasks and algorithms and are not publicly available, while our system is open-source, task-agnostic, and compatible with multiple algorithms. 

\section{Background}

Reinforcement learning is a setting where an agent learns to control through trial-and-error interactions with its environment. 
The agent-environment interaction is modeled with a Markov Decision Process (MDP), where an agent interacts with its environment at discrete timesteps.
At the current timestep ${t}$, the agent is at state $S_t \in \mathcal{S}$, where it takes an action $A_t \in \mathcal{A}$ using a probability distribution $\pi$ called a \emph{policy}: $A_t\sim\pi(\cdot | S_t)$. 
At the subsequent timestep ${t+1}$, the agent receives the next state $S_{t+1}$ and reward $R_{t+1}$ according to a transition probability density function $S_{t+1}, R_{t+1} \sim p(\cdot, \cdot | S_t, A_t)$.
An episode ends when the agent arrives at a terminal state.
The state space $\mathcal{S}$ and the action space $\mathcal{A}$ of the tasks used in this paper are continuous. 

\subsection{Soft Actor-Critic}
Soft Actor-Critic (SAC) is an off-policy RL algorithm where the policy is trained to maximize a trade-off between the expected return and the policy's entropy (Haarnoja et al.\ 2018). 
SAC learns a policy $\pi_\phi(A_t|S_t)$ by maximizing the following objective:

\begin{align*}
    J_\pi (\phi) &= \mathbb{E}_{S_t \sim \mathcal{D}} \left[ \mathbb{E}_{A_t \sim \pi_\phi} [\alpha \log{\pi_\phi (A_{t} | S_{t})} - Q_\theta (S_{t}, A_{t})] \right],
\end{align*}
which is based on an entropy-augmented version of the standard RL objective (Ziebart 2010).
Here $\phi$ denotes the policy parameters, and $Q_\theta (S_{t}, A_{t})$, parameterized by $\theta$, denotes an estimate of the Soft-Q function $q_\pi$ defined as $q_\pi(s, a) = E_\pi\left[\sum_{t=0}^{T-1} R_{t+1} - \alpha\sum_{t=1}^{T-1}\log\pi(A_t|S_t)\big|S_0=s, A_0=a\right]$.

The parameters of the soft Q-function are trained to minimize the following objective:
\begin{align*}
    J_Q (\theta) &= \mathbb{E}_D \left[ \frac{1}{2} \left( R_{t+1} + \gamma V_{\Tilde{\theta}}(S_{t+1})-Q_\theta (S_t, A_t)  \right)^2 \right],
\end{align*}
where $V_{\Tilde{\theta}}$ is defined as
\begin{align*}
    V_{\Tilde{\theta}}(S_{t}) &= \mathbb{E}_{\pi} \left[ Q_\theta (S_{t}, A_{t}) - \alpha \log{\pi_\phi (A_{t} | S_{t})} \right],
\end{align*}
with $\Tilde{\theta}$ being the parameters of the target network.
In this paper, we use the undiscounted episodic formulation. 
In other words, $\gamma$ is set to 1.

\subsection{Proximal Policy Optimization (PPO)}
PPO (Schulman et al.\ 2017) is an on-policy policy gradient algorithm that alternates between collecting data via interactions with the environment and optimizing the following clipped surrogate objective:
\begin{align*}
 L_{\theta}^\text{CLIP} = &\mathbb{E}_{S_t,A_t \sim \pi_{\theta_{old}}} [ \min (\rho_\theta(A_t|S_t) {H}_{\theta_\text{old}}(S_t, A_t), \\ 
 & \quad\text{clip}(\rho_\theta(A_t|S_t), 1 - \epsilon, 1 + \epsilon) {H}_{\theta_\text{old}}(S_t, A_t))],
\end{align*}
where $\rho_\theta(A_t|S_t) = \frac{\pi_\theta(A_t \vert S_t)}{\pi_{\theta_\text{old}}(A_t \vert S_t)}$ is the action probability ratio between the new policy and the old policy, and $H_{\theta_\text{old}}(S_t, A_t)$ is an estimate of the advantage function. 
The function $\text{clip}(\rho_\theta(A_t|S_t), 1 - \epsilon, 1 + \epsilon)$ clips the probability ratio to be no more than $1 + \epsilon$ if the advantage is positive and no less than $1 - \epsilon$ if the advantage is negative. 
The policy is optimized by running several iterations of stochastic gradient ascent at each update. 

\section{TASKS}
\label{section:tasks}
For this paper, we develop two visuomotor robotic tasks, named \emph{UR5-VisualReacher} and \emph{Create-Reacher}, that are not easy to solve using simple RL approaches since they require tens of thousands of images to learn good state representations and control policies.
The robots we used in this paper are the \textbf{UR5} industrial robot arm and iRobot \textbf{Create2} mobile robot. 
All environments are designed using the guidelines proposed by Mahmood et al.\ (2018a) to reduce the computational overhead and take advantage of the multiprocessing capability of modern computers for handling different transmission rates.
\subsection{UR5-VisualReacher}
This environment is the same task proposed by Yuan and Mahmood (2022). 
It is an image-based task that aims to move a UR5 arm's fingertip to a random target designated by a red (the target color) blob on a monitor. 
The movement of the UR5 arm is restricted within a bounding box to prevent collisions and accidental damage.
The action space is the desired angular velocities for the five joints between $[-0.7, 0.7]$rad/s. 
The observation vector includes joint angles, joint angular velocities, three consecutive images of dimension $160\!\times\!90\!\times\!3$ taken by a camera, and the previous action.
Every 40ms, the SenseAct environment process actuates the five joints and receives the next observation by sending low-level control commands. 
The SenseAct communicator process transmits joint data every 8ms and image data every $33.\Bar{3}$ms to the environment process.
Since this environment is inherently non-Markovian, we stack three images to provide the partial history to the learning agent. 
Each episode lasts 4 seconds (i.e., 100 timesteps). The reward function is designed to encourage the following behaviors: 1) moving the fingertip closer to the target while keeping the target centralized, and 2) avoiding abrupt twisting of the joints. 
Specifically, the reward function is defined as follows (Yuan \& Mahmood (2022)):
\[
    r_t=\alpha \frac{1}{hw}M \circ W-\beta \left(\bigg|\pi-\sum_{n=1}^{3}\omega_n\bigg|+\bigg|\sum_{n=4}^{5}\omega_n\bigg|\right),
\]
where $h$ is the height and $w$ is the width of the image observation, $M$ is a mask matrix whose element $M_{i,j}$ is 1 if the target color is detected at location $(i,j)$ and 0 otherwise, $W$ is a constant weight matrix whose elements range from 0 to 1 to encourage centralization of the target, and $\omega$ is the vector of the five joints' angles. 
The coefficient $\alpha$ and $\beta$ weigh the importance of the two behaviors. 
In this paper, we set $\alpha=800$ and $\beta=1$. During a reset, a new random target will be displayed on the monitor for the next episode, and the UR5 arm will be set to a predefined posture. 
The environment runs on a Lenovo Ideapad Y700 laptop.

\begin{figure*}
\centering
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{LocalAgent.pdf}
        \caption{}
        \label{fig:local_agent}
    \end{subfigure}\hfill%
    \begin{subfigure}{.975\columnwidth}
        \includegraphics[width=\columnwidth]{RemoteAgent.pdf}
        \caption{}
        \label{fig:remote_agent}
    \end{subfigure}
\caption{The SAC-Agent in the Remote-Local Mode. 
The lengths of the transition queues on the local computer and the remote computer are set to the maximum episode length to avoid losing data.}
\label{fig:remote_local}
\end{figure*}

\subsection{Create-Reacher}
This image-based task aims to move Create2 as soon as possible to a target designated by a green piece of paper attached to a wall. 
The target is thought to be reached if it occupies more than 12\% pixels of the current image. 
The size of the image is $160\!\times\!120\!\times\!3$. 
The movement of Create2 is limited by an arena of size $110 \text{cm} \times 70 \text{cm}$ built by wood and cardboard. 
The action is the desired speeds for the two wheels, limited to $[-150, 150]$mm/s to prevent damage to Create2. 
The observation vector includes the values of the six wall sensors, three consecutive images taken by a camera, and the previous action. 
Every 45ms, the SenseAct environment process actuates the two wheels and receives the next observation by sending low-level control commands. 
The SenseAct communicator process transmits the sensory data every 15ms and the image data every $33.\Bar{3}$ms to the environment process. 
Similar to UR5-VisualReacher, we stack three images to give the partial history to the learning agent. 
Each episode has a maximum length of 30 seconds. 
The current episode will terminate earlier if Create2 reaches the target. The reward is $-1$ per step to encourage shorter episodes.
The reset routine will bring Create2 to a random location in the arena. 
During a reset, Create2 first moves backward and reorients itself in a random direction. 
Our reset routine also handles recharging when the battery life falls below a specified threshold. 
This environment runs on a Jetson Nano computer.

\section{
The proposed ReLoD system
}
\label{section: the system}

\begin{table*}[ht]
  \begin{center}
   
    \begin{tabular}{|c|c|c|c|c|}
       
       
        \hline
          &  \textbf{Remote-Local} & \textbf{Remote-Only} & \textbf{Local-Only} & \textbf{Period of Iteration}\\ \hline
        \textbf{Agent Interface Process (AIP)} & Local & Local & Local & action cycle time \\ \hline 
        \textbf{Action computation}  & Included in AIP & Included in LIP  & Included in AIP & action cycle time \\ \hline 
        \textbf{Local-Send Process} & Local & N/A  & N/A & action cycle time\\ \hline 
        \textbf{Local-Receive Process} & Local & N/A  & N/A & Every $k$ iterations of LIP's loop\\ \hline 
        \textbf{Learner Interface Process (LIP)} & Remote & Remote & Included in AIP & action cycle time\\ \hline 
        \textbf{Replay Buffer Process} & Remote & Remote & Local & hardware dependent \\ \hline 
        \textbf{Update Process} & Remote & Remote & Local & hardware dependent \\ \hline
    \end{tabular}
  \end{center}

  \caption{A list of all processes and their distributions between the local and the remote computers in various operational modes of the ReLoD system.}
  \label{tab:1}
\end{table*}
In this section, we describe our proposed ReLoD system, which distributes the computational components of a real-time RL agent between a local computer and a remote computer.
Most deep RL systems have three computationally expensive components: 1) Action Computation, 2) Policy Updating, and 3) Minibatch Sampling. 
ReLoD spawns parallel processes for each of the three components and runs them in an asynchronous, distributed manner to better utilize the computational resources of the local and remote computers.
It also creates additional inter-process communication sockets to enable this distributed and parallel architecture.
ReLoD supports three different modes of operation: \textit{Remote-Only}, \textit{Remote-Local}, and \textit{Local-Only.}
The mode of operation determines how the various processes are distributed between the local computer and the remote computer.
ReLoD is designed to work with various RL algorithms.
Users can easily add more RL algorithms to the system by implementing the agent interface. 
In this paper, we implement two state-of-the-art RL algorithms: SAC and PPO. 
We call the resulting agents \emph{SAC-Agent} and \emph{PPO-Agent}, respectively. 

Fig.\ \ref{fig:remote_local} outlines the structure of the SAC-Agent in the Remote-Local mode.
There are three processes, called \textit{Agent-Environment Interface}, \textit{Local-Send}, and \textit{Local-Receive}, that run in parallel on the local computer. 
The Agent-Environment Interface process computes the actions to take in the environment (except in Remote-Only mode given in Table  \ref{tab:1}).
Three more processes, called \textit{Learner Interface}, \textit{Replay Buffer}, and \textit{Update},  run in parallel on the remote computer. 
The Replay Buffer process samples mini-batches, and the Update process updates policies.
By distributing the three most computationally expensive components of SAC on two different computers, ReLoD takes advantage of both computers.
On the other hand, in the Remote-Only mode, all agent computations happen on the remote computer, including action computation, and the local computer relays observations and actions between the remote computer and the robot.
More details about the process distribution for all three modes of the SAC-Agent are given in Table \ref{tab:1}.

Note that the learning algorithm determines what processes to use in each mode.
For example, the Replay Buffer and the Update processes are specific to our asynchronous SAC implementation adapted from Yuan and Mahmood (2022). 
The PPO-Agent does not have them due to the nature of the algorithm. It updates policies in the Learner Interface process instead.









\section{EXPERIMENTS \& RESULTS}
\label{section:expts}

\begin{figure*}
\centering
    \begin{subfigure}{.32\textwidth}
        \includegraphics[width=\columnwidth]{iros_ur5_reacher.pdf}
        \caption{}
        \label{fig:ur5_sac}
    \end{subfigure}
    \begin{subfigure}{.32\textwidth}
        \includegraphics[width=\columnwidth]{ur5_reacher_all_runs_ppo.pdf}
        \caption{}
        \label{fig:ur_ppo}
    \end{subfigure}
        \begin{subfigure}{.32\textwidth}
        \includegraphics[width=\columnwidth]{yufeng_ours_comparison.pdf}
        \caption{}
        \label{fig:yufeng_ours}
    \end{subfigure}
    \caption{Comparison of the learning performance across three operational modes of ReLoD on UR5-VisualReacher. 
    The wide lines for each mode are averaged over five independent runs.
    The real experience time in our plots does not include any additional components required to run an experiment, such as resetting the environment or recharging the robot. 
    Fig.\ \ref{fig:ur5_sac} shows the comparison of the learning performance of the SAC-Agent. 
   
    Fig.\ \ref{fig:ur_ppo} shows the comparison of the learning performance of the PPO-Agent. 
    Fig.\ \ref{fig:yufeng_ours} shows the comparison of the Local-Only mode of the SAC-Agent with an off-the-shelf asynchronous SAC implementation proposed by \protect Yuan and Mahmood (2022) on a powerful workstation.}
    \label{fig:ur5_curves}
\end{figure*}
In this paper, we first compared the performance of Local-Only SAC between a resource-limited local computer and a powerful local workstation, referred to as \textit{Local-WStation-Only}, to evaluate the impact on performance due to resource limitations.
The default hyper-parameters provided by Yuan and Mahmood (2022) were used in this paper.
We used the IdeaPad Y700 laptop as the resource-limited computer.
It has an Intel i7-6700HQ CPU, an NVidia GTX 960M GPU, and 16GB memory.
We reduced SAC's mini-batch size to 64 on the laptop to fit the hardware constraints. 
The maximum allowable buffer size was also reduced compared to the workstation, within which size of 16000 performed well.

Considering that heavily resource-limited computers like Jetson Nano are commonly used as the local controlling computers for small robots, we also aimed to investigate how SAC would have performed on Jetson Nano.
However, when ReLoD was configured to Local-Only SAC, Jetson Nano could not run the system fully onboard as its hardware and software architectures were not optimized for updating network weights.
Jetson Nano took about half a second to complete one network update on average and became unresponsive after a couple of updates.
To overcome the architectural limitations and get an empirical upper bound of performance on such a heavily resource-limited computer, we emulated the capability of Jetson Nano with the laptop by restricting its update rate and available memory.
Thus, two Local-Only variants of SAC were tested on the laptop: 
\begin{itemize}
    \item \emph{Local-Laptop-Only}: This variant fully utilizes the computational resources of the laptop. 
    The policy is updated at the fastest allowable speed, that is, back-to-back.
    \item \emph{Local-JNanoEm-Only}: This variant emulates the capability of Jetson Nano. 
    Since the average time required for one SAC update on Jetson Nano is $500ms$ and the action cycle time of UR5-VisualReacher is $40ms$, the policy is updated once per 12 steps on the laptop to best match the capability of Jetson Nano. The buffer size is set to 16000, a maximum that fits Nano's 4GB RAM.
\end{itemize}
We used UR5-VisualReacher for this comparison as mounting the laptop to Create2 was impractical.
The workstation for UR5-VisualReacher has an AMD Ryzen Threadripper 2950 processor, an NVidia 2080Ti GPU, and 128G memory.
The results are shown in Fig \ref{fig:ur5_sac}.

Fig \ref{fig:ur5_sac} shows that the performance of SAC on the resource-limited laptop dropped by about 28\% on average compared to that of SAC on the workstation.
Moreover, the Local-JNanoEm-Only variant struggled to learn a comparable policy within the same time frame.
Note that SAC's Local-WStation-Only configuration attained the highest return among all our results.

Since tethering a workstation to a mobile robot is inconvenient in most cases, we investigated how to utilize the resource of a remote workstation to compensate for the performance loss due to the resource-limited local computer. 
We used the most common setting in practice in which the robot is tethered to a small resource-limited computer and wirelessly connected to a powerful workstation.
To get better coverage of RL algorithms, two typical algorithms, SAC and PPO, were tested.
PPO was adapted from Achiam (2018), and we extended it to support learning with images.
The default hyper-parameters provided by Achiam (2018) were used for PPO.

SAC is resource-demanding since it updates the policy every step and requires a large replay buffer to store all its past experiences. PPO is relatively simpler since it only updates the policy every few thousand steps, and its buffer needs to hold only a few episodes generated on-policy.
The SAC-Agent was tested on UR5-VisualReacher and Create-Reacher, while the PPO-Agent was only tested on UR5-VisualReacher.
As the local computer, we used a Jetson Nano 4GB for Create-Reacher and the laptop for UR5-VisualReacher.
The workstation for Create-Reacher has an AMD Ryzen Threadripper 3970 CPU, an NVidia 3090 GPU, and 128G memory.

The learning curves of the three modes of the SAC-Agent on UR5-VisualReacher and Create-Reacher are shown in Fig.\ \ref{fig:ur5_sac} and Fig.\ \ref{fig:create_sac}, respectively.
Note that for the same reason mentioned above, SAC-Agent's Local-Only mode was not run for Create-Reacher.
Fig \ref{fig:ur5_sac} reveals a counterintuitive result that SAC's Remote-Only mode, referred to as \emph{Remote-WStation-Only} in the Figures, barely improved its performance over the Local-Only mode, which used a resource-limited laptop.
Meanwhile, the Remote-Only mode also exhibited higher variance in the overall learning performance on both tasks.
This is surprising since this mode fully utilizes all resources of the remote workstation.
We attribute the relatively poor performance of the Remote-Only mode to the variable latency in communicating actions over WiFi, which will not occur in the other two modes, as the Remote-Only mode computes actions on a remote computer.
Although the Remote-Local mode also encounters latency due to WiFi communication, the effect is much less severe as it only involves the transfer of policies and buffer samples, while action computation still happens locally.
Our results are in line with Mahmood et al.'s (2018a)
results
showing that delays occurring closer to robot actuations can be substantially more detrimental to performance compared to delays occurring further from actuations.
A real-time learning system can get affected by delays at least through two pathways: inference for action computation and learning update.
When a delay occurs within the system, it may spare the more important inference pathway, as in our case. 
Whereas when a delay occurs closer to the robot actuation, that is, at the periphery of the learning system, it affects both pathways.

Fig.\ \ref{fig:ur5_sac} and Fig.\ \ref{fig:create_sac} suggest that SAC-Agent's Remote-Local mode, referred to as \emph{Remote-WStation-Local-Laptop} and \emph{Remote-WStation-Local-JNano} in the Figures, consistently compensated for the performance loss on both tasks.
As Fig.\ \ref{fig:ur5_sac} indicates, the highest average return attained by SAC-Agent's Remote-Local mode is more than 90\% of the highest average return attained by SAC in the Local-WStation-Only configuration, whose performance is the best.
However, the Remote-Local mode only benefits computationally intensive algorithms like SAC, as Fig.\ \ref{fig:ur_ppo} shows that PPO's performance is nearly the same across the three modes.

By comparing Fig.\ \ref{fig:ur5_sac} and Fig.\ \ref{fig:ur_ppo}, we notice that SAC significantly outperformed PPO on complex robotic tasks.
All distribution modes of the SAC-Agent except the Local-JNanoEm-Only attained more than twice the return attained by PPO in a shorter time frame.
Nevertheless, when all computations of SAC happen on a computer incapable of performing frequent policy updates, indicated by Local-JNanoEm-Only, SAC's performance degrades substantially to a degree comparable to that of PPO on the same computer.









\begin{figure}
    \centering
    \begin{subfigure}{0.33\textwidth}
        \includegraphics[width=\columnwidth]{iros_create_reacher.pdf}
    \end{subfigure}
    \caption{Comparison of learning performance across two modes of SAC-Agent on Create-Reacher. 
    The wide lines for each mode are averaged over five independent runs.}
    \label{fig:create_sac}
\end{figure}

Finally, on a powerful tethered workstation, we compare the performance of Local-Only SAC to that of the system proposed by Yuan and Mahmood (2022), which is well-tuned for a single computer. 
This test is to determine if the performance of the Local-Only mode of our system is comparable to the off-the-shelf RL algorithm implementation given adequate resources. 
The environment was UR5-VisualReacher, and five independent runs were performed for each system.
The comparison results are shown in Fig.\ \ref{fig:yufeng_ours}. 
Both methods achieved similar overall learning performance. 
Thus, our proposed system can leverage the power of deep RL algorithms effectively, and as a result, it is also suitable for conventional RL research.

\section{Conclusion}
In this paper, we introduced the ReLoD system for learning to control robots by a real-time RL agent that distributes its computations between a local and a remote computer.
In our experiments with ReLoD, we demonstrated that when SAC ran on a resource-limited computer, its performance could be dramatically reduced. 
Moreover, solely deploying all computations of SAC on a wirelessly connected remote workstation may not improve performance due to latency.
On the other hand, our results suggest that performing local action computation and remote policy updating compensates substantially for SAC's performance loss.
However, this distribution may not benefit all RL algorithms since PPO's performance was nearly unaffected by the distribution mode.

ReLoD provides a useful tool and guidance for real-time RL in vision-based robotic control.
Computationally intensive algorithms like SAC can best utilize ReLoD's Remote-Local mode. 
Moreover, ReLoD's Local-Only mode is suitable for conventional RL research as its performance was shown to be on par with a well-tuned single-computer system.
However, due to the latency in communicating actions, ReLoD's Remote-Only mode should only be used when the local computer cannot compute actions within the action cycle time.


















\section{Acknowledgements}
We gratefully acknowledge funding from the CCAI Chairs program, the RLAI laboratory, Amii, and NSERC of Canada. We also thankfully acknowledge the donation of the UR5 arm from the Ocado Group.

\addtolength{\textheight}{-4cm}  
                                 
                                 
                                 
                                 
                                 












\section{References}
\hangindent=2em
\hangafter=1
\noindent Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., \& Riedmiller, M. (2018). Maximum a posteriori policy optimisation. In \textit{International Conference
on Learning Representations.}

\hangindent=2em
\hangafter=1
\noindent Achiam, J. (2018). Spinning Up in Deep Reinforcement Learning.

\hangindent=2em
\hangafter=1
\noindent Bloesch, M., Humplik, J., Patraucean, V., Hafner, R., Haarnoja, T., Byravan, A., . . . others (2021). Towards real robot learning in the wild: A case study in bipedal locomotion. In \textit{5th Annual Conference on Robot Learning.}

\hangindent=2em
\hangafter=1
\noindent Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., . . . others (2018). Using simulation and domain adaptation to improve efficiency of deep
robotic grasping. In \textit{2018 IEEE international conference on robotics and automation (ICRA) (pp. 4243–4250).}

\hangindent=2em
\hangafter=1
\noindent Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., ... \& Ng, A. (2012). Large scale distributed deep networks. \textit{Advances in neural information processing systems}, 25.

\hangindent=2em
\hangafter=1
\noindent Gupta, A., Eppner, C., Levine, S., \& Abbeel, P. (2016). Learning dexterous manipulation for a soft robotic hand from human demonstrations. In \textit{2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)} (pp. 3786–3793).

\hangindent=2em
\hangafter=1
\noindent Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., \& Levine, S. (2019). Learning to walk via deep reinforcement learning. In \textit{Robotics: Science and Systems.}

\hangindent=2em
\hangafter=1
\noindent Haarnoja, T., Zhou, A., Abbeel, P., \& Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In \textit{International conference on machine learning} (pp. 1861–1870).

\hangindent=2em
\hangafter=1
\noindent Kenneally, G., De, A., \& Koditschek, D. E. (2016). Design principles for a family of direct-drive legged robots. \textit{IEEE Robotics and Automation Letters}, 1(2), 900–
907.

\hangindent=2em
\hangafter=1
\noindent Krishnan, S., Boroujerdian, B., Fu, W., Faust, A., \& Reddi, V. J. (2019). Air learning: An ai research platform for algorithm-hardware benchmarking of autonomous aerial robots. \textit{arXiv preprint arXiv:1906.00421.}

\hangindent=2em
\hangafter=1
\noindent Lambert, N. O., Drew, D. S., Yaconelli, J., Levine, S., Calandra, R., \& Pister, K. S. (2019). Low-level control of a quadrotor with deep model-based reinforcement
learning. \textit{IEEE Robotics and Automation Letters}, 4(4), 4224–4230.

\hangindent=2em
\hangafter=1
\noindent Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., \& Srinivas, A. (2020). Reinforcement learning with augmented data. \textit{Advances in Neural Information Processing Systems}, 33, 19884–19895.

\hangindent=2em
\hangafter=1
\noindent Levine, S., Kumar, A., Tucker, G., \& Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. \textit{arXiv preprint
arXiv:2005.01643.}

\hangindent=2em
\hangafter=1
\noindent Mahmood, A. R., Korenkevych, D., Komer, B. J., \& Bergstra, J. (2018a). Setting up a reinforcement learning task with a real-world robot. In \textit{2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)} (pp. 4635-4640). IEEE.

\hangindent=2em
\hangafter=1
\noindent Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., \& Bergstra, J. (2018b). Benchmarking reinforcement learning algorithms on real-world robots. \textit{In Confer-
ence on robot learning} (pp. 561–591).

\hangindent=2em
\hangafter=1
\noindent Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon,
R., De Maria, A., . . . others (2015). Massively parallel methods for deep reinforcement learning. \textit{arXiv preprint arXiv:1507.04296.}

\hangindent=2em
\hangafter=1
\noindent Peng, X. B., Andrychowicz, M., Zaremba, W., \& Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. In \textit{2018 IEEE international conference on robotics and automation (ICRA)} (pp. 3803–3810).

\hangindent=2em
\hangafter=1
\noindent Ramstedt, S., \& Pal, C. (2019). Real-time reinforcement learning. \textit{Advances in neural information processing systems}, 32.

\hangindent=2em
\hangafter=1
\noindent Schulman, J., Wolski, F., Dhariwal, P., Radford, A., \& Klimov, O. (2017). Proximal policy optimization algorithms. \textit{arXiv preprint arXiv:1707.06347.}

\hangindent=2em
\hangafter=1
\noindent Smith, L., Kostrikov, I., \& Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. \textit{arXiv preprint arXiv:2208.07860.}

\hangindent=2em
\hangafter=1
\noindent Vasan, G., \& Pilarski, P. M. (2017). Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning. In \textit{2017 International Conference on Rehabilitation Robotics (ICORR)} (pp. 1457–1464).

\hangindent=2em
\hangafter=1
\noindent Yuan, Y., \& Mahmood, A. R. (2022). Asynchronous reinforcement learning for real-time control of physical robots. \textit{In 2022 IEEE international conference on
robotics and automation (ICRA).}

\hangindent=2em
\hangafter=1
\noindent Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University. 


\newcommand{\goodbreak\hangindent=.3cm \noindent}{\goodbreak\hangindent=.3cm \noindent}

\newpage









\end{document}