\section{Introduction}\label{sec:intro}
Environmental modeling is a common application in field robotics, where a robot is used to create a model of the environment, such as a spatial field.
One application is in ocean sampling, where one can create models of sea-surface temperature, or algae distributions.
Informative sampling is a proven technique for spatial field modeling~\cite{Krause2007,Singh2007,Low2008}.
In informative sampling a model is created of the field using, for example, Gaussian Process (GP) regression.
Information-theoretic metrics, such as entropy or mutual information, are then used to decide where the robot should sample next.
When sampling is run on-line, incorporating data sampled by the robot, this is called adaptive sampling.

In this work, we investigate how humans select waypoints, when given the same information as a robot sampling a field, and we compare the modeling performance between the manual and autonomous waypoint selection methods.
We construct a graphical user interface where humans can iteratively set waypoints for a robot, and where they get to see what data is revealed along the way.
The humans are given the instruction to guide the robot to best model the underlying field.
Simulations are run for robotic adaptive sampling, with one random sampling approach and two informative sampling approaches. 
Results show that for scenarios that correspond to the robot's on-board model assumptions, the robot performs better than the average human, though similar to the best human.
For scenarios that do not correspond to the modeling assumptions, the robot performs similar to the average human, and random sampling performs on average as well as sampling based on GP models.

\section{Related Work}\label{sec:rel_work}

Borji and Itti~\cite{Borji2013} performed a similar study where they compared human search behavior with various search strategies, including Bayesian Optimization methods and Gaussian Processes.
Their study was more extensive in terms of number of search strategies considered and the number of participants recruited.
However, in their study, the task was simplified to seeking a maximum of a 1D function. As a stopping criteria, participants judged when they had found the maximum. 
The study results showed that humans are surprisingly good at this task, and that Bayesian Optimization methods based on Gaussian Processes came closest to the human performance. Furthermore, they concluded that the participants focus on exploring uncertain regions, in active tasks.
This supports our study into comparing adaptive sampling by humans to sampling with robots that use Gaussian Process models.

Chien et al.~\cite{Chien2010} compared manual (human) path planning with automated path planning for search and rescue scenarios.
They used a deterministic roadmap planner, maximizing information gain, and making local decisions.
This automated approach was evaluated against data from human subject experiments. Results showed that for coverage achieved and regions explored, there was no significant difference between human and automated path planning.
The automated path planning did lead to a higher count for victims found, though the manual path planning required less teleoperation. 
Overall they concluded that a human path planner could be replaced with an automated path planner without loss in performance.
We are interested to see if their results hold in our environmental modeling scenarios, given the differences in modeling approach and application domain.

Other studies have looked at deducing human strategies and human-like behavior to create planners for robots. 
These studies actively try to imitate human or animal behavior.
Henry et al.~\cite{Henry2010} used Gaussian Process regression to estimate the density and flow of crowds of people. They then employed maximum entropy inverse reinforcement learning to learn cost weights for the features, i.e. the density and flow of crowds, to understand how to navigate through crowded spaces. 
Sexton and Ren~\cite{Sexton2016} developed a crowdsourcing game to learn human search strategies. They made the assumption that human actions stem from Bayesian Optimization (Gaussian Processes), and used maximum likelihood estimation to learn the model covariance function.

In contrast to these kinds of studies, we are not attempting to learn strategies from our human subject experiment. 
Instead, we are comparing performance of the human and adaptive sampling strategies over time, and we gather explicit feedback from the participants about their strategies through a questionnaire. 
We wish to better understand how well humans perform at adaptive sampling if given the same information as a robot, what strategies they use, and which approaches are able to learn a good model more quickly.
The results of this project could help identify the potential benefits of using autonomous sampling techniques. This is especially useful for applications where the manual setting of waypoints by a human operator is infeasible or undesirable due to communication constraints or because the human operator has to control too many robots at the same time.

\pagestyle{plain}

\vspace{.2cm}
\section{Gaussian Process Regression}\label{sec:theory}

Gaussian Process (GP) regression is a standard method for spatial field modeling~\cite{Rasmussen2006}. 
We briefly recap the theory here:
A GP model is specified by its prior mean and covariance function, also known as the kernel.
A standard approach is to use a zero mean prior and the isotropic squared exponential kernel.
The squared exponential kernel is given by~\cite{Rasmussen2006}:
\begin{equation}
  k({\bf x},{\bf x}' ) = \sigma_{f}^2 \exp \left\{ - \frac{1}{2 l^2} | {\bf x} - {\bf x}' |^2 \right\}
\end{equation}
where ${\bf x}$ and ${\bf x}'$ are locations of a training sample, $\sigma_{f}^2$ is the signal variance (or amplitude), and $l$ is the kernel's length scale. 
$\sigma_{f}^2$ and $l$ are the GP's hyperparameters. 
The hyperparameters are typically estimated from the data, by using, for example, maximum likelihood estimation~\cite{Rasmussen2006}.

To make predictions based on a GP model, one calculates the predictive mean and variance for test locations:
\begin{equation}
  {\bf y_* ({\bf X_*}, {\bf X}, {\bf y})} = K({\bf X_*}, {\bf X}) K({\bf X}, {\bf X})^{-1}  {\bf y}
\end{equation}
\begin{dmath}
  {\bf \Sigma_*} ({\bf X_*}, {\bf X}) = K({\bf X_*}, {\bf X_*}) - \\ K({\bf X}_*, {\bf X}) K({\bf X}, {\bf X})^{-1} K({\bf X}, {\bf X}_*)
\end{dmath}
where ${\bf X}$ are the training locations, ${\bf y}$ are the training samples, and ${\bf X_*}$ are the test locations.
The entropy for every test location can then be calculated using the standard equation for entropy:
\begin{equation}\label{eq:entropy}
  H = \frac{1}{2} \ln ( 2 \pi \exp \sigma^2 )
\end{equation}

For our robotic adaptive sampling approaches, we test three  different approaches for waypoint selection: one random and two informative sampling approaches.
The first informative approach uses the GP and selects waypoint locations based on maximum entropy.
Note that the entropy does not depend on the measured values, and only depends on the locations of the training samples.
Therefore, we also test another selection method, where we combine the entropy with the predictive mean:
\begin{equation}\label{eq:entropyplusmean}
  {\bf G} = \frac{{\bf H}}{\max({\bf H})} + \alpha \frac{\bm{\mu_{*}}}{\max(\bm{\mu_{*}})}
\end{equation}
where ${\bf G}$ is the new acquisition function, ${\bf H}$ is the entropy for all test locations, $\bm{\mu_{*}}$ is the predictive mean for all test locations, and $\alpha$ is an exploration-exploitation parameter which determines the importance of the predictive mean in the waypoint selection.
We set $\alpha = 0.25$.


\vspace{.2cm}
\section{Experimental Set-Up}\label{sec:setup}

We use the Matlab GPML libraries~\cite{gpml} both for running GP regression on the sampled data, as well as for generating simulation scenarios.

\subsection{Scenarios}\label{sub:scenarios}

Figure~\ref{fig:scenarios} shows the 12 scenarios created for this study.
The first six scenarios are random samples from a Gaussian Process model, whose hyperparameters were set based on averages from prior field data to $l = \exp \{-7.81\}$ and $\sigma_f^2 = \exp \{1.68\}$.
In doing so, we obtain the same smoothness parameters as observed fields.
The latter six scenarios are Gaussian mixture models (GMMs), where the number of Gaussians, their locations and their orientations are all randomly sampled.
Random noise is added up to $5\%$ of the maximum data value.
Data values are ranged from $0$ to $20$ as a proxy for Chlorophyll measurements ($\mu g/L$), which indicates algae abundance.
For all scenarios, data are simulated over a space of $400$x$200\,m$, along a grid with $10\,m$ grid spacing.

\begin{figure}[!b]
  \centering
  \includegraphics[width=\columnwidth]{scenarios.png}
  \caption{Simulated fields: the first six scenarios are randomly sampled from a GP, the latter six are randomly generated Gaussian mixture models. The colorscale ranges from dark blue, low values to yellow, high values.}
  \label{fig:scenarios}
\end{figure}

\subsection{Humans}

For the human subject experiment, we created a graphical user interface (GUI) using python.
The GUI loads an image of the scenario, and creates a black overlay over the scenario.
The user then iteratively selects waypoints, after which the underlying data is revealed between the previously selected and currently selected waypoints.
To simplify data collection and viewing, the black overlay is a grid of squares of the same size as the simulated data grid.
This means that sampled data can easily be indexed from the generated data file.
Furthermore, the revealed squares are large enough for humans to easily observe.

\begin{figure*}[!t]
  \centering
  \begin{minipage}{\textwidth}
    \centering
    \includegraphics[width=.9\textwidth]{blank_screen_screenshot.png}
    \vspace{.1cm}
  \end{minipage}
  \begin{minipage}{\textwidth}
    \centering
    \includegraphics[width=.9\textwidth]{human_path.png}
  \end{minipage}
  \caption{GUI used by human subjects. The scenario is hidden by a black overlay (top),  which is removed as the human iteratively selects waypoints and creates a path for the robot to traverse (bottom).}
  \label{fig:gui}
\end{figure*}

Figure~\ref{fig:gui} shows the created GUI both at the initial state, and after a user has selected waypoints up to the maximum revealed number of squares, i.e. the path length.
The GUI includes a description of the task, such that the instructions are the same for all participants. 
It also includes a colorbar to indicate the meaning of the colors, and it shows how many squares can still be revealed.
The total number of squares revealed is limited to 190, which corresponds to 1900-2660 meters of sampling, depending on whether the path is completely horizontal or vertical, or diagonal.
This length was chosen because it is approximately 40\% of the time required to run both a north-south and east-west $20\,m$ grid spaced lawnmower survey over the area.
Without a time limit, every model would end up being equally good.
Though the path length was limited, participants were not restricted in time regarding how long they took to decide where to place the waypoints. 
Similarly, computation time for the robot was not considered, though it is on the order of seconds on an Intel i7 quad-core machine with 32GB RAM.

We recruited 11 human participants for this study: 7 men and 4 women, aged between early $20$s and mid $40$s. 
Nine participants were computer science and robotics students and professors, one participant works in oceanography and marine biology, and one participant works in student affairs.
None of the participants had run this study before, nor had they seen the scenarios.
All participants ran all scenarios in the order given in Figure~\ref{fig:scenarios}.
Participants were asked to follow the instructions on the screen. 
No feedback was given about their performance, throughout the experiment. 
Some participants asked questions about what their strategy should be, and were told that they should choose waypoints so as to  collect data that would create the best model of the underlying field.

All participants were also given a survey after running the experiment, which included the following questions:
\begin{itemize}
\item Describe your strategy when choosing waypoints, if you had one.
\item Did the data revealed along the path change the way you chose waypoints?
\end{itemize}

\subsection{Robot}

For the robot experiments, 
three approaches were tested: (1) random waypoints, (2) GP model with entropy optimization, (3) GP model optimizing for entropy plus predictive mean.
Equations~\ref{eq:entropy}-\ref{eq:entropyplusmean} showed the calculations for approaches (2) and (3).
Given that there is no sensor or sampling noise, a single run was done for each approach.
The robots created GP models whose kernel is a combination of the isotropic squared exponential covariance function and a white noise kernel, such that there are three hyperparameters: the length scale $l$, signal variance or amplitude $\sigma_f^2$ and signal noise $\sigma_n^2$.
The log of the hyperparameters are initialized to $-7.5, 0.5$ and $1.0$, respectively, based on prior simulations~\cite{Kemna2017}.
The length scale, which is set in longitude/latitude degrees, roughly corresponds to a length of $55\,m$ at a latitude of 34.086.
The hyperparameters are re-estimated when every batch of samples is received and a new waypoint needs to be calculated.

\section{Results}\label{sec:results}

For evaluation, we use two methods of reconstructing the fields using the sampled data.
The first method is a spline-based method, V4, which is one of Matlab's standard interpolation methods, which also allows for extrapolation.
The second method is GP regression, using the GPML Matlab libraries~\cite{gpml}, with a standard isotropic squared-exponential covariance function.
Hyperparameters are estimated on the collected data for every model.
We evaluate performance in terms of the root mean squared error (RMSE) between the ground truth (simulated scenarios) and the learned models.

\begin{figure*}[!t]
\centering
\includegraphics[width=.63\textwidth]{happ_v42.png}
\caption{Robot versus human RMSE boxplots (median, 25th, 75th percentiles) for each field, using a spline-based method (V4) for field interpolation.}
\label{fig:happ_v4}
\end{figure*}

\begin{figure*}[!t]
\centering
\includegraphics[width=.63\textwidth]{happ_gp2.png}
\caption{Robot versus human RMSE boxplots for each field, using GP regression for field interpolation.}
\label{fig:happ_gp}
\end{figure*}

Figures~\ref{fig:happ_v4} and~\ref{fig:happ_gp} show the results of the experiments, with human performance as boxplots (median, 25th, 75th percentiles), and robot performance as points.
For the V4 interpolation, shown in Figure~\ref{fig:happ_v4}, we see that the robot was able to perform better than the average human in most cases, though the best human was able to outperform the robot in all cases, except scenarios 10 and 11.
For the GP interpolation, we see the RMSE greatly declines for the initial 6 scenarios. 
This result is logical because these were generated from a GP model.
For the first six scenarios, where the robot optimizes on a GP model, it always performs well when using entropy optimization and approximately as good as the best human.
For the latter six scenarios, the robot performance is similar to the average human performance for all methods.

Table~\ref{tab:avg_rmse} shows the modeling error averaged over all scenarios, for all sampling and modeling methods. 
This table shows that models created using GP regression were on average better than models created with V4 interpolation, due to the great improvement for scenarios $1$-$6$ with GP modeling.
Overall, for the robot performance, the best approach was the one that used a GP model with entropy for optimization. 

{\setlength{\tabcolsep}{1em}
\bgroup
\def\arraystretch{1.2
\begin{table}[!t]
\centering
\begin{tabular}{|l|l|}
  \hline
  {\bf Method} & {\bf Average RMSE}\\
  \hline
   {\bf GP model} & \\
   Human & 0.048\\
   GP + entropy & {\bf 0.029} \\
   GP + entropy + mean & 0.037\\
   Random & 0.056\\
  \hline
  {\bf V4 model} &\\
   Human & 0.064\\
   GP + entropy & {\bf 0.042}\\
   GP + entropy + mean & 0.048\\
   Random & 0.083\\
  \hline
\end{tabular}
\caption{Average RMSE over all scenarios, for each method of sampling and modeling.}
\label{tab:avg_rmse}
\end{table}
\egroup
}

We further evaluate the modeling performance over time, from the start of each run until all samples are collected.
We evaluate the timed performance for the GP interpolation, given the overall better performance compared to V4 interpolation.
We divide the 190 samples into batches of 20 samples (10 for the last group), and create a GP model at every batch.
We calculate the RMSE between the GP model and the ground truth.
Because of the difference in the generation method for the scenarios, we split this evaluation between fields $1$-$6$ and fields $7$-$12$.
Figures~\ref{fig:rmse_time1} and~\ref{fig:rmse_time2} show the RMSE boxplots (median, 25th and 75th percentiles) over time, for the two groups of fields.

Figure~\ref{fig:rmse_time1} shows that overall the performance of both humans and the robot using the GP with entropy criterion are the most consistent for the six scenarios that were sampled from a GP.
For the initial 20 samples (number of samples axis label $n = 1$), performance is quite similar across all approaches, and none are much better than random.
After 40-60 samples have been gathered ($n = 2$-$3$), the GP with entropy criterion is on average better than the humans at improving the model. 
This advantage disappears somewhat over time, though it has a consistently good model across all scenarios towards the end of all simulations.
The random approach is overall the worst in consistency and performance.
The GP with entropy plus mean has more variance in performance across the scenarios, and is less consistent overall, though occasionally better than the other approaches.

\begin{figure*}[!t]
\centering
  \includegraphics[width=.7\textwidth]{rmse_timesteps_fields1-6.png}\\
\caption{RMSE boxplots (median, 25th, 75th percentiles) evaluated over time, 20 samples per time step, for the first six scenarios, averaged over the GP sampled scenarios.}
\label{fig:rmse_time1}
\end{figure*}

\begin{figure*}[!t]
\centering
  \includegraphics[width=.7\textwidth]{rmse_timesteps_fields7-12.png}\\
\caption{RMSE boxplots evaluated over time, 20 samples per time step, for the last six scenarios, averaged over the GMM scenarios. \vspace{-.2cm}}
\label{fig:rmse_time2}
\end{figure*}

Figure~\ref{fig:rmse_time2}, showing RMSE performance over time for the GMM scenarios, shows variable performance across scenarios for all evaluated methods.
Human performance is a lot more consistent for the first $20$-$40$ samples ($n = 1$-$2$), and stays more consistent till the end.
During initial sampling, i.e. $20$-$80$ samples ($n = 1$-$4$), the robotic sampling techniques are on average slightly better at collecting informative samples for the model.
Overall, the difference between the robot approaches is not significant, and the GP-based approaches do not perform better than random sampling initially, though on average they are slightly better after the final $20$-$30$ samples ($160$-$190$, $n = 9$-$10$).

\subsection*{Survey results}
Ten of the eleven participants filled out the survey.
Table~\ref{tab:strategies} lists the most commonly used strategies.
The most commonly employed strategy was to try and cover the whole area. The next most common strategies were to seek out maxima and to sample                                                                                                   boundary regions rather than inside the extrema.
Two participants are counted for two strategies, because both started every scenario with coverage sampling and then refined the sampling afterward with a different strategy: one in combination with finding maxima and the other in combination with finding boundaries.
Out of the 10 participants who filled out the survey, 8 participants said they used the information they were given, i.e. the points revealed. 
The other two participants reported they did not use this information partially or completely. These participants optimized waypoint selection for coverage.

{\setlength{\tabcolsep}{1em}
\bgroup
\def\arraystretch{1.2
\begin{table}[!t]
\centering
\begin{tabular}{|l|l|}
\hline
{\bf Strategy} & {\bf Number of participants}\\
\hline
Coverage & 6/10\\
Find maxima & 3/10\\
Find boundaries & 3/10\\
\hline
\end{tabular}
\caption{Strategies employed by participants.\vspace{-.4cm}}
\label{tab:strategies}
\end{table}
\egroup
}

\section{Discussion \& Future Work}\label{sec:disc}

The results of this study show that when the robot's modeling assumptions match the characteristics of the scenario, the robot performance is better than the average human. 
However, in most cases the robot was unable to outperform the best human.
When the scenario does not fit the modeling assumptions well, the robot achieves similar performance to the average human.
The results evaluating RMSE over time confirmed this observation, and also showed that the robot and human performance for the GMM scenarios were not significantly better than random sampling.
This is surprising and interesting, because these scenarios are commonly used as simple algae bloom or underwater plume models.
The results suggest that the use of GP models with entropy-based optimization is not the most effective for these scenarios.
It would be interesting to further explore the GMM scenarios, to see if different modeling and path planning methods could further improve robot performance.

The current study had eleven participants with predominantly computer science and robotics backgrounds. 
It would be interesting to run this experiment with a wider range of participants, including subject experts, for example oceanographers, biologists, or people who often process field data, as well as more participants that do not have computer science or robotics backgrounds. 
If such a wide pool of participants were recruited, we could also investigate the differences in performance between different backgrounds.

\section{Conclusions}
Adaptive informative sampling approaches are commonly used in robotics for spatial field modeling. 
In this paper, we evaluate the modeling performance between an adaptive sampling algorithm and human participants.
We developed a GUI where the human participants could iteratively select waypoints and reveal data, to obtain similar amounts of information that a robot might collect.
Our goal was to better understand how well humans can choose waypoints for adaptive sampling if given the same amount of information, and which strategies they employ.
Results showed that for scenarios that were generated under different assumptions than the model, the GP models with entropy-based criteria and the humans performed no better than random sampling.
When the modeling assumptions corresponded well to the characteristics of the actual field, the robot performance using the GP model and entropy selection criterion, was the most consistent and the best on average.
However, some humans were still able to outperform the robot.
The most commonly used strategy by humans was to optimize for coverage, followed by optimizing for maxima and exploring boundary zones.

\section*{Acknowledgment}

The authors would like to thank everyone who participated in the experiment!

\bibliographystyle{IEEEtran}

