\section{Homeostasis}
\label{sec:homeostasis}

 Mechanisms that maintain the biochemical environment of the body have been studied at 
great length since Claude Bernard~(\citeyear{bernard_lesphenomenesdelavie}) observed that the 
internal environment (or `{\em milieu interne}') was essential to 
survival~\citep{bernard_lesphenomenesdelavie,cannon_organizationhomeostasis}.
 
 Since then, many have argued that cognition is a highly-developed form of homeostasis~\citep{lewontin_adaptationspopulationsvarying, 
ashby_designforabrain, godfrey_complexityfunctionmind}, and that cognition is closely linked with autonomy~\citep{varela_autonomyandautopoiesis, 
froese_autonomyareview, boden_autonomywhatisit, vernon_artificialcognitivesystems}. Given the links between cognition, homeostasis and autonomy, 
we can take inspiration from cognition to guide our development of either autonomous or, in this case, homeostatic systems.

 From a robotics standpoint, the ability to maintain a robot's internal environment (battery charge, internal temperature etc.) while still completing 
its objectives is a key challenge. Artificial homeostasis has the potential to provide this ability, paving the way for autonomous systems that are 
able to effectively balance conflicting decisions without external intervention.
 
 The mechanisms for homeostasis within the human body are driven by interactions between the nervous, endocrine, and immune 
systems~\citep{widmaier_vandershumanphysiology}. Previous attempts to develop artificial homeostasis have made use of artificial counterparts for each 
of these systems (in the form of artificial neural networks, artificial endocrine networks, and artificial immune systems). 
\citet{neal_timidityusefulemotional, vargas_artificialhomeostaticsystem} take this approach, focussing predominantly on neuro-endocrine networks to 
produce a robot controller that can change its high-level behaviour based on artificial hormones in its internal environment. The work by 
\citet{stradner_analysisimplementationartificial, schmickl_modellinghormoneinspired} follows a similar approach, developing a robotic system that makes 
use of hormone secretion and diffusion as a mechanism for signalling within a robot. They adapt the behaviour using evolutionary methods, and apply it 
to the problem of object avoidance.

 The work presented in this paper takes a different approach. Rather than focussing on the biological basis for homeostasis, this paper views 
homeostasis as a cognitive process. When the body is working to maintain its internal environment, it responds to changes in external stimuli as 
detected through low-level signals. It must then decide which high-level action to take in order to maintain that internal environment. From this 
perspective, homeostasis could be considered similar to the action selection problem or a decision-making 
problem~\citep{tyrrell_computationalmechanismsactionselection}. In other words, how does the body {\em decide} which action to take next in order to 
maintain its internal environment?

In this paper, we present \cah{}, a robot control architecture based on \cites{cohen_tendingadamsgarden} definition of cognition in the immune system. By 
linking memory, learning, and decision-making, a robot is able to rapidly adapt to---and sustain itself in---previously-unseen environments. 

\begin{quote}
 \begin{minipage}{\linewidth}
 \noindent We show that the \cah{} architecture can:
\begin{itemize} 
 \item Enable a robot to learn from previous experiences and use them to influence future behaviour
 \item Provide homeostatic behaviour to a robot
 \item Balance two conflicting requirements to provide homeostatic behaviour to a robot
\end{itemize}
\end{minipage}\end{quote}

Section~\ref{sec:architecture} describes the \cah{} architecture in detail, including a novel collective decision-making algorithm that does not 
require distributed sensing or leaders within the swarm. This provides the ability for a swarm of robots (or an individual robot simulating the swarm) 
to make multiple successive decisions without human intervention. 

Section~\ref{sec:robotics} presents experimental results showing that a robot running the \cah{} architecture is able to balance conflicting decisions. 
To demonstrate this, the robot is placed in an arena with a charging station within a high-temperature region. In order for the robot to re-charge, it 
has to venture into the high-temperature region, and balance overheating with running out of charge.



















 
 

 
\section{\cah{}: The cognitively-inspired homeostatic architecture}
\label{sec:architecture}

The similarities between cognition, autonomy and homeostasis have been discussed at great length, most notably 
in~\citep{varela_autonomyandautopoiesis, froese_autonomyareview, boden_autonomywhatisit, vernon_artificialcognitivesystems}. Cognition and 
autonomy have many different definitions. The large number of definitions (in excess of 20 separate types and 
definitions~\citep{schillo_taxonomyautonomymultiagent, vernon_artificialcognitivesystems}) gives rise to the problem of how to develop 
`autonomy' or `cognition' in a robotic system. While it might be possible to construct a system that meets the requirements of one definition, 
there are a large number of other definitions that will contradict it. The work in this paper has been based on one specific definition, which 
has evidence in distributed biological systems~\citep{cohen_cognitiveparadigm, cohen_cognitiveprinciplechallenges, 
cohen_tendingadamsgarden, mitchell_selfawarenesscontrol}. 
 \citet{cohen_tendingadamsgarden} proposed three properties of cognitive systems:
 \begin{quote}
   ``Cognitive systems, I propose, differ strategically from other systems in the way they combine three properties:
  \begin{enumerate}
    \item They can exercise options; {\em decisions}. 
    \item They contain within them images of their environments; {\em internal images}.
    \item They use experience to build and update their internal structures and images; {\em self-organization}.'' (p.~64, emphasis original)
  \end{enumerate} 
 \end{quote}

 The \cah{} (\cahLongUpper{}) architecture, shown in fig.~\ref{fig:cah}, takes \cites{cohen_tendingadamsgarden} definition of cognition and uses 
it as the basis for homeostasis in a robot. A robot running the \cah{} architecture is able to perform basic cognitive tasks---making decisions, 
learning about the environment, and applying previous experience to influence future decisions. These cognitive tasks enable the robot to 
exhibit homeostatic behaviour and adapt to new environments rapidly, in contrast to previous 
approaches~\citep{vargas_artificialhomeostaticsystem, stradner_analysisimplementationartificial}.

\begin{figure}[h!]
 \centering 
 \includegraphics[width=0.8\linewidth]{CDM_architecture.jpg}
 \caption[Diagram of the \cah{} architecture]{High-level architecture of the \cah{} system. The internal sensors (battery 
charge, temperature) drive the behaviour of the cognitive decision making{} (\cdm{}) simulation. As the agents within the \cdm{} react to the internal sensors, 
counting the proportion of agents in each attractor source within the \cdm{} can be used to drive the binary signals (black arrow) to the CMM input. 
The CMM (trained to recall which colour lamp is mounted above charging stations and high-temperature areas) then recalls which colour to search for 
according to the input that is triggered by the \cdm{}. The output from the CMM is then fed into the light-seeking circuits that perform basic 
chromotaxis to seek out or flee from the appropriate colour in the environment. }
 \label{fig:cah}
\end{figure}

The \cah{} architecture consists of three main components: the \cdm{} (\cdmLong{}) component, a CMM (Correlation Matrix Memory), and some basic 
light-seeking circuitry. The \cdm{} runs a novel decision-making algorithm (details in section~\ref{sec:decisionmaking}), and provides action 
selection based on the robot's internal sensor values.


Through the CMM, the robot is able to learn about its environment by associating changes in the robot's external sensor values with changes in 
its internal sensor values. Once trained, the CMM can then recall the appropriate high-level behaviour for the robot, based on the decisions 
made by the \cdm{}. Details of CMMs and their use in \cah{} are provided in section~\ref{sec:cmm}.


The rest of this section provides details of each of these components, and how they contribute to the working of the \cah{} architecture.




\subsection{Decision making}
\label{sec:decisionmaking}

 The \cdm{} is a major part of the \cah{} architecture, providing action selection based on 
internal sensor values. The \cdm{} is used to decide which of the low-level signals should 
affect the high-level behaviour of the robot. It works by simulating an abstract virtual 
environment formed of attractors and a swarm of agents. Each attractor has an associated 
coefficient that is linked to an internal sensor. As the sensor value varies, the `strength' 
of the attraction also varies. 

The swarm of virtual agents are collected together into a flock that makes decisions 
following \cites{cohen_tendingadamsgarden} definition of decision-making:
 \begin{quote}
   ``[A] decision emerges\ldots from a match between an environmental case and an internal 
motive. Decisions are associations.'' (p.\,69)
 \end{quote}

The swarm of agents respond to changes in the virtual environment, linking the behaviour of 
this virtual swarm to changes in the real environment outside the robot. The movement of the 
swarm of agents between attractors, therefore, provides the action selection required for the 
high-level behaviour of the robot by reacting to the low-level sensor values. In this paper, 
the attractors are connected to the charge level and temperature sensors on the robot.

The layout of this virtual environment could be varied in a number of different ways, 
but---other than varying the strength of the attraction---we keep the environment static. For 
this paper, we use an environment of 140x140 patches and centre the attractors at (58, 34) 
and (58, 110), with a spread of 15 patches. The swarm of agents is initialised at (15, 72). 
Through the use of emergent identities~\citep{stovold_preservingswarmidentity}, multiple 
swarms could be supported in the same environment but this is outside the scope of this 
paper.




We define the following terms: 
 \begin{itemize}
   \item `context' is the state of the environment as perceived by an agent 
(\citeauthors{cohen_tendingadamsgarden} `environmental case')
   \item `motive' is the likelihood of picking a certain actions, should an appropriate 
context arise
  \end{itemize} 
 In an empty environment, therefore, the actions of the swarm can only be a result of the 
inherent predisposition of certain actions within the swarm, and so are representative of the 
motive of the swarm. The internal motive of the swarm is provided by each agent projecting 
`virtual goals' ahead of them. The details of this algorithm are provided in 
section~\ref{sec:algorithm}.
 
 By combining the internal motive of the swarm with the context that the swarm is in, basic 
decision-making can be observed. As the swarm traverses the virtual environment, it reacts to 
contextual cues---provided by attractor sources---and decides whether to follow its internal 
motive, or to follow the environmental context. This approach to decision-making allows for 
multiple, successive decisions to be made without intervention from any external influence. 
This is key to its use in the \cah{} architecture, where the component is used to repeatedly 
make decisions based on continually-changing sensor values.

The \cdm{} differs from previous collective decision-making algorithms as it does not rely on 
`distributed sensing'. Previous approaches \citep{schmickl_beeclustswarmalgorithm, 
garnier_collectivedecisionmaking} rely on the distributed nature of the swarm to sample the 
environment, then aggregate at a certain point in space to make a decision. Once a decision 
has been made, however, the agents are all collected together at a single point in the 
environment. Without a way to recognise when a decision has been made, robots have no way to 
make subsequent decisions. Most existing collective decision-making algorithms instead rely 
on an external influence to redistribute them across the environment (typically a human 
operator). By collecting the agents into a flock, the \cdm{} algorithm avoids this problem 
and---unlike \citep{yu_collectivedecisionmaking}---the approach of the CDM does not require 
`leaders' within the swarm to guide it to a decision.

The modular nature of the \cah{} architecture means that the \cdm{} can be replaced by a 
different component to provide different behaviour to the robot. In this paper, we compare 
the \cdm{} component with a control case to test the efficacy of the \cdm{} to make 
successive decisions. The control case in the presented experiments uses a simple threshold 
component in place of the \cdm{}. The threshold component provides output signals based on 
sensor readings, for example, the charge in the battery dropping below a certain fixed 
threshold.


\subsubsection{\cdm{} algorithm} \label{sec:algorithm} As mentioned above, the \cdm{} simulates a virtual environment that contains attractors and a swarm of 
agents collected into a flock. This section describes the algorithm that the \cdm{} agents follow.

We defined context and motive in the previous section. \cites{cohen_tendingadamsgarden} definition of decision-making in a cognitive system is reduced to a function of context and motive:
 \begin{equation*}
   \label{eqn:decision}
   Decision = f(motive, context)
 \end{equation*}

The agents in the swarm move according to three forces: the flock force, the motive force, and the context force. These three forces---described in 
detail below---combine to provide a final directional force that the agents follow. The flock force is provided by 
\cites{olfatisaber_flockingmultiagent} algorithm, and works to keep the flock in a lattice formation. The motive force is provided by `virtual 
goals' projected by each agent ahead of the flock. The context force provides environmental cues to the agents, resulting from attractors in the 
environment.

The overall algorithm, combining each of the three forces listed above can be described by a series of vector-adjustments, $u_i$, for an agent $i$:
 \begin{equation}
  u_i = f_i^g + f_i^\gamma
 \end{equation}
 \noindent where $f_i^g$ acts to form a lattice structure among the neighbours in a flock 
(flock force), and $f_i^\gamma$ provides `navigational feedback' towards a goal (combining 
motive and context forces). This definition of $u_i$ varies from 
\cites{olfatisaber_flockingmultiagent} original definition through the removal of the 
directional term $f_i^d$, relying instead on the navigational feedback provided through 
$f_i^\gamma$.

\paragraph{Flock force} The flock force, $f_i^g$, provides the cohesive force that pushes the agents into a lattice formation. It is defined by 
\citet{olfatisaber_flockingmultiagent} as:
 \begin{equation}
        f_i^g = -\nabla_{q_i}V(q)
 \end{equation} where $V(q)$ is an attractive/repulsive force based on the distance to the nearest neighbouring agents, and $q_i$ is the two-dimensional position of agent $i$. $f_i^g$ provides the force required to keep an agent $i$ a set distance from its neighbouring agents. The full details of the flocking algorithm are provided in \citep{olfatisaber_flockingmultiagent}.


By restricting the agents to only local interactions, the algorithm is applicable to both simulated and real (robotic) platforms. In order to 
provide $f_i^\gamma$ to the swarm without global information, each agent projects a `virtual goal' ahead of itself, which collectively provides the 
navigational feedback required to prevent the flock from disintegrating \citep{olfatisaber_flockingmultiagent}. This virtual goal is the key 
component of the `motive force'.

\paragraph{Motive force} The motive force is provided through a `virtual goal' in the environment. This is calculated by each agent projecting forward by a predetermined distance, \param{d}, and using that position in the environment as its goal for a set period of time, \param{G$^{\param{update}}$}. These parameters, \param{d} and \param{G$^{\param{update}}$}, are set at the start of each simulation run, and do not vary throughout the run or between agents. Between them, these parameters provide a weighting towards motive or context for the agents (i.\,e.~ a weighting between the motive force and the context force).

Specifically, an agent $i$ calculates its virtual goal, $G_i$, by taking the average heading of its neighbours, $N_i$, within a pre-defined radius, \param{r}. Let the average heading of the neighbours of agent $i$ be $\phi_i$, defined as:
 \begin{equation}
   \phi_i = \dfrac{1}{|N_i|} \sum_{j\in N_i} \theta_j
 \end{equation}
 \noindent then projecting forwards by the predefined distance, \param{d}, gives the virtual goal for an agent $i$ as the coordinate pair $G_i = (x_{G_{i}},y_{G_{j}})$:
 \begin{align}
   x_{G_{i}} =&\, x_i + d\cdot cos(\phi_i) \\
   y_{G_{i}} =&\, y_i + d\cdot sin(\phi_i)
 \end{align}
 The goal is recalculated periodically so that the motive reflects the current state of the agent, including influences from the environment. This update period is parameterised as the virtual goal update interval (\param{G$^{\param{update}}$}). As the interval is increased, the agent is weighted further towards the motive than the context (and vice-versa), as information from the context will influence the position of the virtual goal less often.

\paragraph{Context force} The environment is empty other than any attractor sources included. An attractor, $j$, exerts a force on all agents in the environment. The force at any point can be described by the Gaussian probability density function, with mean $\mu_j$, standard deviation $\sigma_j$, and distance to the centre of the attractor from agent $i$, $(q_i - \mu_j)$.
 \begin{equation}
 \label{eqn:gaussian}
  f_i^{context}(\mu_j,\sigma_j,q_i) = \dfrac{1}{\sqrt{2\sigma_j^2\pi}}\, e^{-\dfrac{(q_i - \mu_j)^2}{2\sigma_j^2}}
 \end{equation}
The agent calculates $f_i^\gamma$ based on the coordinates of $G_i$ and the gradient formed by the attractors in the environment:
 \begin{equation}
 \label{eqn:fgamma}
  f_i^\gamma = f_i^{motive}(q_i,p_i,q_{G_i},p_{G_i}) + f_i^{context}(\mu_j,\sigma_j,q_i)
 \end{equation}
 \noindent Each agent multiplies the attractor gradient by a global parameter \param{ctx$\_$mult} as it senses it, in order to provide a weighting between $f_i^{motive}$ and $f_i^{context}$. See table \ref{tab:dm:gamma} for a summary of each term used in the definition of $f_i^\gamma$.

\begin{table}[h!]
 \centering
 \begin{tabular}{l|l}
  \textbf{Variable}             & \textbf{Description}                          \\ \hline\hline
  $q_i$                         & Two-dimensional position of agent $i$         \\ \hline
  $p_i$                         & Velocity of agent $i$                         \\ \hline
  $G_i$                         & Position of virtual goal for agent $i$        \\ \hline
  $\mu_j$                       & Centre-point of attractor $j$                 \\ \hline
  $\sigma_j$                    & Spread of attractor $j$                       \\
 \end{tabular}
 \caption[Variables used in definition of $f_i^\gamma$]{Descriptions of the variables used in equation \ref{eqn:fgamma}.}
\label{tab:dm:gamma}
\end{table}

\begin{table}[tbh]
 \centering
 \begin{tabular}{l|l|l|l}
  \textbf{Parameter}                    & \textbf{Description}                          & \textbf{Value}  & \textbf{Units}  	\\ \hline\hline
  \param{ctx$\_$mult}                   & Multiplier for attractor gradient             & 185   & N/A		\\ \hline
  \param{d}                             & Distance from agent to virtual goal           & 30    & patches	\\ \hline
  \param{G$^{\param{update}}$}  		& Update period for virtual goal                & 25    & timesteps	\\ \hline
  \param{r}                             & Radius used to calculate neighbourhood        & 8     & patches 	\\
 \end{tabular}
 \caption[Typical parameter values for the \cdm{} simulation]{Typical parameter values and descriptions for the \cdm{} simulation. `Patches' and `timesteps' are generic terms for coordinate space in the simulation and discretised simulated time, respectively. }
 \label{tab:parameters}
\end{table}

\subsection{Correlation Matrix Memories}
\label{sec:cmm}

Associative memory is a mechanism for associating stimuli with responses. In biological systems, the memory results from temporal associations that 
emerge between two sets of interacting neurons~\citep{palm_onassociativememory, palm_towardstheorycellassemblies}. As the stimulus is presented to 
the first set of neurons, the response of those neurons is sent to the second set of neurons, which subsequently respond. This second response is 
similar every time, indicating that the same response will occur if given the same stimulus.


The CMM (Correlation Matrix Memory)~\citep{kohonen_correlationmatrixmemories} is a matrix-based representation of this neural network. The matrix 
represents the binary weights from a fully-connected, two-layer artificial neural network (one input layer, one output layer; see 
fig.~\ref{fig:cmm:nn}). As such, the network in fig.~\ref{fig:cmm:nn} would be represented by the CMM, $\mathcal{M}$, with $k$ input--output pairs 
$\mathcal{I}$ and $\mathcal{O}$ in fig.~\ref{fig:cmm:maths}.

\begin{figure}[h!]
 \centering
 \subfloat[Basic CMM architecture (left), where $\mathcal{M}$ represents the matrix of binary weights, $\mathcal{I}$ and $\mathcal{O}$ represent the input--output pair corresponding to the neurons $a,b,c$ and $d,e,f$ in (b), respectively. \label{fig:cmm:maths}]{ \centering \parbox{\linewidth}{ %
 \begin{displaymath}
 \begin{array}{lc|cr}
 \begin{matrix}
   \begin{pmatrix}
     ~ \\
         \mathcal{I} \\
         ~
    \end{pmatrix}
   &
        \begin{pmatrix}
         ~ & ~ & ~ \\
         ~ & \mathcal{M} & ~ \\
         ~ & ~ & ~
        \end{pmatrix}
   \\
        ~ \\
   &
        \begin{pmatrix}
         ~ & ~\mathcal{O} & ~~
        \end{pmatrix}
  \end{matrix}
 & ~~ & ~~ &
  \begin{matrix}
    \begin{pmatrix}
        a \\
        b \\
        c
        \end{pmatrix}
  &
        \begin{pmatrix}
      ad & ae & af \\
          bd & be & bf \\
          cd & ce & cf
        \end{pmatrix}
  \\
        ~ \\
  &
        \begin{pmatrix}
          ~d & ~e & ~f~
        \end{pmatrix}
  \end{matrix}
 \end{array}
 \end{displaymath}  }
} \\
 \subfloat[CMM associative memory neural network. The binary weights between the two layers are represented by the matrix $\mathcal{M}$ in (a).
\label{fig:cmm:nn}] { \centering %
  \includegraphics[width=0.5\linewidth]{cmm_nn.jpg} }
 \caption[CMM associative memory neural network]{}
 \label{fig:cmm}
\end{figure}

Before training, the initial matrix $\mathcal{M}$ is filled with zeros (as there are no associations stored in the network). As the $k$ 
binary-valued input--output pairs are presented to the network, associations are built up in the matrix, $\mathcal{M}$. These associations are 
stored as 1s in the matrix, corresponding to coincident 1s in both input and output vectors. For example, in fig.~\ref{fig:cmm:maths}, if $a$ and 
$e$ were both 1 then $ae$ would be set to 1 after training.

\subsubsection{CMM in \cah{}}

In the \cah{} architecture, we use a CMM to associate changes in the environment with changes in internal motive, following on from 
\cites{cohen_tendingadamsgarden} definition of cognitive decision-making:
 \begin{quote}
   ``[A] decision emerges\ldots from a match between an environmental case and an internal motive. Decisions are associations.'' (p.\,69)
 \end{quote}
 \noindent He continues:
 \begin{quote}
   ``Decision-making is positive action; instead of passively receiving what the environment imposes, the cognitive system exerts its will\ldots in choosing among alternatives''~(p.\,69)
  \end{quote}
 The associations made between the environmental case (referred to throughout this paper as `context') and internal motive is made explicitly 
through the CMM, but also implicitly in the \cdm{} algorithm described above. 


The output from the \cdm{} component (i.\,e.~ the robot motive) is passed as an input to the CMM. In the \cah{} architecture the CMM stores an `internal 
image'---or imprint---of the environment for the robot to use. As such, the CMM offers the robot the capacity to make associations between changes 
in internal and external sensor values that occur at the same time. For example, if we mark a charging station with a blue light, the robot can 
sense the increase in blue light at the same time as an increase in battery charge. By associating these two signals, the CMM offers the ability to 
recall which colour to search for in order to find a charging station.

Fig.~\ref{fig:cmm:sensors} shows how the CMM is constructed for two internal sensors (charge, $c$, and temperature, $t$) and for the three colour 
components from the external light sensor (red, green, and blue). When one of the sensors passes a threshold, the corresponding binary value 
switches from 0 to 1. If this occurs on one of the internal sensors at the same time as one of the external sensors, then the corresponding value in 
the association matrix is set to 1 as well, storing this association between the two sensors. Re-presenting one of the internal sensor inputs (e.\,g.~ 
`charge'), by setting it to 1 again will recall the association. This will cause the closest stored associations to be recalled, and the appropriate 
values set in the output vector. This allows the system to test which colours have been associated with which inputs.

\begin{figure}[h!]
 \centering
\begin{displaymath}
\begin{array}{lc|cr}
 \begin{matrix}
  \begin{pmatrix}
    c \\
    t
  \end{pmatrix}
 &
  \begin{pmatrix}
        cr & cg & cb \\
        tr & tg & tb
  \end{pmatrix}
 \\
  ~ \\
 &
  \begin{pmatrix}
    ~r & ~g & ~b~
  \end{pmatrix}
 \end{matrix}
 & ~~ & ~~ &
 \begin{matrix}
   \begin{pmatrix}
         1 \\
         0
   \end{pmatrix}
&
   \begin{pmatrix}
        ~0 & ~0 & ~1 \\
        ~0 & ~0 & ~0
   \end{pmatrix}
  \\
        ~ \\
  &
        \begin{pmatrix}
        ~0 & ~0 & ~1~
        \end{pmatrix}
 \end{matrix}
\end{array}
\end{displaymath}
 \caption[Basic CMM for the \cah{} architecture]{Basic CMM architecture (left), where $c$ and $t$ correspond to the internal sensors for charge and 
temperature, and $r$, $g$, and $b$ correspond to the red, green, and blue colour components from the light sensor, respectively. The vectors 
$\icol{c\\t}$ and $(r\,g\,b)$ are provided as stimuli to the association matrix. Any points in the association matrix that has a 1 on both input 
vectors is set to 1, storing the association between the two. An example association is shown on the right, as would occur after the robot finds a 
charging station under a blue light.}
 \label{fig:cmm:sensors}
 \end{figure}

 
 
 
 
 
 
 







\section{Results}
\label{sec:robotics}


 \begin{minipage}{\linewidth}
\noindent By comparing with a control case, we demonstrate that the \cah{} architecture can\ldots
\begin{quote}
\begin{itemize} 
 \item[\textbf{\resQu{1}}:] \ldots enable a robot to learn from previous experiences and use 
them to influence future behaviour
 \item[\textbf{\resQu{2}}:] \ldots provide homeostatic behaviour to a robot
 \item[\textbf{\resQu{3}}:] \ldots balance two conflicting requirements to provide 
homeostatic behaviour to a robot
 \end{itemize}
\end{quote}
 \end{minipage}\newline

This control case uses a replacement for the \cdm{} that reacts to its internal state 
directly with a fixed threshold. 


 \subsection{A robot running the \cah{} architecture can learn from previous experiences}
 \label{sec:cah:sim}
 
 The \cah{} architecture enables the robot to adapt to its environment through the use of a 
CMM. The CMM associates large changes in internal sensors with large changes in external 
sensors. For example, if the robot discovers a charging station under a green lamp, the CMM 
will associate green light with charging. While, in principle, the \cah{} architecture is 
able to adapt on-line, this functionality is disabled in all experiments presented in this 
paper. This ensures that any variation between test and control cases are only due to 
swapping out the \cdm{}. This section aims to demonstrate that a robot running the \cah{} 
architecture can learn from its experience in an environment, and put it into practice in a 
simulated environment ahead of being used in real robot experiments.
 
 The simulation is shown in fig.~\ref{fig:cah:simulator}. This simulation is constructed in 
NetLogo~\citep{wilensky_netlogosimulator} and has the entire \cah{} architecture implemented. 
The signals provided by the \cdm{} to the rest of the architecture, however, are provided 
using switches on the simulator interface. The simulation consists of a single, zero-mass 
agent that roams around a simple environment of different light gradients. In place of the 
\cdm{}, we provide a mechanism for signalling that the agent has a certain need. This `need' 
is usually provided by the decision-making ability of the \cdm{}. By signalling that the 
agent has a certain need, the \cah{} architecture provides a mechanism for the agent to find 
the corresponding part of the environment. For example, `needs charge' recalls the 
appropriate colour of light from the CMM and searches for that colour in order to 
recharge.\footnote{we don't provide the literal term `needs charge' to the \cah{} 
architecture, the system instead provides a signal that indicates it is running low on 
charge.}
  
 \paragraph{Simulated training} The agent roams around the simulated environment during its 
training phase, learning about the environment. Once this has been completed the system is 
ready to be tested. In this case, if the training is successful, the agent should head 
towards the green light when the `needs charge' motive is switched on.
  
 Fig.~\ref{fig:cah:simulator} shows a trace of the agent moving around the environment. Once the `needs charge' switch is enabled (green arrow), the 
agent heads towards the green light source. Once this motive is disabled (blue arrow), the agent leaves the light source.
 
 This shows that, in principle, the architecture has the capacity to learn about its environment.
  



\begin{figure}[h!]
 \centering
 \includegraphics[width=0.7\linewidth]{netlogo_screenshot.jpg}
 \caption[Screenshot of the proof-of-principle simulation]{Screenshot of the proof-of-principle simulation, along with trace of the post-training agent searching for the green light source. The white arrows are the trace of the agent position, the green and blue arrows signify when the `needs charge' motive switch is enabled and disabled.}
 \label{fig:cah:simulator}
\end{figure}

 The post-training behaviour of the simulation shows that the \cah{} architecture is able to search for a specific part of the environment, based on 
high-level inputs such as a `needs charge' signal. The architecture translates this signal back to a colour, based on previous experience of the 
environment. In other words, the architecture recalls `green' as it learned that its batteries were charged when under a green light, which it 
subsequently seeks out.
will be provided by the \cdm{} algorithm when implemented into a robot.

 Having shown that the architecture behaves as expected in a simulated environment, the next step is to implement this architecture on a robotic 
platform, and test whether this behaviour is consistent. 

 \paragraph{Robot-based training} The \cah{} architecture is implemented onto a robot (details of the robot platform and experimental setup are 
provided in appendix~\ref{app:setup}). As before, the robot is allowed to roam around the environment during its training phase, learning about the 
environment. The environment (shown in fig.~\ref{fig:robot:cdm:arena}) consists of two coloured lamps (one red, one blue), and the \cah{} is set up 
to simulate charging its batteries under the blue light, and to simulate an increase in temperature under the red light.

 As the robot roams, the CMM associates high values from the internal sensors with high values from the external sensors. The threshold for external (colour) sensor values is 600 acu; for charging stations, 500 acu; for high-temperature areas, 300 acu (see appendix~\ref{app:setup} for details of the colour unit `acu'). Once the robot has roamed across both areas of the environment with lamps in, the robot is taken from the arena and the contents of the CMM downloaded.

 Due to the small size of the CMM used in this paper (3x2 matrix), the training phase can be tested with relative ease. In the following matrices, the following key can be used to determine whether the training has been successful, $r,g,b$ correspond to the red, green, and blue external sensors, and $c,t$ correspond to the charge and temperature internal sensors:
\begin{displaymath}
 \begin{matrix}
   \begin{pmatrix}
	cr & cg & cb \\
	tr & tg & tb
   \end{pmatrix}
 \end{matrix}
\end{displaymath}
After exposing the robot to an environment with one blue lamp over a charging station, the CMM is trained correctly as:
\begin{displaymath}
 \begin{matrix}
   \begin{pmatrix}
	~0 & ~0 & ~1 \\
	~0 & ~0 & ~0 
   \end{pmatrix}
 \end{matrix}
\end{displaymath}
After exposing the robot to an environment with both a red lamp over a high-temperature area and a blue lamp over a charging station, the CMM is trained correctly as:
\begin{displaymath}
 \begin{matrix}
   \begin{pmatrix}
	~0 & ~0 & ~1 \\
	~1 & ~0 & ~0 
   \end{pmatrix}
 \end{matrix}
\end{displaymath}


\subsection{\cah{} architecture provides homeostatic behaviour to a robot}
\label{sec:robot:results:two_sensor}

This section considers how a robot running the \cah{} architecture is able to alter its high-level behaviour according to its low-level needs. As the internal state of the robot varies, the \cdm{} signals to the rest of the \cah{} architecture what it needs in order to keep the state within its limits. 

The \cdm{} works to keep the internal sensors (charge and temperature) within certain limits. If the temperature gets too high or the charge too low then the robot will fail. The sensors are simulated so that the internal behaviour of the robot can be measured (if the battery actually ran out of charge, the data about internal behaviour might be lost or corrupted). The values used for different variables in our experimental setup are given in table~\ref{tab:robot:internal_sensors}.

\begin{table}[h!]
 \centering
 \begin{tabular}{l|c|c}
	~		 				& \textbf{Temperature}	& \textbf{Charge} 	\\ \hline\hline
	\param{Start value} 	& 10.0					& 5.0			\\ \hline
	\param{+ve Delta} 		& 1.8					& 0.8			\\ \hline
	\param{-ve Delta} 		& 0.5					& 0.1			\\ \hline
	\param{Fail Point}		& 55.0					& 0.1			\\ \hline
	\param{Limits} 			& 10.0, 60.0			& 0.01, 15.0 		\\ \hline
	\param{Control threshold}	& N/A				& 2.5
 \end{tabular}
 \caption[Parameter values for the basic homeostasis experiment]{Parameter values used in the basic homeostasis experiment. The values have been picked to result in a challenging, but possible scenario. }
 \label{tab:robot:internal_sensors}
\end{table}


This section looks at a basic scenario: asking whether the robot is able to keep itself charged while attending to another task. This other task is to warm itself under a lamp. By arranging the experimental arena (depicted in fig.~\ref{fig:robot:cdm:arena}) so that the charging station (a blue lamp) is far away from the warming lamp (a red lamp), the robot needs to leave the warming lamp in order to recharge, and vice-versa. This should result in the robot switching from sitting under one lamp to sitting under the other repeatedly and indefinitely. The parameters are chosen such that it is not able to warm itself sufficiently to complete its task before needing to charge again~(thus cooling the robot back down again).


\begin{figure}[h!]
 \centering
 \includegraphics[width=\linewidth]{two_source.jpg}
 \caption[Arena for homeostasis experiment]{Photo of arena with charging and warm areas superimposed, arena size: 224x108cm}
 \label{fig:robot:cdm:arena}
\end{figure} 

\paragraph{Null Hypothesis ($H_0$):} A robot running the \cah{} architecture will survive no longer with the \cdm{} component than with the threshold component.\newline

The design of the architecture described in section~\ref{sec:architecture} was such that the \cdm{} could be replaced by another component that 
provides different signals, based on the values of the internal sensors. The threshold component (described in section~\ref{sec:architecture}) is used 
in this way, and has a simple threshold mechanism. The output switches from 0 to 1 once the charge value drops below the control threshold of 2.5 (see 
table \ref{tab:robot:internal_sensors}, respectively for temperature).

The threshold value for the control system was calculated from the amount of time it takes for the robot to get to the light source it is searching 
for. This was determined empirically as 45 seconds (the upper quartile of the data represented by fig.~\ref{fig:robot:results:attractor_timing}). Using 
this value for traversing the environment, the threshold can be calculated as 2.0. In order to take into account the variability of working in a noisy 
environment, the threshold was increased slightly to 2.5, allowing 54 seconds in the worst case for the robot to find its way between light sources. 
This also helps to reduce the chance of a false positive result by making it easier for the control setup to survive for longer.


\begin{figure}[h!]
 \centering
 \includegraphics[width=0.4\linewidth]{traversal.jpg}
 \caption[Arena traversal times]{Boxplot showing how long it takes to cross the arena between light sources, measured empirically.}
 \label{fig:robot:results:attractor_timing}
\end{figure}

 Fig.~\ref{fig:robot:results:example_graphs} shows the path of one run (a) with the \cdm{} in place. The graph in (b) shows a subset of the internal 
sensor data provided to the \cdm{} component for the duration of the experimental run. The robot was allowed to continue running until it reached 29 
minutes, at which point it would be stopped\footnote{29 minutes comes from the maximum amount of time the camera could record for}. The traces show that 
the robot is able to perform a task while preventing itself from running low on charge.

\begin{figure}[h!]
 \centering 
 \subfloat[Trace of robot path between 5.5 and 10.5 mins superimposed on the environment. The blue and red circles indicate points where the robot stopped to charge or warm itself respectively. The green and white circles indicate the start and end points. The red and blue solid lines indicate that the robot is searching for that colour. The white dotted line is when the robot wanders (not actively searching for anything).  \label{fig:robot:results:example_graphs:trace}] { \centering	%
 \includegraphics[width=.9\linewidth]{cdmTrace.jpg} }			\\
 \subfloat[Graphs showing internal sensor values (top) and \cdm{} outputs (bottom) for the robot between 5.5 and 10.5 mins. The red and blue line show the internal values for temperature and charge as presented to the \cdm{} component. The red and blue blocks of colour in the bottom graph show the output of the \cdm{} component (red indicating to the robot to search for red light, and blue indicating to search for blue light).  \label{fig:robot:results:example_graphs:graphs}] { \centering %
 \includegraphics[width=.9\linewidth]{cdm_examplegraphs.jpg} } 
 \caption[Robot behaviour in the arena, along with internal values]{A short (5 min) excerpt from one of the 29-minute replicates. The trace (a) shows 
how the robot moves between the red and blue areas in the environment as it needs to charge. The graphs in (b) show that as the internal sensor values 
for charge get too low, the robot actively searches for the charging station, and once it is charged sufficiently, it returns to warming itself under 
the red lamp.}
 \label{fig:robot:results:example_graphs}
\end{figure}

Table \ref{tab:robot:results:two-sensor} shows the length of time that the robot survived when set up with the two components described above (up to 
the maximum of 29 minutes). This data is presented 
in a table format, rather than a boxplot due to the relatively few replicates in this experiment. The high variance in the \cdm{} data is unexpected 
and can be explained by the robot getting stuck in a corner early on in the run and failing as a result. These `failed' runs are still included to 
ensure the results are not biased by their removal. Even with only 6 replicates, it is clear that the \cdm{} code can survive for longer than the 
control code. This is confirmed through the Mann--Whitney--Wilcoxon test and Vargha--Delaney A-test, giving $p<0.05$ and $A<0.16$ respectively. From 
this, $H_0$ can be rejected at the 95\% confidence level.

\begin{table}[h!]
 \centering
 \begin{tabular}{r|l|l}
  \textbf{Replicate}	&	\textbf{Test (mins)}	& 	\textbf{Control~(mins)}	\\ \hline\hline
  1				&	3.5	&	29		\\ \hline
  2				&	4	&	2.5		\\ \hline
  3				&	2	&	29			\\ \hline
  4				&	2.5	&	6			\\ \hline
  5				& 	10	&	29			\\ \hline
  6				&	3	&	29			
 \end{tabular}
 \caption[Results from the dual-attractor experimental setup]{Details of how long the robot survived in the environment when using the \cdm{} (test) 
component and the threshold (control) component. Mann--Whitney--Wilcoxon test and Vargha--Delaney A-test give $p<0.05$ and $A<0.16$ respectively, 
rejecting $H_0$ with 95\% confidence.}
 \label{tab:robot:results:two-sensor}
\end{table}



While performing a relatively simple task, the results presented in this section show that the \cah{} architecture is a viable option for providing 
simple artificial homeostasis to a robot. This provides the evidence required to answer the research question \resQu{2}. The results here also suggest 
that the robot is able to make use of its previous experience in the environment, and apply it to survive in the same environment. This provides 
further evidence towards answering research question \resQu{1}. The next section discusses how well the architecture handles more complicated scenarios 
that involve conflicting needs.


\subsection{Conflicting decisions}
\label{sec:robot:results:conflict}


While the work above shows how the \cah{} architecture could be used to control a robot in a simple environment (two distinct light sources), with a 
simple task (survive as long as possible, while performing a basic task), this section addresses research question \resQu{3}: can the \cah{}-controlled 
robot survive when faced with conflicting decisions about what to do?

In order to achieve this, the experimental setup is altered such that the charging station and the higher temperature area are provided by the same 
light source (see fig.~\ref{fig:robot:results:conflict:arena}). The behaviour of the \cdm{} is inverted with respect to temperature, so that the robot 
now tries to avoid getting warm. Therefore, in order to charge its battery, the robot will have to endure warmer temperatures. Once the battery runs 
out, the robot will fail, and once the temperature reaches a set maximum, the robot will fail. This experiment once again makes use of the same 
architecture between test and control cases, except for the \cdm{} component that will be replaced by the threshold component for the control case.


\paragraph{Null Hypothesis ($H_0$):} A robot running the \cah{} architecture will survive no longer with the \cdm{} component than with the threshold component in the presence 
of conflicting needs. \newline

The experimental setup is as shown in fig.~\ref{fig:robot:results:conflict:arena}. This is the same arena as above, except for the removal of the blue 
light from the right-hand side. In this more difficult scenario, the robot is expected to fail more often than it did in the previous setup, as before 
it was possible for the robot to sit and charge indefinitely without failing, whereas here this would result in a failure. After preliminary testing, 
it was evident that the parameter values used previously (given in table~\ref{tab:robot:internal_sensors}) provided a scenario that was too difficult 
for the robot to complete with either \cdm{} or threshold-based architecture (all survival times were under 2.5 minutes). As such, the parameter values 
are adjusted to those shown in table~\ref{tab:robot:conflict:internal_sensors}, to make the robot less likely to overheat straight away, but balancing 
this by making it slower to charge.

\begin{figure}[h!]
 \centering
 \includegraphics[width=\linewidth]{one_source.jpg}
 \caption[Arena for conflicting decisions]{Photo showing the new arena for conflicting decisions experiment, arena size: 224x108cm}
 \label{fig:robot:results:conflict:arena}
\end{figure}

\begin{table}[h!]
 \centering
 \begin{tabular}{l|c|c}
	~		 					& \textbf{Temperature}	& \textbf{Charge} 	\\ \hline\hline
	\param{Start value} 		& 10.0					& 5.0			\\ \hline
	\param{+ve Delta} 			& 1.3					& 0.3			\\ \hline
	\param{-ve Delta} 			& 0.8					& 0.1			\\ \hline
	\param{Fail Point}			& 55.0					& 0.1			\\ \hline
	\param{Limits} 				& 10.0, 60.0			& 0.01, 15.0	 	\\ \hline
	\param{Control threshold}	& 25.0					& 2.5
 \end{tabular}
 \caption[Parameters for conflicting decision experiment]{The parameters used in the `conflicting decisions' experiment. These values have been picked 
to result in a challenging, but possible scenario. }
 \label{tab:robot:conflict:internal_sensors}
\end{table}

Fig.~\ref{fig:robot:results:conflict:example_graphs} shows a trace of the sensor values as provided to the \cdm{} component. The graphs show how the 
robot balances the need to charge its battery while avoiding getting too warm. The robot with \cdm{} component was able to balance the conflicting 
decisions well, failing only once in 13 replicates, whereas the control case failed 6 times out of 13. These results are presented in the boxplots in 
fig.~\ref{fig:robot:results:conflict:boxplots}.


\begin{figure}[H]
 \centering
 \subfloat[Trace of robot path between 14 and 19 mins superimposed on the environment. The blue and red circles indicate points where the robot stopped to charge or cool itself respectively. The green and white circles indicate the start and end points. The blue and red solid lines indicate that the robot is searching for, or fleeing from, the red light, respectively. The white dotted line is when the robot wanders (not actively searching for anything).  \label{fig:robot:results:conflict:example_graphs:trace} ] { \centering %
 \includegraphics[width=.9\linewidth]{conflictTrace.jpg} }		\\
 \subfloat[Graphs showing internal sensor values (top) and \cdm{} outputs (bottom) for the robot between 14 and 19 mins. The red and blue lines show the internal values for temperature and charge, as presented to the \cdm{} component. The red and blue blocks of colour in the bottom graph show the output of the \cdm{} component (red indicating to the robot to flee red light, and blue indicating to search for red light).  \label{fig:robot:results:conflict:example_graphs:graphs} ] { \centering %
 \includegraphics[width=.9\linewidth]{conflict_examplegraphs.jpg} }
 \caption[Robot path trace and graphs from conflict run]{A short (5 min) excerpt from one of the 29-minute replicates. The trace (a) shows how the robot moves in and out of the red area in the environment as it needs to charge and cool down. The graphs in (b) show that as the internal sensor values for charge get too low, the robot actively searches for the charging station, and vice-versa for when the temperature gets too high. Note that, contrary to the behaviour described in fig.~\ref{fig:robot:results:example_graphs}, when the temperature gets too high at around 17 mins, the \cdm{} does not actively push the robot away from the red light, as it is already out of the area by chance. This has the effect of starting to reduce the temperature, reducing the likelihood that the \cdm{} component would push the robot to seek out a cooler area.}
 \label{fig:robot:results:conflict:example_graphs}
\end{figure}

The boxplots in fig.~\ref{fig:robot:results:conflict:boxplots} show the amount of time that the two setups survived (up to the maximum of 29 minutes). 
There are 13 replicates for each. Results from the control code are presented on the left, and from the \cdm{} (test) code on the right.

\begin{figure}[h!]
 \centering
 \includegraphics[width=0.7\linewidth]{conflict_boxplot.jpg}
 \caption[Conflicting decisions experiment results]{Boxplots showing the distribution of survival times from the conflicting decisions experiment. The two distributions are significantly different, with $p=0.03$ and $A=0.30$.}
 \label{fig:robot:results:conflict:boxplots}
\end{figure}


While these results were expected to be less clear-cut than the previous experiment, analysis gives similar results. The Mann--Whitney--Wilcoxon test gives $p=0.03$, and the Vargha--Delaney effect-magnitude test gives $A=0.30$. From this, $H_0$ can be rejected with $95\%$ confidence, resulting in the conclusion that the \cdm{} component is better at balancing conflicting decisions than the threshold component, and providing sufficient evidence to answer research question \resQu{3}. As with the previous experiment, the results here suggest that the robot is still able to make use of its previous experience in the environment, and apply it to survive in the same environment. This provides even further evidence towards answering research question \resQu{1}. 



\section{Conclusion}
\label{sec:conclusion}

This paper has presented a new approach to developing robotic homeostasis. By taking inspiration from cognition, rather than cellular-level biology, a 
simple homeostatic architecture has been developed for a robot which is able to balance conflicting needs. Evidence towards answering three research 
questions is presented. These research questions are summarised below. \newline

\noindent\textbf{\resQu{1}} ({\em Can the \cah{} architecture enable a robot learn to from previous experiences and use them to influence future behaviour?}) Section \ref{sec:cah:sim} presents the testing of the CMM training phase. This process consists of the robot roaming around an environment and associating simultaneous changes in the internal and external sensors. These associated changes represent the previous experiences that the robot has learnt. The behaviour of the robot in sections \ref{sec:robot:results:two_sensor} and \ref{sec:robot:results:conflict} show that the robot is able to successfully recall these experiences and use them to influence its high-level behaviour. \newline


\noindent\textbf{\resQu{2}} ({\em Can the \cah{} architecture provide homeostatic behaviour to a robot?}) Section \ref{sec:robot:results:two_sensor} presents results showing that the robot is able to alter its high-level behaviour according to the value of two internal sensors (temperature and charge). The behaviour of the robot is compared between two versions of the \cah{} architecture: one with the \cdm{} component, one with a simple threshold function (see section \ref{sec:architecture}). The \cdm{} component consistently outperforms the threshold function at providing homeostatic behaviour to a robot, as shown by the survival task in fig.~\ref{fig:robot:results:example_graphs}. \newline

\noindent\textbf{\resQu{3}} ({\em Can the \cah{} architecture balance two conflicting needs to provide homeostatic behaviour to a robot?}) Section \ref{sec:robot:results:conflict} presents results showing that the robot is able to balance two conflicting needs. The robot was required to withstand a high-temperature region in order to gain access to a charging station. The behaviour of the robot is again compared between two versions of the \cah{} architecture as with \resQu{2}, above. The \cdm{} component once again consistently outperforms the threshold function at providing homeostatic behaviour to a robot, while balancing two conflicting needs. \newline


While other approaches to homeostasis have directly mimicked the natural systems of the body~\citep{neal_timidityusefulemotional, neal_oncemoreunto, 
vargas_artificialhomeostaticsystem, stradner_analysisimplementationartificial, schmickl_modellinghormoneinspired}, the \cah{} architecture has been 
developed by taking inspiration from cognition~\citep{cohen_tendingadamsgarden, mitchell_selfawarenesscontrol}.  Combining the \cdm{} 
(cognitive decision making{}) with a CMM (Correlation Matrix Memory) associative memory provides the \cah{} (\cahLongUpper{}) architecture with the ability to alter 
its high-level behaviour according to the low-level state of the robot, while based on previous experiences.

The CMM is trained by coincident `spikes' in the internal and external sensors (e.\,g.~ when the `charge' internal sensor rises at the same time as the `blue' external sensor---this relates to a charging station under a blue light in the real-world environment). This provides adaptivity to the robot, as it is able to learn about new parts of an environment and adapt its behaviour accordingly (for example, if the robot were to find another charging station, it would store the association in the CMM and recall it as before). 

The robot is able to make decisions about its current internal state through the \cdm{} and transfer this to high-level behavioural changes, showing that the \cah{} architecture provides the capacity for homeostasis to the robot. Furthermore, the use of the CMM to recall previous experiences allows the robot to proactively search for, or flee from, specific areas in the environment. This proactivity allows for behaviour that is more realistic for real-world applications.

The modular nature of the \cah{} architecture, in that the \cdm{} can be easily replaced by a different decision-making mechanism, makes the \cah{} architecture more flexible for different applications. It could be possible to run multiple decision-making algorithms as an ensemble, with the combination of their outputs being passed to the CMM. 

There is also potential for online learning about an environment, where the CMM would be training and recalling simultaneously. Furthermore, the sharing of experience between multiple robots working in the same area could be possible by sharing the contents of CMMs between robots.

In summary, cognition is a viable source of inspiration for homeostasis in robots. The resulting architecture shows that there is significant potential from implementing cognitive behaviour into robots, giving the ability to balance conflicting needs more effectively than a conventional threshold mechanism.


















