\section{Introduction}
\label{sec:introduction}

\IEEEPARstart{P}{olicy} optimization is a cornerstone in planning, control and reinforcement learning. Classic approaches include value iteration and policy iteration \cite{dynkin1979controlled, puterman2014markov, bertsekas2011dynamic, bertsekas2019reinforcement, sutton2018reinforcement}. From a theoretical standpoint, a key step is establishing when policy optimization is well-posed, i.e., when optimal policies exist, and when they can be found algorithmically. While such results for value and policy iteration are well established for discrete systems with finitely many states and actions (also known as the tabular setting), relatively few foundational results have been established for continuous control.

In this paper, we study Metric Policy Optimization Problems (MPOPs), which come endowed with metric state and action spaces. Compared to tabular MDPs, several new challenges arise in the continuous setting. Even for deterministic problems, rewards may be unbounded, maxima of functions (over actions) may not exist, and the domains of value functions may be different for different policies. Without addressing these challenges, optimal policies need not exist and value iteration or policy iteration may be impossible.

In order to establish well-posedness of dynamic programming approaches, we define the class of Compactly Restrictable MPOPs (CR-MPOPs). We show that CR-MPOPs arise naturally when imposing forward-invariance constraints on the policy class one optimizes over. As such, CR-MPOPs are well suited for characterizing many systems which rely on forward-invariance for controller design.

Sampled-data design \cite{nevsic1999sufficient} allows us to synthesize policies for continuous-time systems when inputs are passed through a zero-order hold (held constant over fixed frequency time intervals), as is realistic for many physical systems. We leverage recent results \cite{taylor2021sampled, taylor2022safety} to certify the existence of a compact subset of the state space that can be rendered forward-invariant through control. These results are readily applicable to feedback linearizable control affine systems, allowing us to design CR-MPOPs for a wide array of complex systems.

Many frameworks extend the theory of discrete MDPs to general state and action spaces. Most relevant for our work, \cite{dynkin1979controlled} develops semicontinuous MDPs with Borel measurable policy classes, \cite{blackwell1974optimal} and \cite{bertsekas1996stochastic} develop results for analyticially and universally measurable policy classes, respectively, and \cite{feinberg2012average} refines conditions for the well-posedness of value iteration. For a more complete summary, see \cite{yu2015mixed}. Our work focuses on developing classes of policy optimization problems that can naturally connect theoretical results from reinforcement learning with nonlinear continuous-time control, and can be directly translated to checking certain properties in control systems.

Methods that maintain the forward-invariance of subsets of the state space of interest have been well-studied \cite{blanchini1999set, prajna2005optimization, ames2016control}. These and related methods have recently found applications in safe reinforcement learning \cite{Berkenkamp2017SafeRL, rosolia2017learning, cheng2019end, choi2020reinforcement}. Physical systems and robotic platforms have been popular applications for reinforcement learning methods recently, with the majority of methods employing function approximation, discretizations (such as fixed or adaptive gridding or state aggregation), or direct policy search \cite{kober2013reinforcement,levine2016end, lillicrap2015continuous}. 
While policy search methods generally have no guarantees of global optimality, they have received much attention due to the scalability problems of dynamic programming methods (especially after discretization) and convergence problems of approximate dynamic programming (for a more complete discussion of the relative merits of these methods, see \cite[\S 2.3]{kober2013reinforcement}). In contrast, our goal is to identify settings that are compatible with value iteration and to use control theoretic tools to guide solution methods so we can expect good performance.

\subsection{Contributions}
We develop our results in the following phases:
\begin{inparaenum}[1)]
    \item We describe a generic property (Assumption \ref{as:well-posed}) that admits the well-posedness of value iteration for MPOPs (Theorem \ref{thm:well-posed}).
    \item We leverage the forward-invariance of compact sets to design MPOPs and policy classes that satisfy Assumption \ref{as:well-posed}, allowing us to prove well-posedness of a large class of policy optimization problems (Theorem \ref{thm:invariance}). Such problem settings are called CR-MPOPs.
    \item We apply our results on CR-MPOPs to analyze control affine systems with time-sampled control inputs.
    \item We further apply our results to analyze robotic systems with time-sampled control inputs, which comprise a large class of control affine systems. These results translate the generic requirements for well-posedness to concrete requirements on the robotic system.
    \item We finally show that full-state feedback linearizable control affine systems can render a compact subset of the state space forward invariant with continuous controllers (thus satisfying conditions of Theorem \ref{thm:invariance}), demonstrating practically relevant instances of CR-MPOPs.
\end{inparaenum}

\subsection{Notation, Conventions, and Definitions}
We consider nonempty metric spaces throughout this paper and endow each such space with the $\sigma$-algebra generated by its topology, called the Borel $\sigma$-algebra. In this case, open sets and closed sets are measurable sets and continuous functions are measurable functions. A metric space is separable if it contains a countable dense subset. Finite or countable sets can be regarded as separable metric spaces when equipped with the discrete metric \cite[\S 3.A]{kechris2012classical}. For nonempty metric spaces $X$ and $Y$, the set of measurable functions from $X$ to $Y$ is denoted $\mathcal{L}^0(X; Y)$. A bounded measurable function $f: X \to \bb{R}$ is upper semicontinuous if for any $c \in \bb{R}$, the preimage $f^{-1}([c, \infty)) = \{ x \in X: f(x) \geq c \}$ is closed.

The set of all subsets of $Y$ is the powerset of $Y$, denoted $\mathcal{P}(Y)$. A set-valued function $C: X \to \mathcal{P}(Y)$ is called a correspondence.
The graph of $C$ is defined as:
\begin{equation}
    \mathrm{Graph}(C) = \{ (x, y) \in X \times Y: y \in C(x) \}.
\end{equation}
If $C(x) \neq \emptyset$ for all $x \in X$, then a selector of $C$ is a function $f: X \to Y$ satisfying $f(x) \in C(x)$ for all $x \in X$. For any set $B \subseteq Y$, the lower preimage of $B$ under $C$ is defined as:
\begin{equation}
    C^{\ell}(B) = \{ x \in X: C(x) \cap B \neq \emptyset \}.
\end{equation}
The correspondence $C$ is upper hemicontinuous\footnote{Some authors call such correspondences upper semicontinuous; we use hemicontinuous to distinguish correspondences from real-valued functions.} if the lower preimage of each closed set is a closed set \cite[Lemma 17.4]{aliprantis2006infinite}.

A nonempty subset $X_0 \subseteq X$ is a metric space when equipped with the restriction of the metric on $X$ to $X_0 \times X_0$. For any open set $U \subseteq X_0$, there is an open set $V \subseteq X$ such that $U = X_0 \cap V$. For any continuous function $f: X \to Y$, the restriction $\left.f\right|_{X_0}: X_0 \to Y$ is continuous. Similarly, for any measurable set $A \subseteq X_0$, there is a measurable set $B \subseteq X$ such that $A = X_0 \cap B$, and for any measurable function $f: X \to Y$, the restriction $\left.f\right|_{X_0}$ is measurable. 

Consider a nonempty metric space $Z$ and a function $f: X \times Y \to Z$. For $x \in X$ and $y \in Y$, the $x$-section $f_x: Y \to Z$ and the $y$-section $f^y: X \to Z$ are defined as:
\begin{align}
    f_x(y') &= f(x, y'), & f^y(x') = f(x', y),
\end{align}
for all $x' \in X$ and $y' \in Y$. If $f$ is continuous, then $f$ has continuous sections, and if $f$ is measurable, then $f$ has measurable sections. We also define sections for functions on graphs. Consider a correspondence $C: X \to \mathcal{P}(Y)$ with $C(x) \neq \emptyset$ for all $x \in X$, and a function $f: \mathrm{Graph}(C) \to Z$. For $x \in X$, we define the $x$-section $f_x: C(x) \to Z$ as above.

$\bb{N}$ denotes the natural numbers, $\bb{Z}_+$ denotes the nonnegative integers, $\bb{R}_+$ and $\bb{R}_{++}$ denote nonnegative and positive reals, and $\bb{S}^d_+$ and $\bb{S}^d_{++}$ denote $d \times d$ positive semidefinite and definite matrices.

\section{Metric Policy Optimization Problems}
\label{sec:mdp}
In this paper, a deterministic Markov decision process (MDP) will be characterized by a tuple $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$, with state space $\mathcal{S}$, action space $\mathcal{A}$, action-admissibility correspondence $C: \mathcal{S} \to \mathcal{P}(\mathcal{A})$ satisfying $C(s) \neq \emptyset$ for each state $s \in \mathcal{S}$, transition map $f: \mathrm{Graph}(C) \to \mathcal{S}$, reward function $r: \mathrm{Graph}(C) \to \bb{R}$, and discount factor $\gamma \in [0, 1)$. Note while $r$ may be unbounded, $r$ cannot assume the values $\pm \infty$. We refer to an MDP as a metric MDP if $\mathcal{S}$ and $\mathcal{A}$ are nonempty separable metric spaces and $f$ and $r$ are measurable functions. In this paper, we focus solely on metric MDPs, and often refer to them simply as MDPs for brevity.

We will limit our consideration to deterministic, Markovian, and stationary (time-invariant) policies. In this case, a policy $\pi: \mathcal{S} \to \mathcal{A}$ is a selector of $C$; that is, $\pi(s) \in C(s)$ for all states $s \in \mathcal{S}$. The corresponding closed-loop transition map $f_\pi: \mathcal{S} \to \mathcal{S}$ and single-step reward function $r_\pi: \mathcal{S} \to \bb{R}$ are defined as:
\begin{align}
    f_\pi(s) &= f(s, \pi(s)), & r_\pi(s) = r(s, \pi(s)).
\end{align}
for all states $s \in \mathcal{S}$. For any $t \in \bb{Z}_+$, let $f_\pi^t: \mathcal{S} \to \mathcal{S}$ denote the $t$-iterated composition of $f_\pi$. Define the subset $\mathcal{S}_\pi \subseteq \mathcal{S}$ as:
\begin{equation}
    \mathcal{S}_\pi = \left\{ s \in \mathcal{S}: \sum_{t = 0}^\infty \gamma^t r_\pi( f_\pi^t(s) ) ~\mathrm{converges~absolutely} \right\}.
\end{equation}
If $r_\pi$ is bounded, then $\mathcal{S}_\pi = \mathcal{S}$. If $\gamma > 0$, then $f_\pi(\mathcal{S}_\pi) \subseteq \mathcal{S}_\pi$, as a convergent series still converges after the first term is removed. If $\mathcal{S}_\pi \neq \emptyset$, then the corresponding value function $V_\pi: \mathcal{S}_\pi \to \bb{R}$ is defined explicitly and implicitly, for all states $s \in \mathcal{S}_\pi$, as:
\begin{equation}
    V_\pi(s) = \sum_{t = 0}^\infty \gamma^t r_\pi(f_\pi^t(s)) = r_\pi(s) + \gamma V_\pi(f_\pi(s)).
\end{equation}

A set of policies $\Pi$ admits a partial order if $\mathcal{S}_\pi = \mathcal{S}$ for all policies $\pi \in \Pi$. In this case, for policies $\pi, \pi' \in \Pi$, if $V_{\pi}(s) \geq V_{\pi'}(s)$ for all states $s \in \mathcal{S}$, then $\pi \succeq \pi'$.

\begin{definition}[Metric Policy Optimization Problem]
We refer to the pair of an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ admitting a partial order as a \textit{metric policy optimization problem} (MPOP). We call an MPOP well-posed if there is an optimal policy $\pi^* \in \Pi$ with $\pi^* \succeq \pi$ for all policies $\pi \in \Pi$.
\end{definition}

\textbf{Goals of this section.} Our goal is to analyze well-posedness of policy optimization for MDPs. We first describe sufficient conditions for well-posedness of value iteration (Theorem \ref{thm:well-posed}). We then establish conditions under which ill-posed MPOPs can be restricted to well-posed problems by only considering policies that render the same subset of the state space forward-invariant (Theorem \ref{thm:invariance}), resulting in CR-MPOPs. We will show how to apply our results on value iteration to control affine systems in Section \ref{sec:control}.

Throughout this section, we refer to and modify the following example to ground our presentation:
\begin{example}\label{ex:running-example}
Consider the state space $\mathcal{S} = \bb{R}$, action space $\mathcal{A} = \bb{R}$, and the constant action-admissibility correspondence $C = \mathcal{A}$. Additionally, consider the transition function $f: \mathrm{Graph}(C) \to \mathcal{S}$ and reward function $r: \mathrm{Graph}(C) \to \bb{R}$ defined as:
\begin{align}
    f(s, a) &= s + \tanh{(a)}, & r(s, a) = -s^2 - (\tanh{(a)})^2,
\end{align}
for all state-action pairs $(s, a) \in \mathrm{Graph}(C)$, and a discount factor $\gamma \in [0, 1)$. Note that $| f(s, a) | \leq | s | + | \tanh{(a)} | \leq | s | + 1$ for all state-action pairs $(s, a) \in \mathrm{Graph}(C)$. Consider any policy $\pi: \mathcal{S} \to \mathcal{A}$. By induction, we have $| f_\pi^t(s) | \leq | s | + t$ for all states $s \in \mathcal{S}$ and $t \in \bb{Z}_+$. For any state $s \in \mathcal{S}$ and $T \in \bb{Z}_+$, we have: 
\begin{align}
    \sum_{t = 0}^T \gamma^t r_\pi(f_\pi^t(s)) &\geq -\sum_{t = 0}^\infty \gamma^t ( (f_\pi^t(s))^2 + (\tanh{(\pi(f_\pi^t(s)))})^2 )\nonumber\\
    &\geq -\sum_{t = 0}^\infty \gamma^t ( s^2 + 2|s|t + t^2 + 1 )\nonumber\\
    &= -\frac{s^2 + 1}{1 - \gamma} - \frac{2\gamma |s|}{(1 - \gamma)^2} - \frac{\gamma(\gamma + 1)}{(1 - \gamma)^3}.
\end{align}
The sequence of partial sums as $T \to \infty$ is monotone and bounded below, so $V_\pi(s)$ is well-defined. As $s$ was arbitrary, we have $\mathcal{S}_\pi = \mathcal{S}$.
\end{example}

\subsection{Value Iteration}
Consider a policy class $\Pi$ with $\mathcal{S}_\pi = \mathcal{S}$ for each policy $\pi \in \Pi$ and $\sup_{\pi \in \Pi} V_\pi(s) < \infty$ for each state $s \in \mathcal{S}$. Accordingly, we define the optimal (with respect to $\Pi$) value function $V^*: \mathcal{S} \to \bb{R}$ as:
\begin{equation}
    V^*(s) = \sup_{\pi \in \Pi} V_\pi(s),
\end{equation}
for all states $s \in \mathcal{S}$. If a policy $\pi^* \in \Pi$ satisfies $V_{\pi^*} = V^*$, then $\pi^*$ is optimal and the policy optimization problem is well-posed.

Value iteration generates a sequence of approximations of $V^*$. Given an initial guess $V_0: \mathcal{S} \to \bb{R}$, we seek a sequence of real-valued functions $\{ V_n : n \in \bb{N} \}$ satisfying:
\begin{equation}\label{eq:value-iteration-step}
    V_{n + 1}(s) = \sup_{a \in C(s)}{\{ r(s, a) + \gamma V_n(f(s, a)) \}},
\end{equation}
for all states $s \in \mathcal{S}$ and $n \in \bb{Z}_+$.

We now describe conditions under which MPOPs are well-posed and can be solved with value iteration (Theorem \ref{thm:well-posed}). When assumptions are strengthened by requiring $\mathcal{S}$ and $\mathcal{A}$ to be Polish spaces, we will make use of \cite[Theorem 4.1]{feinberg2012average} (the setting of this theorem is called a semicontinuous model in \cite[\S 6.7]{dynkin1979controlled}). We provide a more direct proof of well-posedness without these additional assumptions in the appendix.

\begin{assumption}\label{as:well-posed}
The action admissibility correspondence $C$ has compact values and is upper hemicontinuous, the transition function $f$ is continuous, the reward function $r$ is upper semicontinuous and bounded, and the policy class $\Pi$ is the set of all measurable selectors of $C$. Moreover, for each state-action pair $(s, a) \in \mathrm{Graph}(C)$, there is a corresponding policy $\pi_{s, a} \in \Pi$ satisfying $\pi_{s, a}(s) = a$.
\end{assumption}

\begin{remark}\label{rem:relax-assumption}
If the projection of $\mathrm{Graph}(C)$ onto the action space is compact, then the assumption that for any state-action pair $(s, a) \in \mathrm{Graph}(C)$, there exists a policy $\pi_{s, a} \in \Pi$ with $\pi_{s, a}(s) = a$ can be removed. See Lemma \ref{lem:relax-assumption}.
\end{remark}

\begin{remark}[Tabular MDP]
If $\mathcal{S}$ and $\mathcal{A}$ are finite sets equipped with discrete metrics and $\Pi$ is the set of all selectors of $C$, then these assumptions are immediately met. This follows since the discrete metric renders all sets are open, closed, and compact, and all functions defined on such sets are continuous (and bounded if real-valued).
\end{remark}

\begin{example}\label{ex:running-example-well-posed}
Consider the MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ from Example \ref{ex:running-example}. While $C$ is upper hemicontinuous, it is not compact-valued. While $f$ and $r$ are continuous, $r$ is unbounded. Consider the constant correspondence $C' = [-1, 1]$, and the reward $r': \mathrm{Graph}(C') \to \bb{R}$:
\begin{equation}
    r'(s) = e^{-s^2} - (\tanh{(a)})^2,
\end{equation}
for all state-action pairs $(s, a) \in \mathrm{Graph}(C')$. Note that $\mathrm{Graph}(C') = \mathcal{S} \times [-1, 1]$, so $r'$ is bounded. Since $f$ is continuous, its restriction to $\mathrm{Graph}(C')$ is as well. With $\Pi$ the set of all measurable selectors of $C'$, note that for every pair $(s, a) \in \mathrm{Graph}(C')$, the constant policy $\pi_{s, a} = a$ is a measurable selector of $C'$, so $\pi_{s, a} \in \Pi$. Thus, the MPOP defined by the MDP $(\mathcal{S}, \mathcal{A}, C', \left.f\right|_{\mathrm{Graph}(C')}, r', \gamma)$ and policy class $\Pi$ satisfies Assumption \ref{as:well-posed}.
\end{example}

We now demonstrate well-posedness under Assumption \ref{as:well-posed}.
\begin{theorem}\label{thm:well-posed}
Consider an MPOP characterized by an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ that satisfies Assumption \ref{as:well-posed}. There is an optimal policy $\pi^* \in \Pi$ satisfying $V_{\pi^*} = V^*$, and $V^*$ is the limit of value iteration when the initial guess is bounded and upper semicontinuous.
\end{theorem}

\begin{proof}
If $\mathcal{S}$ and $\mathcal{A}$ are Polish spaces, we can show that the MDP in question satisfies the three conditions of Assumptions (W*) of \cite{feinberg2012average}, in which case, there is an optimal policy and $V^*$ is the limit of value iteration when the initial condition is the zero function (a conclusion of \cite[Theorem 4.1]{feinberg2012average}). The first condition follows since $r$ is upper semicontinuous and bounded above (as it is bounded both above and below), which corresponds to a lower semicontinuous cost function that is bounded below. The second condition is satisfied by the upper hemicontinuity of $C$ (using the sequential definition in \cite[Theorem 17.20]{aliprantis2006infinite}) since $r$ is also bounded below. The third condition is satisfied since the MDP is deterministic (for which transition probabilities are represented by Dirac measures). In this case, we require:
\begin{equation}
    \lim_{n \to \infty}{ \varphi(f(s_n, a_n)) } = \varphi(f(s, a)),
\end{equation}
for any bounded continuous function $\varphi: \mathcal{S} \to \bb{R}$ and any convergent sequence $\{ (s_n, a_n) \in \mathrm{Graph}(C): n \in \bb{N} \}$ converging to $(s, a) \in \mathrm{Graph}(C)$; equality follows since $\varphi \circ f$ is continuous (as the composition of continuous functions).

If $\mathcal{S}$ and $\mathcal{A}$ are not both Polish spaces, we prove well-posedness in the appendix, in which case, $V^*$ is the limit of value iteration when the initial guess is any bounded and upper semicontinuous function.
\end{proof}

We conclude with a lemma mentioned in Remark \ref{rem:relax-assumption}, allowing Assumption \ref{as:well-posed} to be relaxed slightly.

\begin{lemma}\label{lem:relax-assumption}
Suppose an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and policy class $\Pi$ satisfy Assumption \ref{as:well-posed}, suppose the projection of $\mathrm{Graph}(C)$ onto $\mathcal{A}$ is compact, and let $\rho_{\mathcal{A}}: \mathcal{A} \times \mathcal{A} \to \bb{R}_+$ denote the metric on $\mathcal{A}$. For any state-action pair $(s, a) \in \mathrm{Graph}(C)$, the function $\varphi_{s, a}: \mathrm{Graph}(C) \to \bb{R}$ defined as:
\begin{equation}
    \varphi_{s, a}(s', a') = -\rho_{\mathcal{A}}(a', a),
\end{equation}
for all state-action pairs $(s', a') \in \mathrm{Graph}(C)$ admits a maximizing policy $\pi_{s, a} \in \Pi$ for which $\pi_{s, a}(s) = a$.
\end{lemma}

\begin{proof}
Since $\rho_{\mathcal{A}}$ is continuous, so is $\varphi_{s, a}$. Since the projection of $\mathrm{Graph}(C)$ onto $\mathcal{A}$ is compact, $\varphi_{s, a}$ is bounded. If $\mathcal{S}$ and $\mathcal{A}$ are Polish spaces, then the proof of \cite[Lemma 3.4]{feinberg2012average} uses the Arsenin-Kunugui theorem to show the existence of a policy $\pi_{s, a} \in \Pi$ satisfying:
\begin{align}\label{eq:distance-minimizer}
    \varphi_{s, a}(s', \pi_{s, a}(s')) &= \max_{a' \in C(s')} \varphi_{s, a}(s', a')\nonumber\\
    &= -\min_{a' \in C(s')} \rho_{\mathcal{A}}(a', a),
\end{align}
for all states $s' \in \mathcal{S}$. In particular, $\pi_{s, a}(s) = a$ as $\varphi_{s, a}(s, \pi_{s, a}(s)) = -\min_{a' \in C(s)} \rho_{\mathcal{A}}(a', a) = 0$. This requires Assumptions (W*) of \cite{feinberg2012average} to be met, which is verified in the proof of Theorem \ref{thm:well-posed}.

We arrive at the same conclusion when $\mathcal{S}$ and $\mathcal{A}$ are not both Polish spaces in the appendix.
\end{proof}

\subsection{Compactly Restrictable MPOPs}
\label{sec:cr-mmdps}
While Assumption \ref{as:well-posed} allows us to determine when MPOPs are well-posed and can be solved using value iteration, it is not met for many systems of interest. We now outline an alternative assumption that not only mitigates theoretical shortcomings, but also better fits the problem settings for physical systems. Our main result here is Theorem \ref{thm:invariance}, showing how an MPOP satisfying these new assumptions can generate an MPOP satisfying Assumption \ref{as:well-posed} through a systematic transformation. We call the original MPOP a CR-MPOP.

\subsubsection{Motivation}
Consider an MPOP defined by an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ that satisfies Assumption \ref{as:well-posed}. The proof of Theorem \ref{thm:well-posed} requires a bounded reward function $r$ and a compact-valued action-admissibility correspondence $C$. For many examples, these requirements are too strict, as shown in Example \ref{ex:running-example-well-posed}. If $C$ can be modified such that its graph is compact and if $r$ is continuous, then $r$ will be bounded over the graph of the modified correspondence. This guides our systematic construction of a well-posed policy optimization problem from an ill-posed one.

If the graph of the modified correspondence is compact, the projection of the graph onto the state space must also be compact. Therefore, if the state space is not already compact, then the modified correspondence must be defined on a compact strict subset $\mathcal{S}_0 \subset \mathcal{S}$; that is, the modified correspondence must have the form $C_0: \mathcal{S}_0 \to \mathcal{P}(\mathcal{A})$, where $C_0(s) \subseteq C(s)$ is nonempty and compact for all states $s \in \mathcal{S}_0$. Moreover, while restricting admissible actions to compact sets is reasonable for physical systems (as forces, voltages, currents, etc. are bounded by physical constraints), these restrictions must be chosen carefully to enforce the nonemptiness condition. For the restriction of $f$ to $\mathrm{Graph}(C_0)$ to be a well-defined transition function, we require $f(s, a) \in \mathcal{S}_0$ for all pairs $(s, a) \in \mathrm{Graph}(C_0)$. Finally, we must ensure that $C_0$ admits measurable selectors.

Restricting our attention to a compact subset $\mathcal{S}_0$ is also reasonable for many physical systems, as dynamics models are often well-understood only in a bounded set around an operating condition.

\subsubsection{Compact Restriction}
For this construction, we use policies that render compact subsets of the state space forward-invariant. A policy $\pi \in \Pi$ renders a subset $\mathcal{S}_0 \subseteq \mathcal{S}$ forward-invariant if $f_\pi(\mathcal{S}_0) \subseteq \mathcal{S}_0$. By induction, we have $f_\pi^t(\mathcal{S}_0) \subseteq \mathcal{S}_0$ for any $t \in \bb{Z}_+$.

\begin{definition}[Compactly Restrictable Metric Policy Optimization Problem]
An MPOP characterized by an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ is a \textit{compactly restrictable metric policy optimization problem} (CR-MPOP) if there is a nonempty compact subset $\mathcal{S}_0 \subseteq \mathcal{S}$ and a continuous policy $\pi_0 \in \Pi$ rendering $\mathcal{S}_0$ forward-invariant.
\end{definition}

\begin{assumption}\label{as:general}
The action-admissibility correspondence $C$ has closed values and $\mathrm{Graph}(C)$ is closed, the transition function $f$ and reward function $r$ are continuous, and the policy class $\Pi$ is the set of all measurable selectors of $C$.
\end{assumption}

To show how an CR-MPOP satisfying Assumption \ref{as:general} can lead to an MPOP satisfying Assumption \ref{as:well-posed}, we need the following lemma.

\begin{lemma}\label{lem:continuous-on-graph-implies-continuous-sections}
Consider an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ satisfying Assumption \ref{as:general}. For any state $s \in \mathcal{S}$, the $s$-section $f_s: C(s) \to \mathcal{S}$ is continuous. Additionally, for any state $s \in \mathcal{S}$ and any closed set $G \subseteq \mathcal{S}$, the preimage $(f_s)^{-1}(G)$ is a closed subset of $\mathcal{A}$.
\end{lemma}

\begin{proof}
Let $\mathrm{id}: \mathcal{S} \times \mathcal{A} \to \mathcal{S} \times \mathcal{A}$ denote the identity function on $\mathcal{S} \times \mathcal{A}$. For any state $s \in \mathcal{S}$, the $s$-section $\mathrm{id}_s: \mathcal{A} \to \mathcal{S} \times \mathcal{A}$ is continuous, meaning the restriction $\left.\mathrm{id}_s\right|_{C(s)}: C(s) \to \mathcal{S} \times \mathcal{A}$ is continuous. We can write $f_s = f \circ (\left.\mathrm{id}_s\right|_{C(s)})$, implying $f_s$ is continuous. For any closed set $G \subseteq \mathcal{S}$, there is a corresponding closed set $F_{s, G} \subseteq \mathcal{A}$ satisfying $(f_s)^{-1}(G) = C(s) \cap F_{s, G}$. Since $C(s)$ is also a closed subset of $\mathcal{A}$, we conclude that $(f_s)^{-1}(G)$ a closed subset of $\mathcal{A}$.
\end{proof}

We now present our guarantee of the existence of a well-posed value iteration settings under Assumption \ref{as:general}.

\begin{theorem}
\label{thm:invariance}
Consider a CR-MPOP characterized by an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ that satisfies Assumption \ref{as:general}. There exists an MPOP characterized by an MDP $(\mathcal{S}_0, \mathcal{A}_0, C_0, f_0, r_0, \gamma)$ and a policy class $\Pi_0$ satisfying Assumption \ref{as:well-posed}, with $\mathcal{S}_0 \subseteq \mathcal{S}$ rendered forward-invariant by a continuous policy in $\Pi$, $\mathcal{A}_0 \subseteq \mathcal{A}$, $C_0: \mathcal{S}_0 \to \mathcal{P}(\mathcal{A}_0)$ satisfying $C_0(s) \subseteq C(s)$ for all states $s \in \mathcal{S}_0$, $f_0 = \left.f\right|_{\mathrm{Graph}(C_0)}$, $r_0 = \left.r\right|_{\mathrm{Graph}(C_0)}$, and:
\begin{equation}
    \Pi_0 = \{ \left.\pi\right|_{\mathcal{S}_0} : \pi \in \Pi, \pi(\mathcal{S}_0) \subseteq \mathcal{A}_0, ~\mathrm{and}~ f_\pi(\mathcal{S}_0) \subseteq \mathcal{S}_0 \}.
\end{equation}
\end{theorem}

\begin{proof}
For some $n \in \bb{N}$, consider continuous policies $\pi_1, \dots, \pi_n \in \Pi$ satisfying $f_{\pi_i}(\mathcal{S}_0) \subseteq \mathcal{S}_0$ for all $i \in \{ 1, \dots, n \}$. Such policies are guaranteed to exist by the existence of the continuous policy $\pi_0 \in \Pi$. Consider the union of the images of $\mathcal{S}_0$ under each of these policies, defining the compact set $\mathcal{A}_0 = \bigcup_{i = 1}^n \pi_i(\mathcal{S}_0)$.
Since $\mathcal{S}$ and $\mathcal{A}$ are separable metric spaces, so are $\mathcal{S}_0$ and $\mathcal{A}_0$ \cite[Corollary 3.5]{aliprantis2006infinite}.

Define the correspondence $C_0: \mathcal{S}_0 \to \mathcal{P}(\mathcal{A}_0)$ as:
\begin{equation}
    C_0(s) = \{ a \in C(s) \cap \mathcal{A}_0: f(s, a) \in \mathcal{S}_0 \} = (f_s)^{-1}(\mathcal{S}_0) \cap \mathcal{A}_0,
\end{equation}
for all states $s \in \mathcal{S}_0$. 
Since $\pi_1(s), \dots, \pi_n(s) \in C_0(s)$, $C_0(s) \neq \emptyset$  for all $s \in \mathcal{S}_0$.
By Lemma \ref{lem:continuous-on-graph-implies-continuous-sections} and since $\mathcal{S}_0$ is closed (it is compact in a metric space), the preimage $(f_s)^{-1}(\mathcal{S}_0)$ is a closed subset of $\mathcal{A}$. $C_0(s)$ is thus compact, as it is a closed subset of $\mathcal{A}_0$. For any closed set $G \subseteq \mathcal{A}_0$, the lower preimage of $G$ under $C_0$ satisfies:
\begin{align}
    &~(C_0)^\ell(G) = \{ s \in \mathcal{S}_0: f(s, a) \in \mathcal{S}_0 ~\mathrm{for~some}~ a \in C(s) \cap G \}\nonumber\\
    &~\qquad= p_\mathcal{S}( f^{-1}(\mathcal{S}_0) \cap (\mathcal{S}_0 \times G) ),
\end{align}
where $p_{\mathcal{S}}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ denotes the canonical projection onto $\mathcal{S}$. Since $\mathcal{S}_0$ is closed and $f$ is continuous, there is a closed set $F \subseteq \mathcal{S} \times \mathcal{A}$ satisfying $f^{-1}(\mathcal{S}_0) = \mathrm{Graph}(C) \cap F$. Since $\mathrm{Graph}(C)$ is also closed, $f^{-1}(\mathcal{S}_0)$ is a closed subset of $\mathcal{S} \times \mathcal{A}$. Since $G$ is a closed subset of the compact set $\mathcal{A}_0$, $G$ is compact, and since $\mathcal{S}_0$ is compact, the product $\mathcal{S}_0 \times G$ is compact. As a closed subset of a compact set, $f^{-1}(\mathcal{S}_0) \cap (\mathcal{S}_0 \times G)$ is compact. Since the projection operator $p_{\mathcal{S}}$ is continuous, $(C_0)^\ell(G)$ is compact. Therefore, $(C_0)^\ell(G)$ is a closed subset of $\mathcal{S}_0$, and since $G$ was arbitrary, $C_0$ is upper hemicontinuous.

As restrictions of continuous functions, $f_0$ and $r_0$ are continuous, implying $r_0$ is upper semicontinuous. Note that:
\begin{align}
    \mathrm{Graph}(C_0) &= \{ (s, a) \in \mathcal{S}_0 \times \mathcal{A}_0: f(s, a) \in \mathcal{S}_0 \}\nonumber\\
    &= f^{-1}(\mathcal{S}_0) \cap (\mathcal{S}_0 \times \mathcal{A}_0).
\end{align}
Since $\mathcal{A}_0$ is trivially a closed subset of $\mathcal{A}_0$, we have already shown that $f^{-1}(\mathcal{S}_0) \cap (\mathcal{S}_0 \times \mathcal{A}_0)$ is compact, meaning $\mathrm{Graph}(C_0)$ is compact. Therefore, since $r_0$ is continuous, it is bounded. 

The policy class $\Pi_0$ satisfies:
\begin{align}
    &\Pi_0 = \{ \left.\pi\right\vert_{\mathcal{S}_0}: \pi \in \Pi, \pi(s) \in C_0(s) ~\mathrm{for~all~states}~ s \in \mathcal{S}_0 \}\nonumber\\
    &= \{ \pi \in \mathcal{L}^0(\mathcal{S}_0; \mathcal{A}_0): \pi(s) \in C_0(s) ~\mathrm{for~all~states}~ s \in \mathcal{S}_0 \}.
\end{align}
To verify the last equality, first consider a policy $\pi \in \Pi$ satisfying $\pi(s) \in C_0(s)$ for all states $s \in \mathcal{S}_0$. The restriction $\left.\pi\right|_{\mathcal{S}_0}$ is a measurable function from $\mathcal{S}_0$ to $\mathcal{A}_0$ selecting from $C_0$. Conversely, consider a measurable function $\pi: \mathcal{S}_0 \to \mathcal{A}_0$ selecting from $C_0$. Pick any policy $\pi_e \in \Pi$ and define the extension $\bar{\pi}: \mathcal{S} \to \mathcal{A}$
as $\pi$ on $\mathcal{S}_0$ and $\pi_e$ on $\mathcal{S} \setminus \mathcal{S}_0$. For any measurable set $B \subseteq \mathcal{A}$, we have:
\begin{align}
    &\bar{\pi}^{-1}(B) = \{ s \in \mathcal{S}_0: \pi(s) \in \mathcal{A}_0 \cap B \} \cup \{ s \in \mathcal{S} \setminus \mathcal{S}_0: \pi_e(s) \in B \}\nonumber\\
    &~= \pi^{-1}(\mathcal{A}_0 \cap B) \cup (\pi_e^{-1}(B) \setminus \mathcal{S}_0),
\end{align}
which is a measurable set; therefore, $\bar{\pi}$ is a measurable function. Note that $\bar{\pi}(s) = \pi(s) \in C_0(s) \subseteq C(s)$ for all states $s \in \mathcal{S}_0$ and $\bar{\pi}(s) = \pi_e(s) \in C(s)$ for all states $s \notin \mathcal{S}_0$. Since $\bar{\pi}$ is a measurable selector of $C$, we have $\bar{\pi} \in \Pi$, and inclusion follows since $\left.\bar{\pi}\right|_{\mathcal{S}_0} = \pi$.

Finally, since $\mathrm{Graph}(C_0)$ is compact, its projection onto $\mathcal{A}_0$ is compact. Therefore, by Lemma \ref{lem:relax-assumption}, for every state-action pair $(s, a) \in \mathrm{Graph}(C_0)$, there is a policy $\pi_{s, a} \in \Pi_0$ satisfying $\pi_{s, a}(s) = a$.
\end{proof}

\begin{remark}
An MPOP characterized by an MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ and a policy class $\Pi$ that satisfies Assumption \ref{as:well-posed} is trivially a CR-MPOP satisfying Assumption \ref{as:general} if the following sufficient conditions are met:
\begin{inparaenum}[1)]
    \item The state space $\mathcal{S}$ is compact,
    \item The reward function $r$ is continuous,
    \item The policy class $\Pi$ contains a continuous policy.
\end{inparaenum}
\end{remark}

\begin{example}
Consider the MDP $(\mathcal{S}, \mathcal{A}, C, f, r, \gamma)$ defined in Example \ref{ex:running-example} with a policy class $\Pi$ comprised of measurable selectors of $C$, and let $\mathcal{S}_0 = [-1, 1]$. Note that $| f(0, a) | = | \tanh{(a)} | < 1$ for all actions $a \in \bb{R}$. Consider any state $s \in [-1, 0)$. For any nonnegative action $a \in \bb{R}_+$, we have $-1 \leq s \leq f(s, a) = s + \tanh{(a)} < s + 1 < 1$, and $f(s, -a) = s + \tanh{(-a)} \in [-1, s)$ if and only if $-a \in [\tanh^{-1}{(-1 - s)}, 0)$ since $\tanh^{-1}$ is monotonically increasing. By a similar argument for states in $(0, 1]$, we determine that, for all $s \in \mathcal{S}_0$ and $a \in \mathcal{A}$, we have $f(s, a) \in \mathcal{S}_0$ if and only if:
\begin{equation}\label{eq:fi-condition}
    a \in \begin{cases} [\tanh^{-1}{(-1 - s)}, \infty) & -1 \leq s < 0,\\
    \mathcal{A} & s = 0,\\
    (-\infty, \tanh^{-1}{(1 - s)}] & 0 < s \leq 1.
    \end{cases}
\end{equation}

\begin{figure}
    \centering
    \includegraphics[width=0.35\textwidth]{compact_construction.pdf}
    \caption{(Left) The preimage $f^{-1}(\mathcal{S}_0)$ is shown in blue, which contains all state-action pairs mapped into $\mathcal{S}_0 = [-1, 1]$ by $f$. For any state in $\mathcal{S}_0$, the set of actions mapping that state into $\mathcal{S}_0$ is unbounded, therefore not compact. (Right) The graph of $\pi_0$ is shown in orange and the graph of $C_0$ is shown in blue. The graph of $C_0$ is compact, and for every state $s \in \mathcal{S}_0$, the set $C_0(s)$ is compact.}
    \label{fig:compact-construction}
\end{figure}

A correspondence coinciding with the requirements on actions in \eqref{eq:fi-condition} would not have compact values; this is illustrated by the preimage $f^{-1}(\mathcal{S}_0)$ in Figure \ref{fig:compact-construction}. Therefore, consider the continuous policy $\pi_0 \in \Pi$ defined as $\pi_0(s) = -s$ for all states $s \in \mathcal{S}$. For any state $s \in [-1, 0)$, we have $\pi_0(s) = -s > 0 \geq \tanh^{-1}(-1 - s)$.
Similarly, for any state $s \in (0, 1]$, we have $\pi_0(s) < \tanh^{-1}(1 - s)$. Therefore, $f_{\pi_0}(\mathcal{S}_0) \subseteq \mathcal{S}_0$. The image of $\mathcal{S}_0$ under $\pi_0$ is $[-1, 1]$. Therefore, we can define $\mathcal{A}_0 = [-1, 1]$ and $C_0: \mathcal{S}_0 \to \mathcal{P}(\mathcal{A}_0)$ as, for all states $s \in \mathcal{S}_0$:
\begin{equation}
    C_0(s) = \begin{cases}
        [\tanh^{-1}(-1 - s), 1] & s \leq -1 - \tanh{(-1)},\\
        [-1, \tanh^{-1}(1 - s)] & s \geq 1 - \tanh{(1)},\\
        [-1, 1] & \mathrm{otherwise}.
    \end{cases}
\end{equation}
\end{example}

\subsection{Policy Iteration}

We briefly mention the difficulties of policy iteration for MPOPs, as noted in \cite{yu2015mixed}. For a well-posed MPOP, policy iteration generates a monotonically nondecreasing sequence of policies. Given an initial policy $\pi_0 \in \Pi$, we seek a sequence of policies $\{ \pi_n \in \Pi: n \in \bb{N} \}$ satisfying, for all states $s \in \mathcal{S}$ and $n \in \bb{Z}_+$:
\begin{align}\label{eq:policy-improvement-step}
    &r(s, \pi_{n + 1}(s)) + \gamma V_{\pi_n}(f(s, \pi_{n + 1}(s)))\nonumber\\
    &~\qquad\qquad= \sup_{a \in C(s)}{\{ r(s, a) + \gamma V_{\pi_n}(f(s, a)) \}}.
\end{align}
If $f$, $r$, and $\pi_0$ are continuous, then so are $f_{\pi_0}$ and $r_{\pi_0}$ which we can show renders $V_{\pi_0}$ continuous. This implies $r + \gamma V_{\pi_0} \circ f$ is continuous. If $r$ is bounded, then so are $V_{\pi_0}$ and $r + \gamma V_{\pi_0} \circ f$. Since $r + \gamma V_{\pi_0} \circ f$ is upper semicontinuous and bounded, we can show (using \cite{feinberg2012average} or the appendix) that for any state $s \in \mathcal{S}$, the optimization problem:
\begin{equation}
    \sup_{a \in C(s)}{\{ r(s, a) + \gamma V_{\pi_0}(f(s, a)) \}},
\end{equation}
is solved by the next policy $\pi_1$, a measurable selector of $C$. However, we cannot conclude that $r + \gamma V_{\pi_1} \circ f$ is upper semicontinuous and bounded, so the same argument cannot be applied iteratively.

Demonstrating that policy iteration can be applied requires special knowledge that the policy improvement step in \eqref{eq:policy-improvement-step} continues to produce policies with value functions that permit maximization and appropriate selection (for the chosen policy class). Such knowledge is available in linear-quadratic regulator (LQR) problems with linear policy classes (see \cite{tu2019sample} for the discounted case).

\subsection{Computation}
There are 3 central difficulties in computationally implementing value iteration for a well-posed MPOP:
\begin{inparaenum}[1)]
    \item Representing iterates during value iteration,
    \item Determining admissible actions, and
    \item Approximating the update step \eqref{eq:value-iteration-step}.
\end{inparaenum}
Typically, iterates are represented using function approximators such as neural networks \cite{sutton2018reinforcement} or Gaussian process regression models \cite{kuss2003gaussian}. Training such approximators involves sampling sufficiently many states from the state space region of interest. For any given state, the maximization in the update step \eqref{eq:value-iteration-step} is generally a nonconvex optimization, with possible approximations including (projected) gradient ascent from several initial action seeds or maximization over a finite but sufficiently dense sampling of admissible actions. If the function approximation and optimization approximation errors can be controlled, approximate convergence of value iteration can be guaranteed \cite[Proposition 2.3.2]{bertsekas2022abstract}.

Any approximate optimization approach requires checking admissibility of actions, which, in the case of a CR-MPOP, requires checking whether or not a state-action pair is mapped to the correct compact set under the transition map. This necessitates efficient set membership checking, and for nonconvex compact sets, proximity-based membership approximations may be needed.

Finally, in Section \ref{sec:sampled-data-control}, we will use the closure of a reachable set (under a specific policy) to restrict problems with feedback linearizable control affine systems to well-posed MPOPs. To sample a state from the reachable set, we can sample a state from the appropriate set of initial conditions and follow the policy for a number of steps sampled from a geometric distribution (with success probability $1 - \gamma$). The resulting distribution is the discounted state distribution under the policy \cite{silver2014deterministic}, which is supported on the entirety of the reachable set (thus, on a dense subset of its closure).

There are many possible combinations of computational approaches that may be suitable for well-posed MPOPs; we deem the precise best choice outside the scope of this theoretical framing.

\section{Control Affine Systems}
\label{sec:control}
We now show how to apply the results from Section \ref{sec:mdp} to a general class of control systems called control affine systems \cite[\S 13.2.3]{lavalle2006planning}. For state and action space dimensions $d, m \in \bb{N}$, respectively, consider a set $\mathcal{D} \subseteq \bb{R}^d$ and vector fields $f_0, g_1, \dots, g_m: \mathcal{D} \to \bb{R}^d$. Define the matrix-valued function $G: \mathcal{D} \to \bb{R}^{d \times m}$ with columns $g_1, \dots, g_m$. Define $F: \mathcal{D} \times \bb{R}^m \to \bb{R}^d$ as:
\begin{equation}
    F(x, u) = f_0(x) + G(x)u,
\end{equation}
for all $x \in \mathcal{D}$ and $u = (u_1, \dots, u_m) \in \bb{R}^m$. An initial value problem with constant control input $u \in \bb{R}^m$ is characterized by an initial condition $x \in \mathcal{D}$ and an open time interval $I \subseteq \bb{R}$ with $0 \in I$; a corresponding solution is a differentiable function $\phi: I \to \mathcal{D}$ satisfying $\phi(0) = x$ and $\dot{\phi}(t) = \frac{\mathrm{d}}{\mathrm{d}t}\phi(t) = F(\phi(t), u) = f_0(\phi(t)) + G(\phi(t))u$
for all times $t \in I$.

\subsection{Time Sampling}
In contrast to continuous-time control design, we consider sampled-data control design (see \cite{nevsic1999sufficient, taylor2021sampled, taylor2022safety}), in which initial value problems with constant control characterize the evolution of a system over fixed sample intervals, resulting in control input trajectories that are piecewise constant in time. Such assumptions are realistic for physical systems interacting with digital controllers, which measure states and compute control inputs at nearly fixed frequencies. This setting requires the time intervals over which solutions are defined to be sufficiently long, uniformly for all initial conditions and control inputs under consideration. 

Specifically, fix a sample period $h \in \bb{R}_{++}$, and define the subset $\mathcal{S}_h \subseteq \mathcal{D}$ and correspondence $C_h: \mathcal{S}_h \to \mathcal{P}(\bb{R}^m)$ such that the following properties are satisfied:
\begin{itemize}
    \item For every initial condition $x \in \mathcal{S}_h$, the corresponding set of control inputs $C_h(x) \subseteq \bb{R}^m$ is nonempty,
    \item For every  $(x, u) \in \mathrm{Graph}(C_h)$, there is a unique solution to the initial value problem characterized by initial condition $x$, control input $u$, and open time interval $I \subseteq \bb{R}$  with $[0, h] \subset I$, with the solution denoted $\phi_{x, u}: I \to \mathcal{D}$,
    \item $\mathcal{S}_h$ and $C_h$ are maximal (cannot be contained in a superset and containing correspondence)
\end{itemize}
Define $f_h: \mathrm{Graph}(C_h) \to \mathcal{D}$ for all pairs $(x, u) \in \mathrm{Graph}(C_h)$ as:
\begin{equation}
    f_h(x, u) = \phi_{x, u}(h).
\end{equation}

\begin{example}[Linear System]\label{ex:linear-system}
Let $\mathcal{D} = \bb{R}^d$. For matrices $A \in \bb{R}^{d \times d}$ and $B \in \bb{R}^{d \times m}$, suppose
$f_0(x) = Ax$ and $G(x) = B$ for all initial conditions $x \in \bb{R}^d$. Then $\mathcal{S}_h = \bb{R}^d$ and $C_h(x) = \bb{R}^m$ for all $x \in \bb{R}^d$. Moreover, $f_h$ can be expressed explicitly as a linear function in terms of the matrix exponential of $A$.
\end{example}

\begin{example}[Continuously Differentiable and Lipschitz Continuous Vector Fields]\label{ex:diff-and-lip}
Suppose $\mathcal{D} = \bb{R}^d$ and $f_0, g_1, \dots, g_m$ are continuously differentiable and (globally) Lipschitz continuous. For a control input $u \in \bb{R}^m$, since the $u$-section $F^u: \bb{R}^d \to \bb{R}^d$ is a linear combination of continuously differentiable and Lipschitz continuous vector fields, it is continuously differentiable and Lipschitz continuous. Therefore, for any initial condition $x \in \bb{R}^d$ the initial value problem characterized by initial condition $x$, control input $u$, and time interval $\bb{R}$ has a unique solution \cite[Theorem 3.1.3]{perko2013differential}. Therefore, $\mathcal{S}_h = \bb{R}^d$ and $ C_h(x) = \bb{R}^m$ for all initial conditions $x \in \bb{R}^d$. In general, $f_h$ does not have an closed-form representation.
\end{example}

\begin{assumption}\label{as:control-affine-well-posed}
The vector fields $f_0, g_1, \dots, g_m$ are locally Lipschitz continuous. Moreover, there exist subsets $\mathcal{S} \subseteq \mathcal{S}_h$ and $\mathcal{A} \subseteq \bb{R}^m$ and a correspondence $C: \mathcal{S} \to \mathcal{P}(\mathcal{A})$ such that:
\begin{itemize}
    \item For every initial condition $x \in \mathcal{S}$, the set of control inputs $C(x) \subseteq \mathcal{A}$ is nonempty with $C(x) \subseteq C_h(x)$,
    \item $\mathrm{Graph}(C)$ is a closed subset of $\mathcal{S} \times \mathcal{A}$,
    \item $f_h(\mathrm{Graph}(C)) \subseteq \mathcal{S}$.
\end{itemize}
\end{assumption}

Both Examples \ref{ex:linear-system} and \ref{ex:diff-and-lip} satisfy Assumption \ref{as:control-affine-well-posed} with $\mathcal{S} = \mathcal{S}_h$, $\mathcal{A} = \bb{R}^m$, and $C(x) = \mathcal{A}$ for all $x \in \mathcal{S}$. The assumption of local Lipschitz continuity in Assumption \ref{as:control-affine-well-posed} ensures that the restriction of $f_h$ to $\mathrm{Graph}(C)$ is continuous; this follows from \cite[Theorem 3.5]{khalil2002nonlinear} by treating control inputs as parameters.

\subsection{Robotic Systems}
Many robotic systems can be modeled as control affine systems, and we will demonstrate how to show that such systems can satisfy Assumption \ref{as:general}, establishing well-posedness of value iteration for a large class of practically relevant MPOPs. Examples of such systems include manipulators, automobiles, aircraft, and spacecraft \cite{murray1994mathematical, olfati2001nonlinear}.

\subsubsection{Dynamics}
An unconstrained robotic system with $n \in \bb{N}$ degrees of freedom is characterized by an $n$-dimensional $\mathcal{C}^2$ manifold $\mathcal{Q}$ called the configuration manifold. In this work, we consider open subsets $\mathcal{Q} \subseteq \bb{R}^n$ for which we can identify the tangent bundle $T\mathcal{Q}$ with $\mathcal{Q} \times \bb{R}^n$. The inertia matrix function $D: \mathcal{Q} \to \bb{S}^n_{++}$ characterizes the kinetic energy function $T: \mathcal{Q} \times \bb{R}^n \to \bb{R}_{++}$, defined as $T(q, \dot{q}) = \frac{1}{2}\dot{q}^\top D(q)\dot{q}$ for all configurations $q \in \mathcal{Q}$ and velocities $\dot{q} \in \bb{R}^n$. The assumption that $D$ takes positive definite values ensures that no configurations exist that admit arbitrarily large velocities without affecting the kinetic energy of the system. We require $D$ to be differentiable, allowing us to express the matrix-valued function $C: \mathcal{Q} \times \bb{R}^n \to \bb{R}^{n \times n}$ of Coriolis and centrifugal terms\footnote{The use of $C$ for this term is standard, but we will not use it in the same context as an action-admissibility correspondence.} as:
\begin{equation}
    (C(q, \dot{q}))_{ij} = \sum_{k = 1}^n \left(\frac{\partial}{\partial q_j} ( D(q) )_{ik} - \frac{1}{2} \frac{\partial}{\partial q_i} ( D(q) )_{jk} \right) \dot{q}_k,
\end{equation}
for all configurations $q = (q_1, \dots, q_n) \in \mathcal{Q}$, velocities $\dot{q} = (\dot{q}_1, \dots, \dot{q}_n) \in \bb{R}^n$, and indices $i, j \in \{ 1, \dots, n \}$ \cite[Equation 4.21]{murray1994mathematical}. We also require the potential energy function $U: \mathcal{Q} \to \bb{R}^n$ to be differentiable, and we denote external nonconservative forces and torques applied to the system with the vector-valued function $F_{\mathrm{ext}}: \mathcal{Q} \times \bb{R}^n \to \bb{R}^n$. Finally, if the system is controlled with $m \in \bb{N}$ actuators, then $B: \mathcal{Q} \to \bb{R}^{n \times m}$ denotes the actuation matrix function, converting control inputs to forces and torques. With $d = 2n$ and $\mathcal{D} = \mathcal{Q} \times \bb{R}^n$, we define $F: \mathcal{D} \times \bb{R}^m \to \bb{R}^d$ as:
\begin{align}
    F(x, u) &= \begin{bmatrix} \dot{q} \\ D(q)^{-1}( F_{\mathrm{ext}}(q, \dot{q}) - C(q, \dot{q})\dot{q} - \nabla U(q) ) \end{bmatrix}\nonumber\\
    &~\qquad + \begin{bmatrix} 0_{n \times m} \\ D(q)^{-1}B(q) \end{bmatrix}u,
\end{align}
for all $x = (q, \dot{q}) \in \mathcal{Q} \times \bb{R}^n$ and $u \in \bb{R}^m$.

\subsubsection{Regularity}
We now establish regularity conditions that enable us to define a valid transition function that can be used in an MDP. We first establish sufficient conditions such that every initial condition and control input under consideration correspond to initial value problems with unique solutions. We then establish sufficient conditions for these solutions to exist for all nonnegative time.

\begin{assumption}\label{as:robotic-well-posed}
The configuration manifold satisfies $\mathcal{Q} = \bb{R}^n$. The functions $D$, $C$, $\nabla U$, $F_{\mathrm{ext}}$, and $B$ are each locally Lipschitz continuous. There is a strictly positive lower bound $\lambda_{\mathrm{\min}}$ such that $\lambda_{\min} I_n \preceq D(q)$ for all configurations $q \in \bb{R}^n$. There is some closed set $G \subseteq \bb{R}^m$ such that for any control input $u \in G$ there are corresponding constants $c_0 \in \bb{R}_+$ and $c_1 \in \bb{R}_{++}$ such that:
\begin{equation}
    \| F_{\mathrm{ext}}(q, \dot{q}) - \nabla U(q) + B(q)u \|_2 \leq c_0 + c_1 \| \dot{q} \|_2,
\end{equation}
for all configurations $q \in \bb{R}^n$ and velocities $\dot{q} \in \bb{R}^n$.
\end{assumption}

\begin{remark}
If the derivative of $D$ is locally Lipschitz continuous, then so is $D$ itself as it is continuously differentiable, and so is $C$ as it is bilinear in the derivative of $D$ and the velocities. If $D$ is twice continuously differentiable, then the derivative of $D$ is locally Lipschitz continuous, implying $D$ and $C$ are as well. If $U$ is twice continuously differentiable, then $\nabla U$ is locally Lipschitz continuous.
\end{remark}

\begin{remark}
If $B$ is bounded by some $M \in \bb{R}_+$ and there are constants $d_0 \in \bb{R}_+$ and $d_1 \in \bb{R}_{++}$ such that:
\begin{equation}
    \| F_{\mathrm{ext}}(q, \dot{q}) - \nabla U(q) \|_2 \leq d_0 + d_1 \| \dot{q} \|_2,
\end{equation}
for all configurations $q \in \bb{R}^n$ and velocities $\dot{q} \in \bb{R}^n$, then for any $u \in \bb{R}^m$, we have:
\begin{equation}
    \| F_{\mathrm{ext}}(q, \dot{q}) - \nabla U(q) + B(q)u \|_2 \leq d_0 + M \| u \|_2 + d_1 \| \dot{q} \|_2,
\end{equation}
for all configurations $q \in \bb{R}^n$ and velocities $\dot{q} \in \bb{R}^n$. Therefore, choose $c_0 = d_0 + M \| u \|_2$ and $c_1 = d_1$, as well as $G = \bb{R}^m$.
\end{remark}

The proof of \cite[Theorem 9.8b]{rudin1976principles} shows that matrix inversion is locally Lipschitz continuous. Fixing a control input $u \in G$, the $u$-section $F^u$ is locally Lipschitz continuous as it is comprised of sums and products of locally Lipschitz continuous functions. Fix an initial configuration $q_0 \in \bb{R}^n$ and velocity $\dot{q}_0 \in \bb{R}^n$. There is a corresponding maximal open interval $I_{\max} \subseteq \bb{R}$ with $0 \in I_{\max}$ such that the initial value problem characterized by initial condition $(q_0, \dot{q}_0)$, control input $u$, and time interval $I_{\max}$ has a unique solution $\phi: I_{\max} \to \bb{R}^{2n}$ \cite[Theorem 2.4.1]{perko2013differential} (the proof of this theorem applies to locally Lipschitz continuous vector fields, not just continuously differentiable vector fields). Let $\psi: I_{\max} \to \bb{R}^n$ satisfy $\phi(t) = ( \psi(t), \dot{\psi}(t) )$ for all times $t \in I_{\max}$, and note that $\psi(0) = q_0$ and $\dot{\psi}(0) = \dot{q}_0$. Let $I_{\max}^+ = \bb{R}_+ \cap I_{\max}$. The kinetic energy satisfies:
\begin{align}
    &T(\psi(t), \dot{\psi}(t)) - T(q_0, \dot{q}_0)\nonumber\\
    &~ = \int_0^t \dot{\psi}(s)^\top ( F(\psi(s), \dot{\psi}(s)) - \nabla U(\psi(s)) + B(\psi(s))u ) ~\mathrm{d}s\nonumber\\
    &~\leq \int_0^t \| \dot{\psi}(s) \|_2 (c_0 + c_1 \| \dot{\psi}(s) \|_2) ~\mathrm{d}s,
\end{align}
for all times $t \in I_{\max}^+$. Since $T(q, \dot{q}) \geq \frac{1}{2} \lambda_{\min} \| \dot{q} \|_2^2$ for all configurations $q \in \bb{R}^n$ and velocities $\dot{q} \in \bb{R}^n$, we have:
\begin{align}
    &\left(\sqrt{\lambda_{\min}} \| \dot{\psi}(t) \|_2\right)^2 / 2 \leq \left(\sqrt{ 2T(q_0, \dot{q}_0) }\right)^2 / 2\nonumber\\
    &~\qquad\qquad + \int_0^t \frac{c_0 + c_1 \| \dot{\psi}(s) \|_2}{\sqrt{\lambda_{\min}}} (\sqrt{\lambda_{\min}} \| \dot{\psi}(s) \|_2) ~\mathrm{d}s,
\end{align}
for all times $t \in I_{\max}^+$. For any upper bound $t_f \in I_{\max}^+$ with $t_f > 0$, the functions $\phi_1, \phi_2: [0, t_f] \to \bb{R}_+$ defined as:
\begin{align}
    \phi_1(t) &= \sqrt{\lambda_{\min}} \| \dot{\psi}(t) \|_2, & \phi_2(t) = \frac{ c_0 + c_1 \| \dot{\psi}(t) \|_2 }{\sqrt{\lambda_{\min}}},
\end{align}
for all $t \in [0, t_f]$ are continuous and bounded, implying $\phi_2$ is absolutely integrable. Therefore, from \cite[Lemma 17]{ballard2000dynamics} (originally \cite[Lemma A.5]{brezis1973ope}), we have:
\begin{equation}
    \sqrt{ \lambda_{\min} } \| \dot{\psi}(t) \|_2 \leq \sqrt{ 2T(q_0, \dot{q}_0) } + \int_0^t \frac{c_0 + c_1 \| \dot{\psi}(s) \|_2}{\sqrt{\lambda_{\min}}} ~\mathrm{d}s,
\end{equation}
for all times $t \in [0, t_f]$. By the Gronwall-Bellman inequality \cite[Lemma A.1]{khalil2002nonlinear}, we have:
\begin{equation}\label{eq:gb-bound}
    \| \dot{\psi}(t) \|_2 \leq ( \sqrt{ 2T(q_0, \dot{q}_0) / \lambda_{\min} } + c_0 t / \lambda_{\min} ) e^{\frac{c_1}{\lambda_{\min}}t},
\end{equation}
for all times $t \in [0, t_f]$. Since $t_f$ was arbitrary, the bound \eqref{eq:gb-bound} holds for all times $t \in I_{\max}^+$.

We now use \eqref{eq:gb-bound} to show that $I_{\max}^+ = \bb{R}_+$. For contradiction, assume there is some time $t_f \in \bb{R}_{++}$ with $t_f \notin I_{\max}^+$. By upper bounding the right-hand side of \eqref{eq:gb-bound} by its value at the time $t_f$, we conclude that $\dot{\psi}$ is bounded on $I_{\max}^+$. Also:
\begin{align}
    &\| \psi(t) - q_0 \|_2 \leq \int_0^t \| \dot{\psi}(s) \|_2 ~\mathrm{d}s\nonumber\\
    &\qquad~\leq t_f \left( \sqrt{ 2T(q_0, \dot{q}_0) / \lambda_{\min} } + c_0 t_f / \lambda_{\min} \right) e^{\frac{c_1}{\lambda_{\min}}t_f},
\end{align}
for all times $t \in I_{\max}^+$; that is, $\psi$ is bounded on $I_{\max}^+$. Therefore, $\phi$ is bounded on $I_{\max}^+$, contradicting \cite[Theorem 2.4.3]{perko2013differential} (again, the proof of this theorem applies to locally Lipschitz continuous vector fields). This implies $I_{\max}^+ = \bb{R}_+$, meaning for any sample period $h \in \bb{R}_{++}$, there is an open interval $I \subseteq \bb{R}$ with $[0, h] \subset I$.

Thus, any robotic system satisfying Assumption \ref{as:robotic-well-posed} also satisfies Assumption \ref{as:control-affine-well-posed}, no matter which sample period $h$ is chosen.

\subsection{Sampled-Data Control}
\label{sec:sampled-data-control}
We now determine sufficient conditions for the existence of a continuous policy rendering a compact subset of the state space forward-invariant. This construction uses the methods of \cite{taylor2021sampled, taylor2022safety}, based on the original work \cite{nevsic1999sufficient} in sampled-data control.

A control affine system is full-state feedback linearizable if $\mathcal{D}$ is open and there exist a function $\Phi: \mathcal{D} \to \bb{R}^d$ that is a diffeomorphism between $\mathcal{D}$ and an open subset of $\bb{R}^d$, a continuous controller $k_{\mathrm{fbl}}: \bb{R}^d \times \bb{R}^m \to \bb{R}^m$ that accepts an auxiliary input, and a controllable pair $(A, B) \in \bb{R}^{d \times d} \times \bb{R}^{d \times m}$ such that:
\begin{equation}
    \frac{\partial \Phi}{\partial x} F(x, k_{\mathrm{fbl}}(x, v)) = A \Phi(x) + Bv,
\end{equation}
for all  $x \in \mathcal{D}$ and auxiliary control inputs $v \in \bb{R}^m$.

\begin{remark}
Any robotic system satisfying Assumption \ref{as:robotic-well-posed} with $n = m$ and $B(q)$ invertible for all configurations $q \in \bb{R}^n$ is full-state feedback linearizable. The feedback linearizing controller $k_{\mathrm{fbl}}: \bb{R}^d \times \bb{R}^n \to \bb{R}^n$ is defined as:
\begin{equation}
    k_{\mathrm{fbl}}(x, v) = B(q)^{-1} ( C(q, \dot{q})\dot{q} + \nabla U(q) - F_{\mathrm{ext}}(q, \dot{q}) + D(q)v ),
\end{equation}
for all $x = (q, \dot{q}) \in \bb{R}^n \times \bb{R}^n$ and $v \in \bb{R}^n$, and:
\begin{equation}
    F(x, k_{\mathrm{fbl}}(x, v)) = \begin{bmatrix} 0_{n \times n} & I_n \\ 0_{n \times n} & 0_{n \times n} \end{bmatrix} \begin{bmatrix} q \\ \dot{q} \end{bmatrix} + \begin{bmatrix} 0_{n \times n} \\ I_n \end{bmatrix}v,
\end{equation}
for all $x = (q, \dot{q}) \in \bb{R}^n \times \bb{R}^n$ and  $v \in \bb{R}^n$.
\end{remark}

Consider a full-state feedback linearizable control affine system, and suppose $0_d \in \mathcal{D}$ and $\Phi(0_d) = 0_d$. Consider any gain matrix $K \in \bb{R}^{m \times d}$ making $A - BK$ Hurwitz stable with all eigenvalues having strictly negative real parts. Suppose the system satisfies Assumption \ref{as:control-affine-well-posed} and $0_d \in \mathrm{int}(\mathcal{S})$. Additionally, suppose $\pi_0: \mathcal{S} \to \bb{R}^m$ defined as:
\begin{equation}
    \pi_0(x) = k_{\mathrm{fbl}}(x, -K\Phi(x)),
\end{equation}
for all states $x \in \mathcal{S}$ satisfies $\pi_0(x) \in C(x)$ for all states $x \in \mathcal{S}$. Since $k_{\mathrm{fbl}}$ and $\Phi$ are continuous, $\pi_0$ is continuous.

By \cite[Lemmas 3, 4]{taylor2021sampled}, there is a continuous (class $\mathcal{KL}$) function $\beta: \bb{R}_+ \times \bb{R}_+ \to \bb{R}_+$ with monotonically increasing $s$-sections $\beta^s$ for all $s \in \bb{R}_+$ and monotonically nonincreasing $r$-sections $\beta_r$ for all $r \in \bb{R}_+$, as well as some $\overline{R} \in \bb{R}_{++}$ and a bounded open set $N \subset \bb{R}^d$ with $0_d \in N$ and:
\begin{equation}
    \{ x \in \bb{R}^d: \| x \|_2 \leq \sup_{x' \in N} \beta(\| x' \|_2, 0) + \overline{R} \} \subseteq \mathcal{S}.
\end{equation}
Additionally, for any $R \leq \overline{R}$, if the sample period is sufficiently small, we have:
\begin{align}
    \| (f_h)_{\pi_0}^t(x) \|_2 &\leq \beta( \| x \|_2, th ) + R \leq \sup_{x' \in N} \beta(\| x' \|_2, 0) + \overline{R},
\end{align}
for all $x \in N$ and $t \in \bb{Z}_+$. Let $\mathcal{R} \subseteq \mathcal{S}$ denote the set of all states reachable from $N$ when following $\pi_0$, defined as $\mathcal{R} = \{ (f_h)_{\pi_0}^t(x) \in \mathcal{S}: x \in N, t \in \bb{Z}_+ \}$,
and note that $(f_h)_{\pi_0}(\mathcal{R}) \subseteq \mathcal{R}$ and $\mathrm{cl}(\mathcal{R}) \subseteq \mathcal{S}$. Since $(f_h)_{\pi_0}(\mathrm{cl}(\mathcal{R})) \subseteq \mathrm{cl}(\mathcal{R}) \subseteq \mathcal{S}$, we can set $\mathcal{S}_0 = \mathrm{cl}(\mathcal{R})$. The set $\mathcal{S}_0 \subseteq \mathcal{S}$ is closed and bounded since:
\begin{equation}
    \sup_{x \in N} \beta(\|x\|_2, 0) + \overline{R} = \beta( \sup_{x \in N} \| x \|_2, 0 ) + \overline{R} < \infty.
\end{equation}
Therefore, $\mathcal{S}_0$ is a compact set rendered forward-invariant by the continuous policy $\pi_0$, implying the system satisfies Assumption \ref{as:general}.

Finally, we can construct a compact subset of the state space in a similar manner by appealing to \cite[Theorem 3]{taylor2022safety}, employing the notion of practical safety. In this case, if a compact $0$-superlevel set of a family of sampled-data control barrier functions \cite[Definition 7]{taylor2022safety} can be rendered forward-invariant when the transition map is approximated by an appropriate Euler/Runge-Kutta scheme, then an enlarged (but bounded) set can be rendered forward-invariant for the exact transition map when the sample period is sufficiently small. The enlargement can be controlled by decreasing the sample period.

\section{Future Work}
There are many directions for future work. First, our results can be viewed as ``existence'' results, and set the stage for studying computational aspects of policy optimization in metric MDPs. Second, our results are applicable to infinite-dimensional state and action spaces, such as those encountered in the control of systems governed by partial differential equations. This setting provides new challenges, such as how to choose appropriate compact subsets of the state space, as well as further computational considerations. Third, and it would be interesting to also consider stochastic dynamics. Finally, it is important to study the interaction between policy optimization and model learning when a model must be derived from data.

\bibliographystyle{IEEEtran} 
