\section{Introduction}
As more deep neural network (DNN) apps are getting added in 
autonomous vehicles (AVs), it is critically important to 
ensure the timeliness and quality of their execution. 
Major car-makers, like Tesla \cite{tesla2021autopilot} and 
Toyota \cite{toyota}, are expanding the scope of using DNNs 
for camera-based vision apps.
In AVs, a camera continuously monitors environmental 
conditions at a rate of 10 -- 40 
frames per second (FPS) \cite{yang2019re, lin2018architectural}.
Since the results of processing the current image frame 
should be produced before the next frame arrives,
all in-vehicle apps using the image frames captured by a camera must 
meet the same deadline dictated by the camera's frame rate.
However, the interference between concurrent vision apps may lead to miss 
some of their deadlines.
Fig.~\ref{fig:objective} shows a motivating scenario in which two 
apps process image frames from the same camera. 
In such a case, the two apps have the {\em same} 
deadline since they must neither produce outdated data nor 
interfere with the processing of the next image frames.
If the execution of a classification app precedes that of 
a real-time object detection (RTOD) app, the end-to-end 
(i.e., camera-to-detection output) latency of the object 
detection app exceeds its deadline.

\begin{figure}[t]
    \begin{center}
    \vspace*{-3mm}
    \includegraphics[width=1\linewidth]{figures/objective.pdf} 
    \end{center}
    \caption{Example use-case and the objective of {\tt DynaMIX}}
    \label{fig:objective}
    \vspace*{-3mm}
\end{figure}


\begingroup
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1}
\begin{table*}[tp]
    \small
    \caption{Comparison with prior work. ``Multi-tasking Support'' denotes running various apps not constrained by specific DNN models}
    \centering
    \vspace*{-2mm}
    \begin{center}
        \fontsize{9}{12}
        \begin{tabular}{c!{\vline width 1pt}ccc}
        \hlineB{2}
                                                                                               & Multi-Tasking Support & Real-Time Guarantee  & No Binding to Specific Processor \\ \hlineB{2}
        \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}A. Resource Management\\ \cite{han2016mcdnn, huynh2017deepmon, fang2018nestdnn} \end{tabular}} & \multirow{2}{*}{\checkmark}     & \multirow{2}{*}{}    & \multirow{2}{*}{\checkmark}           \\ 
                                                                                               &                       &                      &                             \\\hline
        \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}B. RTOD Execution\\ \cite{liu2020continuous, yang2019re, heo2020real, kang2022dnn} \end{tabular}}      & \multirow{2}{*}{}     & \multirow{2}{*}{\checkmark}    & \multirow{2}{*}{\checkmark}           \\
                                                                                               &                       &                      &                             \\\hline
        \begin{tabular}[c]{@{}c@{}}C. Hardware-based Scheduling\\ \cite{baek2020multi, jiang2020optimized, jeong2022band, han2022microsecond} \end{tabular}            & \multicolumn{1}{c}{\checkmark}  & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{}        \\ \hline
        DynaMIX                                                                                &                \checkmark       &       \checkmark               &             \checkmark                \\\hlineB{2}
        \end{tabular} 
        \vspace*{-4mm}
    \end{center}
    \label{tbl:relatedwork}
\end{table*}


\looseness=-1
However, guaranteeing the deadline of embedded control apps 
like AVs remains a challenging problem.
When the number of concurrent apps changes dynamically, 
the amount of resources (e.g., CPU and memory space) 
they require should also change to meet their deadline.
Unfortunately, most vision apps are computation-intensive, 
and hence state-of-the-art (SOTA) multi-tasking systems
\cite{yang2019re, jiang2020optimized, jeong2022band, han2022microsecond}
focused on concurrent execution of multiple DNN apps
on specific processors without dynamically modifying
their resource requirements.
The most straightforward way to reduce the amount of resources 
required by each app is compressing the DNN model.
Given the tradeoff between accuracy and computation, 
uniformly compressing all the layers in a model 
with a single precision is not efficient.
For example, quantizing a model with high precision may result 
in a deadline miss. Conversely, low precision quantization 
will deteriorate accuracy \cite{krishnamoorthi2018quantizing}, 
which may cause fatal accidents.
This tradeoff calls for a design that optimizes each DNN model 
to be represented in mixed-precision while considering 
{\em total accuracy} (defined as the weighted sum of 
accuracies of concurrent apps) as well as its deadline.

\looseness=-1
In this paper, we propose {\tt DynaMIX}\ (\underline{\tt Dyna}mic
\underline{\tt MIX}ed-precision model construction),
which dynamically changes the bit-width setting of each DNN model 
according to the status of running apps to optimize their 
computation and memory resources.
For the realization of {\tt DynaMIX}, we must address two difficulties 
that make mixed-precision models unsuitable for 
time-critical systems like AVs.
First, we have to determine the optimal degree of compression 
(i.e., optimal bit configuration) for each DNN model at 
runtime. Balancing between latency and accuracy at runtime 
is difficult as both require to run DNN inferences 
as many as the number of compressed models of each app
when the status of running apps changes,
thus incurring high online overhead.
To address this difficulty,
we create measurements-based app performance profiles 
via regression for worst-case latency and most-likely 
inference accuracy according to the available memory size.
The resulting profiles are then used for online resource 
optimization across multiple apps.


Second, we need to reduce the overhead of model loading.
When their optimal bit settings are modified, 
loading the entire mixed-precision models is inefficient 
unless the bit-widths of all their layers are changed. 
However, loading {\em all} possible mixed-precision 
models generated from one DNN model into memory is neither 
efficient nor possible.
To reduce this model switching overhead,
we propose runtime model reconfiguration based on the concept of lazy loading,
commonly used in other domains such as web services and 
operating systems \cite{park2019design, arendt1998system}.
The proposed layer-wise lazy loading helps avoid the need 
for loading the entire model whenever the optimal bit setting 
is changed to use the platform resources efficiently.

{\tt DynaMIX}\ is composed of offline and online stages.
In the online stage, it formulates the resource optimization 
as a constrained nonlinear optimization problem.
The constraints of the optimization include not only 
the deadline but also the offline-generated profiles to 
determine an optimal solution (i.e., memory space for 
each app) that maximizes the total accuracy subject to 
the deadline and the available memory space. 
When a new camera image enters the system and any change 
in the status of concurrent apps is detected, the main 
process adjusts the amount of memory space allocated 
to the concurrent apps.
After determining the new bit configuration with the 
adjusted memory space for each DNN app, the app 
reconfigures its execution path by loading the necessary 
high-precision layers to optimize the amount of 
computation resource. This way, all of them can meet 
the deadline without compromising the total accuracy.

We have evaluated {\tt DynaMIX}\ for running multiple concurrent apps.
Our findings help one or more concurrent apps meet the 
deadline while optimizing the resource allocation. 
The resulting model for each app is shown 
to lose $<2.2$\% accuracy even in the case of an increased 
number of concurrent apps or tight timing constraints.
Furthermore, we have shown the feasibility of {\tt DynaMIX}'s 
resource allocation in various aspects and
demonstrated its effectiveness in handling multi-tasking 
scenarios. We have also identified the cases 
{\tt DynaMIX}\ could not support, or the resulting total accuracy 
was too low to be acceptable for the apps in reality.

In summary, this paper makes the following contributions:

\begin{itemize}
    \item The first resource optimization scheme for 
       concurrent in-vehicle apps to meet their deadline 
       without binding to specific processors;
    \item Formulation of the optimization problem using app 
        performance profiles to find the optimal compression 
        degrees for multiple apps while accounting for 
        their accuracies and worst-case latencies in real time;
	\item Runtime model reconfiguration while lazy loading 
	  only the selected layers when the optimal bit setting 
	  of an app varies, to mitigate the entire model 
	  loading overhead; 
    \item Facilitation of rapid deployment of existing DNN 
      apps without any (re)training required to prevent any 
      significant accuracy drop in runtime adaptive 
      quantization.
   
   
   
   
\end{itemize}



\section{Related Work}
\label{sec:relatedwork}

Prior work on multiple or real-time DNN inferences on embedded 
platforms has evolved into three main branches: 
resource management, RTOD optimization, 
and hardware-based scheduling
(Table~\ref{tbl:relatedwork}).

\subsection{Resource Management for Multiple Vision Apps}

Resource management plays a critical role in ensuring 
the continuous execution of multiple vision apps.
Most resource-management schemes reduce the excessive cost of 
DNN pipeline execution by fully/partially offloading the 
computation to a cloud server or other edge devices. 
MCDNN \cite{han2016mcdnn}, a representative 
resource-management scheme, proposes a runtime scheduler 
to process DNN models (or fragments thereof) 
across both the edge device(s) and the cloud server. 
Although offloading reduces the computation 
load of the edge system, it yields inconsistent 
app quality because of unpredictable cloud accessibility 
or possible privacy leakage.

To mitigate/avoid these problems, \cite{huynh2017deepmon,fang2018nestdnn} 
suggested on-device resource management.
DeepMon \cite{huynh2017deepmon} aims to guarantee 
continuous vision apps by optimizing the convolutional neural 
networks (CNN) on mobile GPUs. It accelerated the convolution 
by reusing the intermediate results via caching.
NestDNN \cite{fang2018nestdnn} proposed a 
filter pruning for resource management.
However, both DeepMon and NestDNN
support only non-real-time tasks, probably because the 
most commonly used real-time tasks (e.g., object detection 
or tracking) require significant amounts of computation 
that the edge system cannot complete in a timely 
manner (e.g., DeepMon shows only 1$\sim$2 FPS).
{\tt DynaMIX}\ was inspired by \cite{huynh2017deepmon, fang2018nestdnn} 
in that both pursue a server-less method.
{\tt DynaMIX}\ reduces the complexity of vision apps via DNN 
compression, and finds the optimal models by exploiting 
the knowledge of app performance.

\subsubsection{Why not adapting NestDNN \cite{fang2018nestdnn} 
to guarantee deadlines?}
Even if NestDNN’s scheduler is revised to meet real-time 
constraints, it cannot guarantee both accuracy and timeliness.
NestDNN leverages a small, fixed number (i.e., five) 
of pruned models for each app, thus sacrificing accuracy 
to create models of various sizes.
If more pruned models were generated
for accuracy, timely execution is difficult because of its 
high overhead of finding an optimal model set for concurrent DNN apps.
To address this, they need to formulate an optimization 
problem using app performance profiles and generate 
lookup tables as we proposed here.
In contrast, {\tt DynaMIX}\ fully supports real-time scenarios.


\subsection{RTOD Execution on Embedded Systems}
\label{sec:rtod}
The great potential of RTOD for improved safety and 
convenience has yielded a large body of work running 
RTOD on embedded systems, particularly AVs. Despite 
the significant recent advances in object detection 
\cite{liu2016ssd, ren2016faster, redmon2016you}, 
RTOD still remains to be a bottleneck in embedded 
real-time systems due to its heavy computation requirement
to localize and classify multiple objects 
captured in a stream of camera images.

\looseness=-1
AdaVP \cite{liu2020continuous} 
built a parallel object detection and tracking 
pipeline to run two apps in parallel; when a new object 
is detected, the DNN setting (i.e., input frame size) 
is adjusted at runtime to increase the tracking accuracy.
Yang {\em et al.} \cite{yang2019re} 
handled simultaneous vision apps with private 
camera streams and identical DNN models; throughput was 
increased by sharing the base architecture and 
processing multiple camera images with multiple threads.
DNN-SAM \cite{kang2022dnn} enabled multiple inferences 
with one RTOD model to process the images produced by 
different cameras. 
It ensures the deadline by splitting the original image 
into different portions according to the criticality levels 
and adjusts the size of each portion via scaling.

\looseness=-1
The authors of \cite{heo2020real} proposed a new RTOD 
system that could change the execution path based on 
a \textit{dynamic deadline} --- the deadline of a RTOD task 
varies with the underlying driving environment, like skipping 
layers or choosing one of the sub-networks of different sizes.
Moreover, they determined the execution path of RTOD 
using per-layer latency (for all layers) and 
dynamic deadline, but did not account for the 
interruptions by other concurrent apps in their 
path-selection decisions.
{\tt DynaMIX}\ was inspired by \cite{heo2020real} 
in that the execution path can be changed dynamically
to account for the execution environmental condition.
{\tt DynaMIX}\ reconfigures the DNN execution pipeline 
once the amount of available
resources for each app changes.

In summary, prior work accelerates the
RTOD task to meet its explicit/implicit deadline 
using model adaptation or multi-threading.
However, the resource contention by multiple threads/apps
may lead to performance degradation, and the real-time model 
adaptation can cause a significant accuracy drop.
More importantly, their requirement of full use 
of hardware resources for RTOD can be problematic when 
applied to a real-time multi-tasking platform.

\subsubsection{Why cannot Heo {\em et al.} \cite{heo2020real}~be 
modified for multi-neural network execution?}
\looseness=-1
The authors of \cite{heo2020real} showed that the WCET model 
for a certain layer can work in multi-tenant systems, 
but they neither showed how to share the limited resources 
between concurrent DNN models nor accounted for the 
overhead of context switching between apps. 
In contrast, {\tt DynaMIX}\ addresses these problems in 
multi-tasking systems.

\subsection{Hardware-based Scheduling for Multi-DNN Inferences}
\looseness=-1
AI-MT \cite{baek2020multi} proposed a new NN accelerator 
architecture and scheduling scheme for multi-tasking platforms.
\cite{jiang2020optimized, jeong2022band} 
coordinated multiple latency-critical DNN tasks 
by using specific DNN accelerators (e.g., FPGA, NPU).
Scheduling jobs across heterogeneous processors 
allowed their real-time execution.
The most recent work in \cite{han2022microsecond} proposed 
a GPU scheduling method to run multiple real-time apps. 
Basically, it schedules memory- and computation-bound jobs 
across heterogeneous or several processors to enhance
concurrency. However, it does not work in the 
representative case of Fig.~1.
It only focuses on scheduling multiple real-time jobs with 
the unmodified DNN models which require a {\em static} 
amount of resources, thus limiting the number of 
concurrent jobs it can handle. 
In contrast, {\tt DynaMIX}\ dynamically adjusts the resource 
consumption of each app. Moreover, {\tt DynaMIX}\ is orthogonal
to these prior solutions, and hence can
run with them together.


\section{Background: Mixed-precision Quantization}
\label{sec:background}
\looseness=-1
Quantization of activation and model parameters can accelerate 
DNN execution by reducing the computational complexity of 
the underlying models. This is predicated on the fact 
that integer operations yield a much higher throughput in 
vectorized computations than floating-point operations 
on most computing platforms.
However, removing some bits in a fully-trained model 
(that is already converged to the lowest loss) causes an 
output perturbation between the full-precision and 
quantized models, thus degrading accuracy significantly.

\looseness=-1
Mixed-precision quantization has been explored as 
a promising solution to this problem by using layers of 
different bit-widths \cite{wang2019haq, cai2020zeroq, 
dong2019hawq, lee2021novel, zhe2019optimizing}. 
Using a higher bit-width at a layer more sensitive to 
quantization can help the layer preserve its original 
values, thus making the model suffer less output 
perturbation and accuracy drop.
Typical mixed-precision quantization algorithms are 
composed of layer sensitivity measurement and layer 
bit-width decision-making.

\subsection{Measurement of Layer Sensitivity}
\label{sec:sensitivity}

{\em Layer sensitivity} represents the extent to which the 
model output changes when a certain layer is quantized.
For the most exhaustive mixed-precision approach, a model 
of $L$ layers with $B$ types of bit-widths yields 
$B^L$-quantized models. Since DNN models are recently 
becoming deeper, mixed-precision will be less attractive.
So, identifying layer sensitivity is an efficient way to 
reduce the large design space for bit allocation.
Well-measured layer sensitivity can also be used to 
calculate the overall perturbation of the resulting 
mixed-precision model, and is a good measure for finding 
the best bit setting.

Layer sensitivity is affected by several factors, such as 
layer position, operation type, connection with other 
layers, and layer parameter size.
It is difficult to define sensitivity by considering 
all these factors in a large model; researchers 
used output perturbation as their sensitivity metric 
by calculating \textit{L2-Norm} or \textit{KL-Divergence}
\cite{zhe2019optimizing, cai2020zeroq}.

\looseness=-1
In this paper, we use the sensitivity metric 
defined in the state-of-the-art mixed-precision quantization 
method ZeroQ \cite{cai2020zeroq}, which is based on 
\textit{KL-Divergence} between the full-precision  
model and the quantized model, as:
\begin{align*}
  {S}_{i}(k)=\frac{1}{N} \sum_{j=1}^{N} \mathrm{KL}
  \big(\mathcal{N} \big(x_{j}\big), ~ \mathcal{\widetilde{N}}_{i}^{k}\big(x_{j}\big)\big)
\end{align*}
where ${S}_{i}(k)$ denotes the sensitivity of quantized 
model $\mathcal{\widetilde{N}}_{i}^{k}(\cdot)$ in which 
the $i$-th layer is quantized into $k$-bits, 
$\mathcal{N}(\cdot)$ indicates the full-precision model, 
and $x$ is a small set of input images of size $N$ used 
for sensitivity measurement.

\subsection{Hardware-aware Bit-width Decision}
\looseness=-1
Mixed-precision quantization has the flexibility of hardware-aware 
model compression. As the most basic method of bit-width decision, 
rule-based schemes have been used based on the knowledge of the 
DNN model and hardware architecture as well as manual effort.
However, increasingly complex models
make such heuristics difficult to apply.
ZeroQ \cite{cai2020zeroq} employed a 
\textit{Pareto frontier}-based method that finds an optimal 
compressed model with minimum output perturbation.
The overall sensitivity for each mixed-precision model is 
computed by summing the sensitivities of all layers in the model.
Although the authors did not reflect inter-layer dependency 
in this process, they showed that such sensitivity calculation 
incurs less computational overhead and produces good 
empirical results.
Additionally, sensitivity and model size are considered in 
ZeroQ as the indirect indicators of accuracy 
and latency. 

However, each processor has a distinct architectural design, 
implying that the best bit settings vary with hardware 
\cite{wang2019haq}.
For instance, a weight parameter layout designed to increase 
reusability can efficiently reduce the latency of conventional 
convolution layer, which can thus have a higher bit-width than 
fully-connected or depth-wise convolution layers.
Thus, with hardware-software codesign, this advanced quantization 
can reduce the computation and resource costs without 
any severe loss of inference accuracy compared to 
hardware-oblivious DNN compression methods (e.g., 
uniform quantization \cite{krishnamoorthi2018quantizing}).

Thanks to its flexibility, mixed-precision can accelerate the 
network model with a negligible accuracy degradation. 
Thus, recent processing engines have been 
released with the function of mixed-precision arithmetic 
with variable bit-widths \cite{nvidia_h100, raha20208}. 
These recent advancements have raised the importance of 
mixed-precision-based methods for DNN acceleration.

Inspired by the reinforcement learning-based bit-width setting 
method \cite{wang2019haq}, we determine the best mixed-precision 
model for each app in a hardware-aware manner.
Instead of using indirect indicators, e.g., FLOPs, model size, and sensitivity,
we use the worst-case latency (to meet 
the deadline), peak memory usage (to consider memory 
capacity), and inference accuracy measured in 
our simulated environment.
We aim to use a mixed-precision approach to find an optimal 
solution that maximizes the total accuracy while making the 
most of limited resources.
Sec.~\ref{sec:runtime-change} and~\ref{sec:architecture} will 
detail how {\tt DynaMIX}\ builds the mixed-precision 
model and chooses an  optimal bit configuration 
for each DNN-based app.

\begin{figure}[tp]
    \begin{center}
    \includegraphics[width=0.9\linewidth]{figures/layerloading.pdf} 
    \end{center}
    \vspace*{-3mm} 
    \caption{Runtime model reconfiguration and execution mechanism.}
    \label{fig:loading}
\end{figure}

\section{Dynamic Model Reconfiguration \& Execution}
\label{sec:runtime-change}
\looseness=-1
At the core of {\tt DynaMIX}\ is finding the most suitable mixed-precision 
model for each app by reflecting the current system condition; 
app processes may otherwise be unstable for DNN models due
to their low accuracies or 
severely underutilize the platform resources. 
Thus, we need to consider various compressed models with different 
bit configurations, their saving and loading of many models. 
However, reloading the resulting entire model each time may 
be neither suitable for real-time apps nor efficient 
when only a few layers need to be loaded.
Thus, we propose to reconfigure the model bit-width 
configuration at runtime by lazy loading FP layers 
in order to reduce the model loading overhead 
and ensure deadline satisfaction.
This dynamic model reconfiguration approach 
is inspired by \cite{tang2022arbitrary, bulat2021bit}
where bit-width decisions are made 
only for a {\em single} DNN model, and
training a special DNN from scratch is necessary 
for runtime adaptive quantization.
In contrast, we develop a solution that determines 
bit configurations for multiple DNNs 
without requiring any additional (re)training.


\looseness=-1
In {\tt DynaMIX}, when the optimal bit setting for each app is 
determined, only the additional layers for the target model
are loaded to compose the path ({\small\circled{7}} in Table~\ref{tbl:navigation}).
Before operating this at runtime, we separate a model into layers 
and store the full-precision (FP) and quantized (INT) versions of 
all layers ({\small\circled{1}}). That is, 2$L$ versions of layers 
are saved for a model consisting of $L$ layers.
Specifically, operation type (e.g., convolution) 
and FP weights are saved for an FP layer; for an INT layer, 
additional parameters required for quantization of FP activations 
are saved with operation type and INT weights.

\noindent\textbf{\textit{Mechanism.}} Model reconfiguration 
consists of 1) loading necessary layers and 2) changing the 
model execution path. Here we deal with a multi-tasking system 
in which the main process is in charge of process monitoring 
and memory allocation, and the subprocess is in charge of 
model execution.
To load only the layers needed at runtime, {\tt DynaMIX}~should first 
hold the fully-quantized models of all possible apps as their 
base models by loading their INT layers and building them 
without changing the original architecture. 
When a new camera image enters to the system and any change in the status of concurrent apps is detected, the main process adjusts the amount 
of memory resource allocated to the concurrent apps 
({\small\circled{5}}).
After determining the new bit setting from the adjusted memory 
space for each app, the necessary FP 
layers required for the new model are loaded ({\small\circled{6}}), and then they 
are executed instead of the existing INT layers. 
After execution, the FP layers are deleted, and each model 
returns to its original base model state.
Fig.~\ref{fig:loading} shows the above mechanism with two 
different cases.
How to build a mixed-precision model while considering the given 
memory space will be elaborated next.



\section{System Architecture}
\label{sec:architecture}

Our main objective is to optimize the allocation of system 
resources, such as CPU time and memory space, for multiple simultaneous apps while meeting the 
deadline, which has not yet been 
addressed adequately because of the high computational 
complexity of real-time CNN models.

\looseness=-1
Representing a model using both high and low precisions 
({\small\circled{2}}) is key to reduction of the computational 
cost of DNNs, thereby enabling multiple DNN apps to share 
limited platform resources.
Central to {\tt DynaMIX}\ is a memory resource allocation algorithm 
that determines the best bit setting for each app to use 
so as to maximize total accuracy ({\small\circled{5})}.
This algorithm uses app profiles that are created offline to 
save the information of a set of mixed-precision models 
({\small\circled{3}}), as a constraint term. 
Thus, optimization of computation resource is triggered
by the apps whose memory requirements change dyamically.
After completing a series of processes for resource optimization, 
the additional layers to load/execute are selected by 
looking up tables ({\small\circled{6}}), 
which were generated offline ({\small\circled{4}}).

To reduce the online burden of computation-heavy processes, 
{\tt DynaMIX}\ is divided into offline and online stages (see 
Fig.~\ref{fig:overview}).
{\small\circled{1}, \small\circled{7}} are discussed in 
Sec.~\ref{sec:runtime-change}, and the remaining components 
are elaborated in the subsequent sections.

\begin{figure}[tp]
    \begin{center}
    \includegraphics[width=0.9\linewidth]{figures/architecture.pdf} 
    \end{center}
    \vspace*{-3mm} 
    \caption{System architecture.}
    \label{fig:overview}
    \vspace*{-2mm} 
\end{figure}

\begingroup
\setlength{\tabcolsep}{6pt}
\renewcommand{\arraystretch}{1.0}
\begin{table}[t]
    \caption{Navigation for system components. PIC means the process in charge of the corresponding component.}
    \begin{center}
    \begin{tabular}{clcc}
        \hlineB{2}
        \textbf{Stage} & \multicolumn{1}{c}{\textbf{Component}} & \textbf{Section} &\textbf{PIC}\\
        \hlineB{2}
        \multirow{4}{*}{Offline}
        &{\small\circled{1}} Per-precision Layer Saving & \ref{sec:runtime-change} & Sub\\ \cline{2-4}
        &{\small\circled{2}} Mixed-precision Model Building & \ref{subsec:compression} & Sub\\ \cline{2-4}
        &{\small\circled{3}} Profile Generation & \ref{subsec:profiling} & Main\\ \cline{2-4}
        &{\small\circled{4}} Lookup Table Generation & \ref{subsec:lookup} & Main\\
        \hlineB{1.5}
        \multirow{3}{*}{Online}
        &{\small\circled{5}} Memory Resource Allocation & \ref{subsec:allocation} & Main\\ \cline{2-4}
        &{\small\circled{6}} Lookup Table Querying& \ref{subsec:lookup} & Main\\ \cline{2-4}
        &{\small\circled{7}}  Model Reconfig. and Execution & \ref{sec:runtime-change} & Sub \\
        \hlineB{2}
    \end{tabular}
    \end{center}
    \label{tbl:navigation}
    \vspace*{-3mm} 
\end{table}
\endgroup

\subsection{Mixed-Precision Model Building} 
\label{subsec:compression}
\looseness=-1
Using the FP and INT versions of all layers, we can build 
different mixed-precision models (based on the identical 
architecture) which show the effects of mixed-precision 
quantization in terms of accuracy, speed, model size, etc.
The main objective of {\tt DynaMIX}~is to find a 
model that maximizes the accuracy with the given resources 
and deadline.
We generate different compressed models of 
various bit configurations (showing different performances), 
and use them for the prediction of performance in accordance 
with the amount of available memory resource.

We develop a sorting-based approach to build models while yielding 
the highest accuracy compared to the degree of compression
(defined as the number of the quantized layers in the model). 
In this approach, the FP layers of the model are 
sorted in descending order of layer sensitivity discussed in 
Sec.~\ref{sec:sensitivity}.
ZeroQ \cite{cai2020zeroq} shows the overall sensitivity of a quantized 
model (where multiple layers are quantized with low bit-widths) 
can be calculated by summing up the sensitivities of all 
quantized layers in that model.
Based on this calculation, starting with the fully-quantized 
model, the INT layers are replaced with their corresponding 
FP layers one-by-one (in descending order of layer sensitivity). 
This means that at all degrees of quantization, the generated 
model would yield the smallest output perturbation from the 
full-precision model, thus leading to the smallest accuracy drop.
This model building process is described in 
Algorithm~\ref{alg:quantization}.
Although this can be presented recursively or iteratively like 
depth-first search, we present it recursively for readability,
and the initial invocation for this procedure is 
\textsc{build}~(Fully-quantized model, $\emptyset$).

Our sorting-based approach yields $L$ compressed models for a 
model consisting of $L$ layers, which effectively removes 
unnecessary and repetitive models among $2^L$ models produced by 
an exhaustive model generation approach.

\looseness=-1
Fig.~\ref{fig:profiles} shows the inference results of the 
compressed models of VGG16 \cite{simonyan2014very}, 
ResNet50 \cite{he2016deep}, and YOLO-v3 \cite{redmon2016you}.
Note latency includes the time for model reconfiguration, 
execution, and restoration.
We used 2 bit-widths, FP32 and INT8, throughout this paper to 
demonstrate the effectiveness of using mixed-precision models 
on app performance, i.e., all layers are
expressed with either precision.
The three graphs in the upper row show latency changes and 
the three graphs in the lower row display accuracy changes as 
a function of the degree of compression.
Considering memory usage and latency of a model may change each time 
we run the model, we measured 100 times for each model and used the 
highest peak memory usage and latency as the worst case.
The results show a latency reduction without sacrificing accuracy 
much when compression degree is increased.
To utilize the performance results of the compressed models in the 
online stage, two profiles and a lookup table are created for 
each model.

\begin{algorithm} [t]
    \caption{Mixed-Precision Model Building} 
    \begin{algorithmic}
    \State $L$  \textbf{:} a set of layers sorted in descending order by 
        layer sensitivity
    \State $\mathcal{S}$  \textbf{:} a set of compressed models ($\mathcal{S} 
       \gets \emptyset$)
      \\
    \Procedure{Build}{model $m$, layer $l$}
        \State $n$ = Replace $l$ of $m$ with FP layer\Comment{\small If $l$ is $\emptyset$, $n \gets m$}
        \State $\mathcal{S} \gets \mathcal{S}\cup n$
        \State $v$: the most sensitive layer in $L$\Comment{\small If $L$ is $\emptyset$, \Return {$\mathcal{S}$}}
        \State remove $v$ in $L$
        \State \textsc{BUILD}($n$, $v$)
        \State \Return {$\mathcal{S}$}
    \EndProcedure
    \end{algorithmic}
    \label{alg:quantization}
\end{algorithm}

\subsection{Profile Generation}
\label{subsec:profiling}
The above process yields a number of mixed-precision models for 
each app, and hence the goal of our memory resource allocation is 
to find 
an optimal set of compressed models for all concurrent apps.
Although such a large number of combinations offers
flexibility to accommodate various real-world cases, the high 
computational cost of finding an optimal set makes it 
unsuitable for real-time resource allocation. 
Our model compression yields $O(L)$ models, and in the case of 
a 10 FPS camera, the end-to-end latency per frame should be 
less than 100 ms \cite{yang2019re}.
Given that $I$ apps are simultaneously running on the platform, 
an excessive amount of time (up to $O(100 \cdot L^{I})~ms$) is 
required to find an optimal combination.

\begin{figure}[tp]
    \begin{center}
    \includegraphics[width=1.0\linewidth]{figures/profiles.pdf} 
    \end{center}
    \vspace*{-3mm} 

    \caption{The results of model compression and profiling on 
    VGG16, ResNet50, and YOLO-v3. Each gray dot indicates a 
    certain compressed model that has a specific bit setting. 
    A yellow line denotes the profile of worst-case latency, 
    thus covering all the compressed models as shown in the gray 
    areas. A blue line means the profile of 
    most-likely accuracy, showing the trend of accuracy.}
    \label{fig:profiles}
    \vspace*{-2mm} 
\end{figure}

\begingroup
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{1.3}
\begin{table}[tp]
\begin{centering}
\caption{Latency and accuracy profiles. X means peak memory usage.}

\vspace*{-2mm} 
\begin{tabular}{>{\centering\arraybackslash}m{1.4cm} >{\centering\arraybackslash}m{1cm} >{\centering\arraybackslash}m{4.3cm}}
\hlineB{2}
\textbf{NN Model}         & \textbf{Type} & \multicolumn{1}{c}{\textbf{Profile}}                                               \\ \hlineB{2}
\multirow{2}{*}{VGG16}    & Latency       & $-0.0001472~x^3 + 0.1135 x~x^2 - 28.16 ~x + 2267$   \\[0ex] \cline{2-3} 
                          & Accuracy      & $-7.781e\mhyphen07~x^3 + 0.0006417~x^2 - 0.1753~x + 108.4$ \\[0ex] \hline
\multirow{2}{*}{ResNet50} & Latency       & $1.021e\mhyphen05~x^3 - 0.01083 ~x^2 + 4.648~x - 683$     \\ \cline{2-3} 
                          & Accuracy      & $1.285e\mhyphen07~x^3 - 0.000181~x^2 + 0.08553~x + 62.48$ \\ \hline
\multirow{2}{*}{YOLO-v3}  & Latency       & $-6.706e\mhyphen06~x^3 + 0.01861~x^2 - 15.5~x + 4241$      \\ \cline{2-3} 
                          & Accuracy      & $6.764e\mhyphen08~x^3 - 0.0001834~x^2 + 0.1676~x + 2.821$ \\ \hlineB{2}
\end{tabular}
\vspace*{-3mm} 
\label{tbl:profiles}
\end{centering}
\end{table}
\endgroup

For real-time optimization, the authors of NestDNN \cite{fang2018nestdnn} 
considered only five candidate models, which come at the cost of 
both resource utilization and accuracy. 
In {\tt DynaMIX}, however, profiles are generated offline for each app 
in the form of polynomials for latency and accuracy by 
applying polynomial regression to the compressed models.
This app performance profiling eliminates the need 
of additional inference processing in the online stage. 
We introduce latency and accuracy profiles, which are used 
in our online memory resource allocation algorithm.

\subsubsection{Latency Profile}
By employing the latency profile of a target DNN, we can estimate the
worst-case latency when the model is compressed to a certain size. 
If some models cannot be covered by the latency profile, the newly 
selected model at runtime can exceed the expected number obtained 
from the latency profile, thus failing to guarantee the deadline. 
Thus, the latency profile should cover all the resulting 
compressed models. The impact of inter-task interference in a 
multi-tasking platform on execution latency is discussed in 
Sec.~\ref{sec:eval_feasibility}.

To create the latency profile that covers all compressed models, 
the compressed models are stored when its latency is 
greater than the latencies of the smaller models, and then 
polynomial regression is applied to the stored models.
We exploit cubic polynomials because latency is not completely 
but fairly proportional to peak memory usage, and high-order 
regression burdens the optimization algorithm (see 
Fig.~\ref{fig:profiles}).
After obtaining a polynomial profile for each model, 
to guarantee the worst-case latency for a given memory space,
we delete those models whose latencies are greater 
than the values estimated 
with the latency profile from a set of all compressed models.
The resulting latency profiles of VGG16, ResNet50, and YOLO-v3 
are provided in Table~\ref{tbl:profiles}.

\subsubsection{Accuracy Profile}
The role of the accuracy profile is different from that of the 
latency profile where we observe the change in accuracy according 
to the degree of compression. For this, we apply regression 
to all of the remaining models after creating the latency profile, 
without deleting models any further.
Accuracy also has a weak positive linear correlation with peak 
memory usage, and thus regression is performed using a cubic 
polynomial (see Fig.~\ref{fig:profiles}). The accuracy profiles 
of VGG16, ResNet50, and YOLO-v3 are provided in 
Table~\ref{tbl:profiles}. 

\subsection{Optimal Resource Allocation}
\label{subsec:allocation}
Once a new camera image enters to the system and any change 
in the status of app processes is detected, 
{\tt DynaMIX}\ allocates the memory resource for concurrent apps 
while maximizing the total accuracy under the given deadline.
The optimization problem in this process is solved by reflecting 
the latency and accuracy profiles generated in the offline 
stage to determine the optimal allocation of memory resource 
for each app. In what follows, we present the development of 
a resource allocation algorithm, which is formulated as a 
constrained nonlinear optimization problem.

{\tt DynaMIX}\ assumes a limited memory capacity, and the timing 
requirement of a real-time app is determined
by camera's frame rate.
Therefore, the problem of resource 
allocation for concurrent apps is formulated as: 

\begin{equation}
\begin{aligned}
\label{eqn:rsc_optimization}
& \arg \max_{m_{i}} && \sum_{i \in I} \lambda_{i}A_{i}\left(m_{i}\right) \\ 
& \textrm{s.t.} && \sum_{i \in I} L_{i}\left(m_{i}\right) \leq D - \varepsilon \\
&&& {\forall{i} \in I}:m_{i} \leq M_{\max } - \mu_{i} \\
&&& {\forall{i} \in I}:m_{i}^{L} \leq m_{i} \leq m_{i}^{U}
\end{aligned}
\end{equation}

\looseness=-1
\noindent
where $I$ is the set of concurrent apps. 
We want to find the optimal memory size
$m_{i}$ for each app $i \in I$ that maximizes total accuracy.
{\em Total accuracy} is calculated as the weighted sum of 
accuracies of all apps, and $\lambda_{i}$ is used to give different 
weights when each app has a different level of importance or the 
degree of change in accuracy during quantization varies with app.
$\lambda_{i}$ is set to 1 in this paper.
For the constraint terms, we consider memory capacity apps can 
use $M_{\max}$, and the given deadline $D$. 
Here, $\varepsilon$ is the average processing time in the main 
process, and $\mu_{i}$ is the memory usage of the base models 
of concurrent apps, which is calculated as 
$\mu_{total}-\mu_{app_{i}}$, where  $\mu_{total}$ is $\mu_{vgg}+\mu_{resnet}+\mu_{yolo}$, and
$\mu_{app_{i}}$ is one of 
$\mu_{vgg}$, $\mu_{resnet}$, and $\mu_{yolo}$.
$\varepsilon$ and $\mu_{i}$ will be detailed in Sec.~\ref{sec:eval_feasibility}
in which we assess the feasibility of solving this problem.
We also use the latency profile $L_{i}$ and accuracy profile 
$A_{i}$ obtained from the profile-generation phase. 
The latency-related constraint term ensures that estimated latency 
$L_{i}\left(m_{i}\right)$ does not exceed the deadline. 
Lastly, we set lower and upper bounds ($m_{i}^{L}$, $m_{i}^{U}$) for 
$m_{i}$ to reduce the search space in solving this optimization 
problem. $m_{i}^{L}$ is set to the smallest peak memory usage of 
app $i$, and $m_{i}^{U}$ is set to the largest peak memory usage 
of app $i$. The optimization result determines the appropriate 
memory space size to be allotted to each app. 

\begin{algorithm} [t]
    \caption{Lookup Table Generation for App $i$} 
    \begin{algorithmic}
    \State $\mathcal{T}$  \textbf{:} lookup table ($\mathcal{T} \gets \emptyset$)
    \State $\mathcal{M}$ \textbf{:} a set of mixed-precision models
    \State \textbf{$z$}  \textbf{:} The number of lookup table entries
    \State \textbf{$max(i)$}  \textbf{:} the largest peak memory usage
    \State \textbf{$min(i)$}  \textbf{:} the smallest peak memory usage
    \State \textbf{$m_{pm}$}  \textbf{:} peak memory usage of model $m$
    \State \textbf{$m_{acc}$}  \textbf{:} accuracy of model $m$
    \\ 
    \State  Sort $\mathcal{M}$ in ascending order by peak memory usage
    \State $val \gets 0, oldkey \gets 0$
    \While {$m \in \mathcal{M}$}
        \State $ key = min(i)
                        + \Bigl\lceil{ \frac{(z-1)(m_{pm} - min(i))}{max(i)-min(i)} \Bigr\rceil}
                        \frac{max(i) - min(i)}{(z-1)} $
                        
        \If{$key == oldkey$}
            \If{$m_{acc}$ is greater than $val_{acc}$}
                \State $val \gets m$
            \EndIf
        \Else
            \State $\mathcal{T} \gets \mathcal{T} \cup (oldkey,  val)$
            \State $val \gets m$, $oldkey \gets key$
        \EndIf
        \State remove $m$ in $\mathcal{M}$
    \EndWhile
    \end{algorithmic}
    \label{alg:lookup}
\end{algorithm}

\subsection{Lookup Table for Final Model Selection}
\label{subsec:lookup}

After resource allocation, {\tt DynaMIX}~should determine which 
compressed models to load into the system.
Given a large number of compressed models for each DNN model, 
searching for the best fitting model sequentially will likely 
take too long for real-time apps.
We thus reduce this search time via lookup tables, which are 
generated for each model in the 
offline stage and used in the online stage.

\subsubsection{Generation of Lookup Table}
\looseness=-1
We present a simple yet efficient method for generating the lookup 
table, which is based on the principle of arithmetic sequences.
We first divide equally the range of peak memory usage of each app.
Second, a new entry is added to the table in the form of a key--value 
pair, where the key is the endpoint of a certain divided section, 
and the value is the bit setting of the model with the highest 
accuracy when the memory size corresponding to that end-point 
is allocated.
Starting from the smallest section, the entry for each section 
is iteratively inserted into the table.
As a result, the resulting lookup tables remain to be of constant 
size (i.e., the preset number of table entries) regardless of the 
complexity of the DNN model.
Fig.~\ref{fig:lookup_table} shows the accuracy of the models of 
which bit settings are stored in the lookup table.
Each graph shows the number of different settings {\tt DynaMIX}\ needs 
to store for each app (e.g., 2 for VGG16). The detailed procedure 
for lookup table generation is described in 
Algorithm~\ref{alg:lookup}.

\begin{figure}[pt]
    \begin{center}
     \includegraphics[width=1\linewidth]{figures/lookup.pdf} 
    \end{center}
    \vspace*{-5mm} 
    \caption{The accuracies of the models stored in the lookup table. 
    The ticks marked on each x-axis denote the sections created by dividing 
    the range of peak memory usage by a fixed number (i.e., the number of 
    lookup table entries). Each of the resulting lookup tables contains 
    50 entries.}
    \label{fig:lookup_table}
    \vspace*{-5mm} 
\end{figure}

\subsubsection{Querying Lookup Tables}
The generated lookup tables allow {\tt DynaMIX}~to select the final model 
quickly after the online resource allocation.

\begin{align*}
{
    key = min(i) + \Bigl\lfloor{ \frac{(z-1)(m_{i}^{\star} - min(i))}{max(i)-min(i)} \Bigr\rfloor}\frac{max(i) - min(i)}{(z-1)}
}
\end{align*}
where $m_{i}^{\star}$ denotes the memory size allocated for 
app $i$, and $z$ is the number of table entries. 
$max(i)$ and $min(i)$ represent the maximum and minimum peak 
memory usages of app $i$. 

After entering the resulting keys into the tables, the final models 
to execute are determined. 
Each subprocess of an app then loads the additional FP 
layers required for the newly selected model and runs 
according to the reconfigured execution path as discussed in 
Sec.~\ref{sec:runtime-change}.

\section{Evaluation}
\label{sec:evaluation}

We now evaluate {\tt DynaMIX}\ by implementing all the components 
in Table~\ref{tbl:navigation} in a simulated environment.
We implemented layer quantization and model execution 
using PyTorch 1.10.2 \cite{pytorch}, which is one of the most 
representative deep learning frameworks.
In compliance with the quantization support by 
PyTorch, INT8 is used as low precision on 
x86-64 CPU for the acceleration of DNN execution.
Instead of using indirect metrics for app performance, 
we measured actual latency by running all the compressed models 
and peak memory usage by using memory profiler \cite{mem_profiler}.
For profile generation, we used the polynomial regression 
utility, polyfit, in the NumPy API.
The resource allocation algorithm was programmed using the 
sequential least square programming utility in SciPy API.
All our experiments were conducted on an AMD Ryzen 7 
5700G processor and 16GB RAM with Ubuntu Linux 20.04 
LTS operating system.
Given the goal of {\tt DynaMIX}, we want to meet the same
deadline of concurrent DNN apps.

\looseness=-1
Following \cite{heo2020real, kang2022dnn}, the target deadline 
is relatively determined by reflecting the performance of 
the original (i.e., full-precision) models measured in our environment.
Even though we experimented in a simulated environment, 
we aim to show {\tt DynaMIX}’s ability of resource optimization
when the state of the running apps changes.


\noindent\textbf{\textit{Multi-tasking Implementation.}}
Our multi-tasking environment consists of the main 
and sub-processes. The main process detects changes in the system,
and allocates platform resources to concurrent apps.
Each subprocess is responsible for the execution of its DNN model. 
Note that the number of subprocesses 
is equal to that of types of possible apps. 
For multi-tasking, we use the Non-Preemptive Shortest Job 
First (SJF) scheduling to avoid the high overhead of context 
switching between DNN-based apps, consider the equal 
priority between apps (attributed to the common deadline), 
and minimize the average waiting time of concurrent apps. 
For example, the three apps introduced throughout 
this paper are scheduled in the order of VGG16, ResNet50, 
and YOLO-v3.

As discussed in Sec.~\ref{sec:runtime-change}, each subprocess 
first builds the fully-quantized model of its app as a base model.
When processing images coming from the camera, 
{\tt DynaMIX}\ behaves differently depending on whether any change 
in the status of concurrent apps is detected or not.
In case a change occurs, the main process determines the optimal 
memory space for each app, and then notifies the corresponding bit 
configuration to one of the subprocesses by sending a message 
(\textbf{\em case~1}). 
Otherwise, the main process does not make resource allocation 
and notifies the last bit configuration to the subprocess 
(\textbf{\em case~2}).
When the subprocess receives the information, it loads the 
necessary layers, runs the model, and then reports to the main 
process that the job is over through shared memory.
Then, the main process 
repeats this procedure until all apps are executed.
In the experiments in Sec.~\ref{sec:scenario_test}, we adjusted 
the status of apps at 1s intervals.
If \textbf{\em case~1} meets the deadline, so does \textbf{\em case~2}.
We therefore show the results of deadline satisfaction 
for \textbf{\em case~1}.

In what follows, we first present the DNN models and datasets 
used in our evaluation, and discuss the feasibility, effectiveness,  
and robustness of {\tt DynaMIX}.

\begin{figure*}[pt]
    \begin{center}
    \includegraphics[width=0.9\linewidth]{figures/mainsubproc.pdf} 
    \end{center}
    \vspace*{-4mm} 
    \caption{Distributions of the latency and peak memory usage of 
    main and sub-process. 
    If the number of subprocesses is 1, YOLO-v3 is loaded and 
    executed. In case of 2, ResNet50 and YOLO-v3 are loaded and 
    executed, and in case of 3, all three apps are loaded and 
    executed. In the last graph, `V', `R', and `Y' stand for 
    `VGG16', `ResNet50', and `YOLO-v3', respectively.
    Each box contains the results of 100 measurements.}
    \label{fig:mainproc}
    \vspace*{-4mm} 
\end{figure*}

\subsection{Neural Networks and Datasets}
\label{sec:eval_networks}

Image classification and object detection apps often 
run simultaneously on AVs.
Furthermore, even for identical tasks, the complexity of model 
varies greatly with the function.
For example, classifying objects in front of a car is more 
difficult than classifying the color of a traffic light.
Considering the diverse real-world cases, the following three 
DNN models were selected for two types of apps.
Although the state-of-the-art models change rapidly, their 
underlying architectures are predicated on the models of 
our choice \cite{yun2019cutmix, ding2021repvgg}.

\subsubsection{VGG16 on CIFAR-10}
CIFAR-10 \cite{cifar10} is made up of 60K images in 10 classes, 
consisting of a low-resolution color image of animals or vehicles.
Classification of CIFAR-10 is considered to be a simple and easy 
problem owing to the small number of classes and the clear visual 
distinction between class objects.
Based on CIFAR-10, we used VGG16,\footnote{Obtained from 
https://github.com/chengyangfu/pytorch-vgg-cifar10} which is 
fully trained on the dataset.
VGG is one of the most popular and representative architectures \cite{lee2021novel, zhe2019optimizing, fang2018nestdnn}, which 
uses small (e.g., 3$\times$3) kernels in all convolution layers.
In particular, VGG16 (consisting of 16 layers) shows nearly 
state-of-the-art inference accuracy on the CIFAR-10 dataset.
As a result of the simplicity of the task, VGG16 shows higher 
accuracy than the other models as shown in 
Figs.~\ref{fig:profiles} and~\ref{fig:lookup_table}.

\subsubsection{ResNet50 on ImageNet}
ImageNet \cite{ILSVRC15} is composed of 1.4M images in 1K classes, 
each of which is a high-resolution color image of animals, plants, 
vehicles, or electronic devices.
Classification on ImageNet is more difficult than on CIFAR-10 
because of the high similarity of features between classes and 
a large number of classes.
The ResNet50 model,\footnote{Obtained from 
https://github.com/pytorch/vision/blob/main/torchvision/} 
fully trained on the ImageNet dataset, was used in this study.
ResNet, released after VGG, has a much deeper neural network 
architecture (a depth of up to 152 layers) than VGG (a depth of 
up to 19 layers), which solves the vanishing gradient problem 
caused when designing deep networks by adding skip connections 
to the network. 
Due to the complexity of ImageNet, the accuracy of ResNet-50 is lower 
than that of VGG16 on CIFAR-10 (see Figs.~\ref{fig:profiles} 
and~\ref{fig:lookup_table}), but currently widely-used ImageNet-based 
models show accuracies of 75 $\sim$ 80\%, and ResNet-50 shows 
nearly state-of-the-art inference accuracy on ImageNet.
More importantly, the ResNet architecture has great potential for 
building the current high-performance deep model \cite{touvron2019fixing}.

\subsubsection{YOLO-v3 on MS COCO}

For the object detection task, we used the YOLO-v3 
model\footnote{Obtained from 
https://github.com/eriklindernoren/PyTorch-YOLOv3} trained on the MS 
COCO dataset \cite{lin2014microsoft}, which is composed of 5K images 
in 91 classes, consisting of images pertaining to our daily life 
such as one or more objects of vehicles and daily necessities.
YOLO is the first object detection architecture that localizes and 
classifies multiple objects in an input image or video in one single 
forward propagation. Thanks to its speed, RTOD has been realized 
in autonomous driving \cite{yang2019re}, and its role in promoting 
safety and convenience continues to grow.
In Figs.~\ref{fig:profiles} and~\ref{fig:lookup_table}, the accuracy 
of YOLO-v3 (measured in ``average precision (AP)'') is lower than 
that of the classification models (measured in ``Top-1'').
This is because AP means not only the accuracy of classification
but also the prediction accuracy of each object position, 
whereas Top-1 only indicates classification accuracy. 
However, YOLO-v3 performs well in balancing accuracy and latency and 
is one of the most commonly used RTOD models in both academia 
and industry \cite{yang2019re, lin2018architectural, kangdnn}.

\subsection{Feasibility of Resource Allocation Algorithm in {\tt DynaMIX}}
\label{sec:eval_feasibility}
Ensuring the feasibility of resource allocation depends
on the following factors: 
1) the overhead and additional memory usage in solving 
Eq.~(\ref{eqn:rsc_optimization}), 2) the reasonableness 
of using latency profiles for the first constraint term  
of Eq.~(\ref{eqn:rsc_optimization}), and 3) the memory 
usage of each app’s base model for the second constraint 
term of Eq.~(\ref{eqn:rsc_optimization}). 
We show the feasibility of the 
proposed algorithm by identifying each of these factors.

\subsubsection{Runtime overhead and memory usage in 
solving Eq.~(\ref{eqn:rsc_optimization})} 
Eq.~(\ref{eqn:rsc_optimization}) entails a cubic 
objective function with cubic and linear constraints. 
Although the size of the search space is reduced by using 
lower/upper bounds, we need to assess its runtime overhead 
to ensure the deadline satisfaction.
Fig.~\ref{fig:mainproc}~(a) shows the number of concurrent 
apps vs.~the latency of solving 
Eq.~(\ref{eqn:rsc_optimization}).
Latency in the single-tasking case was about 
1.7$\times$ smaller than the multi-tasking case.
Based on this observation, we set $\varepsilon$
in Eq.~(\ref{eqn:rsc_optimization}) to 15 ms by considering
the maximum latency of resource allocation.
Meanwhile, Fig.~\ref{fig:mainproc}~(b) shows the peak 
memory usage is found almost constant regardless of 
the number of running apps (the ratio of the smallest
to the largest mean is approximately 1).
So, given that $M_{max}$ is the memory capacity 
for the subprocesses of the apps, not for the main process,
the main process does not require more space and hence
has little effect on $M_{max}$.

For the various cases in real-world,
the latency and peak memory usage values (for each box) in 
Figs.~\ref{fig:mainproc}~(a) and (b) are measured while 
changing deadline and memory capacity.

\subsubsection{Reasonableness of using latency profiles}
Given the latency profile is based on the measurements in isolation, 
the latencies should hold even when multiplexed with other apps.
From Figs.~\ref{fig:mainproc}~(c) and (d), one can observe 
that the latency of each compressed model holds in the non-preemptive 
SJF scheduling-based multi-tasking environment.
Due to space limitation, we only showed two models: 
(c) for the fully-quantized model of YOLO-v3, 
and (d) for its full-precision model.
From these, we conclude that the latency profile can be used 
for the first constraint term.

\subsubsection{Memory usage of each app’s base model}
\looseness=-1
More concurrent apps occupy more memory space by loading 
their base models.
Fig.~\ref{fig:mainproc}~(e) shows the memory used by 
each base model. Based on their mean values, we set 
$\mu_{vgg}$, $\mu_{resnet}$, 
$\mu_{yolo}$, $\mu_{total}$ to 228, 273, 323, 823,
respectively, in the second constraint term.

\begin{figure*}[tp]
    \begin{center}
    \includegraphics[width=0.9\linewidth]{figures/evaluation_multi.pdf} 
    \end{center}
    \vspace*{-5mm} 
    \caption{Resource optimization results. In (a), we marked the 
    margin between peak memory usage by 3 apps and the memory 
    capacity with a blue line; `V' is VGG16, `VR' is the sum of memory 
    used by VGG16 and ResNet50, and `VRY' denotes the total memory 
    space used by all 3 apps. 
    (b) shows the total latency (consisting of model reconfiguration, 
    app execution, etc.) and each deadline (marked in a blue line), 
    and (c) shows the accuracy change of YOLO-v3.}
    \label{fig:multi}
    \vspace*{-4mm} 
\end{figure*}

\subsection{Effectiveness of {\tt DynaMIX}}
\label{sec:scenario_test}
\looseness=-1
To show the efficacy of {\tt DynaMIX}\ in a multi-tasking 
environment, we designed scenarios where the number of concurrent apps 
changes under two types of timing constraints: sufficient and tightened 
deadlines to run multiple DNN apps (see Table~\ref{tbl:scenarios}).
Owing to the space limitation, we selected some deterministic 
yet representative (instead of random) cases where deadline 
or the number of concurrent apps is adjusted 
to show the change of each app performance.

In general, the model execution speed depends strongly 
on the processing engine (e.g., CPU, GPU).
Considering the use of much slower CPU than GPU, we set
the deadlines based on the latencies measured in our 
simulated environment. In particular, we used
minimum (148 ms), median (470 ms), and maximum (702 ms) 
worst-case latencies resulting from YOLO-v3's 
mixed-precision models.
Memory capacity was set to 1600 MiB by considering the maximum 
peak memory usage of those models.
Figs.~\ref{fig:multi}~(a) and (b) show that {\tt DynaMIX}'s 
resource optimization enables deadline satisfaction.


\begingroup
    \setlength{\tabcolsep}{10pt}
    \renewcommand{\arraystretch}{1.0}
    \begin{table}[tp]
        \caption{Evaluation scenarios on a multi-tasking system}
        \begin{tabular}{C{0.7cm}L{6.3cm}}
        \hlineB{2}
        \textbf{Scenario} & \multicolumn{1}{c}{\textbf{Description}}                                          \\ \hlineB{2}
        Initial          & YOLO-v3-based app is working alone ({\small{\fullsmallcircled{0}{midnightblue}{midnightblue}{white}}}).                                        \\ \hline
        1                 & VGG16-based app starts to run in {\small{\fullsmallcircled{0}{midnightblue}{midnightblue}{white}}}.                     \\ \hline
        2                 & Deadline decreases from 700 ms to 500 ms in {\small{\fullsmallcircled{1}{midnightblue}{midnightblue}{white}}}.  \\ \hline
        3                 & Deadline decreases from 500 ms to 250 ms in {\small{\fullsmallcircled{2}{midnightblue}{midnightblue}{white}}}.   \\ \hline
        4                 & ResNet50-based app starts to run in {\small{\fullsmallcircled{0}{midnightblue}{midnightblue}{white}}}.                     \\ \hline
        5                 & Deadline decreases from 700 ms to 500 ms in {\small{\fullsmallcircled{4}{midnightblue}{midnightblue}{white}}}.  \\ \hline
        6                 & Deadline decreases from 500 ms to 250 ms in {\small{\fullsmallcircled{5}{midnightblue}{midnightblue}{white}}}.   \\ \hline
        7                 & VGG16-based app starts in {\small{\fullsmallcircled{4}{midnightblue}{midnightblue}{white}}}.                         \\ \hline
        8                 & Deadline decreases from 700 ms to 500 ms in {\small{\fullsmallcircled{7}{midnightblue}{midnightblue}{white}}}.  \\ \hline
        9                 & Deadline decreases from 500 ms to 250 ms in {\small{\fullsmallcircled{8}{midnightblue}{midnightblue}{white}}}.   \\ \hlineB{2}
        \end{tabular}
        \vspace*{-6mm} 
        \label{tbl:scenarios}
    \end{table}
\endgroup

We assume the initial environment 
({\small{\fullcircled{0}{midnightblue}{midnightblue}{white}}}) 
to be in a state where YOLO-v3 operates alone and there is 
enough time and memory space to run the full-precision model.
In such a case, {\tt DynaMIX}\ assigns 1026 MiB to the app, 
expecting the highest accuracy when the 
accuracy profile is used. 
Although the target deadline (700 ms) is determined while considering 
the full-precision model, {\tt DynaMIX}\ selects a different 
model (INT layer accounts for 21\% of the total)
which runs more quickly and accurately with less resources.
In {\small{\fullcircled{1}{midnightblue}{midnightblue}{white}}}, 
despite sharing resources with VGG16, YOLO-v3 uses the 
same bit configuration used in 
{\small{\fullcircled{0}{midnightblue}{midnightblue}{white}}}.
This phenomenon is attributed to 
the low memory usage and time complexity of VGG16.
Also, given the accuracy of VGG16 is almost constant 
regardless of the degree of compression (see Fig. ~\ref{fig:profiles}),
{\tt DynaMIX}~compresses VGG16 maximally and allocates 
most resources to YOLO-v3.
However, in {\small{\fullcircled{4}{midnightblue}{midnightblue}{white}}}, 
the execution of ResNet50 requires compression of
both ResNet50 and YOLO-v3.
Specifically, the memory space allocated to YOLO-v3 decreases 
from 1017 MiB to 952 MiB, and hence YOLO-v3 uses more low 
precision layers (60\% of the total) than 
{\small{\fullcircled{1}{midnightblue}{midnightblue}{white}}}.
In {\small{\fullcircled{7}{midnightblue}{midnightblue}{white}}},
the degrees of compression of ResNet50 and YOLO-v3 are the same as in
{\small{\fullcircled{4}{midnightblue}{midnightblue}{white}}}
due to the low computation and memory costs of VGG16.

{\small{\fullcircled{2}{midnightblue}{midnightblue}{white}}}, 
{\small{\fullcircled{5}{midnightblue}{midnightblue}{white}}}, 
and {\small{\fullcircled{8}{midnightblue}{midnightblue}{white}}} 
show {\tt DynaMIX}'s resource optimization for concurrent apps when 
the deadlines are tightened (from 700 ms to 500 ms).
{\small{\fullcircled{3}{midnightblue}{midnightblue}{white}}}, 
{\small{\fullcircled{6}{midnightblue}{midnightblue}{white}}}, 
and {\small{\fullcircled{9}{midnightblue}{midnightblue}{white}}} 
show the optimization results with the deadlines tightened 
further (from 500 ms to 250 ms).
In these scenarios, concurrent DNN models are executed 
by building more compact mixed-precision models.
Consequently, a larger margin between the peak memory 
usage and memory capacity is made available in 
Fig.~\ref{fig:multi}~(a).
The overall margins between the measured latency/peak memory usage 
and deadline/memory capacity may seem large, 
because we used the worst-case latency and maximum peak memory 
usage (through iterative measurements) when profiling, 
and hence the margins vary with the execution.

\looseness=-1
Our resource allocation aims to maximize total accuracy
without violating any timing constraint,
and hence there is no significant loss in accuracy 
(see Fig.~\ref{fig:multi}~(c)).
Due to the lack of space, we presented the results of YOLO-v3 only, 
the most influential model for both real-time execution and total 
accuracy drop.
In all the scenarios considered, all three models show 
less than 2.2\% accuracy drops. 
The maximum accuracy drop occurs in YOLO-v3 in 
{\small{\fullcircled{9}{midnightblue}{midnightblue}{white}}}.


\begin{figure}[pt]
    \begin{center}
     \includegraphics[width=0.9\linewidth]{figures/evaluation_baseline.pdf} 
    \end{center}
    \vspace*{-4mm} 
    \caption{Comparison of results when {\tt DynaMIX}~and the baseline are applied to YOLO-v3. In (a), the red pentagon represents the worst-case latency and peak memory usage when using the baseline, each of which is calculated using 100 measurements. (b) shows the latency breakdown of the main models in (a). 
    In the case of the baseline, ``prep.'' means the time required in loading the entire model;
    otherwise, it means the time for model reconfiguration.
    ``etc." includes the time spent in model restoration, interprocess communication.}
    \label{fig:baseline}
    \vspace*{-4mm} 
\end{figure}

\subsection{Performance Comparison}
\noindent\subsubsection{Comparison with baseline}
If a new app is allotted enough resources to run the full-precision 
model in \textbf{\em case~1}, the subprocess will reconfigure the 
DNN pipeline by loading all its FP layers, which may use resources 
less efficiently than the baseline approach (i.e., loading 
the entire FP model at once {\em without} reconfiguration). 
Albeit such inefficiency, we show {\tt DynaMIX}~to make 
efficient use of resources in most cases.

Fig.~\ref{fig:baseline}~(a) shows the peak memory usage and 
worst-case latency of the mixed-precision models of YOLO-v3
(the same as the YOLO-v3 latency graph in 
Fig.~\ref{fig:profiles}), but this time, the peak memory usage 
and worst-case latency resulting from the baseline approach are 
marked with red pentagon, and the latency breakdowns for the 
main cases (marked with red hollow dots) are shown in Fig.~\ref{fig:baseline}~(b).
This analysis suggests several aspects of {\tt DynaMIX}'s effectiveness.
First, \textit{m4} (all layers in the execution path 
are replaced with FP layers) shows inefficient latency and 
memory usage compared to the baseline.
However, 73.7\% of our compressed models showed less 
usage of memory and processing time (the yellow area in Fig.~\ref{fig:baseline}), especially  78.9\% of 
our compressed models showing reduced latencies.
In case of \textit{m1} (all layers are in low precision), latency 
and memory usage were reduced by 4.3$\times$ and 1.5$\times$, respectively, compared to the baseline.
Additionally, when about 22.7\% of the layers were quantized 
(\textit{m3}), it showed the same level of latency as the 
baseline, and when about 30.7\% of the layers were quantized 
(\textit{m2}), it showed the same level of memory usage.
From these results, we conclude {\tt DynaMIX}~reacts well to the given 
deadlines by optimizing the computation and memory resources required in running apps. 

\noindent\subsubsection{Comparison with NestDNN}
NestDNN \cite{fang2018nestdnn} also uses a DNN pruning method 
to run concurrent vision apps (non-real-time). 
They create a fixed number of small models according to 
the degree of pruning. They demonstrated feasibility in terms of 
memory usage, accuracy, and energy consumption.
Energy consumption is not in our scope and the app's memory usage 
is discussed by using its model size as an indirect metric. 
So, we compare {\tt DynaMIX}\ with NestDNN only for accuracy.

\looseness=-1
Table~\ref{tbl:nestdnn} shows the differences between the 
generated models when the two methods are applied to VGG16.
Here, {\tt DynaMIX}~is demonstrated: 1) for all the 
generated mixed-precision models (ALL) and 
2) for the models saved in the lookup table (LUT).
While NestDNN uses only five compressed models 
for online resource allocation, 
we can handle more models by inserting the app profiles
into the optimization problem and 
reducing the time of searching for models via 
lookup tables.
Furthermore, all of the compressed models generated by {\tt DynaMIX}~show 
higher accuracy than NestDNN, because its pruning scheme
can incur significant accuracy drop since the distribution of 
model parameters is changed. 
Although the authors recover the 
accuracy to some extent via retraining, the accuracy is not 
fully recovered due to the diminished capacity of the model.
The higher accuracy of {\tt DynaMIX}~enhances the feasibility, but 
shows a lower compression ratio.
However, the degree of compression can be increased by using 
lower precision (e.g., INT4), and if different low precisions 
are used simultaneously, we can increase both the compression 
degree and accuracy.

\begingroup
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.0}
\begin{table}[pt]
\centering
\caption {Comparison between NestDNN \cite{fang2018nestdnn} and {\tt DynaMIX}. Max compression ratio means the ratio of the uncompressed model size to the smallest compressed model size.}
\vspace*{-4mm} 
\begin{tabular}{cccc}\\ \hlineB{2}
\textbf{}                                   &\cite{fang2018nestdnn}     & {\tt DynaMIX}~(All)         & {\tt DynaMIX}~(LUT)         \\ \hlineB{2}
\# Compressed models                        & 5                         & 17                   & 2                  \\ \cline{1-4}
Min accuracy                                & 83                        & 92.47               & 92.47               \\\cline{1-4}
Max accuracy                                & 89                        & 92.51               & 92.51               \\ \cline{1-4}
Max compression ratio                       & 7.8                       & 4.0                 & 4.0                 \\ \hlineB{2} 
\end{tabular}
\vspace*{-5mm} 
\label {tbl:nestdnn}
\end{table}
\endgroup

\subsection{Robustness of {\tt DynaMIX}}
\label{sec:robustness}
Since system error in AVs can cause exigent situations, we must 
ensure system robustness in terms of system availability and 
quality --- two critical factors of system failure.
The resource optimization mechanism in {\tt DynaMIX}\ fails to yield 
results if the deadline is set below a certain level.
Furthermore, heavily compressed models can be used to meet 
the short deadline but may lead to fatal accidents
due to their low accuracy. 
Considering these factors that affect 
system availability and quality, we can evaluate system robustness.

Below we show the robustness of {\tt DynaMIX}\ using the total accuracy 
resulting from lookup tables.
Since the robustness may be different in the platforms with 
different memory capacity, we visualized the relationship between 
deadline, memory capacity, and total accuracy in 
Fig.~\ref{fig:robustness}.
The black cells indicate the points where system gives invalid 
results for three reasons: when available bit-widths cannot meet the 
deadline, when memory capacity cannot support {\tt DynaMIX}, or when the 
total accuracy is degraded by more than a predefined threshold 
(3\% in this paper). In the remaining cases, {\tt DynaMIX}\ guarantees 
acceptable quality for users.

As our environment supports INT8 and FP32 operations, total 
accuracy does not decrease by more than 3\%, but the 
lower bounds of deadline and memory capacity appear high
when the black area is considered.
The ultra-low-bit integers, such as INT4 and INT2, reduce the lower 
bounds, thus allowing {\tt DynaMIX}\ to operate without errors under 
tighter deadlines and less memory capacity.
Therefore, we can improve system robustness by using the various 
bit-widths the system supports.

\begin{figure}[pt]
    \begin{center}
     \includegraphics[width=0.6\linewidth]{figures/evaluation_heatmap.png} 
    \end{center}
    \vspace*{-5mm} 
    \caption{Total accuracy while changing the deadline and memory 
    capacity. The lighter the color, the higher the total accuracy. 
    Black cells denote invalid results caused by the inability of 
    optimization or a significant drop in accuracy.}
    \label{fig:robustness}
    \vspace*{-4mm} 
\end{figure}

\section{Conclusion and Future Work}
\label{sec:conclusion}
We have proposed a resource optimization framework, called {\tt DynaMIX},
that allows multiple DNN-based real-time apps to meet the same 
deadline of apps (dictated by camera's frame rate),
which is critically important for CPS apps.
{\tt DynaMIX}~creates the app performance profiles offline,
and reconfigures the mixed-precision DNN models at runtime 
while reflecting dynamically varying resource requirements 
of the running apps according to their status so as to
share the platform resources.
Our implementation in a simulated environment has 
demonstrated its feasibility and effectiveness.

We assume enough computation resources
for the apps so that their models can fit in memory. 
How to relax this assumption is part of our future work.
We would also like to apply {\tt DynaMIX}\
in mixed criticality system where multiple apps of different 
criticality levels can run.
Although its current version focuses on CNN-based apps, 
{\tt DynaMIX}~can be extended to support various DNN-based apps including 
language models, facilitating the execution of many 
real-time apps on AVs.

\bibliographystyle{ieeetran}
