




















\section{Appendix}
\label{appendix}

\subsection{Additional Evaluation Results}

\begin{table}[H]
    \centering
    \begin{tabular}{|c|c|c|c|}
    \hline 
    Measure & Monitoring & Analysis \\ \hline \hline
    Avg. Response Time & 172.97ms & 136.30ms \\ \hline
    Avg. KS-Test & -- & 0.1467 \\ \hline
    Avg. Wasserstein Distance & -- & 23.9742 \\ \hline
    \end{tabular}
    \caption{Aggregated measurements and metrics (across 100 repetitions) about the calibration accuracy for CoCoME's "bookSale" service}
    \label{table:cocome:metrics}
\end{table}

\input{externals/wasserstein/wasserstein_cocome_all.tex}

\input{externals/comparisons/wasserstein_teastore_parametric.tex}

\begin{table}[H]
    \centering
    \begin{tabular}{|c|c|c|c|}
    \hline 
    Measure & Monitoring & PM & NPM \\ \hline \hline
    Avg. Response Time & 94.92ms & 86.62ms & 102.25ms \\ \hline
    Avg. KS-Test & -- & 0.1267 & 0.1239 \\ \hline
    Avg. Wasserstein Distance & -- & 19.1764 & 16.2669 \\ \hline
    \end{tabular}
    \caption*{PM=Parameterized Model, NPM=Non-Parameterized Model}
    \caption{Aggregated measurements and metrics (across 100 repetitions) about the calibration accuracy for TeaStore's "train" service using a load of \textbf{20 users}}
    \label{table:teastore:param:20}
\end{table}

\begin{table}[H]
    \centering
    \begin{tabular}{|c|c|c|c|}
    \hline 
    Measure & Monitoring & PM & NPM \\ \hline \hline
    Avg. Response Time & 154.19ms & 184.85ms & 103.87ms \\ \hline
    Avg. KS-Test & -- & 0.1609 & 0.2575 \\ \hline
    Avg. Wasserstein Distance & -- & 39.6138 & 51.3531 \\ \hline
    \end{tabular}
    \caption*{PM=Parameterized Model, NPM=Non-Parameterized Model}
    \caption{Aggregated measurements and metrics (across 100 repetitions) about the calibration accuracy for TeaStore's "train" service using a load of \textbf{40 users}}
    \label{table:teastore:param:40}
\end{table}
\section{Running Example}
\label{Running Example}
We introduce a motivating example that illustrates our approach and was used to evaluate it (see Sec. \ref{evaluation}). The example is part of the TeaStore case study \cite{teastore}: a website to buy different kinds of tea. In this case study, the Recommender component is responsible for calculating the recommendations for a certain shopping cart using the services \textit{‘train’} and \textit{‘recommend’}. The \textit{‘train’} service derives information from the previous orders and prepares the data for the \textit{‘recommend’} service. Because there are different strategies to recommend a list of related items, the developers implemented four versions of \textit{‘recommend’}  and \textit{'train'} along different development iterations. 
These implementations have different performance characteristics. Performance tests or monitoring can be used to discover these characteristics for the current state, i.e., for the current deployment and the current workload. However, predicting the performance for another state (e.g., different deployment or workload) is  expensive and challenging because it requires setting up and performing several tests for each implementation alternative.
In our example, answering the following questions is challenging based on APM: “Which implementation would perform better if the load or the deployment is changed?” or “How well does the \textit{'train'} service perform during yet unseen workload scenarios?” An example for the latter question would be an upcoming offer of discounts, where architects expect an increased number of customers and also a changed behavior  of customers in that each customer is expected to order more items.

AbPP can answer these questions faster using simulations instead of the expensive tests if an up-to-date PM is available.

Regardless how the model will be built and updated (reverse engineering extraction or manual/ automatic update), all available approaches recalibrate the whole model by monitoring all parts of the source code instead of recalibrating only the model parts affected by the last changes in source code. For example, the changes in the implementation of \textit{'train'}  belong to the last part of the code which is represented as an internal action ‘trainForRecommender’ by modelling the behaviour using SEFF (see Fig. \ref{fig:seff:train}). This means that these changes have only impact on the RD of ‘trainForRecommender’ and all other PMPs are valid (e.g., RD of preprocessor internal action). Recalibrating the whole PMPs loses potential previous manual changes and causes unnecessary monitoring overhead. Additionally, not all the available approaches detect the parametric dependencies (e.g., the RD of ‘trainForRecommender’ is related to the number of ordered items). 
 



\begin{figure*}
\vspace{-0.2 cm}
	\centering
    \includegraphics[width=0.88\textwidth]{images/MbDevOpsPipline-Final.pdf}
    \vspace{-0.2 cm}
    \caption{Model-based DevOps pipeline}
    \vspace{-0.5 cm}
    \label{pipeline}
\end{figure*}
\vspace{-0.1 cm}
\section{Continuous Integration of Performance Model}
\label{CIPM}
This section provides an overview of the \ac{CIPM} approach, before describing how it is embedded in a continuous software engineering approach in \Cref{MbDevOps} and before providing more details on the CIPM activities in Sections VI-VII.

Our approach \ac{CIPM} incrementally extracts and calibrates archi\-tecture-level \acp{PM} with parametric dependencies after each source code commit. 
To do so, CIPM updates the PM continuously to keep it consistent with the running system, i.e., the deployed source code and the last measurements.

CIPM consists of four main activities: 
 \begin{enumerate}
     \item Performance  model update and adaptive instrumentation: CIPM analyzes the source code changes, updates the architectural PM and static behavior  model  based on the co-evolution approach \cite{langhammer2017a} and  instruments the changed parts of code to calibrate the new/ updated related parts of the architecture as will be discussed in \Cref{adaptive instrumentation}.
     \item Monitoring: CIPM collects the required measurements either during testing or executing the system in production. 
     \item Incremental calibration of \ac{PM} (Sec. \ref{incremental calibration}):  CIPM performs the \textit{Dev-time calibration} of behavior, i.e., \ac{PMPs} (Sec. \ref{Dev-time}) considering the parametric dependencies (Sec. \ref{parametric dependencies}) and  the \textit{Ops-time calibration}, i.e., updating the deployment and usage parts of \ac{PM} (Sec. \ref{Run-time calibration}).
	\item Self-validation: 
	CIPM automatically starts a simulation and calculates the variation between the simulation results and the monitoring data to show the estimation error before performing \ac{AbPP}.
 \end{enumerate}
\label{Preserving the consistency}
To realize these four activities, CIPM extends the co-evolution approach  with a monitoring process to keep the PM consistent with the last up-to-date measurements. The static structure and behavior update of PM  is already provided by the co-evolution approach. For the adaptive instrumentation, we (a) extend the \ac{VSUM} with an \ac{IM} that describes and manages the instrumentation points and (b) define the consistency rules between the IM and source code. The rules define how \vitruv has to respond to the changes in code by adaptive instrumentation of the changed parts with predefined monitoring probes. For the monitoring, we (c) define a \ac{MM} that describes the resulting monitoring records.  These records belong to specific types that are responsible for collecting the necessary monitoring information to calibrate the SEFF elements. Finally, for the incremental calibration and estimation of parametric dependencies, we (d) define the consistency rules between MM and PM, which analyze the monitoring data, calibrate the PMPs and update the deployment and usage parts of PM.
 
With our approach, we can also handle overlapping commits that occur while the monitoring and calibration of a previous commit are still ongoing by managing multiple copies of the \ac{PM} and keeping track of which model instance belongs to which version of the source code.  

\begin{comment}
\end{comment}

\vspace{-0.1 cm}
\section{The Model-based DevOps Pipeline
\label{MbDevOps}
DevOps practices aim to close the gap between the development and operations and to integrate them into one reliable process.
We extend the DevOps practices to integrate and automate the \ac{CIPM} in a \ac{MbDevOps} pipeline. This enables \ac{AbPP} during DevOps-oriented development. The next paragraphs explain the  pipeline processes. The lower case numbers refer to the process's number in Fig. \ref{pipeline}.

\begin{comment}
\ac{MbDevOps} integrates CIPM in the development stage to update the architectural PM and instrument the code (\nth{1} CIPM activity). Then, \ac{MbDevOps} calibrates PM (\textit{Dev-time calibration}) using measurements monitored (with CIPM \textit{Monitoring}) by performing only the necessary performance tests. After the self validation of PM (\nth{4} CIPM activity), one will have an up-to-date PM to predict the performance and answer the what-if questions (e.g., how performance would be if  we use specific resources). This achieves the first goal of \ac{MbDevOps}: An up-to-date PM can speed up the evaluation of performance requirements and can lower the effort and cost needed to set up the test environment to perform such prediction using performance tests.
\end{comment}
\begin{comment}
After the deployment in production environment, \ac{MbDevOps} monitors the system in specific time intervals during the operation (CIPM \textit{Monitoring}), and updates the extracted PM continuously: potential re-calibration of some PMPs (the first part of CIPM \textit{calibration} activity) and responding to run-time changes, e.g., changes in usage or available resources (\textit{Ops-time calibration}). This will provide an up-to-date run-time PM and achieves the second goal of MbDevOps: supporting the improvement and planning stages of \ac{CSE};  performing  model-based experiments to  assure the performance  will support the improvement, e.g., experiments to support model-based auto scaling. Additionally, having an up-to-date descriptive PM supports the planning of the next version (first stage of \ac{CSE}). This is due to the advantages of models, like increasing the understandability of the current version, modelling and evaluating different design alternatives, and answering what-if questions.  
\end{comment}

The \ac{MbDevOps} pipeline starts on the ``development'' side with \(continuous\:integration_{1}\:(CI)\)   process \cite{ci1} that merges the source code changes of the developers (cf.~ Fig. \ref{pipeline}). \vitruv responds to the changes in source code with the first CIPM activity: \(updating \:the \:architectural\;PM_{1.1}\)  and \(generating\: the\: required\: instrumentation\: points\:in\:IM_{1.2}\)  (cf. Sec. \ref{co-evaluation approach}).
Next, \(adaptive\: instrumentation_{2}\) instruments the changed parts of source code using the instrumentation points from \ac{IM} (Sec. \ref{adaptive instrumentation}). The following process is the \(performance\:testing_{3}\), which integrates the second CIPM activity 'Monitoring' to generate the necessary measurements for calibration. The pipeline divides the measurements into training and validation set. Afterwards, the \(Dev{-}time\: calibration_{4}\) (the first part of \nth{3} CIPM activity) enriches the \ac{PM} with \ac{PMPs} using the training set. \Cref{incremental calibration} describes the incremental calibration process whereas \Cref{parametric dependencies} explains how we parametrize the \ac{PMPs} with the influencing input parameters. After the calibration, the pipeline starts  the \(self{-}validation\:with\:test\:data_{5}\) process (the \nth{4} CIPM activity), which uses the validation set to evaluate the calibration accuracy.
If the model is deemed accurate, developers can  use the resulting \ac{PM} to answer the performance questions using \(AbPP_{6}\). If not, they can change the test configuration to recalibrate \ac{PM}. Answering the performance questions using \ac{AbPP} instead of the test-based performance prediction reduces both  the effort and cost of setting up the test environment to perform this prediction.
 
The ``operations'' side of \Cref{pipeline} starts on the \(continuous\: deployment_{7}\) in the production environment. CIPM \(Monitoring_{8}\) in the production environment generates the required run-time measurements for the  next process: \(Ops{-}time\: calibration_{9}\) (the second part of the \nth{3} CIPM activity). The ops-time calibration calibrates and updates both usage model and deployment model (Sec. \ref{Run-time calibration}). Next, the \(self{-}validation\:with\:monitoring\:data_{10}\)     (the \nth{4} CIPM activity) validates the estimated \ac{PMPs}. If the \ac{PM} is not accurate enough, the \(adaptive\:recalibration_{11}\) process  (Sec. \ref{incremental calibration})  recalibrates the inaccurate model parts using monitoring data. 
Finally, the developers can perform more \(model-based\: analysis_{12}\) on the resulting model, e.g., model-based auto scaling. Additionally, having an up-to-date descriptive PM supports the \(development\) \(planning_{13}\). This is due to the advantages of models:  increasing the understandability of the current version, modelling and evaluating design alternatives and answering what-if questions.
\vspace{-0.3 cm}
\section{Adaptive Instrumentation}
\label{adaptive instrumentation}
The goal is to instrument the parts of source code, which have been changed and their changes may affect the validity of related PMPs. In our running example, only the source code of 'trainForRecommender' is instrumented to provide the required measurements to update the RD, whereas there is no need to instrument the loop or the code of ‘preprocess’, because these parts of code are not changed, consequently the old estimations of the PMPs remain valid.

To automate the adaptive instrumentation,  we use the consistency rules between the source code model and IM, which responds to changes in method body (e.g., adding/ updating statements) by instrumenting the related parts. The rules reconstruct the \ac{SEFF} using a reverse engineering tool  \cite{krogmann2012ashort} and map code statements to their \ac{SEFF} element (e.g., internal actions). 
Then, they create the required probes (e.g., service probe, internal action probe, loop probe or branch probe) that refer to the \ac{SEFF} elements whose code statements have been changed. 
We define the following monitoring record types that are related to the aforementioned probe types  using \ac{IRL}  (cf. Section~\ref{kieker}).


\begin{itemize}
\item Service call record to monitor the following:\\
- the input parameters properties (e.g., type, value, number of list elements, etc.) that should be considered later as candidates for parametric dependency investigation,\\
- the caller of this service execution to learn the parametric dependency between the input parameters of both caller and callee services.\\
- the allocation context that captures where the component offering this service is deployed.
\item Internal-action record type to monitor the response time of the internal actions.
\item Loop record to monitor the number of loop iterations.
\item Branch record to monitor the selected branch.
\end{itemize} 
More details about the records types are  in \cite[Chapter~4.3.3]{jaegers}.


Finally, we implemented a model-based instrumentation to generate the instrumented source code as a \vitruv view. This view combines the information from two models: the source code model and the \ac{IM}.
The instrumentation starts with generating the instrumentation code for each probe in the IM according to the probe type. Then it injects the instrumentation codes into a copy of the source code. To detect the correct places for the instrumentation codes, the instrumentation process uses the relations stored in \vitruv correspondence model, i.e., the relation between probes and SEFF elements, the relation between SEFF elements and their source code statements and relation between the original source code and the generated copy (instrumented source code).

\vspace{-0.1 cm}
\section{Incremental Calibration}
\label{incremental calibration}
The following subsections explain the calibration types.
\vspace{-0.1 cm}
\subsection{\ac{Dev-time} calibration}
\label{Dev-time}
The \ac{Dev-time} changes that we consider in this paper are the source code changes that may have an impact on performance, i.e., changes in a method body


On one hand the \textit{incremental} calibration of the SEFF branches, loops or external call arguments requires only to instrument the related source code and to analyze the resulting measurements (e.g., loop iteration number, the selected branch transition, and the values of external call parameters). The goal of this analysis is to detect whether there are dependencies to the service input parameters and express these PMPs sequentially as stochastic expression (cf. \Cref{loop,branches,arguments}).

On the other hand, the incremental calibration of the internal actions with \ac{RD} is challenging because we aim to estimate the \ac{RD} of internal actions incrementally without high monitoring overhead. The existing \ac{RDE} approaches either estimate the RDs at the service level  \cite{spinner2015evaluating} or require expensive fine-grained monitoring \cite{Brosig2009Automated,krogmann2009ck}. Therefore, we propose in the following paragraph a light-weight \ac{RDE} process that is based on adaptive instrumentation and monitoring to allow for an incremental \ac{RDE}.

Our incremental \ac{RDE} aims to estimate the \ac{RDs} in the case of adaptive monitoring, i.e., monitoring the changed parts of source code. In our running example, monitoring the code of ‘trainForRecommender’ to reestimate  its RD. For this goal, we extend the approach of Brosig et al.  \cite{Brosig2009Automated}.   
\paragraph {Basis: Non-incremental estimation of resource demands}
Brosig et al. approximate the \ac{RDs} with measured response times in the case of low resource utilization, typically 20\%. Otherwise, they estimate the \ac{RD} of internal action \(i\) (a part of SEFF, see \Cref{palladio}) for resource \(r\) (\(D_{i,r}\)) based on service demand law \cite{menasce2004} shown in equation (1). Here, \(U_{i,r}\) the average utilization of resource r due to executing internal action \(i\) and \(C_{i}\) is the total number of times that internal action \(i\) is executed during the observation period of fixed length \(T\):

\begin{equation}
D_{i,r} = \frac{U_{i,r}}
{{C_{i}}/{T}} = \frac{U_{i,r} \cdot T }
{C_{i}}
\end{equation}
Brosig et al. measure the \(C_{i}\) and estimate \(U_{i,r}\) by using the weighted response time ratios of the total resource utilization, which is not applicable in our adaptive case where not all internal actions are monitored. Therefore, we extend their approach to estimate \(U_{i,r}\) and as a result \(D_{i,r}\) based on the available measurements and the old \ac{RDs} estimations.

\paragraph{Incremental estimation of resource demands}
\label{ird}
Our new approach distinguishes internal actions into two categories based on whether they have been modified in the source code commit preceding the incremental calibration. We denote internal actions whose corresponding code regions have been modified in the preceding source code commit as \ac{MIAs}, e.g., '\textit{trainForRecommender}' in our running example, -- for these code regions, the consistency rules will generate instrumentation probes and the adaptive monitoring  produce monitoring results. We denote internal actions whose corresponding code regions have not been changed in the preceding source code commit as \ac{NMIAs}, e.g., '\textit{preprocess}' in our running example -- monitoring data for these code regions has already been observed in a previous iteration and, consequently, we have already an estimation of their RDs.

 Based on the fact that the total utilization \(U_{r}\) is measurable and the utilization due to executing \ac{NMIAs} can be estimated based on the old estimations of RDs, we can estimate \(U_{r, MIAs}\) and estimate the RD for each internal action \(i \in MIAs\) accordingly as it will be explained in following paragraphs.



To estimate (\(U_{r, NMIAs}\)),  we  estimate which internal actions \(nmi \in NMIAs\) are processed in this interval and how many times $nmi$ are called (\(C_{nmi}\)). For that, we analyze the service call records (see \Cref{adaptive instrumentation}) to determine which services are called in an observation period \(T\) and which parameters are passed. Then we traverse the service's control flows (i.e. their SEFFs) to get NMIAs and predict their RD using the input parameters. This requires evaluating branches and loops of the control flow to decide which branch transition we have to follow and how many times we have to handle the inner control flow of loops. 
Our calibration, adjusts the new or outdated branches and loops using the monitoring data (as will be described in Sections \ref{loop} and \ref{branches}) before starting this incremental RDE. Thus, we make sure that we can traverse the SEFFs control flow. Consequently, we can sum up the predicted RDs for all calls of the \ac{NMIAs} and divide the result by \(T\) to  estimate the \(U_{r,NMIAs}\) based on the utilization law as shown in the equation 2:
\begin{equation}
U_{r,NMIAs} = \frac{ \sum\limits_{nmi \in NMIAs} \: \sum\limits_{k \leq C_{nmi}} { D_{nmi_{k},r}}}{T}
\end{equation} 

 Accordingly, we estimate the utilization due to executing the \ac{MIAs} (\(U_{r,MIAs}\))  using the measured \(U_{r}\) and the estimated \(U_{r,NMIAs}\) as shown in equation (3):
\begin{equation}
U_{r,MIAs}= U_{r}-U_{r,NMIAs} 
\end{equation}

Hence, we can estimate the utilization \(U_{i,r}\) due to executing each internal action \(i  \in MIAs\) using the weighted response time ratios as shown in equation (4), where \(R_{i}\) and \(C_{i}\) are the average response time of \(i\) and its throughput. \(R_{j}\) is the average response time of the internal action \(j \in MIAs\)  and \(C_{j}\) is the number of executing it in \(T\). 
\begin{equation}
U_{i,r} =U_{r,MIAs} . \frac{R_{i} \cdot C_{i}}
{ \sum\limits_{j \in MIAs} {R_{j}} \cdot C_{j} } 
\end{equation}

Using \(U_{i,r}\) we can estimate the resource demand for  \(i\) (\(D_{i,r}\)) based on the service demand law presented in equation (1).


In the case that the host has multiple processors, our approach uses the average of the utilizations as \(U_{r}\).

Note that we assume that each internal action is dominated by a single resource. If this is not the case, we have to follow the solution proposed by Brosig et al., to measure processing times of individual execution fragments, so that the measured times of these fragments are dominated by a single resource \cite{Brosig2009Automated}.    
To differ between the \ac{CPU} demands and disk demands, we suggest detecting the disk-based services in the first activity of CIPM using specific notation or based on the used libraries. 

 \vspace{-0.1 cm}
\subsection{Ops-time calibration}
\label{Run-time calibration}
The task of the Ops-time calibration is to update the usage models as well as the deployment model according to the run-time measurements.
To achieve that, we use the iObserve approach, which analyzes the measurements and aggregates them to detect changes concerning the deployments and the user behavior.
For this, we extended our monitoring records so that all information required by iObserve is available. This allows us to integrate the usage model extraction and the identification of deployments from iObserve. We do not need the \ac{RAC}  from iObserve, because the mapping information is implicitly presented in our monitoring records, e.g., the records that track the execution of a service include explicitly the ID of the associated service in the architectural model.
\vspace{-0.1 cm}
\section{Parametric dependencies}
\label{parametric dependencies}
This work estimates how \ac{PMPs} depend on input data and their properties (such as number of elements in a list or the size of a file).
We begin by estimating the dependencies of branches and loops, because the incremental \ac{RDE} require traversing the SEFF control flow to estimate the utilization of \ac{NMIAs}.
Currently, we investigate a linear, quadratic, cubic and square root relations using Weka library \cite{Hall2009}. However, we are working on optimizing our estimation using genetic algorithms. The following sections   explain the analysis that we follow to estimate the parametric dependencies of \ac{PMPs}.


\vspace{-0.1 cm}
\subsection{Resource demands}
\label{rd}
\begin{comment}
As described in \Cref{ird} we first collect the \ac{RDs} of \(NMIAs\) to estimate their resource utilization (\(U_{NMIAs}\)). To achieve that, we iterate through all service call records and traverse their control flow.  If we come across a branch, we evaluate the estimated branch transition using the input parameters to decide which branch transition we have to follow in the next step. Similarly, if we get a loop, we evaluate also its predicted iterations count model  by the input parameters to estimate how many times we have to handle its inner control follow. We skip the external calls, because we iterate anyway all service calls in the given observation time \(T\). Every time we get a parametrized internal action, we evaluate its RD model.
Consequentially, we can estimate the resource utilization of them (\(U_{NMIAs}\)) as described in \Cref{ird}.
\end{comment}
To learn the parametric dependency between the resource demand of an internal action \(i\) and input parameters \(P\), we first estimate the resource demand on resource \(r\) for each combination of the input parameters (\(D_{i,r}(P)\)) using the proposed incremental RDE as described in \Cref{ird}. 


Second, we adjust the estimated \ac{RDs} of an internal action using the processing rate of the resource, where it is executed, to extract the resource demand independently of the resource' processing rate  \(D_{i}(p)\). 

Third, if the input parameters include enumerations, we perform additional analysis to test the relation between RDs and enumeration values using decision tree. If a relation is found, we build a data set for each enumeration value. Subsequently, we perform the regression analysis as it will be described in the following paragraph. Otherwise, we create one data set for each internal action that includes the estimated \ac{RDs} and their related numeric parameters. The goal is to find the potential significant relations by the regression analysis of the following equation:
\vspace{-0.15cm}

\begin{gather*}
D_{i}(P) = (a*p_0 + b*p_1 + \dotsb + z*p_n + \\
\qquad \qquad a_1*p_0^2 + b_1*p_1^2 + \dotsb + z_1*p_n^2 +\\
\qquad \qquad a_2*p_0^3 + b_2*p_1^3 + \dotsb + z_2*p_n^3  +\\ 
\qquad \qquad \qquad \qquad a_3*\sqrt{P_0} + b_3*\sqrt{p_1} + \dotsb +z_3*\sqrt{p_n}  +  \textbf{C}) \vspace{-0.1cm}


 \end{gather*}

\( p_{0}\), \( p_{1}\).. \( p_{n}\) are the numeric input parameters and the numeric attributes of objects that are input parameters.

\(a\)... \(z\), \(a_1\)... \(z_1\), \(a_2\)... \(z_2\) and \(a_3\)... \(z_3\) are the weights of the input parameters and their transformations using quadratic, cubic and square root functions. \(C\) is a constant value.

Fourth, we perform the regression analysis algorithm to find the weights of the significant relations and the constant \(C\).

Fifth, we replace the constant value \(C\) with a stochastic expression that describes the empirical distribution of \(C\) value instead of the mean value delivered by the regression analysis. This step is particularly important when no relations to the input parameters are found. In that case, the distribution function will represent the \ac{RD} of internal action better than a constant value. To achieve that, we iterate on the resulting equation that includes the significant parameters and their weights, to recalculate the value of \(C\) for each \ac{RD} value and their relevant parameters. Then, we build a distribution that represents all measured values and their frequency. 

Finally, we build the stochastic expression of \ac{RD} that may include the input parameters  and the distribution of \(C\). 

\vspace{-0.2 cm}\subsection{Loop iterations count}
\label{loop}
To estimate how the number of loop iterations depends on input parameters, we need both the loop iterations' count and the input parameters for each service call. To achieve that, we use our loop records that log the loop iterations' count, every time a loop finishes (see \Cref{adaptive instrumentation}). These records refer to the service call record that contains the input parameters.

The reason why we use additional records to count loop iterations instead of counting the total amount of enclosing service calls, is that the loop may have a nested branch or loop, which does not allow one to infer the correct count of loop iterations \cite{jaegers}.

To estimate the dependency of the loop iteration count on input parameters, we
combine the monitored loop iterations with the integer input parameter into one data
set. To do so, we filter out all non-integer parameters and take into account their integer properties like number of list elements or size of files. Then, we add transformations (quadratic and cubic) of parameters to the data set to test more relations. Finally, we use regression analysis to estimate the weights of the influencing parameters. Due to the restriction that the loop iterations count is an integer number, we have to ensure, that the output value is always an integer value, which is not always the case. Therefore, we have to approximate the non-integer weights or express them as a distribution of integer values. For instance, we can express the value 1.6 using a Palladio distribution function of an integer variable which takes the value \(1\) in \(40\%\) of all cases and the value \(2\) in \(60\%\) of cases. 
Similar to the fifth step of parametrized \ac{RDE} in \Cref{rd}, we replace the constant value of the resulting stochastic expression with an integer distribution function. This will be especially useful, when no relation to the input parameters is found. 
\begin{comment}
To estimate the parametrized loop iterations count, we need both the loop iterations' count and the input parameters by each execution call. To achieve that, we use the predefined loop records that log the loop iterations' count, every time a loop finishes (see \Cref{adaptive instrumentation}). These records have a reference of the service call record that contains the input parameters.

The reason why we use additional records to count loop iterations instead of counting the total amount of enclosing service execution, is that the loop may have a nested branch or loop, which does not allow one to infer the correct count of loop iterations \cite{jaegers}.

To estimate the dependency of a loop in relation to input parameters, we
combine the monitored loop iterations with the integer input parameter into one data
set. Therefore, we filter out all non-integer parameters and take into account their integer properties like number of list elements or size of files. Then, we add transformations (quadratic and cubic) of parameters to the data set to test more relations. Finally, we use regression analysis to estimate the weights of the influencing parameters. Due to the restriction that the loop iterations count is an integer number, we have to ensure, that the output value is always an integer value, which is not always the case. Therefore, we have to approximate the non-integer weights or express them as a distribution of integer values. For instance, we can express the 1.6 value using Palladio distribution function of an integer variable which takes the value \(1\) in \(40\%\) of all cases and the value \(2\) in \(60\%\) of cases. 
Similar to the fifth step of parametrized \ac{RDE} in \Cref{rd}, we replace the constant value of the resulting stochastic expression with an integer distribution function. This will be especially useful, when no relation to the input parameters is found. 
\end{comment}
\vspace{-0.2 cm}
\subsection{Branch transitions}
\label{branches}
To estimate the parameterized branch transitions, we use the predefined branch monitoring records that log which branch transitions are chosen in addition to a reference to the enclosing service call.

We monitor each branch instead of predicting the selected transition according to the external call execution enclosed in the branch due to potential nested control flows (e.g., nested branches), where we cannot infer the selected transition
\cite{jaegers}.

The used monitoring records allow us to build a data set for each branch, which includes the branch transitions and the input parameters.
To estimate the potential relations, we use the J48 decision tree of the Weka library, an implementation of the C4.5 decision tree \cite{Quinlan:1993:CPM:152181}. We filter out the non-significant parameters based on cross-validation. Finally, we transform the resulting tree into a boolean stochastic expressions for each branch transition. 
If no relation is found, the resulting stochastic expression will be a boolean distribution representing the probability of selecting a branch transition.

\vspace{-0.2 cm}\subsection{External call arguments} 
\label{arguments}
This step predicts the parameters of an external call in relation to the input parameters of the calling service.

For each parameter of an external call, we check whether it is constant, identical to one of input parameters, or depend on some of them. To identify the dependencies, we  apply linear regression in the case of numeric parameter and build a decision tree in the case of boolean/ enumeration one. For the remaining  types of parameters, we build a discrete distribution.

Because the relation between input parameters and external calls parameters may be more complex, we are working on optimizing our estimation  using a genetic search similar to the work of Krogmann et al. \cite{krogmann2009ck}.

\section{Conclusion and future works}
\label{Conclusion}
Applying \ac{AbPP} in the agile and DevOps process promises to detect performance problems by simulation instead of the execution in a production environment. 
We presented the continuous integration of the architectural performance model in the development process based on the static and dynamic analysis of the changed code. Our approach keeps the PM continuously up-to-date. We also presented the \ac{MbDevOps} pipeline which automates the incremental calibration process. 

Our calibration estimates the parametrized PMPs incrementally and uses a novel incremental resource demand estimation based on adaptive monitoring. Moreover, our calibration updates both the usage and deployment models to support model-based run-time performance management like auto-scaling.

We tested the functionality using two case studies.
The evaluation showed the ability to calibrate the \ac{PM} incrementally. We evaluated the accuracy of our calibration by comparing the simulation results with the monitoring data. In these case studies, the incrementally calibrated models and their parametric dependencies were accurate and the used adaptive monitoring significantly reduced the monitoring overhead and the calibration effort.

In this work, we have automated only the part of the proposed \ac{MbDevOps} pipeline that is responsible for testing, incremental calibration and self-validation.

In future work, we aim to automate the whole pipeline to integrate the \ac{CI} process with \vitruv.
 Moreover, we will extend the available approaches to integrate existing source code and models into \vitruv platform \cite{leonhardt2015a, Mazkatli2018MiSE}.

 Besides, we will extend our parametrized calibration to test for more complex dependencies of external call parameters. It is also planned to extend the monitoring records to update the system model (composition of the components) automatically. 


\section{Evaluation}
\label{evaluation}

This section introduces the evaluation goals, setup, environment and the results of the evaluation:
\vspace{-0.3 cm}
\subsection{Evaluation Goals}
The evaluation considers three aspects of our approach:
\begin{itemize}[leftmargin=*]
    \item Accuracy of the incremental calibration: We calculate the accuracy of the calibrated models by using the monitoring data as reference and also consider an evolution scenario to include multiple calibration iterations. In particular, we analyze whether the parametric dependencies are recognized by the calibration pipeline.
    \item The time the calibration pipeline needs to derive changes and apply them to the model.
    \item Quantification of the overhead introduced by the monitoring. This allows us to make statements on whether the approach can be applied in practice.
\end{itemize} 

\vspace{-0.4 cm}
\subsection{Experiment Setup}
\label{evaluation:setup}
The general procedure for evaluating our approach is structured as follows:\\
    1. We monitor the application over a defined period of
    time (e.g., 60 minutes) and perform a load test at the same time to artificially simulate user interactions. \\
    2. We run the monitoring in combination with the load test two times. First, we apply a fine-grained monitoring, whose results are passed to the
    calibration pipeline for adjusting the architecture model (training set). Second, we only observe service calls (coarse-grained monitoring). These data is used to estimate the accuracy of the calibrated model (validation set). This procedure prevents the monitoring overhead from falsifying the evaluation results. During the calibration we also recorded the execution times of the different parts in the pipeline and the overhead which is caused by the monitoring. \\
    3. We adjust the usage model according to the validation set and perform a fixed number of simulations of the updated architecture
    model (100 repetitions). A single execution would not be sufficient here, since the simulation samples from stochastic distributions and one result may not be representative. Afterwards we compare the results of the
    simulations with the validation set. Both are distributions,
    therefore we use the \ac{KS-Test} \cite{KSTest}, the Wasserstein metric \cite{majewski_et_al:LIPIcs:2018:9327} and conventional statistical measures, such as the quartiles, to determine the 
    similarity. The lower the \ac{KS-Test}, the higher is the accuracy of the updated architecture model. We are aware that the KS test is not the optimal choice in our specific case, which is why we also used alternative metrics such as the Wasserstein distance to confirm the results. The Wasserstein metric, on the other hand, is rather a distance measure that quantifies the effort needed to transfer one distributions into the other. If we want to compare the accuracy of two simulation distributions (using the monitoring data as reference), this metric is very well suited, but as an absolute number it is difficult to interpret. Therefore, we just used this metric as an additional measure to confirm the results.
    \begin{comment}
    The self-validation executes this step automatically, but only one simulation is triggered. Therefore, we show more reliable results (the results of 100 simulations using an additional validation set) supplementary to the self-validation results.
    \end{comment}
\input{externals/seff_booksale.tex}
\vspace{-0.3 cm}
\subsection{Evaluation Environments}
We evaluated our approach using two case studies: \ac{CoCoME}~\cite{cocome} and
TeaStore \cite{teastore}. %
 \ac{CoCoME}  is a trading system which is designed for the use in supermarkets. It supports several processes like scanning products at a cash desk or processing sales using a credit card. We used a cloud-based implementation of \ac{CoCoME} where the enterprise server and
the database run in the cloud.
\begin{comment}
TeaStore is a micro-service based reference application
\cite{teastore}. It implements a web shop where you can buy
different kinds of tea. Besides, the TeaStore supports user
authentication and automatic recommendations. Amongst others, it is designed to
evaluate approaches in the field of performance modelling \cite{teastore}.


\end{comment}

Additionally, we evaluated our approach using the aforementioned running example of  TeaStore case study (cf. \Cref{Running Example}). 
Because of the fact that \ac{CoCoME} and TeaStore are not implemented using the \vitruv platform, we instrument the changed parts of the source code manually. In future work, we will integrate earlier approaches \cite{leonhardt2015a} into the current \vitruv platform to import existing source code \cite{leonhardt2015a} into \vitruv VSUM, which allows using the automatic change-based adaptive instrumentation.
\subsection{Model Accuracy Evaluation}
\label{evaluation:accuracy}


First, with the help of CoCoME, we validated the basic functionality and performed initial measurements to assess the accuracy of the adapted models. In the second step, we used the TeaStore case study to evaluate the incremental calibration and ensure that parametric dependencies were considered in the generated models.
\subsubsection{CoCoME case study}
In the evaluation scenario using \ac{CoCoME}, we suppose that the ``bookSale'' service  is newly added. According to our assumption, we instrument this service and all services
that are subsequently called by it. 
The ``bookSale'' service is responsible for processing a sale after a user submitted his payment. As input, this service receives a list of the purchased products including all information related to the purchase. In the next step, we performed a load test which simulates user purchases. We used the resulting monitoring data to calibrate our initial architecture model.

The service consists of several internal and external actions and two loops. We also monitored and calibrated the behavior  of all external actions to make the resulting model more accurate.
Figure \ref{fig:seff:booksale} visualizes the structure of the calibrated "bookSale" service
of \ac{CoCoME}. For reasons of simplification, we focused on the structure and omitted certain details.
As described in the experiment setup, we performed 100 simulations and
compared the distribution of the monitoring with the results of the simulations.
Table \ref{table:cdf} shows the quartiles
for both the monitoring and the simulation results for a single example execution
\input{externals/cdf_cocome_table.tex}
In addition, a KS test value of \textbf{0.1321} and a Wasserstein distance of \textbf{13.7726} were obtained by comparing the two distributions. It is well visible that the two distributions are very close to each for the first, second and third quartile. The discrepancy in the minimum and maximum values of the distributions arises because we limited the time spent for simulation.
The average of the KS tests over all 100 simulation runs 
is approximately \textbf{0.1467}, the minimum \textbf{0.1248} and the maximum
\textbf{0.1774}. From these values it follows that our approach provided a model which represents the monitored behavior  in our scenario well.

\subsubsection{TeaStore case study}
For the TeaStore case study, we examined the "Recommender" part (cf. \Cref{Running Example}).
\begin{comment}
The Recommender is divided into a service that calculates the
recommendations for a certain shopping cart (recommend service) and a service
that derives information from the previous orders (train service).
\end{comment}
We monitored the "train" service and calibrated it. Furthermore, we modified the application so that the "train" service is executed when a user placed an order. This causes the number of users to have a direct impact on the service response times, as the number of orders in the database grows faster and therefore a higher number of orders have to be processed within the service.

\begin{comment}
Normally the "train" service only receives the past orders and the associated ordered items as input. We
extended it and added a parameter (enum) indicating which Recommender
implementation is used. This allows us to specify a single \ac{SEFF} that is valid for all Recommender implementations.
\end{comment}
Figure \ref{fig:seff:train} shows the simplified layout of the \ac{SEFF}. Except for the last Internal Action, the behavior  is exactly the same for all Recommender types.

We also considered an evolution scenario for the TeaStore to evaluate the incremental calibration. This means that the calibration of the models is executed several times. In our considered evolution step, new Recommender implementations are added or existing ones were removed.

In the following, we compared the accuracies of a parameterized and a non-parameterized model. 
In the second step, we analyzed the behavior and accuracy of the incrementally calibrated models when executing evolutionary scenarios (e.g., Recommender implementations are added or removed).

In order to confirm the recognition of parametric dependencies, we first monitor the service as previously described in section \ref{evaluation:setup} fine granular (F1) and subsequently coarse granular (C1). We \textbf{only} use the resulting fine granular monitoring data (F1) to calibrate two models: one with parametric dependencies using our incremental calibration and one without parametric dependencies using the distributions of the observed values. 
The results confirmed that our calibration was able to detect the parametric dependencies, e.g. the RD of the internal action \textit{trainForRecommender} (the \nth{4} action in train SEFF) is related to the type of recommendation implementation as well as to the number of ordered elements.
To compare the accuracy of the parameterized model with the non-parameterized model, we predict the performance for a what-if scenario, i.e. what the performance would be if the load is changed from 20 concurrent users to 40 concurrent users. To have a baseline, we measure the performance by applying the load of 40 users (performing coarse granular monitoring (C2)). Then we compare the simulation results with the monitoring data, i.e. (C1) as reference for the first load (20 users) and (C2) for the second load. 
Table \ref{table:train:kss} shows the comparison of the KS tests and the Wasserstein distances between the parameterized and the non-parameterized model. It can be seen that both models are very accurate for the standard load of 20 users. However, it can be stated that the non-parametric model loses significantly more accuracy than the parameterized model, when the load changes to 40 users.
\input{externals/comparisons/teastore_kstest_table.tex}
\begin{comment}
We confirmed this first impression by using Wasserstein distance as an alternative metric. We included the results in the Appendix in Figure \ref{fig:comp:ws:train} and we also summarized the results there in tabular form (see table \ref{table:teastore:param:20} and table \ref{table:teastore:param:40}).
\end{comment}

In order to evaluate the incremental calibration, we simulated an evolution scenario for the TeaStore application. At the beginning of the scenario, the TeaStore contains a single implementation of the "train" service. Then we perform 15 minutes of fine-granular monitoring and use the resulting data to calibrate the model. After that we monitor the service for another 15 minutes coarse granular. The coarse granular monitoring data is used as reference for the estimation of the accuracy of the calibrated model. In the next step, we add another implementation of the "train" service and again, just like before, we perform a fine granular and a coarse granular monitoring for 15 minutes each. If all implementations of the "train" service have already been added, one of them will be randomly removed again. We repeat this procedure over several iterations and calculate the accuracy of the calibrated model for each iteration. In addition, the database fills up more and more over time, which leads to increasing response times. This adds an extra level of difficulty to the calibration process.

\input{externals/evolution/evolution_table.tex}
We performed 10 consecutive iterations and calculated the accuracy for each of them. As before, we calculated the KS test and confirmed the results using Wasserstein distance.
Table \ref{table:evol} lists the aggregated results. We omitted the values of some iterations because these were always very similar to the previous ones.
The values are slightly higher than before where we did not apply any changes to the application. However, it can be seen that the accuracy of the models is almost constant throughout the evolutionary steps. From this, it can be concluded that the automated calibration can handle changes of the observed services in the TeaStore.

\vspace{-0.3 cm}
\subsection{Calibration Performance}
For both \ac{CoCoME} and TeaStore we computed how much time the individual parts of the pipeline took to complete. For the TeaStore, we also distinguished between the two states within the evolution scenario (before and after the first evolution step). The results are shown in table \ref{table:exec}.
\begin{table}[h]
    \centering
    \caption{Execution times of the pipeline parts}
    \begin{tabular}{|c|c|c|c|}
    \hline 
    Measure & CoCoME & TeaStore (1) & TeaStore (2) \\ \hline \hline
    Record Count & 220706 & 294237 & 27085 \\ \hline \hline
    Load Records & 4.664s & 5.142s & 0.774s \\ \hline
    RD Calibration & 9.168s & 4.996s & 0.789s \\ \hline
    UM Adjustment & 2.365s & 0.194s & 0.165s \\ \hline
    Self-Validation & 1.729s & 1.440s & 1.327s \\ \hline \hline
    \textbf{Total} & 17.213s & 10.826s & 2.199s \\ \hline
    \end{tabular}
    \label{table:exec}
\end{table}

We can see that the required time strongly depends on the number of monitoring records and the complexity of the observed service. It has already been shown for iObserve that the derivation of the usage model scales linearly with the number of monitoring records \cite{heinrich2015runtime}. In all considered cases, the execution of the whole pipeline took less than 20 seconds.
It can also be observed that the adaptive monitoring limits the number of monitoring records and accordingly the second iteration of the TeaStore monitoring produces far less records, which results in significantly lower execution times.
To gain detailed insights about the performance of the pipeline, we also examined the behavior for an increasing number of monitoring records. For this purpose, we generated monitoring data with the help of \ac{CoCoME} case study and executed the pipeline several times. The results showed that the execution time of the pipeline scales linearly up with the number of monitoring records, plus an additional constant overhead (see Table \ref{table:perf}).
\input{externals/performance_details_table.tex}
\subsection{Monitoring Overhead}
Kieker only introduces a small overhead, the recording and storing of one record takes on average less than 5 microseconds \cite{kieker}. However, due to the fine-grained instrumentation, a single service call can cause a lot of monitoring records.

We also measured the overhead caused by the monitoring. For CoCoME's "bookSale" service the average monitoring overhead was \textbf{1.31ms}  (fine-grained), \textbf{252\boldmath$\mu$s}  (coarse-grained) and for TeaStore's Recommender.train service the measured overhead on average amounts to \textbf{1.81ms}  (fine-grained), \textbf{88\boldmath$\mu$s}  (coarse-grained). We can notice that the coarse-grained monitoring has almost no influence on the execution times. The fine-grained monitoring overhead scales up with the complexity of the service. In our case, the overhead of around 1ms had no drastic impact on the performance, since the execution times of the services are significantly higher.

The evolution scenario described for the TeaStore shows very well that the adaptive monitoring helps to significantly reduce the monitoring overhead. In the first iteration we instrument the statements of all SEFF elements shown in Figure \ref{fig:seff:train} - (1), (2), (3) and (4). For each additional iteration we just instrument the changed internal action (4). In this case we saved \textbf{at least 75\%} of the monitoring overhead. This is reflected in the times required for calibration, which are considerably lower compared to the first iteration (see \autoref{table:exec}).

In general there is a trade off between the monitoring overhead and the accuracy of the model, but our approach tries to balance them by using the incremental calibration in combination with the adaptive monitoring. Consequently, the calibration time and the monitoring overhead can be kept as small as possible.
\section{Foundations}
\label{foundation}
\subsection{Palladio}
\label{palladio}
Palladio is an approach to model and simulate architecture-level \acp{PM} and has been used in various industrial projects~\cite{reussner2016a,becker2008a}. Within Palladio, the so-called \ac{PCM} defines a language for describing \acp{PM}: the static structure of the software (e.g. components and interfaces), the behavior, the required resource environment, the allocation of software components and the usage profile. The \ac{PCM} \ac{SEFF} \cite{becker2008a} describes the behavior  of a component service on an abstract level using different control flow elements: \textit{internal actions} (a combination of internal computations that do not include calls to required services), \textit{external call actions} (calls to required services), \textit{loops} and \textit{branch actions}. SEFF loops and branch actions include at least one external call. Other loops and branches in the service implementation are ignored and combined into the  internal actions to increase the level of abstraction.

To predict the performance measures (response times, \ac{CPU} utilization and throughput) the architects  have to enrich the SEFFs with \ac{PMPs}.
Examples of \ac{PMPs} are resource demands (processing amount that internal action requests from a certain active resource, such as a \ac{CPU} or hard disk), the probability of selecting a branch, the number of loop iterations and the arguments of external calls.

Palladio uses the \ac{StoEx} language \cite{koziolek_modeling_2016} to define \ac{PMPs} as expressions that contain random variables or empirical distributions.
\ac{StoEx} allows parameter characterization (e.g. determining NUMBER\_OF\_ELEMENTS, VALUE, BYTESIZE and TYPE) and to express calculations and comparisons (e.g., file.BYTESIZE$<=$5*max.VALUE).

\input{externals/seff_train.tex}

\subsection{Co-evolution approach with \textsc{\textit{Vitruvius}}\xspace}
\label{vitruv}
\label{co-evaluation approach}
 The co-evolution approach of Langhammer et al. \cite{langhammer2015a,langhammer2017a}   helps software developers and architects to keep the architecture model and the code consistent when their software system evolves. It defines change-driven consistency preservation rules that propagate changes in source code to the architecture model and vice versa using model-based transformations.

These rules are defined based on \vitruv  \cite{kramer2013b,burger2014diss}, a view-based framework that encapsulates the heterogeneous models of a system and the semantic relationships between them in a \ac{VSUM} in order to keep them consistent. 
\vitruv defines \emph{mappings} and \emph{reactions} languages that describe consistency rules on the metamodel-level. These rules describe the consistency repair logic for each kind of changes, i.e., which and how the artifacts of a metamodel have to be changed to restore the consistency after a change in a related metamodel has occurred. 

Using \vitruv, Langhammer et al. implemented their approach to keep Java source code (using an intermediate model \cite{heidenreich2010a})  and a \ac{PCM} model consistent. The defined consistency rules update the structure of a \ac{PCM} model  (i.e. components and interfaces) and its behavior  (in term of \acsp{SEFF}, but \textit{without} \ac{PMPs}) as a reaction to changes in the source code. Similarly, changes in PCM model are propagated to the Java source code.

\subsection{Kieker}
\label{kieker}
Kieker \cite{kieker} is an APM tool that captures and analyzes execution traces from distributed software systems. It allows one to describe specific probes (data structures of the monitored information) using the \ac{IRL}  \cite{jung2013model}. It also supports dynamic and adaptive monitoring by activating or deactivating probes during run time.




\subsection{iObserve}
\label{iObserve}

iObserve \cite{heinrich2017a} aims to increase the human comprehensibility of run-time changes by updating an architecture model accordingly.  The authors defined specific monitoring records using \ac{IRL} and map the resulting measurements to the corresponding parts in the \ac{PCM} using the so-called \ac{RAC} \cite{heinrich2014}. iObserve detects changes concerning the deployment and the user behavior  and updates the related parts of the \ac{PCM} to analyze performance and privacy aspects \cite{heinrich2016b}. However, iObserve does not support updating component behavior  models (SEFFs) with \ac{PMPs}. 


\section{Introduction}
\label{Introduction}
In many application domains, software is nowadays developed iteratively and incrementally, e.g., following agile practices.
This means that there usually is no dedicated architecture design phase, but architectural design decisions are made continuously throughout the development. For performance analysis, this means that there is not a single point in time at which the performance analysis is carried out, but a performance analysis is required whenever performance-critical architecture design decisions have to be made. 

In such iterative and incremental development projects, developers typically use \ac{APM} \cite{apm,BeEiFeGrRhJaShHoViWaWi2019-ICPE-DevOpsSurvey} to assess the current performance of their software. However, \ac{APM} cannot predict the impact of design decisions or to answer scalability and sizing questions in a large and distributed environment, where the resources may not be accessible and a huge amount of load generators would be needed. 

Software  performance  engineering \cite{SPEwoodside,conni} uses models to assess the performance of a software system. It supports the evaluation of various architecture, design, implementation and workload choices. Compared to \ac{APM}, such model-based performance prediction \cite{mbpp} allows to predict future alternatives before implementing them \cite{conni}.
In particular, \ac{AbPP} approaches~\cite{reussner2016a} reflect the architecture of the software system in a \ac{PM} in order to easily study design alternatives~\cite{martens2010c}. 

The problem of \ac{AbPP} in iterative-incremental development is that modelling is a time-consuming process. In particular, calibrating \ac{PMPs} (e.g., \ac{RDs}, branch probabilities, and loop counts) is challenging, because the \ac{PMPs} may depend on impacting factors such as input data, properties of required resources or usage profile. Ignoring these so-called \emph{parametric dependencies}~\cite{becker2008a}) will lead to inaccurate performance predictions. Thus, keeping the \ac{PM} and in particular the parametrized \ac{PMPs} consistent with the iteratively and incrementally evolving source code over time requires repeated manual effort.

To address the high effort to create performance models when source code is available, researchers have suggested several approaches to extract \acp{PM} automatically.
However, most of these approaches \cite{BrHuKo2011-ASE-AutomExtraction,WaStKoKo2017-QUDOS-PMXBuilder, Brunnert13,langhammer2016automated,spinner2016reference} have three shortcomings: First, they require instrumentation and execution  of the whole system under study to extract the \ac{PM}, which causes a high overhead and is not feasible at high frequency, e.g., after each source code commit. Second, they do not consider parametric dependencies. 
Notable exceptions are the approach by Krogmann et al.~\cite{krogmann2009ck,krogmann2012ashort} or Grohmann et al.~\cite{GrEiElMaKiKo2019-MASCOTS-DependencyIdentification}, which also extract parametric dependencies (but monitor the whole system to calibrate the model without keeping the manual changes).
Third, they reconstruct the whole architectural PM from scratch. Thus, they cannot keep manual changes of the architecture model, which is problematic if the architecture model shall be used as an architecture knowledge base and is enhanced manually with e.g. the rationale of architecture decisions~\cite{kramer2012b}. 

In this paper, we present our approach, called \ac{CIPM}, to incrementally extract and calibrate architecture-level \acp{PM} with parametric dependencies after each source code commit. 
Our approach builds upon the Palladio approach~\cite{reussner2016a} for modelling and simulating architecture-level \acp{PM}. We have extended an incremental extraction of such architectural \acp{PM}~\cite{langhammer2017a} with incremental calibration that considers the parametric dependencies and is based on adaptive monitoring. The goal of the approach is to keep the \ac{PM} up-to-date automatically to allow \ac{AbPP}. An initial version of this idea has been presented in a workshop publication~\cite{Mazkatli2018Qudos} without evaluation.

The contributions of our paper are twofold: 
\begin{enumerate} 
\renewcommand{\labelenumi}{(\Alph{enumi})}
\item \textit{Incremental Dev-time calibration:} We propose a novel incremental calibration at \ac{Dev-time}  that responds to source code changes by adaptive instrumentation of the changed parts of the code and uses the resulting measurements from performance tests or the production system to estimate the \ac{PMPs} incrementally.

For this purpose, we propose a novel incremental \ac{RDE} that is based on adaptive monitoring.
Our calibration uses statistical analysis to learn potential dependencies, e.g., regression analysis for resource demands and decision trees for the estimation of  branch transitions.

\item \textit{Model-based DevOps pipeline:}
We extend agile DevOps practices \cite{brunnert2015a}  to integrate the \ac{CIPM} activities and thus to reduce the effort for \ac{AbPP}. 
To do so, we propose a model-based DevOps pipeline and implement a part of it.
In addition to the above-mentioned incremental \ac{Dev-time} calibration, we also include an existing approach for calibration at \ac{Ops-time} in the pipeline. This \ac{Ops-time} calibration responds to changes in deployment and usage profile and updates the respective parts of the \ac{PM}. We include also prototypical self-validation steps.




\end{enumerate}
We evaluate our contributions using two case studies \cite{cocome,teastore}.
The evaluation confirmed that the incremental calibration was able to detect parametric dependencies while significantly reducing the monitoring overhead. Moreover, we showed that the calibrated performance models are accurate by comparing the simulation results with monitored data.


The next section gives an overview of the foundations. \Cref{Running Example} introduces a motivating example. The overall CIPM process is presented in \Cref{CIPM}. \Cref{MbDevOps} describes how to embed CIPM in the model-based DevOps pipeline. The next three sections provide detail on the CIPM activities: \Cref{adaptive instrumentation} presents the adaptive instrumentation, \Cref{incremental calibration} describes the incremental calibration and \Cref{parametric dependencies} describes how we estimate the parametric dependencies. Our evaluations are discussed in \Cref{evaluation}. The related work follows in \Cref{related works}. The paper ends with our conclusions and future work 
(\Cref{Conclusion}).
 
\section{Related Work}
\label{related works}
A number of approaches for constructing the architectural model based on static (e.g., \cite{becker2010reverse}), dynamic, or hybrid analysis exist.
Walter et al. \cite{WaStKoKo2017-QUDOS-PMXBuilder} propose a  tool to extract an architectural PM as well as performance annotation based on analysing monitoring traces.
Similarly, other works \cite{Brunnert13,spinner2016reference,BrHuKo2011-ASE-AutomExtraction} extract PM based on dynamic analysis. Krogmann et al.  \cite{krogmann2009ck,krogmann2012ashort} extract parametrized PCM  based on hybrid analysis.
Langhammer et al. introduce two reverse engineering tools \cite [P.~140]{langhammer2017a} \cite{langhammer2016automated}  that extract the behavior  of the underlying source code based on hybrid analysis.

The above-mentioned approaches do not support the iterative extraction of  the architectural PMs or the incremental calibration. If they were used in an iterative development, they would re-extract and recalibrate the whole  models after each iteration. This would cause a high monitoring overhead. Moreover, this process ignores the potential manual modifications of the extracted PM that may be formerly applied.


 In addition to the work of Krogmann et al. \cite{krogmann2009ck,krogmann2012ashort}, the works of Ackermann et al.~\cite{AcGrEiKo2018-DependencyModeling} and Curtois et al.~\cite{Courtois2000}  also consider the characterization of parametric dependencies in performance models, while Grohmann et al.~\cite{GrEiElMaKiKo2019-MASCOTS-DependencyIdentification} focus on the identification of those from monitoring data. However, our work considers the parametric dependencies during the incremental calibration of the architectural PMs.
 
Model extraction approaches derive the resource demands either based on coarse-grained monitoring data \cite{spinner2015evaluating,LibReDE} or fine-grained data 
\cite{Brosig2009Automated,willnecker2015comparing}. The latter approaches give a higher accuracy by estimating \ac{PMPs} but have a downside effect because of the overhead of instrumentation and the monitoring. Our approach reduces the overhead by the automatic adaptive instrumentation and monitoring.

Spinner et al. \cite{SpGrEiKo2019-JSS-ModelLearning} propose an agent-based approach to updated architectural performance models. In contrast to our work, they focus on model updates at run-time.

\ac{DPE} \cite{WaEiGrOkKo2018-ICPE-Tools-for-DPE-Tutorial} technically enables to answer concerns based on measurement-based performance evaluation and model-based performance predictions. However, existing solutions do not provide a technical integration into the build pipeline. Also, \ac{DPE} does not answer how to keep PMs updated.


\section{Evaluation}
\label{evaluation}

This section introduces the evaluation goals, setup, environment and the results of the evaluation:
\vspace{-0.3 cm}
\subsection{Evaluation Goals}
\label{goals}
The goal is to evaluate the following three aspects of the approach and answer the  Evaluation-Questions (EQs):
\begin{itemize}[leftmargin=*]
\item \textbf{Accuracy of the incremental calibration}: \\
\textbf{EQ1.1:} How accurate are the incremental calibrated PMs?
\\
\textbf{EQ1.2:} Is the accuracy of the incremental calibrated PM stable over the incremental evolution? 
\item \textbf{Accuracy of the parametric dependencies}: \\
\textbf{EQ2:} Does the estimation of parametric dependencies improve the accuracy of PMs?
\item \textbf{Performance of the calibration pipeline:} \\ \textbf{EQ3:} How long does the pipeline need to update PMs?
  \item \textbf{Monitoring overhead:} \\\textbf{EQ4.1:} How much is the required monitoring overhead?
  \\\textbf{EQ4.2:} How much can the adaptive instrumentation reduce the monitoring overhead?

\end{itemize} 
\vspace{-0.2 cm}
\subsection{Experiment Setup}
\label{evaluation:setup}
The evaluation is structured as follows:\\
    1. We monitor the application over a defined period of
    time (e.g., 60 minutes) and perform a load test at the same time to artificially simulate user interactions. \\
    2. We run the monitoring in combination with the load test two times. First, we apply a fine-grained monitoring, pass the results to the
    calibration pipeline for adjusting the architecture model (training set). Second, we only observe service calls (coarse-grained monitoring). These data are used to estimate the accuracy of the calibrated model (validation set). This procedure prevents the monitoring overhead from falsifying the evaluation results. We also record the execution times (overhead) of the pipeline parts
    3. We adjust the usage model according to the validation set and perform a fixed number of simulations of the updated architecture
    model (100 repetitions). Repetitions are necessary, as the simulation samples from stochastic distributions may not be representative. Afterward, we compare the results of the
    simulations with the validation set. Both are distributions,
    therefore we use the \ac{KS-Test} \cite{KSTest}, the Wasserstein metric \cite{majewski_et_al:LIPIcs:2018:9327}, and conventional statistical measures to determine the 
    similarity.
    The lower the \ac{KS-Test} value, the higher is the accuracy of the updated architecture model. In order to limit the dependence on a single metric, we also included the Wasserstein distance to confirm the results. The Wasserstein metric is distance measure that quantifies the effort needed to transfer one distribution into the other. If we want to compare the accuracy of two simulation distributions (using real monitoring as reference), this metric is well suited, but reports absolute numbers that  are  difficult to interpret. We therefore combined both metrics in our analysis.
\vspace{-0.3 cm}
\subsection{Evaluation Environments}
We evaluated our approach using two case studies: \ac{CoCoME}~\cite{cocome} and
TeaStore \cite{teastore}. 
 \ac{CoCoME}  is a trading system which is designed for the use in supermarkets. It supports several processes like scanning products at a cash desk or processing sales using a credit card. We used a cloud-based implementation of \ac{CoCoME} where the enterprise server and
the database run in the cloud.

\begin{comment}
TeaStore is a micro-service based reference application
\cite{teastore}. It implements a web shop where you can buy
different kinds of tea. Besides, the TeaStore supports user
authentication and automatic recommendations. Amongst others, it is designed to
evaluate approaches in the field of performance modelling \cite{teastore}.
\end{comment}

Additionally, we evaluated our approach using the aforementioned running example of the TeaStore case study (cf. \Cref{Running Example}). Compared to the original, we modified the application so that the "train" service is executed when a user places an order. This causes the number of users to have a direct impact on the service response times.

Because \ac{CoCoME} and TeaStore are not implemented using the \vitruv platform, we instrument the changed parts of the source code manually. In future work, we will use approaches for importing existing source code into \vitruv \cite{leonhardt2015a}, which allows the automatic adaptive instrumentation.

\vspace{-0.2cm}
\subsection{Evaluation of the Incremental Calibration }
\label{Evaluation incremental Calibration}
This section evaluates the accuracy of our calibration for the following incremental evolution scenarios.
\subsubsection{BookSale Calibration}
The ``bookSale'' service of CoCoMe case study is responsible for processing a sale after a user submitted his payment. As input, this service receives a list of the purchased products including all information related to the purchase. The service consists of several internal and external actions and two loops.
Figure \ref{fig:seff:booksale} visualizes the structure of the calibrated "bookSale" service
of \ac{CoCoME}. For reasons of simplification, we focused on the structure and omitted certain details. 
In this evaluation scenario, we supposed that the ``bookSale'' service  is newly added.
According to our assumption, we instrumented this service and all services that are subsequently called by it. 
We performed a load test which simulates user purchases. We used the resulting monitoring data to calibrate our architecture model.

\paragraph{Results}  Table \ref{table:cdf} shows the quartiles for both the monitoring and the simulation results for a single example execution. Besides, a KS test value of \textbf{0.1321} and a Wasserstein distance of \textbf{13.7726} were obtained by comparing the monitoring and the simulation distributions. It is well visible that these  distributions are very close to each for the first, second and third quartile. The discrepancy in the minimum and maximum values of the distributions arises because we limited the time spent for simulation.  The KS tests average over all 100 simulation runs is approximately \textbf{0.1467} (min \textbf{0.1248}, max \textbf{0.1774}). These results confirm the PM accuracy (EQ1.1).
\vspace{-0.1cm}
\input{externals/cdf_cocome_table.tex}

\subsubsection{Train Calibration}
\label{Train Calibration}
\begin{comment}
Similar to \textit{"BookSale"} evaluation scenario, we assume that the first version of 'train' service of "Recommender" (described in \Cref{Running Example}) is newly added.
Therefore we instrument the statements of all SEFF elements shown in Fig. \ref{fig:seff:train} - (1), (2), (3) and (4). Then we monitored and calibrated them.
Due to lack of space and the similarity to the results of the parametric dependency scenario (Table \ref{table:train:kss}, \nth{2} column), we show the accuracy results of train in Sec. \ref{whatIf}\footnote{See \url{https://sdqweb.ipd.kit.edu/wiki/CIPM} for more detail on evaluation.}.

Similar to \textit{"BookSale"} evaluation scenario, we assume that the first version of 'train' service of "Recommender" (cf. \Cref{Running Example}) was newly added.
Therefore we instrument the statements of all SEFF elements shown in Fig. \ref{fig:seff:train} - (1), (2), (3) and (4). Then we monitored and calibrated them.
The resulting Ks Test value (0.18) and Wasserstein value (29.9) confirmed the accuracy of train, which is evaluated again in another scenario in Sec. \ref{whatIf} (see Table \ref{table:train:kss}, \nth{2} column).
\end{comment}
Similar to ”BookSale” evaluation scenario, we assumed that ’train’ service (cf. \Cref{Running Example})  was newly added. Therefore, we instrumented and monitored the statements of all SEFF elements shown in Fig. \ref{fig:seff:train} in two cases: (1) for the first implementation of ’train’ and (2) after adding an enumeration parameter that refers to the used implementation of 'train' to represent the four implementations in one SEFF (cf. Sec. \ref{whatIf}). Then we calibrated the  SEFFs of these two cases. Due to lack of space we show only the accuracy results of the second case in (Table \ref{table:train:kss}, \nth{2} column)\footnote{See \url{https://sdqweb.ipd.kit.edu/wiki/CIPM} for more detail on evaluation.}.
\subsubsection{TrainForRecommender Calibration}
\label{TrainForRecommender}
To evaluate that the accuracy of PM is stable (EQ1.2), we simulated an evolution scenario that assumes that the developers change the implementation of the \textit{'train'} service to examine the four implementation alternatives (cf. \Cref{Running Example}) in four subsequent source code commits $c1$, ..., $c4$. According to our assumption, except of the "trainForRecommender" internal action, the behavior  of \textit{'train'} (\Cref{fig:seff:train}) stays the same.
To evaluate this scenario, we instrumented only the "trainForRecommender" internal action for the first implementation of \textit{'train'} ($c1$).
We performed 15 minutes of fine-granular monitoring and used the resulting data to calibrate the model. Next, we monitored the 'train' service for another 15 minutes of coarse granular. The coarse granular monitoring data was used as reference for the estimation of the accuracy of PM. In the next step,  we instrumented the "trainForRecommender" internal action for the second implementation of \textit{'train'} ($c2$) and performed a fine granular and a coarse granular monitoring for 15 minutes each. We repeated the same procedure for commits $c3$ and $c4$. 

To simulate additional commits, we changed the train implementation by randomly replacing the strategy with another one. We repeated this procedure six more times to simulate six additional iterations and calculate the accuracy of the calibrated model for each iteration. In addition, the database filled up over time, which leads to increasing response times, increasing the difficulty for the calibration process
\vspace{-0.15 cm}
\paragraph{Results}
\input{externals/evolution/evolution_table.tex}
Table \ref{table:evol} lists the aggregated results. 
The values are slightly higher than without applying changes to the application. However, it can be seen that the accuracy of the models is almost constant throughout the evolutionary steps. From this, we conclude that the automated calibration is able to handle changes of the observed services.
\vspace{-0.25cm}
\subsection{Evaluation of the Estimation of Parametric Dependencies}
 \label{whatIf}
This scenario  evaluates the recognition of the parametric dependencies and compares the accuracy of the AbPP using the parameterized model (i.e., calibrated with our approach) with the non-parameterized model (i.e., calibrated ignoring parametric dependencies). For this goal, we extended the SEFF shown in \Cref{fig:seff:train} and added an enumeration input parameter that refers to the recommender type and determines which implementation is used within the "trainForRecommender" internal action. This allowed us to represent all different implementations in one SEFF. This means the RD of "trainForRecommender" can be related to the recommender type. Then, we calibrated the SEFF and predicted the performance for increasing the load from 20  to 40 concurrent users.  
First, we monitored the service fine granular (F1) and subsequently coarse granular (C1) by applying the load of 20 users. We \textbf{only} used the resulting  monitoring data (F1) to calibrate a parameterized model using our approach and the non-parameterized model using the distributions of the observed values. Then, we compared the simulation results of both models with the monitoring data (C1). Then, we performed AbPP for the higher load (40 users). As baseline, we measured the performance using coarse granular monitoring and applying the load of 40 users (C2).  Finally, we compared the AbPP results  of both models with the monitoring data (C2).
\paragraph{Results}
The results confirmed that our calibration was able to detect the parametric dependencies. Hence, the RD of \textit{trainForRecommender} is related to the type of recommendation implementation and the number of elements.

Table~\ref{table:train:kss} shows the comparison of the KS tests and the Wasserstein distances between the parameterized and the non-parameterized model. It can be seen that both models are very accurate for the standard load of 20 users. However, the non-parametric model loses significantly more accuracy than the parameterized model, when the load changes to 40 users. This confirms that the estimation of the parametric dependencies improves the accuracy of PMs (EQ1.1, EQ2).
\vspace{-0.15 cm}
\input{externals/comparisons/teastore_kstest_table.tex}	

\vspace{-0.2 cm}
\subsection{Evaluation of the  MbDevOps Pipeline}
This section evaluates the performance of the implemented part of MbDevOps Pipeline (monitoring, incremental calibration and self-validation) and answers EQ3. 
For that, we computed how long the individual pipeline parts took to complete. We used the three evolution scenarios described in \Cref{Evaluation incremental Calibration}. The results are shown in Table \ref{table:exec}.
\vspace{-0.15 cm}\begin{table}[h]
    \centering
    \caption{Execution times of the pipeline parts}
    \begin{tabular}{|c|c|c|c|}
    \hline 
    Measure & bookSale & train  & trainForRecommender  \\ \hline \hline
    Record Count & 220706 & 294237 & 27085 \\ \hline \hline
    Load Records & 4.664s & 5.142s & 0.774s \\ \hline
    PM Calibration & 9.168s & 4.996s & 0.789s \\ \hline
    UM Adjustment & 2.365s & 0.194s & 0.165s \\ \hline
    Self-Validation & 1.729s & 1.440s & 1.327s \\ \hline \hline
    \textbf{Total} & 17.926s & 11.772s & 3.055s \\ \hline
    \end{tabular}
    \label{table:exec}
\end{table}
\vspace{-0.15 cm}

We can see that the required time strongly depends on the number of monitoring records and the complexity of the observed service. Except for the self-validation, the execution time of all parts of the pipeline depends on the number of records (see Table \ref{table:perf}). In all considered cases, the execution of the whole pipeline took less than \textbf{20} seconds.

We observe that adaptive monitoring limits the number of monitoring records. Accordingly, the second iteration of TeaStore monitoring produces far fewer records, which results in significantly lower execution times.
To gain detailed insights about the performance of the pipeline, we also examined the behavior for an increasing number of monitoring records. For this purpose, we generated monitoring data using \ac{CoCoME}  and the load test that we also used before. Thereafter, we executed the pipeline  several times, with an increasing number of monitoring records as input. The results in Table \ref{table:perf} showed that the execution time of the pipeline scales linearly with the number of monitoring records in this case.
\input{externals/performance_details_table.tex}
\vspace{-0.25 cm}
\subsection{Evaluation of Monitoring Overhead}
To answer the EQ4.1 and EQ4.2, we measured the overhead caused by the monitoring for the incremental calibration scenarios in \Cref{Evaluation incremental Calibration}. 

The average monitoring overhead for \textit{bookSale}  was \textbf{1.31ms}  (fine-grained), \textbf{252\boldmath$\mu$s}  (coarse-grained) and for \textit{train}  \textbf{1.81ms}  (fine-grained), \textbf{88\boldmath$\mu$s}  (coarse-grained). We note that the coarse-grained monitoring has a negligible influence. The fine-grained monitoring overhead scales with the complexity of the service. In our case, the overhead of around 1ms had no drastic impact on the performance, since the execution times of the services are significantly higher.

The evolution of \textit{TrainForRecommender} calibration scenario (cf. \Cref{TrainForRecommender}) shows very well that the adaptive monitoring helps to significantly reduce the monitoring overhead.
Comparing to \textit{train} (cf. \Cref{Train Calibration}) where all SEFF elements are instrumented, we saved \textbf{at least 75\%} of the monitoring overhead by instrumenting only the changed internal action.
This is reflected in the calibration times, which are considerably lower compared to the first iteration (see \autoref{table:exec}) (EQ4.2).
In general, there is a trade off between the monitoring overhead and the accuracy of the model. However, our approach tries to balance them by using the incremental calibration in combination with the adaptive monitoring in order to minimize calibration time and monitoring overhead.





