

\subsection{Reinforcement Learning} \label{ssec:RL}
As a reward-based system, reinforcement learning (RL) strives to maximize the rewards of the interactions between an agent and its environment \cite{b28}. In response to feedback from the environment, the agent learns about its behavior, which is then attempted to improve due to its actions. Reward learning problems can be solved by devising policies (e.g., mapping between states and actions) that maximize the accumulation of rewards. In a reinforcement learning problem, five important entities are involved: the state, the action, the reward, the policy, and the value. Generally, reinforcement learning problems are modeled with Markov decision processes.

\subsection{Q-Learning} \label{sssec:QL}
During reinforcement learning, agents learn policies as they transition between states using the Q-learning algorithm \cite{b37}. In order to determine the optimal set of policies, we must assess all possible actions related to the different states of the agent. The value of this algorithm is maintained continuously by updating the Q-value according to the next state of the algorithm and the greedy action. A Q-function is essentially a function that accepts various arguments, such as state vectors, action vectors, rewards vectors, and learning rates. The discount factor is then calculated for the Q-value. However, because Q-learning-based systems require high dimensionality, they do not perform well in large state spaces.

\subsection{Deep Q Network (DQN)}
\label{sssec:DQN}
Several techniques have been developed to deal with significant space state problems, including Deep Q-Network (DQN), a networked Q-learning algorithm combining reinforcement learning with a class of artificial neural networks known as Deep Q Networks. Depending on the outcome of an action, the environment grants a positive reward or penalizes it with a negative reward. In addition to updating the weights of the DNN, the reward is also used to improve the performance of this machine learning algorithm. 

In practice, when playing Atari games, the DQN algorithm has shown to be able to achieve impressive results. The purpose of this algorithm is to integrate DNNs with Q-Learning algorithms to determine self-aware decision policies ${\pi}$ used for mapping relations between states ${S}$ and actions  ${A}$ such that ${A=\pi(S)}$ \cite{b38}.

\subsection{Formulation}
As shown in Fig \ref{fig:SSL}, the Markov decision process has ${S}$ representing all possible states and ${A}$ representing all possible actions. It is possible to retrieve the possible rewards from any state and action pair ${(s, a)}$ by using the distribution ${R}$. In the Markov decision process, ${P}$ represents the transition probability distribution for the next state, and  ${\gamma}$ represents the discount factor. Future rewards are affected (discounted) by the discount factor. Since an agent faces uncertainty during the process, the more rewards it counts as it tries to predict future outcomes. We want to determine which action should be taken in every state according to a policy that determines the action to take. Furthermore, the complexity of the problem influences the number of states (which affects uncertainty), so our policy function will be complex.  

RL and DRL algorithms were proposed for learning policies with the maximum cumulative reward. In order to determine the optimal policy, we need to: \begin{equation}
\pi^*=\arg \max _\pi \mathbb{E}\left[\sum_{t>0} r^{t r_t \mid \pi}\right]
\end{equation}
Using ${\pi^*}$ as the optimal policy, ${r}$ as rewards, and ${t}$ as the time step.

Value functions represent the expected cumulative reward of being in a particular environment state. It typically refers to the quality of a state and is defined as:  
\begin{equation}
V^\pi(s)=\mathbb{E}\left[\sum_{t \geq 0} r^{t r_t \mid s_0=s, \pi}\right]
\end{equation}
A value function is defined by ${V^\pi(s)}$ based on policy ${\pi}$ for ${s}$, and an expected cumulative reward value is defined by ${\mathbb{E}}$.  

We take action and state into account when calculating the Q-value function. In this case, the expected cumulative reward is calculated by being in a specific state, taking a specific action, and following the policy ${\pi}$.  
\begin{equation}
Q^\pi(s, a)=\mathbb{E}\left[\sum_{t \geq 0} r^{t r_t \mid s_0=s, a_0=a, \pi}\right]
\end{equation}
In this case, ${Q^\pi(s, a)}$ represents the Q-value, ${a}$ represents the chosen action from among all the possible actions (${A}$), and ${s}$ represents a specific state from among all the possible states (${S}$). 

We can retrieve the greatest expected reward from a particular state and action by using ${Q^*(s, a)}$ as the optimal Q-value.  
\begin{equation}
Q^*(s, a)=\max _\pi \mathbb{E}\left[\sum_{t \geq 0} r^{t r_t \mid s_0=s, a_0=a, \pi}\right]
\end{equation}

In the case where the optimal state-action values are identified (${Q^*\left(s^{\prime}, a^{\prime}\right)}$), ${Q^*}$ satisfies the Bellman equation at the next time step:  \begin{equation}
Q^*(s, a)=\mathbb{E}_{s^{\prime} \sim \varepsilon}\left[r+\gamma \max _{a^{\prime}} Q^*\left(s^{\prime}, a^{\prime}\right) \mid s, a\right]
\end{equation}
${\gamma}$ is the discount factor, and ${\varepsilon}$ represents the MDP environment. 

The most effective strategy is to maximize the expected value. The agent uses optimal policy ${\pi^*}$ to decide how to proceed in any state by the specification of ${Q^*}$. Using the Bellman equation as an iteration update, we can find the optimal policy through a value iteration algorithm: \begin{equation}
Q_{i+1}(s, a)=\mathbb{E}\left[r+\gamma \max _{a^{\prime}} Q_i\left(s^{\prime}, a^{\prime}\right) \mid s, a\right]
\end{equation} 

Based on the two equations above, we can rewrite ${Q^*(s, a)}$ as follows:
\begin{equation}
Q^*(s, a)=\mathbb{E}\left[r+\gamma \max _{a^{\prime}} Q_i\left(s^{\prime}, a^{\prime}\right) \mid s, a\right]
\end{equation} 

When neural networks are used in DQN, we get the action-value function ${Q^{\pi}(s, a)}$  from the policy ${\pi}$.
\begin{equation}
    Q(s, a ; \boldsymbol{\theta}) \approx Q^{*}(s, a)
\end{equation}
It is assumed that ${\theta}$ denotes the model parameters. The optimal Q-values are estimated using these weights.  

Agents interact with the environment through greedy algorithms. This algorithm provides a means for balancing "exploration" (random action $a\in A$) with "exploitation" (action $a$ with ${\rm{argma}}{{\rm{x}}_a}\,Q(s,a;\theta )$) and generating a series of experiences defined as $(s_t,a_t,r_t,s_{t+1})$. ER buffer $B$ stores these experience samples. A random sample of experience samples is then taken from ER buffer, which creates a new model based on the randomly generated experience samples.
  
This initial learning stage is accelerated by a dynamic-greedy algorithm generating action distributions to obtain various experience samples in ER buffer $B$, which are stored in it. Changes in parameters can be obtained by following these steps:  
\begin{equation}
\epsilon=\left\{\begin{array}{l}
\epsilon \times \epsilon_{\text {decay }}, \text { if } \epsilon \geq \epsilon_{\min } \\
\epsilon_{\min }, \text { otherwise }
\end{array}\right.
\end{equation}

Q-values can be estimated with an error when using a neural network. Due to this fact, the Bellman equation's error should be minimized. It is possible to minimize the error of the Bellman equation by using training loss functions. As a result, it is a measure of how far ${Q^(s, a)}$ is from the target value ${y_{i}}$ as follows:  
\begin{equation}
L_{i}\left(\boldsymbol{\theta}_{i}\right)=\mathbb{E}\left[\left(y_{i}-Q\left(s, a ; \boldsymbol{\theta}_{i}\right)\right)^{2}\right]
\end{equation}
The model's parameters are updated using ${L_{i}}$ as an objective function representing the mean-square error (MSE). Based on the principle that ${y}$ is the target ${Q}$ value function, the following formula can be used to obtain it: 
\begin{equation}
y_{i}=r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \boldsymbol{\theta}_{i-1}\right)
\end{equation}

The optimal policy is determined by updating the objective function using gradient descent. Accordingly, the gradient updating is calculated as follows: 
\begin{equation}
\nabla_{\boldsymbol{\theta}_{i}} L\left(\boldsymbol{\theta}_{i}\right)=\mathbb{E}_{\sim U}\left[\left(y-Q\left(s, a ; \boldsymbol{\theta}_{i}\right)\right) \nabla_{\boldsymbol{\theta}_{i}} Q\left(s, a ; \boldsymbol{\theta}_{i}\right)\right]
\end{equation}
${\sim U}$ indicates the random uniform sampling.
\subsection{Double Deep Q Network (DDQN)} \label{sssec:DDQN}
There are many similarities between DDQN and DQN. The difference between DDQN and DQN lies in that DDQN employs two neural networks, both of which implement current Q functions and the other which implements target Q functions. It is essential to realize that the target Q function is a copy of the current Q function with a delayed timing loop created after a certain number of training runs are completed. 

It is possible to perform gradient descent over objective functions without recursively relying on training networks, thus preventing the moving target effect from occurring in gradient descent \cite{b39}.

We use this approach in the following form:
\begin{equation}
Q^*(s, a)=(1-\alpha) \cdot Q(s, a)+\alpha\left(r+\gamma \cdot \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)\right)
\end{equation}
Where $\alpha$ is the learning rate, $\gamma$ is the discount factor, and $\max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)$ is the estimate of optimal future value.
\subsection{Classification Markov Decision Process (CMDP)}
Because DRL solves a sequential decision-making problem, it is infrequently used for classification tasks. It has been developed that a standard classification problem is defined as a sequential decision-making problem by the classification Markov decision process (CMDP).

Consider the CMDP framework for the tuple $\{S, A, P, R\}$. Classification problems can be described as sequential decision-making problems in which${S}$, ${A}$, ${P}$, and ${R}$ correspond to state space, action space, transition probability, and reward function, respectively.

\textbf{State space ${S}$.} Training samples determine it. In order to determine the initial state of the agent, a sample will be taken at the beginning of each episode. The training data sequence is generated based on the order of samples collected when the environment starts a new episode.

\textbf{Action space ${A}$.} A list of identified actions is provided, and each action is classified as either an attack or a normal label. Consequently,  ${A}$ consists of  $\{1,2, \ldots, K\}$, whereas ${K}$ represents the entire data set.

\textbf{Transition probability ${P}$.} Transition models are described as follows: $P: S \times A \longrightarrow S$. CMDP has a deterministic transition probability. The agent will receive the next state according to the order in which the samples are received in the mini-batch after obtaining the current state and taking the appropriate action. A mini-batch of samples is then processed sequentially until all samples' problems are resolved.

\textbf{Reward function ${R}$.} Rewarding the agent for its actions promotes the agent's learning of the policy and enables it to make appropriate decisions. If the state label is correctly detected, this work results in a positive reward from the environment; otherwise, a negative reward. Therefore, the reward function is designed as follows: 
\begin{equation}
R\left(s_{t}, a_{t}, l_{t}\right)= \begin{cases}1, & \text a_{t}=l_{t} \\ -1, & \text  a_{t} \neq l_{t}\end{cases}
\end{equation}
$l_{t}$ indicates whether the label is a fault or a normal label.

\begin{figure*}
\centering

  \includegraphics[clip, trim=1cm 20cm 0cm 2cm,width=0.7\textwidth,height=4cm, scale=0.2]{5.pdf}
  \caption{ An overview of the Recommender Agent-User interaction in MDP.}\label{fig:SSL}
\end{figure*}



\subsection{Setup Details}

\textbf{Dataset.}\label{AA}
Data from an actual working steam turbine was used in this study. A sampling rate of one minute per day was used to collect the turbine's data over 124 days. It was determined that 207,361 data were collected in total. However, 190,635 data could be used after preprocessing and removing outlier data from the non-dominant operating mode. The normal category comprises 121,279 data, and the fault category comprises 69,356 data. It contains 31 features, including condenser pressure, pressure on inlet valves, steam flow value, active and reactive power, and others. In this case, the problem was caused by the leakage of one of the pressure valves, failing the entire system. Data are collected in the dominant operating mode of the system, known as the high-pressure mode, and they are sufficiently comprehensive to provide a good understanding of the system's functioning.

\textbf{Parameters.}\label{AA}
With the CMDP framework, we set the mini-batch size to 128, initial $\alpha$ to 1 with a decay rate of 0.9999, and initial $\epsilon$ to 0.9 with a decay rate of 0.99. Additionally, we set the $\gamma$ to 0.001 and the number of iterations to 1000, with the minimum $\alpha$ and $\epsilon$ setting to 0.0001. As part of the training process, we also use an Adam optimizer with learning rates of 0.001. Table 1 shows the parameters of the neural networks, and both the current and target networks have the same structure. In hidden neurons, we use the LeakyReLU activation function. There is no reason to worry about initializing a neural network with an LReLU. It is possible for a ReLU network never to learn if neurons are not activated at the beginning.

At each step of the algorithm execution, $70\%$ of the data is used for training, while the rest is used to test the algorithm. Every time we train the algorithm on daily data, we return a higher-performing model which returns the highest number of recall criteria in all iterations (on test data) to increase the speed of analysis. In order to demonstrate the importance of the class with less data, we consider average recall as a macro.

\textbf{Preprocessing.}\label{AA}
In the available dataset, some rows have a "drop-it" title. Data such as this should be deleted as it is considered outlier data. It is then necessary to normalize the data between 0 and 1 by using a Min-Max scaler as follows:

\begin{align}
{{\rm{x}}_{scaled}} = \frac{{{\rm{x}} - {\rm{ }}{{\rm{x}}_{min}}}}{{{{\rm{x}}_{max}} - {\rm{ }}{{\rm{x}}_{min}}}}.
\label{formula3}
\end{align}
Where $x$ and $x_{scaled}$ assigns features before and after normalization, and $x_{min}$ and $x_{max}$ assigns minimum and maximum value of features based on training data before normalization.

\subsection{Results and Discussion}
We analyze the dataset for all days at once with the framework specifications DQLAP and the parameters mentioned in the previous section. According to the results, we achieved $91\%$ in all performance metrics. According to this performance level, the deep reinforcement learning algorithm's analysis is close to that of an expert. 

By using the daily update to improve the policy instead of all the data at once, we reach $94\%$ in all performance metrics. As a result, the daily update policy is more closely related to the expert, and the algorithm is more similar to the expert's daily feedback. 

Our results show that the neural network does not perform as well as the previous two models outside the designed framework. 

Furthermore, the prediction speed of the network in the framework (per iteration) is faster than that of the same network outside it. Our analysis can be sped up by using a buffer and creating action parameters that interact with the environment.
\begin{table} [!htb]
	\caption{Comparison of designed systems}\label{tab:transfer}

	\renewcommand{\arraystretch}{1.5}



	\label{tab:reg}
	\centering


	\resizebox{0.48\textwidth}{!}{%
		\setlength\tabcolsep{3pt}
		\begin{tabular}{cccccccccc}
			\hline
			method & Accuracy& Precision & Recall& F1 Score \\ \hline
			MLP Network  &0.90&0.91&0.90&0.90\\
			DDQN without Update Policy  &0.91&0.91&0.91&0.91\\
			DDQN with Update Policy &\textbf{0.94}&\textbf{0.94}&\textbf{0.94}&\textbf{0.94}\\
			\hline		
			
		\end{tabular}
	}
\end{table}
\subsection{Visualization}
We map the entire data to 2D space using PCA to understand the recommender system's performance and compare it with the announced labels. This will allow us to analyze the data visually. Fig \ref{fig:v1} and \ref{fig:v2} depict this representation. Fault data is labeled 0, and normal data is labeled 1. The expert and DQLAP labels are the same in most distribution areas, as shown in Fig \ref{fig:v1} and Fig \ref{fig:v2}. There is evidence that DQLAP has declared more normal labels in the above cluster and that the distribution of those areas has become more coherent. By improving our framework, we can improve this cluster's coherence and classification accuracy.

 As seen in Fig \ref{fig:v1}, there is a yellow part of the data distribution in the middle of the page, indicating that the expert declared that area normal. The data distribution seems closer to non-normal or fault, and DQLAP has also categorized it as a fault.

Consequently, the differences between the expert's opinion and the prediction framework appear primarily due to the closeness of the distributions of the two classes. DQLAP's division form seemed better than the expert opinion in some distribution parts.

\begin{figure}[H]
\centering

  \includegraphics[scale=0.6]{download1.png}
  \caption{Scatter plot for two-dimensional representation in the spatial distribution of data with the label suggested by the expert.}\label{fig:v1}
\end{figure}

\begin{figure}[H]
\centering

  \includegraphics[scale=0.6]{download.png}
  \caption{Scatter plot for two-dimensional representation in the spatial distribution of data with the label suggested by our framework.}\label{fig:v2}
\end{figure}

\section{Introduction}
\input{section_Introduction}


\section{Prior Art}
\input{section_prior_art}

\section{Background}

\input{section_background}


\section{Proposed Method}
\input{proposed_method}

\section{Simulations}
\input{simulations}







\section{Conclusion}
This paper presents a recommender framework combining reinforcement learning with a turbine system's neural networks. To solve our problem, we first defined it as a Markov Decision Process, then analyzed and compared it with the feedback of an expert by obtaining suitable parameters for the network and the reinforcement learning agent. The results show that the proposed algorithm has the closest performance to an expert's opinion if we analyze the data daily and update the policy accordingly. Furthermore, our system will perform the best analysis with less label knowledge by learning the distribution space of data and, based on that, announce its forecast for the next few days to help expert man and reduce time waste, and cost, along with detecting faults that are unlikely to occur. Moreover, this method does not require extracting features. DQLAP is better at predicting each iteration than a similar network outside the framework, and the analysis can be performed at more speed every day.

\bibliographystyle{unsrt}  
