\section{Introduction}
\extrafootertext{This is an extended version of a paper appearing in the Proceedings of the 34th Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-22).}
\input{sections/intro}

\section{Relevance of AI to Application}
\input{sections/related}

\section{Application and AI Approach}
\label{sec:approach}
\input{sections/approach}

\section{Experiments}
\label{sec:experiments}
\input{sections/experiment}

\section{Path to Deployment}
\label{sec:deploy}
\input{sections/deploy}

\section{Discussion and Limitations}
\input{sections/discuss}


\section*{Acknowledgments}
This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344. The authors acknowledge financial support through the Point Source Carbon Capture Program at the U.S. Department of Energy, Office of Fossil Energy and Carbon Management. This work was conducted as part of the Carbon Capture Simulation for Industry Impact (CCSI2) project. We thank Phan Nguyen for helpful comments.



\subsection{Application and CFD Data Description}
\label{sec:data}
Our application is IA-based design optimization of CCSs. In particular, we seek a means to find CCSs associated with high IA, a key determinant of CO$_2$-capture efficiency. A typical (but slow) approach is to predict IA using the volume fraction fields resulting from a CFD simulation of a \emph{representative column model} (RCM) (Figure \ref{fig:RCM}), which is a representative section of the bench-scale column of our CCS.

This involves simulating the counter-current solvent and gas flow in the 3D RCM (containing 6.5 million data points) then solving continuity and momentum equations with the Volume of Fluid (VOF) method \citep{fu2020investigation}:

\begin{equation}
\frac{\partial \rho}{\partial t}+\nabla \cdot \rho \mathbf{u}=0,
\end{equation}
\begin{equation}
\frac{\partial(\rho \mathbf{u})}{\partial t}+\nabla \cdot(\rho \mathbf{u u})=-\nabla p+\mu \nabla^{2} \mathbf{u}+\rho \mathbf{g}+\mathbf{F}_{\sigma},
\end{equation}
where $\rho$ is density, $\mu$  is viscosity, $\mathbf{u}$ is velocity, $p$ is pressure, $\mathbf{g}$ is gravity, and $\mathbf{F}_{\sigma}$ is the surface tension arising at the gas-liquid interface. The density and viscosity are calculated by a volume fraction average of liquid ($\alpha$) and gas phase (1-$\alpha$). The evolution of $\alpha$ is governed by the transport equation:
\begin{equation}
\frac{\partial \alpha}{\partial t}+\nabla \cdot (\mathbf{u} \alpha)=0.
\end{equation}

Running many such CFD simulations to thoroughly explore the CCS configuration space (which involves: packing type, e.g., Gyroid vs. Schwarz-D; inlet velocity $v$; solvent viscosity/surface-tension; etc.) is computationally infeasible. However, it may be possible to use existing CFD data to train an AI surrogate model to quickly and accurately simulate dynamics of CCSs with never-before-seen configuration settings. During IA-based optimization, this surrogate's faster speed would allow more thorough exploration of the CCS configuration space, improving the chance of finding a CCS design with better CO$_2$-capture performance.

To evaluate the utility of AI surrogates for our application, we measure the speed and accuracy of surrogate simulations of CCSs with values of the solvent inlet velocity parameter not seen during training. Our data is $N=50$ CCS CFD simulations, each with a unique inlet velocity (Figure \ref{fig:interpolate}). $N_{train}=40$ simulations are used for training, while the remaining $N_{test}=10$ simulations are for evaluation.

As a first step, we worked with CFD data from a 2D domain, derived from a vertical slice of the larger 3D CCS domain (Section \ref{sec:deploy} discusses extension to 3D). Like the 3D domain, the 2D domain with 150,073 polygonal mesh grid points  has its counter-current flow solved via the VOF method. Starting from an initial state, a particular CCS configuration on this 2D domain reaches a steady state in 5000 timesteps (Figure \ref{fig:2D} shows a single timestep), using a wall-clock time $W_{\mathrm{CFD}} = \SI{3600}{\second}$ when run on 96 CPU cores. 

Before training surrogates, the simulation data is temporally downsampled to 500 timesteps. As our AI approach uses CNNs that operate on uniform grids, the simulation data is then spatially downsampled via linear interpolation to obtain uniform grid data $g_t^{(i)} \in G$, where $G = \mathbb{R}^{k \times k}$, $i \in {1,2,...,50}$ denotes the simulation index, and $t \in \{1,2,...,T\}$ is the timestep with $T=500$ (we omit $i$ and $t$ when referring to generic elements of $G$). We report results for $k=64$ but saw similar performance with $k=128$.

Note that we train surrogate models only on volume-fraction field dynamics because our application is focused on IA, which is a function of the volume-fraction field $g_t^{(i)}$.


\subsection{AI Approach to Our CFD Simulations}
\label{sec:surrogates}


\begin{figure*}[t]
\centering
\includegraphics[width=1\textwidth]{figs/df_diagram/df_diagram_v3.pdf} 
\caption{DF surrogate simulation (``full rollout'', $w=T-1$) of CCS's 2D volume fraction field, with IA computation at $t=T$.}
\label{fig:NNs}
\end{figure*}

Here, we discuss surrogate simulation with DF and our innovations. DF is a data-driven, NN-based, non-intrusive surrogate approach. In our application, surrogate input is the timestep $t \in \mathbb{N}$, a \emph{frame} $g_t \in G$ that represents the volume fraction field at that timestep, and the inlet velocity $v \in \mathbb{R}$ of the CCS. Surrogate outputs $\widehat{g}_{t+j}$ are volume fraction fields at times $t+j$. (Note, we use hats to denote predicted quantities.) DF achieves this as follows (see Figure \ref{fig:NNs}): 

\begin{enumerate}
    \item \textbf{Encode the frame:} Reduce the dimensionality of the frame $g_t$ via an \emph{encoder network} $E: G \rightarrow \mathbb{R}^c$, where $c$ is the latent space's dimension. The output of $E$ is a \emph{latent vector} denoted by $l_t$.
    \item \textbf{Add simulation-configuration details to latent vector:} Replace the last two elements of $l_t=E(g_t)$ with $[v,t]$. 
    \item \textbf{Temporally advance the latent vector:} Use a separate \emph{latent integration network} (LIN) $I: \mathbb{R}^c \rightarrow \mathbb{R}^c$ to predict the change $\Delta l_t = l_{t+1}-l_t$ and thereby obtain the next latent vector $\widehat{l}_{t+1}=l_t + I(l_t)$; perform this step $w$ times.
    \item \textbf{Decode the new latent vector(s):} Feed $\widehat{l}_{t+j}$ to the \emph{decoder network} $D:\mathbb{R}^c \rightarrow G$ to obtain $\widehat{g}_{t+j}$.
\end{enumerate}

$E$ and $D$ are trained together as an autoencoder, more generally as a latent vector model $\mathrm{LVM}: G \rightarrow G \times \mathbb{R}^c$, which encodes then decodes a $g_t$ and provides the corresponding latent vector (or encoding) $l_t$ as output. Notably, by inserting the simulation-specific information $[v,t]$ into the encoder's latent code rather than predicting it, we remove the supervised learning of these values done in \citep{kim2019deep} in exchange for allowing the decoder and LIN to train with the exact simulation parameters $[v,t]$. Given a trained LVM, the DF approach computes the latent vector $l$ for every training simulation frame $g$. The LIN $I$ is then trained to predict the deltas among these $l$ within a \emph{rollout} window  ($\Delta l_\tau, \tau \in \{t,...,t+w-1\}$ with $w$ as the window size) given only $l_{t}$ (and thus also $[v,t]$) as input. 

Once $I$ is trained, it can be iteratively applied to perform a full rollout that produces $\widehat{l}_{t}$ and  $\widehat{g}_t$ for $t \in \{1,...,T\}$, given only $g_1$ and $v$ as input. Full rollouts are useful when using trained surrogates to replace CFD simulation (see Figure \ref{fig:NNs}).

In Section \ref{sec:experiments}, we study how surrogate performance changes in response to modifying the training-time rollout window size $w$ and other variables. Additionally, we evaluate larger modifications to the DF approach: joint training of the LVM and LIN (end-to-end training), and new LVM/LIN architectures (e.g., transformers).

\paragraph{Latent vector models} Like the original DF approach, we use a CNN autoencoder (comprising both $E$ and $D$) as an LVM. The encoder network $E$ is four Conv2D layers  (with stride 2, padding 1, kernel size 3, and layer-specific channel counts $[128, 256, 512, 1024]$) followed by a linear layer that maps from the final convolutional feature map to $\mathbb{R}^c$. The decoder network $D$ completes a symmetric architecture for the LVM, with a linear layer followed by four ConvTranspose2D layers (using the layer-specific channel counts $[512, 256, 128, 1]$) so that the output has $g$'s shape.

We also consider replacing the CNN autoencoder with an LVM derived from an SVD of the training data ($B=U\Sigma V^\top$, where $B\in \mathbb{R}^{k^2\times N_{train}T}$). Specifically, encoding then decoding $B$ via multiplication with the truncated matrix $U_{\cdot,1:c}$ gives the best rank-$c$ approximation $\hat{B}$ \citep{eckart1936approximation}. This provides a strong baseline for the larger and slower CNN LVM, which also compresses frames to size $c$ but uses nonlinear transformations to make possible decodings $\hat{B}$ with rank $m{>}c$ and thus smaller error.

Finally, we explore the usage of vision transformers (ViTs) as LVMs. As in \citet{dosovitskiy2021an}, we split each input frame into 16$\times$16 patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard transformer encoder. Then, the transformer decoder takes as input the latent vectors from the encoder and an extra learnable embedding. The learnable embedding's state at the output of the transformer decoder serves as the frame representation and is linearly transformed to generate the reconstructed frame.


\paragraph{Latent integration networks}
As in DF, we use an MLP to learn $I$, the latent integration network (LIN). We explore several modifications to the proposed MLP LIN. We modify the LIN's depth and width, and augment the input dimension to exploit more history (i.e., more frames as input) such that $I : \mathbb{R}^{s\times c} \rightarrow \mathbb{R}^c$, where sequence length $s$ is the number of prior timesteps the LIN uses as input (when $t<s$, the $s-t$ remaining rows/timesteps of the input space are initialized at $\mathbf{0}$ and filled gradually as the LIN produces them).

In addition to MLPs, we evaluated other competitive sequence models, such as CNNs \citep{oord2016wavenet}, LSTMs \citep{hochreiter1997long}, and transformers  \citep{Vaswani2017AttentionIA}, as LINs. For the LSTM and transformer networks, we use PyTorch implementations: the transformer is a series of six 8-head transformer encoder layers with hidden dimension $c$, while the LSTM network is a series of LSTM layers with hidden dimensions $H=[h_1,...,h_n]$. To test CNNs, we created AutoRegressive CNN (ARC) LINs that comprise 1D convolutional layers with output channel counts $[h_1, h_2, ..., h_n, c]$, the first layer using $c$ input channels, and all layers using kernel size 3 and padding 1. The output layer of each LIN is a linear layer that maps to $\mathbb{R}^c$, except for the ARC LIN's output layer, which uses a learnable linear combination of the rows from the final feature map to compute $\Delta l_t \in \mathbb{R}^c$. Finally, with the ViT LVM, we used a single linear layer for the LIN, as the performance bottleneck when using the ViT was the LVM.

\paragraph{Performance optimization and evaluation}
When training LVMs, we optimize the relative error loss $L_{\mathrm{RE}}(\widehat{g},g) = \frac{\norm{\widehat{g}-g}}{\norm{g}}$ averaged across the $K$ volume fraction fields in a batch, as we found this performed better than RMSE and preliminary results showed no quantitative benefit of the gradient-matching loss term used in DF. Similarly, to train the LIN, we optimize the relative error loss $L_{\mathrm{RE}}(\widehat{l}_t,l_t)$ averaged over the $K$ $w$-timestep latent vector rollouts (i.e., there are $Kw$ unique relative errors each batch). When using end-to-end training, the $Kw$ rolled-out latent vectors are decoded to frames $\widehat{g}_t$, then $L_{\mathrm{RE}}(\widehat{g_t},g_t)$ is computed and gradients are taken with respect to the parameters of $E$, $I$, and $D$.
We show the effect of training the LIN with RMSE (instead of $L_{\mathrm{RE}}$) and with different values of $w$ and $s$ in Section \ref{sec:sweep}.

While our loss functions aim to ensure accurate reconstruction of $g_t$, our ultimate goal is IA-based optimization of CCS designs. Thus, we are primarily interested in the ability of DF-inspired surrogates to estimate steady-state IA correctly, given only the initial conditions $g_1$ and inlet velocity $v$ of a new, unseen CCS. After training our models, we measure the steady-state IA relative error $
L_{\mathrm{RE}}\left(IA(\widehat{g}_{T}^{(i)}), IA(g_{T}^{(i)})\right),$
where $i$ indexes the $N_{test}$ previously-unseen test simulations, $\widehat{g}_{T}^{(i)}$ is derived from a full rollout, and $IA : G \rightarrow \mathbb{R}$ is the integral of surface areas for which the volume fraction is 50\% (on a 2D volume fraction field, such surfaces correspond to volume fraction isocurves).

Then, we report the mean error: $\mathrm{Error}_{\mathrm{IA}} = \frac{1}{N_{test}} \sum_{i=1}^{N_{test}} L_{\mathrm{RE}}\left(IA(\widehat{g}_{T}^{(i)}), IA(g_{T}^{(i)})\right)$.

While this mean, final-timestep IA relative error is a critical metric for our application, we are also interested in whether surrogates could potentially replace CFD simulations for purposes other than IA estimation. Thus, to test the ability of surrogates to mimic CFD volume fraction dynamics at all timesteps, we use $\widehat{g}_{t}^{(i)}$ from full rollouts on the test data to compute average volume fraction relative error: $\mathrm{Error}_{\mathrm{VF}} = \frac{1}{N_{test}T} \sum_{i=1}^{N_{test}} \sum_{t=1}^{T} L_{\mathrm{RE}}\left(\widehat{g}_{t}^{(i)}, g_{t}^{(i)}\right)$. 

Finally, we use the ratio of the wall-clock time of the CFD simulation to that of a surrogate's full rollout (which is run on four NVIDIA P100 GPUs) to compute the relative speedup offered by our AI approach: $S_W = \frac{W_{\mathrm{CFD}}}{W_{\mathrm{AI}}}$. 


\subsection{LVM Architecture Comparison}
\label{sec:lvm_comparison}
To test how the DF approach might benefit our application, we first consider LVM error of three model classes, including two new approaches that we introduced to DF. Specifically, we compare the performances of a CNN-based autoencoder (used in DF); a ViT \citep{dosovitskiy2021an} with similar parameter count; and SVD, a faster, linear approach.

\sloppy To measure performance, we average the frame reconstruction and IA errors, $L_{\mathrm{RE}}(\widehat{g},g)$ and $L_{\mathrm{RE}}(IA(\widehat{g}_{T}), IA(g_{T}))$, across the test data. As SVD is linear, we expect it to be less accurate than CNNs/ViTs, which are powerful approximators that can represent nonlinear functions. However, ViTs and CNNs require careful optimization to perform well. This is especially true for ViTs, which lack the inductive biases of CNNs (e.g., translation equivariance and locality), harming their performance when data is scarce  \citep{dosovitskiy2021an} as it is here. Thus, SVD may potentially outperform CNNs or ViTs. Consistent with the difficulty of training ViTs without extra/sufficient data, we found SVD outperformed our ViT on the LVM reconstruction task  (Figure \ref{fig:LVM}). However, SVD did not outperform the CNN. The ability of CNNs but not ViTs to outperform SVD reinforces the (particularly important in our low-data regime) point that image-processing inductive biases provided by convolutional layers offer a significant benefit. As \textbf{CNN-based autoencoders produced the best LVMs}, we use them in our remaining experiments except where we indicate use of a ViT LVM.

\begin{figure}[t]
    \centering
        \includegraphics[width=1\columnwidth]{figs/lvm_error_big.pdf}
\caption{Mean test error for our LVMs at up to five $c$ values.}
\label{fig:LVM}
\end{figure}

\subsection{LIN Architecture Comparison}
\label{sec:lin_comparison}

Given the latent vectors from the CNN LVM, we sought to determine the best LIN architecture. While the original DF used an MLP for $I$, other architectures such as LSTM, CNN, and transformer offer competitive performance in sequence modeling. To test whether these other architectures may outperform MLPs on our data, we trained each architecture using several settings for $c$, $s$, $w$, $b$ (batch size), $H$, and the loss function (see Section \ref{sec:sweep} to see the effect of some of these factors). Then, with this set of trained models, we compared the best $\mathrm{Error}_{\mathrm{VF}}$ and $\mathrm{Error}_{\mathrm{IA}}$ of each architecture. 

\begin{table}[ht]
\centering
\begin{widetabular}{\columnwidth}{lccc}
\toprule
   LIN &  $\mathrm{Error}_{\mathrm{IA}}$  &  $\mathrm{Error}_{\mathrm{VF}}$  & $S_W$\\
\midrule
ARC &0.07 (0.00) &0.49 (0.00) &4800 \\ LSTM &0.08 (0.01) &0.49 (0.01) &2700 \\ MLP & \textbf{0.06} (0.00) & \textbf{0.47} (0.00) &5400 \\ Transformer &0.08 (0.04) &0.51 (0.00) &4300 \\
\bottomrule
\end{widetabular}
\caption{For each LIN architecture, the best $\mathrm{Error}_{\mathrm{IA}}$ and $\mathrm{Error}_{\mathrm{VF}}$ found via hyperparameter sweeps (standard deviations in parentheses) with wall-clock time speedup $S_W$.}
\label{tab:LIN}
\end{table}

While an MLP had the best performance across multiple runs ($0.06$ mean $\mathrm{Error}_{\mathrm{IA}}$), we found that  \textbf{the best single model for our IA-based optimization application was a transformer}, with $\mathrm{Error}_{\mathrm{IA}}$ of just $0.04$ (Table \ref{tab:LIN}). Each architecture in Table \ref{tab:LIN} has roughly \SI{1e5} parameters. For each performance measurement, the best models were found by optimizing $L_{\mathrm{RE}}$ rather than RMSE, except for the best transformer according to $\mathrm{Error}_{\mathrm{IA}}$, which was found by optimizing RMSE. In addition, the best models trained with $s=6$, except for the the best transformer and MLP models according to $\mathrm{Error}_{\mathrm{IA}}$, which trained with $s=1$. Finally, all models attained their best performance with $50\leq w \leq 300$.

The surrogate speedups are significant (up to 5400x) and partly stem from their 10x larger timesteps (see Section \ref{sec:data}). 


\subsection{End-to-End Training/Fine-tuning}
\label{sec:e2e}

The original DF trained LIN and LVM separately. Here, we investigate whether end-to-end (E2E) training improves performance. 
We applied E2E fine-tuning to a pretrained CNN LVM + ARC LIN surrogate and E2E training to a ViT LVM + linear LIN surrogate, then compared against classic training (Table \ref{tab:e2e}). Consistent with the benefit of LIN training that directly optimizes $L_{\mathrm{RE}}(D(\widehat{l}),g)$, we found both \textbf{E2E approaches significantly improved $\mathrm{Error}_{\mathrm{VF}}$ and $\mathrm{Error}_{\mathrm{IA}}$}.

\begin{table}[ht]
\centering
\begin{widetabular}{\columnwidth}{lcccc}
\toprule
 Surrogate & Training style &  $\mathrm{Error}_{\mathrm{IA}}$  &  $\mathrm{Error}_{\mathrm{VF}}$  \\
\midrule
CNN+ARC & Classic DF &                0.07 (0.00) &        0.52 (0.01)	 \\
CNN+ARC & E2E DF &                \textbf{0.06} (0.00) &        \textbf{0.48} (0.00) \\
\midrule
ViT+Linear & Classic DF &            0.53 (0.25)   &    0.77 (0.08) \\
ViT+Linear & E2E DF &                0.36 (0.08) &     0.59 (0.03)   \\
\bottomrule
\end{widetabular}
\caption{The effect of end-to-end training/fine-tuning.}
\label{tab:e2e}
\end{table}



\subsection{Hyperparameter Study}
\label{sec:sweep}

Here, we illustrate the effects of key DF hyperparameters, noting when optimal settings deviate from those in the original DF model. Settings of factors not shown in a table are held constant and are discussed in Appendix \ref{sec:appendix}. 

\paragraph{Deeper LINs and larger rollout windows have higher performance} We found that, for our data, the wide MLP LIN with $H=[1024,512]$ used in the original DF performed worse than a narrower, deeper MLP LIN with $H=[128,128, 128]$ (Table \ref{tab:w}). Since the narrower, deeper LIN offered superior performance with far fewer parameters, we used  $H=[128,128, 128]$ for our other experiments. Additionally, we found that increasing the window size, which allows the model to see and learn to control accumulation of error from repeated LIN applications during training, is helpful up to a point: as shown by the high error at $w=499$, too many LIN applications make training difficult.

\begin{table}[ht]
\centering
\begin{widetabular}{\columnwidth}{lcccccc}\toprule 
         & \multicolumn{6}{c}{$\mathrm{Error}_{\mathrm{IA}}$} \tabularnewline \cmidrule(lr){2-7} 
        H &    $w$=20  &  50  &  150 &  200 &  300 &  499 \tabularnewline
        \midrule  \relax 
        1024, 512      & 0.33 &0.08 &0.09 &0.10 &0.11 &2.53 \tabularnewline \relax  & (0.14) & (0.00) & (0.00) & (0.00) & (0.00) & (0.26) 
        \tabularnewline \relax 
        128, 128, 128   &   0.64 &0.08 &0.08 &0.07 & \textbf{0.06} &2.25 \tabularnewline \relax  & (0.11) & (0.01) & (0.00) & (0.00) & (0.00) & (0.18)
        \tabularnewline \bottomrule
\end{widetabular}
        \caption{Effect on $\mathrm{Error}_{\mathrm{IA}}$ of window $w$ and layer sizes $H$. }
\label{tab:w}
\end{table}

\paragraph{Relative error outperforms RMSE as a LIN loss function, longer sequence lengths can be helpful}
Optimizing the relative error loss ($L_{\mathrm{RE}}$) of $\widehat{l}$ rather than the RMSE significantly improved $\mathrm{Error}_{\mathrm{VF}}$ and led to the best  $\mathrm{Error}_{\mathrm{IA}}$ (Table \ref{tab:loss}). Use of $L_{\mathrm{RE}}$ as an optimization target may lead to better $\mathrm{Error}_{\mathrm{VF}}$ because each of these functions computes a relative error; i.e., $L_{\mathrm{RE}}$ has an advantage over RMSE when high relative error in latent space translates to high relative error in frame space. The fact that $L_{\mathrm{RE}}$ also led to the best $\mathrm{Error}_{\mathrm{IA}}$ may be attributable to the more accurate frame reconstructions obtained when using this loss.

Increasing the sequence length $s$ tended to improve $\mathrm{Error}_{\mathrm{VF}}$ (Table \ref{tab:loss}), regardless of optimization target (except with the transformer). Thus, for most LINs, to minimize discrepancies between CFD and AI surrogate simulations ($\mathrm{Error}_{\mathrm{VF}}$), our results support combining the relative error loss $L_{\mathrm{RE}}$ with $s>1$. Notably, \citet{kim2019deep} use RMSE and $s=1$ (with the MLP LIN).





\begin{table}[ht]
\centering
\begin{widetabular}{\columnwidth}{llcccc} 
\toprule             & {} & \multicolumn{2}{c}{$\mathrm{Error}_{\mathrm{IA}}$} & \multicolumn{2}{c}{$\mathrm{Error}_{\mathrm{VF}}$} \tabularnewline \cmidrule(lr){3-4} \cmidrule(lr){5-6}    
\relax
LIN & $s$ & $L_{\mathrm{RE}}$ & RMSE & $L_{\mathrm{RE}}$&  RMSE   \tabularnewline \midrule  \relax 
ARC&1 &0.10 (0.02) &0.09 (0.00) &0.52 (0.01) &0.55 (0.00) \tabularnewline \relax &6 &0.12 (0.03) &0.12 (0.02) &0.52 (0.01) &0.53 (0.00) \tabularnewline \relax LSTM&1 &0.11 (0.01) &0.09 (0.02) &0.53 (0.03) &0.54 (0.00) \tabularnewline \relax &6 &0.11 (0.01) &0.10 (0.01) &0.52 (0.00) &0.54 (0.00) \tabularnewline \relax MLP&1 & \textbf{0.06} (0.00) &0.08 (0.00) &0.53 (0.00) &0.55 (0.00) \tabularnewline \relax &6 &0.09 (0.00) &0.10 (0.03) & \textbf{0.49} (0.00) &0.52 (0.00) \tabularnewline \relax Transformer&1 &0.09 (0.01) &0.08 (0.04) &0.52 (0.00) &0.55 (0.00) \tabularnewline \relax &6 &0.09 (0.03) &0.14 (0.08) &0.53 (0.00) &0.68 (0.11) \tabularnewline
\bottomrule
\end{widetabular}
\caption{The effect on $\mathrm{Error}_{\mathrm{IA}}$ and $\mathrm{Error}_{\mathrm{VF}}$ of LIN architecture, input sequence length $s$, and training loss.}
\label{tab:loss}
\end{table}

