\section*{\LARGE \centering Appendix}

\section{Technical details} \label{app:tech}

\subsection{Networks}
\begin{figure}[b!]
    \centering
    \includegraphics[width=\linewidth]{images/DensGen_VelGen-part_crop.png}%
    \caption{Our full density estimator network $\mathcal{G_{\rho}}$.
    Lower left: the velocity generator part $\mathcal{G_{\mathrm{\mathbf{u}}}}_l$ that is reused for each scale.
    C is a convolution, RB a ResBlock, NN a nearest neighbour sampling.
    A trapezoid shape indicate a stride of 2. Strided convolutions use a filter size of 4, the rest 5, all use ReLU activation. The number of filters and spatial resolutions are given in the image.}
    \label{fig:fulldensgen}
\end{figure}
The full density generator $\mathcal{G_{\rho}}$ is detailed in \figref{fig:fulldensgen}, the velocity generator part $\mathcal{G_{\mathrm{\mathbf{u}}}}_l$ consists only of the 6 layers which are shared between all scales (see also \figref{fig:velgen}). Like $\mathcal{G_{\rho}}$, it uses a kernel size of 5 and relu activation, except for the input and output convolutions which have a kernel size of 1.
Any ResBlocks that change the number of filters from their input have a kernel size 1 convolution on their skip path. If the number of filters stays the same this convolution is omitted.
We use the simple Layer Normalization of \citet{layernorm} after every convolution except the last in a network.

\paragraph{Discriminator}
We reuse the discriminator of \citet{GlobT} which is a simple stack of convolutional layers. However, we remove the crop and rotation in the data augmentation to allow the discriminator to see the whole domain and consider the spatial orientation of observed features.

\subsection{Losses}
\paragraph{CFL magnitude loss}
For increased stability during advection we enforce the CFL condition on the velocity by using a magnitude loss that only affects velocity components larger than 1:
\begin{equation}
    \mathcal{L}_\text{CFL} = \sum_{i=1}^3 \max\left(\mathrm{\mathbf{u}}_i^2 - 1, 0\right),
\end{equation}
where $i$ are the x-, y-, and z-components of $\mathrm{\mathbf{u}}$.

\paragraph{Smoothness loss}
The first order smoothness loss is also defined per component:
\begin{equation}
    \mathcal{L}_\text{smooth} = \sum_{i=1}^3 \left[ \left( \pder{\mathrm{\mathbf{u}}_i}{x} \right)^2 + \left( \pder{\mathrm{\mathbf{u}}_i}{y} \right)^2 + \left( \pder{\mathrm{\mathbf{u}}_i}{z} \right)^2 \right],
\end{equation}
where the spatial derivative is approximated with finite differencing in practice.


\subsection{Training}
Our method is implemented in 
TensorFlow version 1.12 under python version 3.6 and trained on a Nvidia GeForce GTX 1080 Ti 11GB. For training we use the Adam optimizer. For leaning rate decay we use the following formula:
\begin{equation}
    \frac{\texttt{learning\_rate}}{1+ (\texttt{iteration} - \texttt{offset})\texttt{decay}}
\end{equation}

\paragraph{Data}
Our synthetic dataset consists of 30 randomized simulations of buoyant smoke plumes, simulated with Mantaflow at a resolution of 128x192x128 (2x higher than our training resolution) for 150 time steps. We omit the first 45 frames as startup and train with the remaining 105 frames. The volumes are rendered with the same renderer we use during training to obtain the input images.

For the real world data we use the raw captures of first 30 scenes of the SalarFlow dataset as input images and omit the first 20 frames to ensure that there is enough density visible, leaving 130 frames per scene. We downsample the images by a factor of 10 to 192x108 to approximately match the projected grid resolution.

Each Network and method has been trained specifically for the corresponding dataset.

\paragraph{Density training}
$\mathcal{G_{\rho}}$ is trained with $\loss_{\dens} = \loss_{\target} + \texttt{2e-4} \loss_{\disc} + \texttt{1e-3} \loss_{z}} %{\loss_{\rm{center}}$
and a learning rate of 2e-4 with a decay of 2e-4, the decay is offset by -5000 iterations.
We start at a grid resolution of 8x12x8 with 2 UNet levels. The resolution grows after 8k, 16k and 24k iterations by a factor of 2, adding a level of the UNet every time, thus reaching a maximum grid resolution of 64x96x64 with 5 levels.
New levels are faded in over 3k iterations, starting 2k iterations after growth, by linearly interpolating between the up-sampled previous level and the current level. After fade-in only the output of the highest active level remains.
The image resolution grows in conjunction with the density.

\paragraph{Velocity training}
For $\mathcal{G_{\mathrm{\mathbf{u}}}}$ we use $\loss_{\dens} = \loss_{\target} + \texttt{1e-3} \loss_{\dproxy}  + \texttt{2e-6} \loss_{\disc} + \texttt{1e-3} \loss_{z}} %{\loss_{\rm{center}} + 0.1 \mathcal{L}_\text{CFL} + \texttt{1e-4} \mathcal{L}_\text{smooth}$
with a
learning rate of 4e-4 with the same decay of 2e-4, offset by -5000 iterations.
The velocity training also starts at a grid resolution of 8x12x8 with 2 levels of $\mathcal{G_{\mathrm{\mathbf{u}}}}_l$ and with only 1 advection step (i.e. 2 frames). The density estimation and advection always happen at the maximum resolution of 64x96x64 and the velocity is up-scaled accordingly before it has reached this resolution.
Before increasing the resolution, we extend the sequence, first to 3 frames after 1500 iterations and then to the final 5 frames after 3k iterations. Then we grow the resolution of the velocity estimation after 4k, 8k and 12k iterations, each time adding a new level of $\mathcal{G_{\mathrm{\mathbf{u}}}}_l$.
The residual of a newly added velocity level is linearly faded in over 1500 iterations, starting 500 iterations after growth.
A full training run of our model takes 6 days on average using a single GTX 1080 Ti GPU.


\subsection{Rendering}
We use the differentiable volumetric ray-marcher of \citet{GlobT} for our method which discretizes the rendering equation
\begin{equation}
    \mathcal{R}(\rho, L, c) = \int_n^f L(x) e^{-\int_n^x \rho(a)da} dx,
\end{equation}
where $L$ gives the lighting at point $x$ and $c$ are camera parameters resulting in pixel rays from $n$ to $f$, by stepping along the ray to solve the integrals.
The single-scattered lighting is realized by solving the rendering equation in the same way and storing intermediate step values to create the lighting volume $L$.
A background image $\imggt_{B}$ can be added by attenuating it with the total density along the ray, thus the rendering becomes
\begin{equation}
    \mathcal{R}(\rho, L, c) = \int_n^f L(x) e^{-\int_n^x \rho(a)da} dx + e^{-\int_n^f \rho(a)da} \imggt_{B}.
\end{equation}
We modify the renderer to only render inside of the cell center of the outer cell-layer instead of their outer border to avoid border artifacts at the inflow, which are likely caused by the interpolation to 0 outside the domain.

\subsection{Projection into the volume}
The operation $\render^{-1}$ transfers an image or 2D tensor into the volume by
evenly distributing it along the pixel rays into the volume from a given view.
This is achieved by inverting the ray-marching procedure. While marching along the ray the current pixel value is distributed to the 8 voxels around the current position, weighted by the distance to the voxel. The weights are the same one would use for linear interpolation at that point.
Afterwards, the accumulated voxel-values are normalized with the number of contributions per voxel to obtain a smooth result.
This procedure is also used for the gradient backwards pass of $\mathcal{R}$.


\subsection{Advection}
The MacCormack advection \citep{selleUnconditionallyStableMacCormack2008} is a second order transport scheme using Semi-Langrangian advection as its base, which itself is implemented as a backwards-lookup using the velocity vectors and interpolating the transported field at the lookup location. With $\mathcal{A}_{SL}$ as the Semi-Langrangian base, MacCormac is defined as 
\begin{equation} \label{eq:advMC}
\begin{split}
    \snext{\hat{\rho}} &:= \mathcal{A}_{SL}(\scurr{\dens}, \scurr{\vel})\\
    \scurr{\hat{\rho}} &:= \mathcal{A}_{SL}(\snext{\hat{\rho}}, -\scurr{\vel})\\
    \snext{\dens} &:= \snext{\hat{\rho}} + 0.5(\scurr{\dens} - \scurr{\hat{\rho}}).
\end{split}
\end{equation}
To avoid extreme values the results are limited to be within the range of the interpolants encountered in the first Semi-Langrangian advection.

\paragraph{Boundary and inflow handling}
All boundaries are open, meaning that advections that transport density from outside the domain use values from the nearest boundary layer.
Inflow of densities is thereby automatically handled by the open boundaries which works well in our application but is theoretically limited as an initial density at the boundary layer is needed to have any inflow and the inflow can never exceed the already existing density values due to the linear interpolation in the advection.

\paragraph{Vector potentials and curl}
Generating vector potentials and defining the velocity as their curl, as we do in our velocity estimation, can lead to 
discontinuous velocity fields 
if linear interpolation is used
to interpolate vector potential fields between resolutions \citep{curlflow}. 
Since the curl is based on spatial derivatives, and the derivatives of linear interpolation are not continuous,
the resulting velocity can have discontinuities which
can lead to grid aligned artifacts visible in the results.
To circumvent this issue we use the higher order quadratic B-splices as interpolation kernels to transfer vector potentials
from coarse to finer resolutions of our multi-scale network.
This leads to a noticeably smoother velocity, as can be seen from the advected densities in \figref{fig:curlinterpolation}.
\begin{figure}
    \centering%
    \mygraphicsB{width=0.492\linewidth,trim=20 50 70 260, clip}{images/Own_supp_SF80/img_col_b0000_cam02_0040.png}{}\hfill%
    \mygraphicsB{width=0.492\linewidth,trim=20 50 70 260, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0040.png}{}%
    \caption{Left: the grid-aligned artifacts are caused by discontinuous velocities. Right: higher order interpolation of the potentials forces the velocities to be smoother, resulting in smoother densities after advection.
   
    }%
    \label{fig:curlinterpolation}%
\end{figure}

\section{Addtional results}

\subsection{Ablations}
We run additional ablations for the losses $\mathcal{L}_\text{smooth}$ and $\mathcal{L}_\text{CFL}$ when training $\mathcal{G_{\mathrm{\mathbf{u}}}}$. The corresponding metrics are in \tabref{tab:metricsVelAbl}.

\begin{table}[]
    \caption{Quantitative ablation of different regularization terms when training $\mathcal{G_{\mathrm{\mathbf{u}}}}$ on the real world ScalarFlow dataset. The training diverges without any regularization (first row), leading to empty volumes. Without magnitude regularization ($\mathcal{L}_\text{CFL}$, second row) the training itself is stable, but the evaluation produces velocities that quickly remove all density from the volume after a few time steps, again leaving empty volumes. Lower values are better for all metrics under consideration.}
    \vspace{0.2cm} 
    \label{tab:metricsVelAbl}
    \centering
    \begin{tabular}{l|ccc|lllll}
        \hline
        \  &&&& Input & Side & & & Random\\
        Algorithm & $\mathcal{L}_\text{smooth}$ & $\mathcal{L}_\text{CFL}$ & $\loss_{\dproxy}$ & RMSE & RMSE & LPIPS & FID & FID\\\hline
        Ours & & & & - & - & - & - & -\\
        Ours & \checkmark & & & - & - & - & - & -\\
        Ours & & \checkmark & & \textbf{.0227} & .0341 & \textbf{.117} & \textbf{136} & \textbf{84}\\
        Ours & \checkmark & \checkmark & & .0589 & .0670 & .191 & 170 & 123\\
        Ours & \checkmark & \checkmark & \checkmark & .0229 & \textbf{.0333} & \textbf{.116} & \textbf{134} & 102\\\hline
    \end{tabular}
\end{table}


\subsection{Density prototypes}
We visualize some of the density prototypes $\proxy{\dens}$ for the ScalarFlow dataset in \figref{fig:suppOwnSF80proxy}. These are intended as a rough volumetric guidance via $\loss_{\dproxy}$ and not seen as the real solution.
\begin{figure}
    \centering
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0020.png}{0°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/Own_dProxy_SF80/img_lit_b0000_cam00_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0020.png}{90°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/Own_dProxy_SF80/img_col_b0000_cam02_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0020.png}{30°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/Own_dProxy_SF80/img_sdw_b0000_cam04_0120.png}{}\\%
    \caption{The density prototypes $\proxy{\dens}$ of ScalarFlow scene 80 used in the volumetric guidance $\loss_{\dproxy}$ when training $\mathcal{G_{\mathrm{\mathbf{u}}}}$. These are less detailed then the densities resulting from advection and show some artifacts around the border of the plume.}
    \label{fig:suppOwnSF80proxy}
\end{figure}

\subsection{ScalarFlow data}
Additional results of our method evaluated on ScalarFlow data can be found in \threefigref{fig:suppOwnSF80}{fig:suppOwnSF81}{fig:suppOwnSF82}~ as well as in the supplemental videos.

\begin{figure}
    \centering
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0020.png}{0°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=65 45 35 100, clip}{images/OwnQ_supp_SF80/img_lit_b0000_cam00_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0020.png}{90°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 95, clip}{images/OwnQ_supp_SF80/img_col_b0000_cam02_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0020.png}{30°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 35 95, clip}{images/OwnQ_supp_SF80/img_sdw_b0000_cam04_0120.png}{}\\%
    \caption{Our final model evaluated on ScalarFlow scene 80.}
    \label{fig:suppOwnSF80}
\end{figure}

\begin{figure}
    \centering
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0020.png}{0°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF81/img_lit_b0000_cam00_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0020.png}{90°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF81/img_col_b0000_cam02_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0020.png}{30°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF81/img_sdw_b0000_cam04_0120.png}{}\\%
    \caption{Our final model evaluated on ScalarFlow scene 81.}
    \label{fig:suppOwnSF81}
\end{figure}

\begin{figure}
    \centering
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0020.png}{0°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=90 45 35 120, clip}{images/OwnQ_supp_SF82/img_lit_b0000_cam00_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0020.png}{90°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=20 50 70 120, clip}{images/OwnQ_supp_SF82/img_col_b0000_cam02_0120.png}{}\\%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0020.png}{30°}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0040.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0060.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0080.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0100.png}{}\hfill%
    \mygraphicsB{width=0.164\linewidth,trim=60 55 35 120, clip}{images/OwnQ_supp_SF82/img_sdw_b0000_cam04_0120.png}{}\\%
    \caption{Our final model evaluated on ScalarFlow scene 82.}
    \label{fig:suppOwnSF82}
\end{figure}


\section{Introduction}
Estimating motion is a fundamental problem, and is studied for a variety of settings in two and three dimensions \citep{OFE_SPN, Hur_2021_CVPR, Gregson_2014}. It is also a highly challenging problem, since the motion $\mathrm{\mathbf{u}}$ is a {\em secondary} quantity that typically can't be measured directly and has to be recovered from changes observed in transported markers $\rho$.
We focus on volumetric, momentum-driven materials like fluids, where in contrast to the single-step estimation in optical flow (OF), motion estimation typically considers multiple coupled steps to achieve a stable {\em global transport}.
Furthermore, in this setting the volume distribution of markers $\rho$ is usually unknown and needs to be reconstructed from the observations in parallel to the motion estimation.

So far, most research is focused on the reconstruction of single scenes. Classic methods use an optimization process working with an explicit volumetric representation \citep{ScalarFlow, tomofluid, GlobT}
while some more recent approaches optimize single scenes with neural fields \citep{mildenhall2020nerf, Chu_2022}.
As such an optimization is typically extremely costly, and has to be redone for each new scene, training a neural network to infer an estimate of the motion in a single pass is very appealing.
Similar to most direct optimizations, existing neural network methods rely on multiple input views to simplify the reconstruction \citep{Qiu_GP_2021}. However, this severely limits the settings in which inputs can be captured, as a fully calibrated lab environment is often the only place where such input sequences can be recorded.

The flexibility of motion estimation from single views makes them a highly attractive direction, and physical priors in the form of governing equations make this possible in the context of fluids \citep{eckertCoupledFluidDensity2018, GlobT}. Nonetheless, despite using strong priors, the single viewpoint makes it challenging to adequately handle the otherwise fully unconstrained depth dimension.
We target a deep learning-based approach where a neural network learns to represent the underlying motion structures, such that almost instantaneous, single-pass motion inference is made possible without relying on ground truth motion data.
The latter is especially important for complex volumetric motions,
as reference motions of real fluids can not be acquired directly. Instead, one has to work with reconstructions or even simulated data, suffering from a mismatch between the observations and the synthetic motion data.
While obtaining multiple calibrated captures for training is feasible, using additional views only for losses results in issues with the depth ambiguity of single-view inputs.





In this work, we address the challenging  problem of training neural networks to infer 3D motions from monocular videos in scenarios where no 3D reference data is available.
To the best of our knowledge, we are to first to propose an end-to-end 
approach, denoted by \textit{Neural Global Transport} (NGT) in the following, which
\begin{itemize}
\item[(i)] yields a neural network to estimate a global, dense 3D velocity from a 
single-view image sequence without requiring any 3D ground truth as 
targets. Among others, this is made possible by a custom 2D-to-3D UNet architecture.
\item[(ii)] We address the resulting depth-ambiguity problem using a new approach with differentiable rendering and an adversarial technique.
\item[(iii)] A single network trained with the proposed approach generalizes across a range of different inputs, vastly outperforming optimization-based approaches in terms of performance.
\end{itemize}

   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   


\section{Related work}
\paragraph{Optical flow}
Flow estimation is of great interest in a multitude of settings, from 2D optical flow over scene flow to the capture and physically accurate reconstruction of volumetric fluid flows. 
Operating on a pair of 2D images, optical flow estimates a motion that maps one to the other \citep{Sun:14}.
In this setting, multi-scale approaches in the form of spatial pyramids are a longstanding technique to handle displacements of different scales \citep{Glazer1987}. More recent CNN-based methods also employ spatial pyramids \citep{dosovitskiy2015flownet,OFE_SPN} and can learn to estimate optical flow in an unsupervised fashion \citep{ahmadi2016,Yu2016, UPFlow}.
\paragraph{Scene flow}
Scene flow \citep{vedula1999three} bridges the gap to 3D
where earlier approaches using energy minimization or variational methods\citep{zhang2001sceneflow, huguet2007sceneflow}
have been surpassed by CNNs that bring superior runtime performance while retaining state-of-the-art accuracy \citep{Ilg_2018_ECCV,saxena2019deepsceneflow} and can also be trained without the need for ground truth data \citep{Lee2019,Wang_2019_CVPR}. Flow estimation from a single input is of particular importance as it vastly simplifies the data acquisition and several methods for monocular scene flow have been proposed \citep{Brickwedde_2019_ICCV,Yang_2020_CVPR,Luo2020}. These can be un- or self-supervised \citep{Ranjan_2019_CVPR,Hur_2020_CVPR} and benefit from using multiple frames \citep{Hur_2021_CVPR}.
\paragraph{Fluid flow}
Fluid flows are traditionally extremely difficult to capture and methods ranging from
Schlieren imaging~\mycite{Dalziel00:BOS,Atcheson2008,Atcheson:2008:OEF} and
particle imaging velocimetry (PIV) methods~\mycite{Grant97:PIV,Elsinga:06,Xiong:2017:rainbowPIV}
over laser scanners~\mycite{Hawkins:timeVary,fuchs2007density} to structured light~\mycite{Gu:13} and light path~\mycite{Ji_2013_CVPR} approaches all require specialized setups.
The use of multiple commodity cameras simplifies the acquisition \citep{Gregson_2014,ScalarFlow} and allows for view-interpolation to create additional constrains \citep{tomofluid}.
Few works have attempted to solve monocular flow estimation in the fluids setting. For single-scene optimization \citet{eckertCoupledFluidDensity2018} constrain the motion along the view depth, while \citet{GlobT} use an adversarial approach to regularize the reconstruction from unseen views. \citet{Qiu_GP_2021} have proposed a network that can estimate a long-term motion from a single view, but still require 3D ground truth for training.

\paragraph{3D reconstruction}
In the context of fluids it is common to also reconstruct an explicit representation of the transported quantities.
Such 3D reconstruction is
typically addressed for clearly visible surfaces
\citep{musialski2013survey,koutsoudis2014multi}
where some of the algorithms that have been proposed can incorporate deformations \citep{Zang2018,katoNeural3DMesh2018}.
In this context, volumetric reconstructions make use of voxel grids \citep{papon2013voxel,moon2018v2v}, or more recently
neural network representations \mycite{sitzmann2019deepvoxels,lombardiNeuralVolumesLearning2019,sitzmann2019scene,mildenhall2020nerf}. 
Learned approaches have likewise been used to recover volumetric scene motions, e.g., for moving human characters \citep{mescheder2019occnet} or to separate static and dynamic parts \citep{Chu_2022}.
For coupling with visual observations, several differentiable rendering frameworks have been proposed \citep{kato2020rendersurvey,Zhang2021voldiffrender}. These where used to constrain unseen views of static volumes \citep{henzlerSingleimageTomography3D2018,henzlerEscapingPlatoCave2019} and to reconstruct objects from single images as SDF \citep{SDFDiff}.
Considering fluids, corresponding approaches where used in the context of SPH \mycite{Schenk2018}, for the initialization of a water wave simulator~\mycite{Hu2020}, to realize style transfer onto 3D density field~\mycite{Kim2019,Kim2020}, and for monocular flow reconstruction \citep{GlobT}.














\section{Method}


In this work we focus on the task of estimating 
a global 3D motion $\mathrm{\mathbf{u}}$ of a volumetric quantity $\rho$
over time from a sequence of single view images $\scurr{\imggt}$. 
To that end, we train a generator network $\mathcal{G_{\mathrm{\mathbf{u}}}}$ to estimate the motion $\scurr{\vel}$ between two consecutive time steps $\scurr{\dens}$ and $\snext{\dens}$.
In this context, a simple, supervised method would match the output to a reference velocity $\snext{\vgt}$ given a cost function such as an $\mathcal{L}_2$ loss:
\begin{equation} \label{eq:velRefSupervised}
    \min_{\theta} \SEfn{\mathcal{G_{\mathrm{\mathbf{u}}}}(\scurr{\dens}, \snext{\dens}; \theta) - \snext{\vgt}}.
\end{equation}
However, as ground truth
velocity targets are often not available, unsupervised methods instead leverage a model for the underlying transport process. Given a differentiable discretization of the transport $\mathcal{A}$
we can learn the motion in an unsupervised fashion by requiring that $\scurr{\dens}$ should match $\snext{\dens}$ when $\mathcal{A}$ is applied:
\begin{equation} \label{eq:velRefUnsupervised}
    \min_{\theta} \SEfn{\mathcal{A}(\scurr{\dens}, \mathcal{G_{\mathrm{\mathbf{u}}}}(\scurr{\dens}, \snext{\dens}; \theta)) - \snext{\dens}}.
\end{equation}

As the image sequence $\gt{\image}$ represents projections of volumes that leave the formulation of
\eqref{eq:velRefUnsupervised} to be highly under-constrained, we impose further priors to ensure the training converges to desirable, unimodal solutions.
Assuming that an initial state $\sfirst{\dens}$ is known, we define a multi-step transport as
$\scurr{\dens} = \mathcal{A}^{\iframes}(\sfirst{\dens}, \mathrm{\mathbf{u}}) = \mathcal{A}(\dots \mathcal{A}(\sfirst{\dens}, \sfirst{\vel}) \dots, \mathrm{\mathbf{u}}^{\iframes-1})$,
and the target $\snext{\dens}$ above is replaced by an image-based loss from unsupervised 3D reconstruction \citep{tomofluid,GlobT}.
This loss is realized via a differentiable image formation model $\mathcal{R}$ such that the inferred volumetric state can be compared to the observed target image $\snext{\imggt}$.
In line with $\mathcal{R}$, the network $\mathcal{G_{\mathrm{\mathbf{u}}}}$ is likewise conditioned on the target image $\snext{\imggt}$, which yields the combined learning objective for our approach:
\begin{equation} \label{eq:minTargetLoss}
    \min_{\theta} \SEfn{\mathcal{R}(\mathcal{A}(\scurr{\dens}, \mathcal{G_{\mathrm{\mathbf{u}}}}(\scurr{\dens}, \snext{\imggt}; \theta))) - \snext{\imggt}}.
\end{equation}

While this formulation has the central advantage that it yields a globally coupled, end-to-end process that does not require any volumetric targets, its under-constrained nature
leads to a very challenging environment for learning tasks:
In addition to the well-studied aperture problem of optical flow \cite{Beauchemin1995OF}, the 3D nature of our setting introduces the problem of \emph{depth-ambiguity}: A single density observed in a pixel of $\snext{\imggt}$ can be satisfied by an arbitrary distribution of densities along the corresponding viewing ray direction. 
In contrast to the aperture problem, simple regularizers such as smoothness do not suffice to address depth ambiguity. Below, we explain our solution which uses an adversarial architecture.


\subsection{Depth ambiguity}
For each target color along a ray, an infinite number of
distributions of density along this ray exists, 
e.g. larger and further away vs. smaller and closer.
The existence of multiple parallel observations, such as the pixels of an image, likewise does not guarantee uniqueness.
One approach to address this issue is to refer to multiple observations in the loss during training, as would be available from a calibrated multi-view capture setup.
However, since the network has no way to infer the unique density distributions pertaining to these side targets from its single input view, such an approach typically leads to learning an suboptimally averaged solution.
Instead, we address this depth-ambiguity by expanding the simple target loss
\begin{equation} \label{eq:TargetLoss}
    \loss_{\target} = \SEfn{\mathcal{R}(\snext{\dens}) - \snext{\imggt}}
\end{equation}
from \eqref{eq:minTargetLoss}, where $\snext{\dens} = \mathcal{A}(\scurr{\dens}, \mathcal{G_{\mathrm{\mathbf{u}}}}(\scurr{\dens}, \snext{\imggt}; \theta))$,
with the adversarial loss used by \citet{henzlerEscapingPlatoCave2019} and \citet{GlobT} to restrict $\rho$ to a plausible appearance without imposing a specific solution.
Specifically, we follow previous work \citep{GlobT} and make use of 
a discriminator $\mathcal{D}$ with an RaLSGAN objective \citep{rGAN} 
\begin{equation} \label{eq:DiscLoss}
\begin{split}
    \loss_{\disc}(\snext{\dens}, l) &= \mathbb{E}_{p_{\rm{data}}} \left[ \left( \mathcal{D}(\snext{\imggt}) - \mathbb{E}_{p_{\rm{view}}(\omega)} \mathcal{D}(\mathcal{R}(\snext{\dens}, \omega)) - l \right)^2 \right]\\
    &+ \mathbb{E}_{p_{\rm{view}}(\omega)} \left[ \left( \mathcal{D}(\mathcal{R}(\snext{\dens}, \omega)) - \mathbb{E}_{p_{\rm{data}}} \mathcal{D}(\snext{\imggt}) + l \right)^2 \right],
\end{split}
\end{equation}
where $\omega$ are randomly sampled view directions and the label $l$ is 1 or -1 when training $\mathcal{D}$ or using it as loss, respectively.
To focus on compact volumes and 
to prevent unnecessary scattering of densities along the viewing direction,
we add a depth-Loss $\loss_{z}} %{\loss_{\rm{center}}$,
assuming that the observed densities are concentrated around position $c_z$ along the view's depth direction, the z-axis in our case.
Thus, for every position $p$ we regularize $\rho$ depending on the distance to a center position $c_z$
\begin{equation} \label{eq:CenterLoss}
    \loss_{z}} %{\loss_{\rm{center}} = \snext{\dens}(p)^2  ((c_z-p_z) 2/r)^2,
\end{equation}
where $p_z$ is the projection of $p$ on the primary view direction.
As we choose $c_z$ to be in the center of the grid, we normalize with half the grid resolution $r$.
To further constrain and stabilize the estimated velocity we introduce a prototype volume $\snext{\dproxy}$ generated from a single input image (see \secref{sec:dens-est}) as 3D target for the transported density. This again takes the form of a simple $\mathcal{L}_2$ loss:
\begin{equation} \label{eq:ProxyLoss}
    \loss_{\dproxy} = \SEfn{\snext{\dens} - \snext{\dproxy}}.
\end{equation}



\subsection{Estimating Volumetric Motions}
\begin{figure}
    \centering%
    \hfill%
    \includegraphics[width=0.42\linewidth]{images/NeuralGlobT_symb2_crop.png}\hfil
    \includegraphics[width=0.56\linewidth]{images/VelGen-proxy_symb2_legend_crop.png}\hfil
    \caption{Left: An overview over the complete NGT framework. We generate an initial density volume $\sfirst{\dens}$ that is advected by the velocity to form a sequence. Density estimates are used in addition to the single input image to guide and stabilize the velocity generation. Velocity training is done end-to-end over the whole sequence.
    Right: Our multi-scale velocity estimator $\mathcal{G_{\mathrm{\mathbf{u}}}}$, shown for 3 resolution scales. The inputs contain information about the current ($\iframes$) and next ($\iframes+1$) time step. Each scale generates a residual velocity potential which is used to advect the inputs of step $\iframes$ before generating the next residual. The final velocity is divergence free due to using the curl $\nabla\times$.}
    \label{fig:velgen}
\end{figure}
In addition to the current state $\scurr{\dens}$ and the target image $\snext{\imggt}$, as indicated by \eqref{eq:minTargetLoss}, our generator network also receives $\scurr{\imggt}$, $\scurr{\dproxy}$ and $\snext{\dproxy}$ as input.
The images are projected into the volume using inverse ray-marching denoted by $\render^{-1}$, before being concatenated to the density fields.
This way,
the inputs trivially match the target and rendering losses, are coherent between frames, and
the network can compare the back-projections and prototype volumes to estimate the 3D velocity based on the current state $\scurr{\dens}$.
Thus, the velocity estimation becomes 
\begin{equation}
    \scurr{\vel} = \mathcal{G_{\mathrm{\mathbf{u}}}}(\scurr{\dens}, \render^{-1}(\scurr{\imggt}), \scurr{\dproxy}, \render^{-1}(\snext{\imggt}), \snext{\dproxy}).
\end{equation}
The architecture of $\mathcal{G_{\mathrm{\mathbf{u}}}}$ further follows established multi-scale approaches of optical flow and scene flow \citep{OFE_SPN}, as shown in \figref{fig:velgen}. A residual velocity $\mathrm{\mathbf{u}}^l$ is generated at several resolution scales $l$ with the per-resolution network component denoted as  $\mathcal{G_{\mathrm{\mathbf{u}}}}_{l}$. Our $\mathcal{G_{\mathrm{\mathbf{u}}}}$ further utilizes weight sharing between scales \citep{UPFlow}, such that self-similar structures can be reused.
In combination, velocity generation for a single time step takes the recursive form of
\begin{equation}
   
   
    \scurr{\vel}_{l} = \mathcal{G_{\mathrm{\mathbf{u}}}}_{l}(\mathcal{A}(\dws{\scurr{\dens}} \oplus \render^{-1}(\gstep{\image}{\fcurr}) \oplus \dws{\scurr{\dproxy}}, \ups{\scurr{\vel}_{l-1}}), \render^{-1}(\gstep{\image}{\fnext}), \snext{\dproxy}, \ups{\scurr{\vel}_{l-1}}) + \ups{\scurr{\vel}_{l-1}},
    \label{eq:MSvel}
\end{equation}
where $\oplus$ is concatenation along the channel dimension, $\ups{\scurr{\vel}_{l-1}}$ is the up-sampled velocity of the previous scale, $\scurr{\vel}_{0} = 0$ and final velocity $\scurr{\vel}_{L} = \mathcal{G_{\mathrm{\mathbf{u}}}}_{L}(\dots)$, 
$\ups$ denotes up-sampling using quadratic B-spline kernels, and $\dws$ is down-sampling using average pooling.
The inputs from the current time-step $t$ are warped such that $\mathcal{G_{\mathrm{\mathbf{u}}}}_{l}$ only needs to estimate the residual velocity.



Even with the constraints of equations~\ref{eq:TargetLoss} to~\ref{eq:ProxyLoss},
considerably large solution spaces exist that reach the target state, e.g., unphysical solutions that create mass.
For this reason we include additional constraints on magnitude, smoothness and divergence.
For stability we enforce the CFL condition on the residual of each velocity scale with an $\mathcal{L}_2$ loss on velocity magnitude,
while smoothness is constrained for the combined velocity on each scale.
To respect mass conservation the velocity field has to be divergence free, i.e. $\nabla \cdot \mathrm{\mathbf{u}} = 0$.
Instead of adding a divergence-freeness loss like most previous work, we use a hard constraint 
\citep{kim2019deep}
that is compatible with the residual multi-scale approach:
we define $\mathrm{\mathbf{u}}$ as the curl ($\nabla \times$) of a vector potential as $\mathrm{\mathbf{u}}=\nabla \times \mathcal{G_{\mathrm{\mathbf{u}}}}$.
As the curl is a linear operator,
$\nabla \times (sF + G) = s \nabla \times F + \nabla \times G$, with $F,G$ being vector fields and $s$ a scalar,
\Eqref{eq:MSvel} and the output of $\mathcal{G_{\mathrm{\mathbf{u}}}}$ are treated as a vector potential.





\subsection{Density estimation}\label{sec:dens-est}
Our motion estimation additionally hinges on a second network $\mathcal{G_{\rho}}$, whose role it is to estimate individual volumetric densities $\scurr{\dproxy}$ from a single image $\scurr{\imggt}$:
\begin{equation}
    \scurr{\dproxy} = \mathcal{G_{\rho}}( \scurr{\imggt}; \Phi).
\end{equation}
We employ this network for the prototype density loss \ref{eq:ProxyLoss}
and to infer the initial distribution of markers with $\sfirst{\dens} := \sfirst{\dproxy} = \mathcal{G_{\rho}}( \sfirst{\imggt}; \Phi)$, which was previously assumed to be known.
As the challenges of the single view estimation are similar for density and for velocity, using a generator network in the already established GAN setting is a natural choice and
we can reuse the losses as described above: $\mathcal{L}_{\mathcal{G_{\rho}}} = \loss_{\target} + \loss_{\disc}(\mathcal{G_{\rho}}, -1) + \loss_{z}} %{\loss_{\rm{center}}$.
$\mathcal{G_{\rho}}$ employs a custom mixed 2D/3D UNet-architecture. The encoder processes 2D information, while the skip connections
are realized via $\render^{-1}$.



\subsection{Implementation and training}
We implement NGT with TensorFlow in Python.
The $\mathcal{R}$ and $\mathcal{A}$ operations are custom:
For $\mathcal{R}$, used in both the target loss (\eqref{eq:TargetLoss}) and $\loss_{\disc}$, we use a differentiable volume ray-marching \citep{GlobT}, which supports attenuation, single-scattering, and the use of background images, but is limited to 
isometrically scattering densities and fixed lighting.
For the advection operator $\mathcal{A}$ we employ a discrete, differentiable MacCormack advection scheme.
First, we train $\mathcal{G_{\rho}}$ alone on individual frames, where we grow the resolution from 8 to 64 during training.
Then we freeze $\mathcal{G_{\rho}}$ and train $\mathcal{G_{\mathrm{\mathbf{u}}}}$
with up to 5 advection-steps.
We again grow the resolution of the velocity estimation network, while the estimation of $\sfirst{\dens}$ and the advection happen on the maximum resolution, using up-scaled velocities until $\mathcal{G_{\mathrm{\mathbf{u}}}}$ reaches the maximum resolution.
$\mathcal{D}$ is trained during both density and velocity training.
We refer to appendix \ref{app:tech} for more details.









\section{Evaluation}
\begin{figure}%
    \centering%
    %
    \mygraphicsB{width=0.14\linewidth,trim=0 60 0 60, clip}{images/torus_ref-c_front.png}{Reference, fix}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 60 0 60, clip}{images/torus_mv-nD-c_front.png}{fix, multi, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_mv-nD-d_front.png}{var, multi, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_sv-nD-d_front.png}{var, single, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_mv-D-d_front.png}{var, multi, $\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_sv-D-d_front.png}{var, single, $\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_ref-d_front.png}{reference, var}\\%
    \mygraphicsB{width=0.14\linewidth,trim=0 60 0 60, clip}{images/torus_ref-c_side.png}{reference, fix}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 60 0 60, clip}{images/torus_mv-nD-c_side.png}{fix, multi, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_mv-nD-d_side.png}{var, multi, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_sv-nD-d_side.png}{var, single, $\neg\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_mv-D-d_side.png}{var, multi, $\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_sv-D-d_side.png}{var, single, $\mathcal{D}$}\hfill%
    \mygraphicsB{width=0.14\linewidth,trim=0 65 0 55, clip}{images/torus_ref-d_side.png}{Reference, var}%
    \caption{
   
    Depth-ambiguity ablation study with the shape dataset:
    In the top row all versions closely match the input view while the side view (bottom) shows severe degradation when increasing depth ambiguity by varying the position of the objects (fix $\rightarrow$ var) or using only the input view for $\loss_{\target}$ (multi $\rightarrow$ single). 
    Providing additional views alongside the discriminator does not yield further improvements.
    $\mathcal{D}$ by itself can recover a plausible configuration, even if the density does not exactly match the unknown reference location.
    }%
    \label{fig:smokeDisc}%
\end{figure}

We first present two ablations to illustrate the importance of handling depth ambiguity, and of the prototype density volumes.
We then evaluate the method in comparison to a series of learned and optimization-based methods for a synthetic and a real-world dataset.


\subsection{Depth ambiguity} \label{sec:abl.depthAmbiguity}
In \myreffig{smokeDisc} we show the effect of depth ambiguity.
If there is no depth variation,
a network can easily recover the object when using multiple target views in the loss (column 2), leading to low volumetric and perceptual errors (\tabref{tab:metricsAbl}).
If the object's position is randomized (column 3),
\begin{wraptable}{}{0.65\linewidth}
	\vspace{-3mm}%
    \caption{Quantitative evaluation of the depth-ambiguity issue as seen in \figref{fig:smokeDisc}.
    Side uses 7 evenly distributed views.
    Lower values are better for all metrics under consideration.
    } 
    \vspace{0.2cm} 
    \label{tab:metricsAbl}
    \centering
    \begin{tabular}{l|llll|l}
        \hline
        \  & Input & Side & &  & $\rho$ \\% & Random
        Version & RMSE & RMSE & LPIPS & FID & RMSE\\\hlin
        fix, multi, $\neg\mathcal{D}$ & .0080 & .0106 & .0200 & 48 & .260 \\\hline
        var, multi, $\neg\mathcal{D}$ & .0160 & \textbf{.0509} & .1237 & 120 & \textbf{.636} \\
        var, single, $\neg\mathcal{D}$ & \textbf{.0042} & .0681 & .1345 & 98 & .969 \\
        var, multi, $\mathcal{D}$ & .0151 & .0525 & .1239 & 80 & \textbf{.642} \\
        var, single, $\mathcal{D}$ & .0064 & .0694 & \textbf{.1140} & \textbf{68} & .930 \\\hline
    \end{tabular}
\end{wraptable}
the network can no longer settle on a clear position as the added variation in depth is not evident from the single input.
Thus, the network is presented with multiple solutions for the same input over the course of the training, severely deteriorating the quality and drastically increasing the FID by 2.5 times.
Simply removing the multiple targets improves the FID at the cost of LPIPS, volume error, and visual quality.
Adding the discriminator to the multi-target setup improves the perceptual metrics, but visually the result still looks blurry.
Keeping $\mathcal{D}$ but removing the multiple targets further improves the perceptual quality, sharpening the results and improving FID to 68, at the cost of volumetric accuracy.
In the end, the multiple target views hinder the discriminator
and lead to averaged solutions that are favoured by simple metrics such as RMSE, but have lower perceptual scores.
While the discriminator can not recover the true shape and depth location of the sample,
as indicated by the relatively high volume error,
the compact density it produces nonetheless represents a very good starting point for the motion estimation, given the single view input.

\subsection{Prototype volumes $\proxy{\dens}$} \label{sec:abl.warpLoss}
\begin{figure}
    \centering
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 270, clip}{images/abl-no-proxy/front_40}{Frame 40, 0°, $\neg \proxy{\dens}$}\hfill%
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 270, clip}{images/abl-no-proxy/side_40}{Frame 40, 90°, $\neg \proxy{\dens}$}\hfill%
    \ \
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 270, clip}{images/abl-proxy/front_40}{Frame 40, 0°, $\proxy{\dens}$}\hfill%
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 270, clip}{images/abl-proxy/side_40}{Frame 40, 90°, $\proxy{\dens}$}\\%
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 80, clip}{images/abl-no-proxy/front_120}{Frame 120, 0°, $\neg \proxy{\dens}$}\hfill%
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 80, clip}{images/abl-no-proxy/side_120}{Frame 120, 90°, $\neg \proxy{\dens}$}\hfill%
    \ \
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 80, clip}{images/abl-proxy/front_120}{Frame 120, 0°, $\proxy{\dens}$}\hfill%
    \mygraphicsB{width=0.246\linewidth,trim=0 25 0 80, clip}{images/abl-proxy/side_120}{Frame 120, 90°, $\proxy{\dens}$}%
    \caption{Adding the prototype volume $\proxy{\dens}$ produced by our 2D-to-3D UNet as guidance (right side) stabilizes the inference over long evaluations and results in better target matching and a better overall density distribution, especially from unseen views. Metrics can be found in \tabref{tab:metricsSF}.
   
    }
    \label{fig:velProxy}
\end{figure}
Using the prototype densities $\proxy{\dens}$ as explicit 3D constraint in the form of $\loss_{\dproxy}$ may at first seem counter-intuitive in the presence of the depth ambiguity.
However, as 
the $\proxy{\dens}$ density volumes are generated from single images,
there is only a single $\proxy{\dens}$ associated with each input. 
This yields a reduction of ambiguity for the task of $\mathcal{G_{\mathrm{\mathbf{u}}}}$ when using the prototypes as loss. 
Furthermore, we provide $\snext{\dproxy}$ as additional input to $\mathcal{G_{\mathrm{\mathbf{u}}}}$, to simplify the task of motion inference.
Overall, 
this highlights the importance and central role of the novel 2D-to-3D UNet architecture.
The addition of $\proxy{\dens}$ to the velocity training
stabilizes the inference of long sequences, as evident from \figref{fig:velProxy} and \tabref{tab:metricsSF}. While there is not much difference in the early frames, the network lacking the stabilization clearly diverges after evaluating 120 frames.



\subsection{Results}
We evaluate our method on both
synthetic smoke flows 
and the real-world captures from the ScalarFlow dataset \citep{ScalarFlow}.
As a representative of simpler network architectures, i.e. without multi-scale handling and the 2D-to-3D UNet, we combine the architecture from \citet{Qiu_GP_2021} with our loss formulation. This combination is denoted by (\emph{RapidGen}).
We further compare to the direct optimization algorithms of \citet{ScalarFlow} (\emph{ScalarFlow}) and \citet{GlobT} (\emph{GlobTrans}). These can achieve better target matching and long-term transport because each scene is optimized over the full trajectory individually, but this comes at the cost of vastly increased (more than $\times 100$) reconstruction times.
Due to the large space of solutions for the single view problem in a setting as chaotic as fluid dynamics,
we evaluate the general appearance of the estimations with the perceptual metric LPIPS \citep{zhang2018lpips}
and the Fréchet Inception Distance (FID) between random views of the estimated sequences and random samples of input images to obtain a measure of likeness in appearance. This allows us to asses how realistic the overall look of the plumes is.



\subsubsection{Synthetic plumes}
\begin{figure}
    \centering
   
   
   
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 190, clip}{images/RW_vel_Synth/img_lit_b0000_cam00_0080}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 190, clip}{images/Own_vel_Synth/img_lit_b0000_cam00_0080}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 190, clip}{images/SF_vel_Synth/img_lit_b0000_cam00_0080}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 190, clip}{images/GlobT_vel_Synth/img_lit_b0000_cam00_0080}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 190, clip}{images/Ref_vel_Synth/img_lit_b0000_cam00_0080}{}\\%
   
    \mygraphicsB{width=0.197\linewidth,trim=0 30 0 190, clip}{images/RW_vel_Synth/img_lit_b0000_cam02_0080}{RapidGen}\hfill%
    \mygraphicsB{width=0.197\linewidth,trim=0 30 0 190, clip}{images/Own_vel_Synth/img_lit_b0000_cam02_0080}{Ours}\hfill%
    \mygraphicsB{width=0.197\linewidth,trim=0 30 0 190, clip}{images/SF_vel_Synth/img_lit_b0000_cam02_0080}{ScalarFlow}\hfill%
    \mygraphicsB{width=0.197\linewidth,trim=0 30 0 190, clip}{images/GlobT_vel_Synth/img_lit_b0000_cam02_0080}{GlobTrans}\hfill%
    \mygraphicsB{width=0.197\linewidth,trim=0 30 0 190, clip}{images/Ref_vel_Synth/img_lit_b0000_cam02_0080}{Reference}\\%
   
   
   
   
   
   
   
   
   
   
   
   
   
    \caption{Qualitative comparison between different approaches using synthetic plume data for time-step 80. Top is the input view which all method match fairly well. Bottom is a 90° side view where the shortcomings of the different approaches become visible. 
   
   
    Due to the overshoot, ScalarFlow densities are shown with a factor or $1/2$.
    }
    \label{fig:CMPsynth}
\end{figure}
\begin{table}[]
    \caption{Quantitative evaluation of different methods trained on a synthetic plumes dataset.
   
    } 
    \vspace{0.2cm} 
    \label{tab:metricsSynth}
    \centering
    \begin{tabular}{l|lllll|lll}
        \hline
        \  & Input & Side & & & Random & $\rho$ & $\mathrm{\mathbf{u}}$ & \\
        Algorithm & RMSE & RMSE & LPIPS & FID & FID & RMSE & RMSE & EPE\\\hline
       
        RapidGen & .0145 & .0277 & .0450 & 105 & 76 & 1.65 & .494 & .851 \\
       
        Ours & .0156 & .0243 & .0444 & 101 & 97 & .431 & 2.06 & 3.29 \\\hline
        ScalarFlow & .116 & .120 & .158 & 135 & 106 & 1.39 & .120 & .104 \\
        GlobTrans & .0122 & .0440 & .0877 & 87 & 30 & .373 & .143 & .137 \\\hline
    \end{tabular}
\end{table}

We first evaluate our method with a synthetic data set consisting of simulated plumes of hot marker densities in front of a black background. While the flows are relatively simple compared to their real-world counterparts, the smooth density content makes it very difficult to estimate motions as the differentiable advection relies on spatial gradients in the density for back-propagation.
For completeness, we measure the accuracy of reconstructing the input image in column one of \tabref{tab:metricsSynth}, for which all methods show a low error.
Our goal is to estimate plausible solutions, rather than an exact match to a reference, as the ambiguity in depth allows for many solutions that match the available inputs.
This can be seen particularly well in the volumetric comparisons, where our result has a good match with the reference density with a RMSE of 0.431, but disagrees in terms of underlying motion, leading to a high difference in the velocities. The corresponding videos still show a calm, swirling motion similar to the reference.
A visual comparison is shown in \figref{fig:CMPsynth}.


Comparing the unseen side targets
reveals the differences between the approaches,
especially in conjunction with the estimated motions, which can be seen in the supplemental videos.
GlobTrans performs best in terms of the perceptual metrics, reaching a low FID of 30 due to having the same large smooth structures as the reference. However, GlobTrans exhibits unnatural, excessive motion in the side view.
RapidGen is closer in appearance to the reference, but lacks a clearly defined plume shape leading to a density error 3.8 times higher than ours. And even though the velocity of RapidGen matches the reference better, the videos show that it consists mainly of a static upwards motion.
ScalarFlow exhibits smooth and natural motions but suffers from missing depth regularization, resulting in an excessive amount of density.
These results highlight the complexity of the single-view motion estimation and the multitude of viable solutions when given only a single image even for synthetic inputs, and illustrate the behavior of the different learned and optimization-based methods. Next, we evaluate their performance with real-world inputs.



\begin{figure}
    \centering
   
   
   
    \mygraphicsTB{width=0.197\linewidth,trim=0 30 0 50, clip}{images/RW_vel_SF/SF_front_120}{RapidGen}{input view 0°}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 30 0 50, clip}{images/own_vel_SF/SF_front_120}{Ours}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 30 0 50, clip}{images/SF_vel_SF/SF_front_120}{ScalarFlow}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 30 0 50, clip}{images/GlobT_vel_SF/SF_front_120}{GlobTrans}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 30 0 50, clip}{images/Ref_vel_SF/SF_front_120}{Reference}\\%
   
    \mygraphicsTB{width=0.197\linewidth,trim=0 40 0 50, clip}{images/RW_vel_SF/SF_far-right_120}{}{side view 300°}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 50, clip}{images/own_vel_SF/SF_far-right_120}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 50, clip}{images/SF_vel_SF/SF_far-right_120}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 50, clip}{images/GlobT_vel_SF/SF_far-right_120}{}\hfill%
    \mygraphicsT{width=0.197\linewidth,trim=0 40 0 50, clip}{images/Ref_vel_SF/SF_far-right_120}{}\\%
    \caption{Qualitative comparison between different approaches using ScalarFlow data for time-step 100.
    Our method closely matches the given input and has a clearly defined shape that matches the general shape of the reference. It is only surpassed by the costly single-scene reconstruction method GlobTrans. RapidGen was adapted to be trained without 3D GT.}
    \label{fig:CMPreal}
\end{figure}
\begin{table}[h!]
    \caption{Quantitative evaluation of different methods trained on the real world ScalarFlow dataset. Lower values are better for all metrics under consideration.} 
    \vspace{0.2cm} 
    \label{tab:metricsSF}
    \centering
    \begin{tabular}{l|lllll|l}
        \hline
        \  & Input & Side & & & Random & Inference time\\
        Algorithm & RMSE & RMSE & LPIPS & FID & FID & 120 steps\\\hline
       
        RapidGen \citep{Qiu_GP_2021}& \textbf{.0227} & .0405 & .173 & 186 & 163 & 3m \\
        Ours w/o $\proxy{\dens}$ & .0589 & .0670 & .191 & 170 & 123 & 4m \\
       
        Ours & .0229 & \textbf{.0333} & \textbf{.116} & \textbf{134} & \textbf{102} & 4m \\\hline
        ScalarFlow \citep{ScalarFlow} & .0254 & .0488 & .152 & 153 & 119 & 6.5h \\
        GlobTrans \citep{GlobT} & .0124 & .0281 & .085 & 102 & 77 & 37.5h \\\hline
    \end{tabular}
\end{table}

\vspace{-0.2cm}
\subsubsection{Real-world captures}
\begin{figure}
    \centering%
   
   
    \mygraphicsTB{width=0.246\linewidth,trim=0 45 0 120, clip}{images/RW_vel_SF/col_side_120}{RapidGen}{$\rho$ from 90°}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 45 0 120, clip}{images/own_vel_SF/col_side_120}{Ours}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 45 0 120, clip}{images/SF_vel_SF/col_side_120}{ScalarFlow}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 45 0 120, clip}{images/GlobT_vel_SF/col_side_120}{GlobTrans}\\%
   
   
    \mygraphicsTB{width=0.246\linewidth,trim=0 85 0 120, clip}{images/RW_vel_SF/sdw_30_120}{}{w/ shadows from 30°}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 85 0 120, clip}{images/own_vel_SF/sdw_30_120}{}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 85 0 120, clip}{images/SF_vel_SF/sdw_30_120}{}\hfill%
    \mygraphicsT{width=0.246\linewidth,trim=0 85 0 120, clip}{images/GlobT_vel_SF/sdw_30_120}{}\\%
    \caption{Visualization of density distributions of different approaches using ScalarFlow data for time-step 100.
    Our method shows similar structures as GlobTrans while being slightly less detailed.
    The single-scale RapidGan network fails to replicate the intricate 3D structures of real world data.
   
   
    }%
    \label{fig:CMPreal2}%
\end{figure}
The real-world data of the ScalarFlow dataset 
introduces the challenges of
fine structures, sensor noise, and real-world backgrounds. 
It also does not provide measured 3D ground truth for comparisons, but the  dataset still provides 5 calibrated views. With the center one being used as input, we use the remaining 4 to quantitatively evaluate the estimations from unseen views. 
Again, the input reconstruction in
\tabref{tab:metricsSF} is good for all methods under consideration.

For the unseen side targets,
our method with an FID of 134 now clearly outperforms RapidGen with 186, as well as the expensive ScalarFlow optimization (153). Only the GlobTrans optimization manages to do better in terms of this metric.
We again evaluate the FID between random views of the estimated sequences and random samples of the dataset which clearly detects the lack of features in the RapidGen result.
It also shows that structure-wise our method surpasses ScalarFlow, which 
takes ca. 97 times longer to produce a single output. While the GlobTrans optimization fares even better, it does so at the expense of a more than 560 times longer runtime. These measurements show that our trained network successfully generalizes to new inputs, and yields motion estimates that are on-par in terms of realism with classic, single-case optimization methods.

\vspace{-0.2cm}
\section{Conclusion}
\vspace{-0.1cm}
We presented \textit{Neural Global Transport} (NGT), a new method to estimate a realistic density and velocity sequence from new single view sequences with a single pass of neural network evaluations.
Our method and implementation currently have several limitations: e.g., it currently only supports
white smoke, while anisotropic scattering, self-illuminaton, or the use of other materials would require an extension of the rendering model.
Also, our transport model relies on 
advection without obstructions.
Here, our method could be extended to support obstacles,
or multi-phase flows.
For even better long-term stability, the method could be extended to pass back information about future time steps for improved guidance.

Nonetheless, we have demonstrated that our networks can be trained with only single view data and short time horizons while still being stable for long sequences during inference. 
We solved the depth ambiguity arising from the single-view setting by using an adversarial loss, and we stabilized the velocity estimation by providing estimated prototype volumes both as input and as soft constraint.
The resulting global trajectories of NGT are
qualitatively and quantitatively competitive with single-scene optimization methods which require orders of magnitude longer runtimes.


\section{Reproducibility Statement}
Our source code is publicly available at \url{https://github.com/tum-pbs/Neural-Global-Transport} and includes the data and configurations necessary to reproduce all results of the paper. Further details about network architectures, training procedures and hyperparameters can also be found in the appendix.



\subsubsection*{Acknowledgments}
This work was supported by the Siemens/IAS Fellowship Digital Twin, and the DFG Research Unit TH 2034/2-1.


\section{Changes}
\begin{itemize}
    \item MAIN
	\item Fig. 1 simplified graphics, added mention of curl operator
	\item stated that we use RaLSGAN because it was used by the previous paper
	\item reworked Eq. 6
	\item 3.2 tried to clarify CFL loss
	\item 3.4 clarified background and smoke color in rendering
	\item Fig. 4 updated left side with new results. refered to metrics table.
	\item Fig. 5 updated own synthetic results, updated metrics in Tab. 1
	\item 4.3.1 added mention of the black background
	\item Fig. 6 \& 7 updated own SF results, updated metrics in Tab. 2
	\item 4.3.2 added mention of the real background
	\item 5 added data and rendering limitation, added future work possibility
	\item 6 refered to appendix
    \item APPENDIX
	\item A.2 magnitude/CFL and smoothness loss formulas
	\item A.4 included background in renderin equation
	\item A.5 added explanation of projection into volume
	\item A.6, Fig. 9 added discussion of interpolation for curl
	\item B.1, Fig. 10 added density prototypes
	\item Fig. 11,12,13 updated with new results
    \item SUPPLEMENT
	\item updated own results where applicable
	\item added new videos for the new appendix figures
\end{itemize}

\section{TODO}
\begin{itemize}
    \item new tests
    \begin{itemize}
        \item quadratic potential/curl interpolation
        \begin{itemize}
            \item SF results are visually improved. image metrics also.
            \item Synth results are visually improved. metrics are mixed.
        \end{itemize}
        \item additional ablations: regularization (like Fig. 3?)
        \begin{itemize}
            \item Discriminator (dens+vel training)
            \item dens center (dens+vel training)
            \item vel smoothness
            \item vel CFL
            \item + metrics?
            \item multi-target (+disc)
        \end{itemize}
        \item joint dens-vel training to further improve proxies and initial guess?
    \end{itemize}
    \item paper corrections, clarity
        \begin{itemize}
            \item intuitive explanations (esp. eq. 6)
            \item missing definitions ($L_u$)
            \item eq. 6, the center loss
            \item changes to vel interpolation
        \end{itemize}
    \item paper additions
    \begin{itemize}
        \item new results, tests, and ablations
        \item visualize proxy volumes (appendix)?
        \item discussion of limitations
    \end{itemize}
    \item response and clarifications to reviewers
    \begin{itemize}
        \item clarifications of misunderstandings/missed parts
        \item changes to paper
    \end{itemize}
\end{itemize}

\section{Reviews}
\begin{itemize}
    \item \myquote{In general the proposed method is technically sound.}
    \item Some things need to be clarified: notation, diagrams (legend). try to simplify where possible.
    \comment{Thanks for pointing out issues with the clarity. We tried to improve it where possible, specifically ...}
    \item new results
    \comment{only difference is potential upscaling.}
    \item The results are bad. error metrics, artifacts. / On the quality of the results:
    \comment{estimation, not accurate reconstruction, also free vel in empty spaces (masked velocity metrics are better, but synth is still worse). improved quality and got rid of artifacts with quadratic interpolation (explain, seems obvious in hindsight). clarifications about what is observed (density via single image) and what is secondary (velocity via density change, ambiguous). To clarify, the density is defined as the result of the transport (as per Eq. ?), only the first density is directly estimated/synthesized by a network. the system {\em has} to use the transport/velocity to make the density match the target. As such, the method can account for errors in a previous state by estimating/synthesizing a velocity that corrects that error. Due to the relatively big freedom in the velocity, the initial guess does hardly matter after a few frames.}
    \item background in rendering:
    \comment{Thanks for pointing that out. The background is actually included in the rendering (from the reference Franz et al.) and present in the target loss $L_{tar}$. Input images are provided to the networks as-is, so if there is a background the networks have to handle that.}
    \item More experiments and ablations are required: for regularization, for multi-target loss.
    \comment{
    no time for velocity ablations. we can do the missing ablations (CFL, smoothness) for the final version?
    }
    \item Other changes made to the paper
    \comment{Reworked the explanations about the center loss $L_c$ (Eq. 6).
    
    }
\end{itemize}
\answer{
We want to thank all reviewers for their efforts and constructive feedback.

We have uploaded a revised version with highlighted changes where we have updated our results and revised several sections for increased clarity. We also added more information and additional comparisons to the appendix.

Specifically, we have addressed the grid-aligned artifacts present in both our synthetic and ScalarFlow results. These artifacts are the result of discontinuities in the velocities at cell boundaries, caused by the combination of linear interpolation and curl. Since the curl is defined as differences of spatial derivatives, applying it to a piecewise linear field results in discontinuities.
The revised version of our paper includes results using quadratic B-splines for interpolation, which removes these artifacts.
No other changes where made to our method: only $G_u$ was trained anew with the B-spline interpolation, retaining all original hyperparameters. This update improves the ScalarFlow metrics across the board, with the random view FID improving from 117 to 102. 

We give more detailed answers to the individual questions of reviewers below.
}






\subsection{UynW}
\myquote{Overall, this work presents an effective and efficient solution to the challenging problem of estimating 3D flow and volumetric densities from a monocular video. Leveraging unsupervised learning and adversarial losses appears to be a promising direction. However, the quality of the estimated 3D flow is not as good. To improve the results, more work needs to be done to adjust the network and losses.}

\begin{itemize}
    \item The quality of the estimated flow is not very good. In Table 1, the errors in estimated flow u is much larger than the other methods on the synthetic dataset.
    \comment{Yes, we need better results, esp. for synthetic. I'm working on it.\newline
    Also consider that we are not trying to match a specific GT as accurately as possible, but to estimate an unobserved quantity in a chaotic and severely under-constrained setting. Thus, there are likely multiple velocities that lead to a desirable result (given the sparse constrains) but do not match a specific reference. The FID and input image matching RMSE are more suitable for evaluation here, the comparison to GT was added for completeness.\newline
    free velocity in empty spaces. restrict metric to areas with density?
    Besides this point we do acknowledge that the quality of our results could be improved, especially in the synthetic case. We already achieved some improvements by using a higher order interpolation kernel when upsampling velocity potentials.}
    \item Looking at the videos in the supp material, there are also noticeable artifacts in the flow motion. Perhaps, the design of network and losses makes the proposed method focus more on synthesizing the densities $\rho$ than predicting the flow $u$?
    \comment{the densities are purely a result of the flow (after the initial guess). After further investigation the artifacts seem to stem from using the curl in combination with a linear interpolation of the velocity potential. Since the curl uses the gradient of the potential, linear interpolation leads to non-continuous velocities, resulting in grid-aligned artifacts. Using a higher order interpolation scheme easily fixes this, as visible in our updated results.}
    \item What are the limitations of this work? This is not discussed in the paper.
    \comment{quite static setup (background, lighting). simplified image formation (raymarching with single scattering).}
    \item Notation: this paper uses a large amount of notation. Some are not properly defined. For example, in Figure 1, the loss $L_u$ is not defined in the paper. Also, the notation $\nabla\times$ is not explained.
    \comment{$L_u$ is any loss on the velocity, only magnitude (CFL) and smoothness regularization in our case. $\nabla\times$ is the curl operator. we can state this more explicitly in the paper.}
    \item Consistency: For example, in Eq.(3), the input to network $G_u$ includes $\rho^t, I^{t+1}$, but it is later shown in Eq.(8) that the input to $G_u$ consists of five terms. This is quite confusing.
    \comment{It is stated in the paragraph above Eq. (8) that we modify the input to $G_u$ from the initial version.}
    \item Grammar errors and typos: page 1: "is a very appealing" page 3: "leadsto"
    \comment{fixed.}
    \item I also suggest that some long sentences to be broken into shorter sentences to improve the readability.
    \comment{I can make a pass and see what I can find, but this is pretty vague.}
\end{itemize}
\answer{
We thank the reviewer for the constructive comments and the positive assessment of our submission.

\paragraph{Volumetric velocity errors} 
It is worth noting that our work does not try to match a single, specific reference but rather addresses the estimation of an unobserved quantity in a chaotic and severely under-constrained setting.
Thus, the high velocity error can be explained by the relatively unconstrained velocity and the multitude of solution that can lead to a sequence matching the input view but deviate from any specific reference velocity.
Furthermore, velocities in regions without density can be arbitrary without changing the observed result.
The estimation can also be shifted in the volume, for which the simple error metrics do not account, leading to an artificially high error.
After all, in our setting of single-view estimation, the unobserved velocity has no direct constraints (other than some regularization) and any velocity that leads to a density matching the image-based constraints is acceptable. We have clarified this in section 4.3.1.

The FID and 
matching input images in terms of RMSE
are thus more suitable for evaluation here. The comparison to GT velocities was added for completeness, but is definitely problematic in our setting.
Rather, the high velocity error, in combination with the lower volume and image errors, are a further indication that multiple velocity solutions that lead to similar density observations exist.
As for the artifacts, we already achieved noticeable improvements by using a higher order interpolation kernel when upsampling velocity potentials, but we believe that there is potential for further hyperparameter tuning to improve the synthetic results.

\paragraph{Artifacts in the flow} We removed these artifacts by switching to a higher order interpolation kernel when up-scaling the velocity potentials. They were not related to the core of the proposed method. Please see the general comment above for the explanation behind this.

\paragraph{Limitations} We use a simplified rendering model that supports isotropic single scattering with white densities. While the use of background images is supported, obstructions, i.e. objects in front of the volume, or objects inside the flow 
are not (yet) supported. These limitations are now discussed in section 5 of our submission.
Furthermore, the paper primarily focuses on rising smoke plumes, other types of flows or inputs from particle image velocitemtries would be an interesting direction for future work.

\paragraph{Notation} We removed the unused loss-notation from Fig. 1.
$\nabla\times$ denotes the curl operator which we clarified in the revised paper.

\paragraph{Consistency}  We extend the input to $G_u$ from the initial version with any unambiguous input available to help guide the velocity estimation. More explanations can be found in the paragraph above Eq. 8.
}

\subsection{8YBe}
\myquote{The paper demonstrates a monocular video based method for smoke volumetric density and velocity reconstruction. Although the method has shown to be working for smoke with strict experiment setup (known foreground and background colors), I’m not sure if it can be extended to handle more general case (eg. smoke with real-world background, or even non smoke foreground). In addition, due to the confusions in the writing, the motivation for the design choice is not clear to me. Last but not least, the experiment is lacking in ablations. As a result, I would recommend weak rejection of the submission.}
\comment{background is assumed to be known, the ScalarFlow data contains a real-world background which the network has to handle. there must be no obstructions in or in front of the volume/smoke.}

\begin{itemize}
    \item Key assumption not explained: The rendering equation does not include the background color. If the background color changes, how to make sure this rendering equation is correct? If there is an additional assumption about the colors of the background and smoke, it should be claimed upfront since it is a very strong assumption
	\comment{(Forgot to include that) The background is assumed to be known and included in the rendering after raymarching c + t*b (currently we use a single static background for SF, black for synthetic). the smoke is assumed to be white.}
    \item The writing needs improvement:
    \begin{itemize}
        \item What does it mean by $c_z$ is the center along z (after Eq.6)? Without defining the start and end points along z for each ray, it is hard to understand it.
		\comment{(good point) It is the center in depth of the volume/domain, which is (somewhat) aligned with the camera central view direction.}
        \item Eq.6: what is p? Is it a 3D point location? If so, how can we define $L_z$ over one single 3D point?
		\comment{it is a 3D point. $L_z$ is already defined per point.
		we might want to define it in terms of camera rays? but this would likely make it more complicated.}
        \item Eq.6: why multiplying 2 inside the second term
		\comment{part of the normalization, s.t. the distance from the center to the border is 1. (otherwise it would be 0.5. shouldn't have much influence besides loss balancing. maybe it makes the center loss a bit sharper.)}
        \item eq.6 what is the intuition of the regularization?
		\comment{(it seems the intuitive explanation got lost when refining this section) the intuition is to keep the reconstructed density centered in view depth. the further the density strays from the center, the higher the penalty/loss.}
        \item What is $R^{-1}$ function in Eq.8
		\comment{projection into volume, mentioned in the paragraph before. We could add an explanation of how this is performed in the Appendix.}
    \end{itemize}
    \item To tackle the ambiguities in the solution, the method assumes the initial state is known. What if the initial state is not accurate? How can the subsequent estimation correct for the inaccurate initial state estimation?
	\comment{currently the initial state can not be corrected once estimated, and the estimator $G_\rho$ is not trained while $G_u$, so the sequence training has no effect on the initial estimation. but the training can be extended to train the initial state and proxies together with the velocity. (we dropped this because we did not use the proxy for the most time)
	The exact shape of the initial guess is not all that important as the velocity can deform the denstiy at each step and thus account for inaccuracies. The exact initial guess is no longer important after the first few frames. (there is a lot of freedom for the velocities in this chaotic/turbulent setup to correct the initial/previous state}
    \item Ablation study is not enough: ablations for the additional regularizations that tackle ambiguities are missing.
	\comment{Ablations for the density regularization can be found in Fig. 3, for the volume prototype in Fig. 4. Unfortunately, we don't have the time to run ablations for the magnitude (CFL) and smoothness regularization for the velocity, but these could be added in a final version if necessary.}
    \item Quality and clarity: The clarity needs to improve (see the weakness above for some confusions) Additional ablation studies on the regularizations are needed.
    \item Originality: Although the monocular 3D and scene flow estimation is not a new topic, the inclusion of the MacCormack Advection to model the inter-frame smoke transport is incrementally novel
\end{itemize}
\answer{
We thank the reviewer for the detailed and helpful comments.

\paragraph{Rendering and background} 
Thank you for pointing this out.
While the background is assumed to be known, it is provided in terms of a simple image. This background handling was always part of the rendering and image loses, and is now clearly explained in our revised version.
We mention the use of background images now in the corresponding sections of the paper and added the details about the background blending to the appendix.

\paragraph{Meaning of Eq. 6} 
The intuition behind Eq. 6 is to keep the estimated density in the center of the volume domain to obtain a more compact result, i.e. we refer to these terms as the center loss.
This is done by increasingly penalizing density the further its location is from a designated center $c_z$.
In general, $c_z$ is a problem specific distance from the camera, for which we choose the center point of the reconstruction domain.
$L_z$ is defined for every point $p$ of the volume, penalizing the density at that point ($\rho(p)$), weighted with its distance to the center. This distance is computed along the primary camera view direction, between $c_z$ and the projected point $p_z$. The distance is further normalized with half the grid resolution $r$, s.t. it is 1 at the domain boundaries.

\paragraph{Projection operator $R$ and its inverse} 
$R^{-1}$ essentially distributes an image (or any 2D feature tensor) into the volumes along the pixel rays of a given camera or view.
This is done by "inverse" ray-marching: marching along the ray, the pixel-value is added to the surrounding voxels at each step (which would be used for interpolation in the forward rendering). These scattered values are then normalized by the distance from the ray (using what would be the interpolation weights in the forward rendering) and the number of contributions per voxel to obtain a smooth result.
We clarified that before Eq. 8 and added an explanation in the appendix.

\paragraph{Initial state accuracy} The initial state $\rho^0$ is not known or provided, but estimated by the network $G_{\rho}$. We assume it to be known in the beginning of the Method-section for the sake of the argumentation, and section 3.3 explains that it is now inferred by $G_{\rho}$.
Currently, the initial state can not be changed once estimated, but the velocity estimation typically has enough degrees of freedom to correct for the inaccuracies via transport and inflow at the boundaries. We found that the exact shape of the initial density guess is not overly important as the velocity can deform the density at each step and thus compensate for errors. 

\paragraph{Additional ablations} New ablations for the density regularization can now be found in Fig. 2, for which we added a quantitative evaluation to the paper in Tab. 1. 
An ablation for the volume prototype can be found in Fig. 3, the corresponding metrics are in Tab. 3. 
We will additionally include ablations for the remaining regularization terms, namely magnitude (CFL) and smoothness regularization for the velocity, in a future version of our submission.
}







\subsection{TTZh}
\myquote{In general, the proposed method is technically sound to me. However, the experiments are somewhat limited. Furthermore, the clarity of the paper needs to be improved. Please see the above sections for more details.}

\begin{itemize}
    \item The framework figures are complicated and hard to understand. Too many notations, and some of them are unexplained. It would be better to simplify the diagrams and add some legends.
    \comment{We tried to simplify Fig 1 without omitting any essential parts and created a legend for the colors used. 
    further simplification might be hard, but maybe the $G_u$-parts can be further abstracted. Maybe the multiple colors are confusing?}
    \item It would be better to give more background and intuitive explanations for some technical designs. For example, why use RaLSGAN objective to train the discriminator, and why would CFL condition help constrain the magnitude?
    \comment{RaLSGAN was used by previous work (Franz et al.) for a very similar task and worked best there. (poor wording on CFL) Maybe we need to clarify the CFL condition: we don't want velocities over 1 to keep the discrete advection stable (CFL condition), thus we add a magnitude regularization that is only active for velocities over 1. I can add the exact loss formula to the appendix.}
    \item It would be better to visualize the learned prototype volume.
	\comment{We added a visualization of the density prototype to the appendix and the supplemental videos.}
    \item The ablation study is confusing. In Figure 3, there are two "fix pos, multi, ¬D"; one of them should probably be "var pos, multi, ¬D." It is unclear from Figure 3 that adding the adversarial loss can sharpen the result. It would be better to provide quantitative results for all ablation studies.
    \comment{There was a typo in Fig 3 which we fixed.
    I could evaluate the existing metrics for the shape ablation.}
    \item The paper mentioned that using a multi-view capture setup does not help the density ambiguity issue and would result in a suboptimally averaged solution. How about using a multi-view+the proposed discriminator? Showing such a comparison could strengthen the evaluation.
	\comment{we tried that and dropped it as it did not change the results. I can try to find that test, eval metrics.
    Theoretically, the additional targets add constrains that appear random given only the input view. The discriminator does not change that.
    would showing this improve our position?}
    \item Some minor typos. Section 3.1 - "leadsto", Section 4.1 - "spacial".
    \comment{fixed.}
    \item Clarity: As mentioned in the weakness part, the diagrams are hard to understand, and some technical designs lack background and intuition. The paper can be understood only by audiences familiar with all related work and techniques.
    \item Novelty: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
    \item Reproducibility: The paper provides some implementation details. However, the training flow seems complicated as it requires multiple stages. To reproduce the experiment, more training details including the range of the hyperparameters tuned, and the final selected hyperparameter values are required.
	\comment{Architecture, training details, and hyperparameters can be found in the appendix. (add that to the reproducibility statement?) }
\end{itemize}

\answer{
Thank you for your constructive feedback.

\paragraph{Framework figures} We agree that the previous version was unnecessarily complicated. We have simplified Fig. 1 as much as possible without omitting any essential parts, and added a legend for the colors used. The removed details are explained in the appendix now. Fig. 2 was removed in favor of the complete $G_\rho$ diagram in the appendix, which also contains explanations for the symbols.


\paragraph{Background and intuitive explanations} 
We use a simple magnitude loss on velocities above 1 to implement a CFL condition. 
This is done for increased stability in the advection. We clarified that section of the paper and added the exact loss formulas to the appendix.
GAN balancing and stability is usually very difficult, so we opted to use the established RaLSGAN which was successfully used in previous work for a similar task.


\paragraph{Learned prototype volumes} It is a good idea to show the prototypes. We added a visualization of the density prototype to the appendix and the supplemental videos to give some intuition for the type of volumes that are inferred.

\paragraph{Ablation study} We added a quantitative evaluation of the ablation in table 1 and extended the discussion in section 4.1. It clearly shows that the discriminator perceptually improves the results, especially when no additional targets are present. E.g., the FID improves from 98 to 68 here.
The density error of 0.93 simply indicates that the discriminator by itself has to way of determining the ground truth depth location.

\paragraph{Multi-view image loss in combination with Discriminator} 
We have added an additional ablation for the case of multi-view targets + discriminator to fig. 2 and table 1 and integrated it into the discussion in section 4.1.
Due to the problem of depth ambiguity, the additional targets add constraints that appear random given only the input view,
as $G_\rho$ can not infer the specific target used from the available input.
Even when combined with the discriminator, the multi-view targets push the generator towards averaged solutions
which conflicts with the GAN objective of finding a specific solution.
In the metrics this becomes evident: 
using multiple targets leads to a lower density RMSE, as RMSE typically favours averaged solutions.
However, the perceptual errors improve (LPIPS from 0.124 to 0.114 and FID from 80 to 68) when removing the additional targets in the presence of the discriminator.

We'd also like to point out that using multi-view targets is very challenging in real world settings as it requires a calibrated capture setup. Hence, we believe the single-view approach is very attractive despite the ambiguity of the solution manifolds.

}








\bibliographystyle{alpha}

\end{document}