\section{Experimental extension} \label{app:exp_ext}

In this section, we describe the experimental setup for \ours, as detailed in Appendix \ref{app:exp_setup}.
We also present a comprehensive set of results, including comparisons with the baselines \functa\ and \gasp. 
Of note, we used the original code from \gasp\ to replicate their results and also trained it on new datasets in our experimental setup. 
For \functa, we report results from the original paper \cite{dupontfuncta} for the datasets shared with our study, and for the remaining datasets, we implemented and generated results independently with our own implementation.

\subsection{Experimental setup} \label{app:exp_setup}
Implementation details for \ours\ are provided in Table \ref{table:impl_details}. We base our choice of architecture on \citep{dupont2021generative} as it provides a baseline setting. We train all of our model configurations with Adam optimizer. Also, in all of our models, we train Planar Flow and MLPs with activation of LeakyReLU with the exception of Sigmoid activation in the final layer of function generator. For datasets \celebahq, \shapesd\ and \polymnist, Discretized Logistic likelihood \cite{kingma2016improved, salimans2017pixelcnn++} is utilized, whilst for \chairs\ and \era, Bernoulli and Continuous Bernoulli \cite{loaiza2019continuous} are employed, respectively.

\begin{table*}[h!]
    \centering
\small{
\setlength\tabcolsep{3pt}
    \centering
    \caption{Implementation details of \ours.
}
    \vskip 0.1in
    \resizebox{\linewidth}{!}{
    \label{table:impl_details}
    \begin{tabular}{@{}crccccc@{}}
        \toprule
         &  & \textsc{CelebA-HQ} & \textsc{Shapes3D} & \textsc{PolyMNIST} & \textsc{ShapeNet} & \textsc{Era5} \\ \midrule
         & dim\_z & 64 & 32 & 16 & 32 & 32\\
         & K & 10 & 5 & 4 & 3 & 3 \\
         & epochs & 1000 & 600 & 600 & 500 & 1 \\
         & bs & 64 & 64 & 256 & 22 & 64 \\
         & lr & 1e-4 & 1e-4 & 1e-3 & 1e-3 & 1e-4 \\
         \midrule
        \multirow{4}{*}{\begin{tabular}[c]{@{}r@{}}PointConv\\ Encoder\end{tabular}} 
        & h\_weights & [16,16,16,16]  & [16,16] & [16,16] &[16,16,16,16]  & [16,16,16,16] \\
         & neighbors & [9,9,9,9,9] & [16,16,16] & [9,9,9] & [9,9,9,9,9] & [9,9,9,9,9] \\
         & centroids & [4096,1024,256,64,1] & [1024,256,64] & [196,49,25] & [4096,512,64,16,16] & [1024,256,64,32,16] \\
         & out\_channels & [64,128,256,512,512] & [32,64,256] & [32,32,32] & [32,64,128,256,16] & [32,64,256,512,32] \\
         & avg\_pooling\_neighbors & [9,9,9,9,None] & None & None & None &  None \\
         & avg\_pooling\_centroids & [1024,256,64,16,None] & None & None & None & None \\
          \midrule
        Categorical Encoder & layers & [64,32] & [32,32] & [32,32] & [32,32] &  [32,32]  \\
         \midrule
        Hypernetwork & layers & [256,512] & [128,256] & [16,32] & [256,512] & [256,512] \\
        \midrule
        \multirow{2}{*}{Generator} & layers & [64,64,64] & [32,32] & [4,4,4] & [64,64,64] & [64,64,64] \\
         & RFF  & $\text{m}=128, \sigma=2$ & $\text{m}=128, \sigma=2$ & $\text{m}=128, \sigma=2$ & $\text{m}=128, \sigma=2$ & $\text{m}=128, \sigma=2$ \\
         \midrule
        Flow & T & 80 & 20 & 3 & 40 & 10 \\ \bottomrule
    \end{tabular}
    }
}
\label{table:implementation_details}
\vspace{-5pt}
\end{table*}

\input{appendices/experiments_extended_generation.tex}
\input{appendices/experiments_extended_recons.tex}
\input{appendices/experiments_extended_vamoh.tex}





\subsection{Generation}
\label{app:generation}


In this section, we present a comprehensive evaluation of the generation capabilities of \ours\ in comparison to \functa\ and \gasp. \cref{tableapp:comparison} contains the obtained values for the FID, Precision, and Recall metrics for the image datasets. These values are presented as the mean and standard deviation computed over five independent sets of generated images.
For each of the metrics, we report the values using real embeddings coming from the training set (tr) and the test set (tst), since we also want to evaluate whether any model is overfitting to the training set, or as wished, it is able to generalize.

Results for the \celebahq\ and \shapesd\ datasets were previously discussed in Section \ref{subsec:generation}. 
Additionally, we include in this Section results for the \polymnist\ dataset, where we observe that \gasp\ obtains the best quantitative results, which might be related to its ability to generate sharp samples. 
Quantitative values for \functa\ and \polymnist\ are not reported as they were not used in the original paper and we found that normalizing flows failed to avoid overfitting to the modulations, even when trying different values of dropout, number of layers, and dimensionality of the hidden space. This led to low quality of the generations, as shown in \ref{figapp:gener_poly}. 

Visual inspection of the generated samples in \cref{figapp:gener_1}, \cref{figapp:gener_2}  illustrates that \ours\ generates high-quality samples, capturing details such as smiles in \celebahq\  (\cref{figapp:gener_celebahq}), variety in chair legs in \chairs\ (\cref{figapp:gener_shapenet}), backgrounds in \polymnist\ (\cref{figapp:gener_poly}), as well as different configurations of temperatures (\cref{figapp:gener_era}).


\begin{table*}[h!]
    \centering
\small{
\setlength\tabcolsep{3pt}
    \centering
    \caption{
    Comparison of FID and Precision and Recall scores of image generation for \ours, \gasp, and \functa.  Low FID, high precision, and high recall indicate the best performance. The best results are highlighted in bold. Note that  for \functa\ and \celebahq\ we just report their FID value, since they do not report the precision and recall.}
    \label{tableapp:comparison}
    \vskip 0.15in
    \resizebox{\linewidth}{!}{
    \begin{tabular}{c ccc ccc ccc}
\toprule
 &  \multicolumn{3}{c}{\celebahq}  &  \multicolumn{3}{c}{\shapesd}  &  \multicolumn{3}{c}{\polymnist}  \\
   \cmidrule(r){2-4}   \cmidrule(r){5-7}   \cmidrule(r){8-10}
  Model  &  $\downarrow$ FID &  $\uparrow$ Precision  &  $\uparrow$ Recall  &  $\downarrow$ FID &  $\uparrow$ Precision  &  $\uparrow$ Recall  &  $\downarrow$ FID &  $\uparrow$ Precision  &  $\uparrow$ Recall      \\
  \cmidrule(r){1-10} 
  \gasp (tst) & \bfseries 17.8 $\pm$ 0.17 & \bfseries 0.83 $\pm$ 0.0 & \bfseries 0.42 $\pm$ 0.01 & 119.35 $\pm$ 0.66 & 0.01 $\pm$ 0.0 & 0.16 $\pm$ 0.02 &\bfseries 88.45 $\pm$ 0.63 & \bfseries 0.08 $\pm$ 0.0 & \bfseries 0.34 $\pm$ 0.02 \\
  \functa (tst) &-  &  -  &  -  & 58.3 $\pm$ 0.14 & 0.07 $\pm$ 0.0 & 0.13 $\pm$ 0.0 &  -  &  -  &  -  \\
  \ours (tst) &72.14 $\pm$ 0.21 & 0.43 $\pm$ 0.01 & 0.0 $\pm$ 0.0 & \bfseries 56.63 $\pm$ 0.58 & 0.09 $\pm$ 0.01 & 0.63 $\pm$ 0.02 & 160.45 $\pm$ 0.88 & 0.06 $\pm$ 0.0 & 0.02 $\pm$ 0.02 \\
  \midrule
    \gasp (tr) & \bfseries 14.01 $\pm$ 0.18 & \bfseries  0.81 $\pm$ 0.0 & \bfseries 0.43 $\pm$ 0.01 & 118.66 $\pm$ 0.64 & 0.01 $\pm$ 0.0 & 0.16 $\pm$ 0.01 & \bfseries 89.21 $\pm$  0.61 & \bfseries 0.09 $\pm$ 0.0 &\bfseries  0.33 $\pm$ 0.02 \\
      \functa (tr)&  40.40  &  -  &  -  & 57.81 $\pm$ 0.15 & 0.06 $\pm$ 0.0 & 0.13 $\pm$ 0.0 &  -  &  -  &  -  \\
  \ours (tr)  & 66.27 $\pm$ 0.18 & 0.65 $\pm$ 0.0 & 0.0 $\pm$ 0.0 &  \bfseries 56.25 $\pm$ 0.57 & 
 \bfseries 0.08 $\pm$ 0.0 & \bfseries 0.64 $\pm$ 0.01 & 161.34 $\pm$ 0.88 & 0.04 $\pm$ 0.0 & 0.02 $\pm$ 0.02 \\
\bottomrule
\end{tabular}
}
}
\vspace{-5pt}

\end{table*}



\begin{figure*}[h!]
\begin{subfigure}{.47\textwidth}
    \centering
    Original resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/gasp_samples_celeba_hq.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/test/vamoh/generation.png} 
    \end{subfigure}%
    \vspace{2pt}
    Super-resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/gasp_super_samples_celeba_hq.png} 
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/test/vamoh/super_generation.png} 
    \end{subfigure}
    \caption{\celebahq}
    \label{figapp:gener_celebahq}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Original resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/gasp_samples_shapenet.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__sample_0.png} 
    \end{subfigure}%
    \vspace{2pt}
    Super-resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/gasp_super_samples_shapenet.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__sample_super_0.png} 
    \end{subfigure}
    \caption{\chairs}
    \label{figapp:gener_shapenet}
\end{subfigure}%
\caption{Comparison of generation quality of the proposed \ours, \functa, and \gasp\ at original resolution (top) and x2 resolution (bottom).
}
\label{figapp:gener_1}
\end{figure*}
\raggedbottom




\begin{figure*}[t!]
\begin{subfigure}{.47\textwidth}
    \centering
    Original resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_Shapes3D_samples_1_1.0.png}
        \vspace{5pt}
        \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/gasp_samples_shapes3d.png} 
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/generation.png}
    \end{subfigure}%
    \vspace{2pt}
    Super-resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_Shapes3D_samples_1_2.0.png}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/gasp_super_samples_shapes3d.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/super_generation.png}
    \end{subfigure}
    \caption{\shapesd}
    \label{figapp:gener_shapes}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Original resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/functa_polymnist_samples_x2.png}
        \vspace{5pt}
        \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/gas_samples_polymnist.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/test/vamoh/generation.png}
    \end{subfigure}%
    \vspace{2pt}
    Super-resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/functa_polymnist_samples_x2.png}
    \raisebox{15pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/gas_super_samples_polymnist.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/test/vamoh/super_generation.png}
    \end{subfigure}
    \caption{\polymnist}
    \label{figapp:gener_poly}
\end{subfigure}%
\caption{Comparison of generation quality of the proposed \ours, \functa, and \gasp\ at original resolution (top) and x2 resolution (bottom).
}
\label{figapp:gener_2}
\end{figure*}
\raggedbottom



\begin{figure*}[t!]
\begin{subfigure}{.47\textwidth}
    \centering
    Original resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{17pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/functa_era5_samples_0_1.0.pdf}
        \vspace{5pt}
        \raisebox{17pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/gasp_samples_era5.pdf}
        \vspace{5pt}
        \raisebox{15pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/vamoh_era5_test_generation.pdf}
    \end{subfigure}%
    \vspace{2pt}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Super-resolution
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{17pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/functa_era5_samples_3_2.0.pdf}
        \vspace{5pt}
        \raisebox{17pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/gasp_super_samples_era5.pdf}
        \vspace{5pt}
        \raisebox{15pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/vamoh_era5_test_super_generation.pdf}
    \end{subfigure}%
    \vspace{2pt}
\end{subfigure}%
\caption{Comparison of generation quality of the proposed \ours, \functa, and \gasp\ at original resolution (top) and x2 resolution (bottom).
}
\label{figapp:gener_era}
\end{figure*}
\raggedbottom




\subsection{Reconstruction}
\label{app:reconstruction}

In this section, we provide a comprehensive set of figures demonstrating the capability of \ours\ in reconstructing data at both the original resolution and double the original resolution. \cref{figapp:recons_celeba} and \cref{figapp:recons} present the results for all datasets under examination. These figures reveal that \ours\ is able to produce high-quality reconstructions and super-reconstructions, with visual fidelity comparable or better to that of \functa, while requiring only a simple forward pass.

Furthermore, in Section \ref{subsec:reconstruction}, the Peak Signal to Noise Ratio (PSNR) is employed to evaluate the quality of the reconstruction, denoted as $\tilde{\bm{Y}}$, of an image, denoted as $\bm{Y}$. The computation of PSNR begins by calculating the root mean squared error (RMSE) as follows:

\begin{equation}
    \text{RMSE} = \sqrt{\frac{1}{D}\sum_{d} ||\yb_d - \tilde{\yb_d}||_2^2},
\end{equation}

then the PSNR is computed as 

\begin{equation}
    \text{PSNR} = 20 \log \left( \frac{255}{\text{RMSE}} \right).
\end{equation}


\begin{figure}[h]
\centering
\begin{center}
    Ground truth
\end{center}
\begin{center}
\includegraphics[width=0.5\textwidth]{figures/celeba/celeba_hq_test.png}
\end{center}
\begin{center}
    Reconstruction
\end{center}
\begin{center}
\includegraphics[width=0.5\textwidth]{figures/celeba/test/vamoh/reconstruction.png}
\end{center}
\begin{center}
    Super-reconstruction
\end{center}
\begin{center}
\includegraphics[width=0.5\textwidth]{figures/celeba/test/vamoh/super_reconstruction.png}
\end{center}
\caption{Comparison of reconstruction quality of the proposed \ours\ and \functa\ on ground truth images at original resolution (top) and x2 resolution (bottom). }
\label{figapp:recons_celeba}
\end{figure}

\begin{figure*}[t!]
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \begin{flushright}
    \includegraphics[width=0.9\columnwidth]{figures/chairs/shapenet_voxels_test_gt.png}
    \end{flushright}
    Reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
    \includegraphics[width=\columnwidth]{figures/chairs/functa_shapenet_voxels_test_recons_0_1.0.png}
    \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__recons_0.png}
    \end{subfigure}%
    \\
    Super-reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
     \includegraphics[width=\columnwidth]{figures/chairs/functa_shapenet_voxels_test_recons_0_2.0.png}
     \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__recons_super_0.png}
    \end{subfigure}
    \caption{\chairs}
    \label{figapp:recons_right}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \begin{flushright}
    \includegraphics[width=0.9\columnwidth]{figures/shapes3d/test/vamoh/gt.png}
    \end{flushright}
    \vspace{2pt}
    Reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_shapes3d_test_recons_0_1.0.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/reconstruction.png}
    \end{subfigure}%
    \vspace{2pt}
    Super-reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_shapes3d_test_recons_0_2.0.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/super_reconstruction.png}
    \end{subfigure}
    \caption{\shapesd}
    \label{figapp:recons_left}
\end{subfigure}%
\\
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \begin{flushright}
    \includegraphics[width=0.9\columnwidth]{figures/polymnist/test/vamoh/gt.png}
    \end{flushright}
    \vspace{2pt}
    Reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/functa_polymnist_val_recons_0_1.0.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/test/vamoh/reconstruction.png}
    \end{subfigure}%
    \vspace{2pt}
    Super-reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/functa_polymnist_val_recons_0_2.0.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/polymnist/test/vamoh/super_reconstruction.png}
    \end{subfigure}
    \caption{\polymnist}
    \label{fig_app:recons_poly}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \begin{flushright}
    \includegraphics[width=0.85\columnwidth]{figures/era5/vamoh_era5_test_gt.pdf}
    \end{flushright}
    \vspace{2pt}
    Reconstructions
    \begin{subfigure}{0.85\columnwidth}
    \raisebox{18pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/functa_era5_test_recons_0_1.0.png}
        \vspace{5pt}
        \raisebox{15pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/vamoh_era5_test_reconstruction.pdf}
    \end{subfigure}%
    \vspace{2pt}
    Super-reconstructions
    \begin{subfigure}{0.85\columnwidth}
    \raisebox{18pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/functa_era5_test_recons_0_2.0.png}
        \raisebox{15pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/era5/vamoh_era5_test_super_reconstruction.pdf}
    \end{subfigure}
    \caption{\era}
    \label{figapp:recons_era}
\end{subfigure}%
\caption{Comparison of reconstruction quality of the proposed \ours\ and \functa\ on ground truth data from the first samples of the test set at original and super-resolution.}
\label{figapp:recons}
\end{figure*}





\subsection{Image completion}
\label{app:im_completion}
In this section, we provide qualitative results of \ours\ for various imputation tasks for different missingness patterns. As it is already highlighted in Section \ref{sec:training} and \cref{app:point_dropout}, we use point dropout to increase the robustness of the encoder against missing data. In \cref{figapp:completion}, we provide results for image completion task on with different datasets. 


\begin{figure*}[h!]
\begin{subfigure}{.42\textwidth}
    \centering
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/missing_occluded_gt.png}
    \vspace{2pt}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/missing_reconstruction.png}
    %
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/missing_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/missing_reconstruction.png}
    %
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/missing_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/missing_reconstruction.png}
    %
    \raisebox{20pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/era5/vamoh_era5_test_miss_gt.pdf}
    \raisebox{18pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/era5/vamoh_era5_test_miss_reconstruction.pdf}
    \caption{Missing a patch (in-painting)}
    \label{figapp:completation_left}
\end{subfigure}%
\hfill
\begin{subfigure}{.42\textwidth}
    \centering
    \raisebox{15pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/half_occluded_gt.png}
    \vspace{2pt}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/half_recostruction.png} 
    %
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/half_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/half_reconstruction.png}
    %
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/half_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/half_reconstruction.png}
    %
    \raisebox{20pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/era5/vamoh_era5_test_half_gt.pdf}
    \raisebox{18pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/era5/vamoh_era5_test_half_reconstruction.pdf}
    \caption{Missing half of the image }
\end{subfigure}%
\\
\begin{center}
    \begin{subfigure}{.42\textwidth}
    \centering
    \raisebox{15pt}{\rotatebox[]{90}{GT}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/out_occluded_gt.png}
    \vspace{2pt}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/celebahq/out_reconstruction.png}
    %
    \raisebox{10pt}{\rotatebox[]{90}{GT}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/out_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/shapes/out_recostruction.png}
    %
    \raisebox{10pt}{\rotatebox[]{90}{GT}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/out_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.91\columnwidth]{figures/image_completion/polymnist/out_recostruction.png}
    \caption{Image out-painting}
    \label{figapp:completation_right}
\end{subfigure}%
\end{center}
\caption{Imputation of different amounts of missing parts using \ours. For each dataset, the top row shows the input (In) and the bottom row shows the reconstructed image. }
\label{figapp:completion}
\end{figure*}






\clearpage
\newpage
\subsection{Entropy and Segmentation}
In this section, we present figures that illustrate the contribution of each generator in the image reconstruction task. Specifically, we use entropy maps to visualize the uncertainty of the posterior probabilities of the categorical latent variable. A higher value in the entropy map indicates that the probabilities are more evenly distributed among the components for a given pixel, while a lower value suggests that a smaller number of components contribute to the reconstruction of that pixel. Additionally, we use segmentation maps to visualize the component with the highest probability per pixel, where different components are denoted by different colors. As an example, Figure \ref{figapp:celeba_entropy} shows that there is a high degree of uncertainty among the components that are responsible for reconstructing the background in the \celebahq\ dataset. Conversely, for simpler datasets such as \polymnist, as shown in Figure \ref{figapp:poly_entropy}, the components are able to differentiate between the object and background in the images. For \shapesd\ dataset results presented in Figure \ref{figapp:shapes_entropy}, the uncertainty is lower in the Regions of Interest with high variations (borders), where only one of the components focuses on generating the shape and shadow of the object, as well as the perspective of the background wall. Thus, we provide evidence that meaningful interpretations can be obtained from our Mixture of HyperGenerators.

\begin{figure}[h]
\begin{subfigure}{.47\textwidth}
  \centering
  Test Recons
\includegraphics[width=\linewidth]{figures/shapes3d/test/vamoh/reconstruction.png}
Test Entropy
\includegraphics[width=\linewidth]{figures/entropy_and_seg/shapes/reconstruction_entropy.png}
Test Segmentation
\includegraphics[width=\linewidth]{figures/entropy_and_seg/shapes/reconstruction_segmentation.png}
  \caption{\shapesd\ Test}
  \label{figapp:shapes_entropy}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
  \centering
  Test Recons
\includegraphics[width=\linewidth]{figures/celeba/test/vamoh/reconstruction.png}
Test Entropy
\includegraphics[width=\linewidth]{figures/entropy_and_seg/celeba/reconstruction_entropy.png}
Test Segmentation
\includegraphics[width=\linewidth]{figures/entropy_and_seg/celeba/reconstruction_segmentation.png}
  \caption{\celebahq\ Test}
  \label{figapp:celeba_entropy}
\end{subfigure}%
\\
\begin{center}
    \begin{subfigure}{.47\textwidth}
  \centering
  Test Recons
\includegraphics[width=\linewidth]{figures/polymnist/test/vamoh/reconstruction.png}
Test Entropy
\includegraphics[width=\linewidth]{figures/entropy_and_seg/polymnist/reconstruction_entropy.png}
Test Segmentation
\includegraphics[width=\linewidth]{figures/entropy_and_seg/polymnist/renctruction_segmentation.png}
  \caption{\polymnist\ Test}
  \label{figapp:poly_entropy}
\end{subfigure}%
\end{center}
\caption{Visualization of used components in mixture of generators of \ours\ for entropy (top) and segmentation (bottom) maps for \shapesd, \celebahq, and \polymnist.}
\label{figapp:entropy}
\end{figure}


\section{Further details about \ours}

\subsection{ELBO derivation}
\label{app_sec:elbo}

Our objective is to maximize the evidence lower bound (ELBO) of a set of $D$ features $\bm{Y}$ given the corresponding set of coordinates $\bm{X}$. To do so, we introduce a family of variational distributions for the continuous and discrete latent variables, namely $\zb$ and $c_d $ where $ d \in [D]$:

\begin{equation}
     q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X}) 
 = q_{\gammabz}(\zb|\bm{Y}, \bm{X}) \prod_{d=1}^{D} q_{\gammabc}(c_{d}|\zb,\bm{y}_{d}, \xb_{d}).
\end{equation}

where 
\begin{equation}
    q_{\gammabz}(\zb|\bm{Y}, \bm{X}) = \normal{\zb | f_{\gammabz} (\bm{Y}, \bm{X}) } \quad \text{ and } 
    \quad q_{\gammabc}(c_{d}| \zb,\bm{y}_{d}, \xb_{d}) = \cat{c_{d}| f_{\gammabc}(\zb,\bm{y}_{d}, \xb_{d})}.
    \label{eq:factorization_appendix}
\end{equation}

Note that these two distributions are parameterized by $\gammabz$ and $\gammabc$, which we aim to learn. We refer to the set of all inference parameters as $\gammab = \{\gammabz, \gammabc\}$. 
Moving on to the generative distribution, it factorizes as


\begin{equation}
    p_{\psi, \phi}(\bm{Y}, \bm{C}, \bm{z} | \bm{X}) =  p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  p_{\psi_c}(\bm{C} |  \bm{z}, \bm{X} )   p_{\psi_z}(\bm{z})
\end{equation}

where $ p_{\psi_c}(\bm{C} |  \bm{z}, \bm{X} ) $ is the prior over the categorical latent variable, $  p_{\psi_z}(\bm{z})$ is the prior over the continuous latent variable (parameterized by a Normalizing Flow, see \cref{app:flow_prior}), and  $p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})$ is the likelihood distribution over the set of features.  More in detail,  the likelihood factorizes over coordinates and we define each of them as a mixture:


\begin{equation}
    p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  =  \prod_{d=1}^{D}  p_{\bm{\theta}}(\bm{y}_{d}|\xb_d, c_d, \zb),
    =\prod_{d=1}^{D} \prod_{k=1}^{K}  
\pi_{dk}^{\indic{c_{d}=k} } \, p_{\thetab_k}(\bm{y}_{d}|\xb_d) \quad \text{ where } \quad  \thetab_k = g_{\phi_k}(\zb).
\label{eq:generative_appendix}
\end{equation}

At this point, it is important to remark that we do not learn the parameters  $\thetab_k$ $k \in [K]$ of the  likelihood directly. Instead, we learn $k$ the parameters of $k$ different hypernetworks $g_{\phi_k}$ $k \in [K]$ that output the corresponding  likelihood parameters. Thus, the set of all the parameters of the generative models are $\phib = \{ \phib_k\}_{k=1}^K$ and $\psib = \{ \psi_z, \psi_c \}$.


With this choice of the inference and generative model, we define the ELBO as 
\begin{equation}
     \Lcal (\bm{Y}, \bm{X}; \psi, \phi, \gammab ) =  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log \frac{p_{\psi, \phi}(\bm{Y}, \bm{C}, \bm{z} | \bm{X})}{ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X}) } } \leq \log p(\bm{Y} | \bm{X}).
\end{equation}

Using the factorization of the generative and inference model in \cref{eq:factorization_appendix,eq:generative_appendix}, we can split it in various terms:
\begin{align}
     \Lcal (\bm{Y}, \bm{X}; \psi, \phi, \gammab ) & =  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log  p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  } \nonumber \\ 
     & +  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log \frac{ p_{\psi_z}(\bm{z})}{ q_{\gammab_z}(\zb |\bm{Y}, \bm{X}) } } \nonumber \\
    &  +  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log \frac{ p_{\psi_c}(\bm{C} |  \bm{z}, \bm{X} )}{ q_{\gammab_c}( \bm{C}|\bm{Y}, \bm{X}) } }
\end{align}

We an marginalize some variables from the expectations.  
\begin{align}
     \Lcal (\bm{Y}, \bm{X}; \psi, \phi, \gammab ) & =  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log  p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  } \nonumber\\
     & +  \E[ q_{\gammab_z}(\zb |\bm{Y}, \bm{X})  ]{\log \frac{ p_{\psi_z}(\bm{z})}{ q_{\gammab_z}(\zb |\bm{Y}, \bm{X}) } } \nonumber\\
    &  +  \E[ q_{\gammab_c}( \bm{C}|\bm{Y}, \bm{X})  ]{\log \frac{ p_{\psi_c}(\bm{C} |  \bm{z}, \bm{X} )}{ q_{\gammab_c}( \bm{C}|\bm{Y}, \bm{X}) } }
\end{align}

And rewriting the last two terms as KL divergences we get:

\begin{align}
     \Lcal (\bm{Y}, \bm{X}; \psi, \phi, \gammab ) & =  \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log  p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  } \nonumber\\
     & - \kld{q_{\gamma_{z}}(\zb | \bm{X}, \bm{Y} )}{p_{\psi_{z}}(\zb)} \nonumber\\
 & - \kld{q_{\gamma_{c}}(\bm{C} |\zb, \bm{X}, \bm{Y} )}{p_{\psi_{c}}(\bm{C} | \zb,  \bm{X})}.
\end{align}


Finally, in the first term, we can marginalize over the categorical latent variable such that we only need to approximate by Monte Carlo expectation over $\zb$:

\begin{align}
      \E[ q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{\log  p_{\thetab}(\bm{Y} | \bm{X},  \bm{C}, \bm{z})  } & =   \E[q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X})  ]{ \sum_{d=1}^{D} \log  p_{\thetab}(\bm{y}_{d}|\xb_d, \zb, c_d)  }   \nonumber\\
      & =  \sum_{d=1}^{D} \E[ q_{\gammab_z}(\zb |\bm{Y}, \bm{X})  ]{   \sum_{k=1}^{K} \log  p_{\thetab_k}(\bm{y}_{d}|\xb_d) \cdot \pi_{dk} } 
\end{align}


Then ELBO becomes
\begin{align}
     \Lcal (\bm{Y}, \bm{X}; \psi, \phi, \gammab ) & = \sum_{d=1}^{D} \E[ q_{\gammab_z}(\zb |\bm{Y}, \bm{X})  ]{   \sum_{k=1}^{K} \log  p_{\thetab_k}(\bm{y}_{d}|\xb_d) \cdot \pi_{dk} }  \nonumber\\
     & - \kld{q_{\gamma_{z}}(\zb | \bm{X}, \bm{Y} )}{p_{\psi_{z}}(\zb)} \nonumber\\
 & - \kld{q_{\gamma_{c}}(\bm{C} |\zb, \bm{X}, \bm{Y} )}{p_{\psi_{c}}(\bm{C} | \zb,  \bm{X})}.
\end{align}









\subsection{Algorithm details}
The training methodology is outlined in Algorithm \ref{alg:minibatch}. As discussed in Section \ref{sec:training}, the training process begins with an initial warming stage, in which a standard prior $p(\zb)$ is utilized to stabilize the optimization of the encoder. The second and primary stage involves the introduction of the learnable prior $p_{\psi_z}(\zb)$.

\begin{algorithm}[H]
   \caption{Minibatch training of \ours}
   \label{alg:minibatch}
\begin{algorithmic}[1]
    \STATE Define hyperparameters, estimator $\widetilde{\mathcal{L}}$ using Eq. \eqref{eq:elbo_ours}
    
    \STATE $\gamma_{z,c}$ (inference), $\psi_{z,c}$ (prior), $\phi_{1:K}$ (hypernetwork) $\gets$ initialize
    \REPEAT
    \STATE $\{\bm{X}, \bm{Y}\}_{1:M} \gets$ random minibatch of $M$
    \STATE $\{\bm{X'}, \bm{Y'}\}_{1:M} \gets$
    apply dropout with probability $p\sim U(0, \alpha)$
    \IF{NF is available}
    \STATE parameterize $p_{\psi}(\zb)$ with planar flow
    \ELSE{}
    \STATE fix $p(\zb)$ with standard Gaussian
    \ENDIF
    \STATE $\mathbf{g} \gets \nabla_{\gamma, \psi, \phi_{1:K}} \widetilde{\mathcal{L}}^M\left(\gamma, \psi, \phi_{1:K}; \{\bm{X}, \bm{Y}\}_{1:M}\right)$ (gradients)
    \STATE $\gamma, \psi, \phi_{1:K}\gets$ Update parameters using gradients $\mathbf{g}$
\UNTIL convergence of parameters $\gamma, \psi, \phi_{1:K}$
\end{algorithmic}
\end{algorithm}

\subsection{Point dropout for conditional generation} \label{app:point_dropout}
In this section, we present experimental evidence supporting the effectiveness of the point dropout strategy for training VAMoH, as discussed in Section \ref{sec:training} of the paper. Figure \ref{fig:point_dropout} illustrates the results of patch imputation on test images from the test set of \textsc{CelebA HQ}. When the model is trained on full images, as shown in Figure \ref{fig:point_dropout_a}, the PointConv encoder is unable to learn from a diverse set of centroids, resulting in uninformative posteriors when partial images are used during testing. However, as shown in Figure \ref{fig:point_dropout_b}, when VAMoH is trained on images that have undergone point dropout, as outlined in Section \ref{sec:training}, improved robustness for conditional generation is obtained.

\begin{figure*}[h!]
\centering
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/point_dropout/without.png}
    \caption{Without point dropout}\label{fig:point_dropout_a}
\end{subfigure}%
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/point_dropout/with.png}
    \caption{With point dropout}\label{fig:point_dropout_b}
\end{subfigure}%
\caption{Imputation of missing patches in test \textsc{CelebA HQ} images using VAMoH. In (a), full images are fed to the model during training. In (b), points of training images are deleted with a dropout probability per batch.}
\label{fig:point_dropout}
\end{figure*}


\subsection{Flow-based prior}\label{app:flow_prior}

In this section, we evaluate the effectiveness of the Planar Flow as a prior for our continuous latent variable in addressing the \emph{hole problem} described in Section \ref{subsec:generative_model}. We first train our model on the CelebA dataset \cite{liu2015faceattributes}, utilizing a fixed standard prior $p(\zb)$. Upon convergence, we sample from the learned prior and present the results in Figure \ref{fig:prior_samples_a}. The standard prior places significant probability mass in regions distant from the aggregated posterior, resulting in poor image quality when samples are drawn from these regions. By reducing the variance and approaching the posterior probability mass, as demonstrated in Figure \ref{fig:prior_samples_b}, we observe improved realism in the generated images. However, this also leads to a lack of diversity in the generated samples. In Figure \ref{fig:prior_samples_c} we include decoded samples from the posterior parameterized by the flexible encoder that reveals diversity for reconstructing images. 

In contrast, by introducing the Planar Flow after a warming stage and training its parameters, we observe a more aligned reconstruction-generation process. This is evident by comparing the samples obtained by sampling from the posterior (Figure \ref{fig:prior_samples_d}) and the learned prior ( Figure \ref{fig:prior_samples_e}), respectively.

\begin{figure*}[h!]
\centering
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/prior/samples_standard.png}
    \caption{Standard prior, $\bm{z}\sim \mathcal{N}(\bm{0}, \bm{I})$}\label{fig:prior_samples_a}
\end{subfigure}%
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/prior/samples_standard_center.png}
    \caption{Standard prior, $\bm{z}\sim \mathcal{N}(\bm{0}, 0.01 \cdot\bm{I})$}\label{fig:prior_samples_b}
\end{subfigure}%
\\
\centering
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/prior/samples_posterior_standard.png}
    \caption{Standard prior,  $\bm{z}\sim q(\bm{z} | \bm{X}, \bm{Y})$}\label{fig:prior_samples_c}
\end{subfigure}%
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/prior/samples_posterior_flow.png}
    \caption{Flow-based prior, $\bm{z}\sim q(\bm{z} | \bm{X}, \bm{Y})$}\label{fig:prior_samples_d}
\end{subfigure}%
\\
\centering
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=0.9\columnwidth]{figures/prior/samples_flow.png}
    \caption{Flow-based prior, $\bm{z}\sim p_{\psi_z}(\bm{z})$}\label{fig:prior_samples_e}
\end{subfigure}%
\caption{Decoding samples from different latent distributions. Images in (a), (b), and (c) are generated/reconstructed by our model after being trained assuming a standard prior over $\zb$. In (d) and (e) we include reconstructions and generations, respectively, when modeling a more flexible prior over $\zb$ by training a Planar Flow.}
\label{fig:prior_samples}
\end{figure*}

\subsection{Logistic likelihood}
Inspired by \cite{kingma2016improved}, we utilize a Discretized Logistic distribution for parameterizing the likelihood of discrete data with high number of categories, such as color channels in the RGB space for images. By doing that, we obtain a smooth and memory efficient predictive distribution for $\yb_d$, as opposed to, for example, a Categorical likelihood with parameterized with 256-way softmax. The likelihood parameterized by each hypergenerator is given by
\begin{equation}
    p_{\bm{\theta}_k}(\yb_d | \xb_d)= \sigma\left(\left(\yb_d+0.5-\bm{\mu}_{dk}\right) / \bm{s}_{k}\right)-\sigma\left(\left(\yb_d-0.5-\bm{\mu}_{dk}\right) / \bm{s}_{k}\right),
\end{equation}
 where the means $\bm{\mu}_{dk}$ are outputs of the hypergenerator, and the scales $\bm{s}$ are learnable parameters. Thanks to the design of VAMoH, our approach using a mixture for the likelihood is in line with \cite{salimans2017pixelcnn++}, and results in a Discretized Logistic Mixture Likelihood of the form
 \begin{equation}
    p(\yb_d | \xb_d)= \prod_{k=1}^K \pi_k \left[ \sigma\left(\left(\yb_d+0.5-\bm{\mu}_{dk}\right) / \bm{s}_{k}\right)-\sigma\left(\left(\yb_d-0.5-\bm{\mu}_{dk}\right) / \bm{s}_{k}\right) \right],
\end{equation}
which allows to accurately model the conditional distributions of the points by using a relatively small number of mixture components, as we show in our paper, and in concordance with \cite{salimans2017pixelcnn++}.




\section{Conclusion}





In this paper, we introduced \ours, a novel VAE-based model for learning distributions of functions that enables efficient and accurate generation of data over continuous coordinate systems.
Notably, \ours\ allows for straightforward conditional generation. In contrast with the alternatives, with a simple forward pass on the model, \ours\ can perform in-painting and out-painting tasks or generate higher-resolution versions of a test set sample, that is, unseen during training.
Our experimental results demonstrate the effectiveness of \ours, both in generation and inference tasks, in a wide range of applications and datasets.


As a future research direction, we aim to enhance \ours\ by exploring methods for achieving disentanglement in the latent space.
This would allow for controlled modifications of generated and reconstructed samples and may open up new possibilities for applications such as data editing at various resolutions.


\section{Experiments}




In this section, we provide a thorough empirical evaluation of \ours. 
We evaluate our model on the tasks of data generation, reconstruction, and imputation, including the superresolution results. 

\myparagraph{Baselines} We compare \ours\ with \gasp\ \cite{dupont2021generative} and \functa\ \cite{dupontfuncta} with a Normalizing Flow as generator of modulation vectors, which we denote simply by \functa. More specifically, \gasp\ is only considered for data generation, since it does not allow for any inference-related task. 








\myparagraph{Datasets} We evaluate \ours\ on  \polymnist\ ($28 \times $28), \celebahq\ ($64 \times $64) \citep{karras2017progressive}, \shapesd\ ($64 \times $64) \citep{3dshapes18}, climate data from the \era\ dataset \cite{era5}, and 3D chair voxels from the \chairs\ dataset \cite{chang2015shapenet}. 
The architecture and hyperparameters used for each dataset are detailed in \cref{app:exp_setup}.





We implemented \ours\ in PyTorch and performed all experiments on a single V100 with 32GB of RAM. 
The code with the model implementation and experiments is available at 
\url{https://github.com/bkoyuncu/vamoh}. In this section, we provide a glimpse into the results of our experiments on all datasets. Additional experiments and their outcomes can be found at Appendix \ref{app:exp_ext}.







\subsection{Generation}

\begin{figure*}[t!]
\begin{subfigure}{.5\textwidth}
    \centering
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
    \includegraphics[width=\columnwidth]{figures/celeba/paper_functa_celeba_samples.png}
    \raisebox{12pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/gasp_samples_celeba_hq.png}
    \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/celeba/test/vamoh/generation.png}
    \end{subfigure}%
    \caption{\celebahq}
\end{subfigure}%
\hfill
\begin{subfigure}{.5\textwidth}
    \centering
    %
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_Shapes3D_samples_1_1.0.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\gasp}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/gasp_samples_shapes3d.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/generation.png}
    \end{subfigure}%
    %
    \caption{\shapesd}
\end{subfigure}%
\caption{Comparison of generation quality at original resolution. \celebahq\ samples of \functa\ obtained from \cite{dupontfuncta}. }
\label{fig:generation}
\end{figure*}
\raggedbottom
\label{subsec:generation}


\begin{figure}[t]
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{0.9\columnwidth}
\raisebox{15pt}{\rotatebox[]{90}{\small \gasp}}\hspace{5pt}
\includegraphics[width=\columnwidth]{figures/celeba/gasp_super_samples_celeba_hq.png}
\raisebox{12pt}{\rotatebox[]{90}{\small \ours}}\hspace{5pt}
\includegraphics[width=\columnwidth]{figures/celeba/test/vamoh/super_generation.png}
\end{subfigure}
\caption{\celebahq}
\end{subfigure}
\hfill
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{0.9\columnwidth}
\raisebox{15pt}{\rotatebox[]{90}{\small \gasp}}\hspace{5pt}
\includegraphics[width=\columnwidth]
{figures/shapes3d/gasp_super_samples_shapes3d.png}
\\
\raisebox{12pt}{\rotatebox[]{90}{\small \ours}}\hspace{5pt}
\includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/super_generation.png}
\end{subfigure}
\caption{\shapesd}
\end{subfigure}
\begin{subfigure}{1\textwidth}
\centering
\begin{subfigure}{0.9\columnwidth}
\vspace{5pt}
\centering
\raisebox{20pt}{\rotatebox[]{90}{\small \gasp}}\hspace{5pt}
\includegraphics[width=0.4\textwidth]  {figures/era5/gasp_super_samples_era5.pdf}
\\
\raisebox{20pt}{\rotatebox[]{90}{\small \ours}}\hspace{5pt}
\includegraphics[width=0.4\textwidth]{figures/era5/vamoh_era5_test_super_generation.pdf}
\end{subfigure}
\caption{\era}
\end{subfigure}
\caption{Comparison of uncurated generated samples at super-resolution.}
\label{fig:super_generation}
\end{figure}
\raggedbottom

In this section, we evaluate \ours\ for the task of synthetic data generation over continuous coordinate systems. Samples are obtained from the learned models in both the original and twice the resolution, referred to as \textit{super resolution}. 





\myparagraph{Metrics}  We use two metrics to evaluate the quality of the generated samples. We report the usually employed Fréchet Inception Distance (FID) \cite{heusel2017gans} as well as the \textit{improved precision and recall} \cite{kynkaanniemi2019improved}, which measures the quality of the generated data (i.e, high precision) and the coverage of the true data distribtuion (i..e, high recall).








\myparagraph{Results} 
We present the summary of the quantitative comparison of \ours\ with both baselines for two image datasets in \cref{table:comparison}.
Our analysis indicates that, quantitatively speaking, there is not a clear winner among the models.
While \gasp\ achieves the highest performance
on the \celebahq\ dataset,  visual inspection of the generated samples in \cref{fig:generation} reveals that the quality of \functa\  and \ours\ is also high.
This aligns with previous studies \cite{dupontfuncta} that have noted that FID may over-penalize blurriness.
Additionally, it is worth noting that \ours\ obtains a recall of 0.0 on this dataset, despite the generated samples displaying diversity in features such as facial expressions and hairstyles, as seen in \cref{fig:generation}. 
In analyzing the performance of \ours\ on the \shapesd\ dataset, we observe a clear superiority compared to other models.
As depicted in \cref{fig:generation}, \ours\ is able to generate objects with diverse shapes and colors, as well as  walls and floors delimited with sharp edges. 
Furthermore, it achieves the highest quantitative metrics, as demonstrated in the right columns of \cref{table:comparison}. 
Despite this, it is important to note that the low precision values obtained by all models require a comprehensive evaluation approach, incorporating both quantitative and qualitative metrics when assessing generative models.




Finally, in \cref{fig:super_generation}, we evaluate the super-resolution capabilities of the models by visually inspecting the generated samples at twice the original resolution. 
We utilize the same latent code as in the generated samples in \cref{fig:generation} for the \celeba\ and \shapesd\ datasets. 
Additionally, to demonstrate the versatility of our \ours\ to model diverse types of data, we also include super resolution samples from the \era\ dataset.
Comparison with super-resolution samples generated by \functa, and complete results for all the datasets can be found in Appendix \ref{app:generation}.









\myparagraph{Discussion}
A significant advantage of \ours\ is its ability to achieve the capability of generation through a single optimization procedure. In contrast, \gasp\ requires solving the min-max GAN optimization, which has been acknowledged to be unstable \cite{Jabbar2020ASO}; and \functa\ requires first learning the SIREN model and modulations for each sample, followed by the training of an additional generative model, such as a normalizing flow.

Overall, these results demonstrate that the generation quality of \ours\ is comparable to existing alternatives.
In the following, we show that \ours\ exhibits superior efficiency and performance in inference tasks in comparison with \functa. 
It is worth noting that we do not compare \ours\ with \gasp, as it is purely a generative model.




\begin{figure*}[t!]
\vspace{0.5cm}
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \vspace{4pt}
    \begin{flushright}
    \includegraphics[width=0.9\columnwidth]{figures/chairs/shapenet_voxels_test_gt.png}
    \end{flushright}
    Reconstructions
    \vspace{10pt}
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
    \includegraphics[width=\columnwidth]{figures/chairs/functa_shapenet_voxels_test_recons_0_1.0.png}
    \vspace{3pt}
    \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__recons_0.png}
    \end{subfigure}%
    \\
    Super-reconstructions
    %
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
     \includegraphics[width=\columnwidth]{figures/chairs/functa_shapenet_voxels_test_recons_0_2.0.png}
     \vspace{3pt}
     \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/chairs/vamoh_shapenet_voxels_test_vamoh__voxels__recons_super_0.png}
    \end{subfigure}
    \caption{\chairs}
    \label{fig:recons_right}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    Ground truth
    \begin{flushright}
    \includegraphics[width=0.9\columnwidth]{figures/shapes3d/test/vamoh/gt.png}
    \end{flushright}
    \vspace{2pt}
    Reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_shapes3d_test_recons_0_1.0.png}
        \vspace{5pt}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/reconstruction.png}
    \end{subfigure}%
    \vspace{2pt}
    Super-reconstructions
    \begin{subfigure}{0.9\columnwidth}
    \raisebox{15pt}{\rotatebox[]{90}{\small\functa}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/functa_shapes3d_test_recons_0_2.0.png}
        \raisebox{12pt}{\rotatebox[]{90}{\small\ours}}\hspace{5pt}
        \includegraphics[width=\columnwidth]{figures/shapes3d/test/vamoh/super_reconstruction.png}
    \end{subfigure}
    \caption{\shapesd}
    \label{fig:recons_left}
\end{subfigure}%
\caption{Comparison of reconstruction quality of the proposed \ours\ and \functa\ on ground truth data from the first samples of the test set at original and super-resolution.}
\label{fig:recons}
\end{figure*}
\raggedbottom


\subsection{Reconstruction}
\label{subsec:reconstruction}
In this section, we evaluate the performance of \ours\ in reconstructing data and compare it with \functa. As before, we conduct the assessment at both the original resolution and at double the resolution, which we refer to as \textit{super-reconstruction}. This latter scenario involves generating features at new coordinate positions.


\myparagraph{Metrics} We use the Peak Signal-to-Noise-Ratio (PSNR) to quantify the quality of reconstructions (See \cref{app:reconstruction} 
 for further details).

\begin{figure}[h!]
\begin{center}
\centerline{\includegraphics[width=0.55\columnwidth]{figures/psnr_boxplot.pdf}}
\caption{Comparison of PSNR (db) of reconstructed images in the test set.}
\label{fig:recons_psnr}
\end{center}
\vskip -0.2in
\end{figure}
\raggedbottom













\begin{figure}[h!]
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{1.05\columnwidth}
\centering
Ground truth
\vspace{5pt}
\\
\includegraphics[width=1\columnwidth]{figures/celeba/celeba_hq_test.png}
Super-reconstruction
\vspace{5pt}
\\
\includegraphics[width=1\columnwidth]{figures/celeba/test/vamoh/super_reconstruction.png}
\end{subfigure}
\caption{\celebahq}
\end{subfigure}
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{0.9\columnwidth}
\centering
Ground truth
\\
\includegraphics[width=0.92\columnwidth]{figures/era5/vamoh_era5_test_gt.pdf}
Super-reconstruction
\\
\includegraphics[width=0.93\columnwidth]{figures/era5/vamoh_era5_test_super_reconstruction.pdf}
\end{subfigure}
\caption{\era}
\end{subfigure}
\caption{Ground truth for the first six test set samples and the corresponding super-reconstruction obtained by \ours\ for \celebahq\ and \era\ datasets. }
\label{fig:recons_super}
\end{figure}
\raggedbottom


\begin{figure*}[t]
\begin{subfigure}{.47\textwidth}
    \centering
    \raisebox{15pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/celebahq/missing_occluded_gt.png}
    \vspace{10pt}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/celebahq/missing_reconstruction.png}
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/shapes/missing_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/shapes/missing_reconstruction.png}
    \caption{Missing a patch}
    \label{fig:completation_left}
\end{subfigure}%
\hfill
\begin{subfigure}{.47\textwidth}
    \centering
    \raisebox{15pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/celebahq/half_occluded_gt.png}
    \vspace{10pt}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/celebahq/half_recostruction.png} 
    \raisebox{10pt}{\rotatebox[]{90}{In}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/shapes/half_occluded_gt.png}
    \raisebox{12pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
    \includegraphics[width=0.93\columnwidth]{figures/image_completion/shapes/half_reconstruction.png}
    \caption{Missing half of the image }
    \label{fig:completation_right}
\end{subfigure}%
\caption{Imputation of different amounts of missing parts using \ours. For each dataset, the top row shows the input with the missing part (In) and the bottom row shows the reconstructed image. }
\label{fig:completion}
\end{figure*}
\raggedbottom



\myparagraph{Results}
\cref{fig:recons} shows reconstructions and super-reconstructions for the first 6 samples of the test set of \shapesd\  and \chairs\ for \functa\ and \ours. 
We observe that both methods produce high-quality reconstructions.
When evaluating the \chairs\ dataset (as shown in \cref{fig:recons_left}), we observe that both models are capable of capturing the fine-grained details of the chairs, such as the various patterns present on the back of the chairs.
Notably, \ours\ achieves more detailed edges in both reconstructions and super-reconstructions for the \shapesd\ dataset (see \cref{fig:recons_right}) as well as overall less blurriness.
A more quantitative assessment is provided in \cref{fig:recons_psnr}, which compares PSNR values obtained for all the test set samples of the image-like datasets. 
We observe that \ours\ achieves similar quality on \polymnist\ and outperforms \functa\ on other datasets. 
This is quite remarkable as \functa\ requires solving an optimization problem per sample, and thus it `overfits' each of the samples. In stark contrast, \ours\ efficiently generates reconstructions with a single forward pass.
Finally, \cref{fig:recons_super} shows the ability of \ours\ to generate  quality super-reconstructions on the \celebahq\ and \era\ datasets. 











\myparagraph{Discussion} Both \ours\ and \functa\ are able to reconstruct and super-reconstruct data with high quality with different data modalities (e.g., images and voxels). On top of this, \ours\ achieves better PSNR values for the  test set, which indicates it reconstructs more details.
Additionally, \ours\ enables more efficient inference, given that it requires a simple forward pass. In contrast, \functa\ requires finding a new modulation for each new sample, which is an optimization in itself. As so, we argue that it might be preferable to rely on \ours, and thus, in the following we keep understanding the capabilities of \ours\ without comparison.







\subsection{Image completion}
We evaluate the performance of \ours\ on the task of image completion.
We consider two scenarios: (i) missing a patch (i.e., image in-painting) and (ii) missing half of the image, as a challenging case. 
\cref{fig:completion} illustrates the results on unseen test samples from \celebahq\ and \shapesd. 
Our results demonstrate that \ours\ can reconstruct high-quality images, even when half of the image is missing, as in \cref{fig:completation_right}.
Thanks to the use of PointConv \cite{wu2019pointconv} as the encoder, \ours\ can infer missing information without the need of inputting dummy values (e.g., zeros) for the features of the missing coordinates. This is something needed in standard VAEs, and allows \ours\ to mitigate any potential biases in the results. 
Complete results for all datasets evaluated, including those for image out-painting, can be found in \cref{app:im_completion}.









\subsection{The flexibility of VAMoH}

In this Section, we provide evidence of the  interpretability of \ours. In particular, we show an example in which the mixture of hypergenerators, with $K=4$, allows for splitting the generation of each pixel into several modes.

In \cref{fig:segmentation}, we show the entropy (middle row) and map (bottom row) of the posterior distribution of the categorical latent variables $\bm{C}$. By examining the entropies per pixel, we observe low values for most pixels, but higher values for the borders. This indicates that the model is uncertain about which components to use at the borders, while it is using fewer components for the rest of the image.
Furthermore, the map values of $\bm{C}$ resemble a segmentation map. Here,  we observe that one hypergenerator (orange) is specialized in filling the colors of different parts of the image, while the others (blue, red, and green) are used for the borders.


\begin{figure}[t]
\centering
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{1.0\columnwidth}
\raisebox{14pt}{\rotatebox[]{90}{Recons.}}\hspace{5pt}
\includegraphics[width=1\columnwidth]{figures/shapes3d/test/vamoh/reconstruction.png}
\end{subfigure}
\end{subfigure}
\\
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{1.0\columnwidth}
\raisebox{14pt}{\rotatebox[]{90}{\small{$\bm{C}_{\text{ent}}$}}}\hspace{5pt}
\includegraphics[width=1\columnwidth]{figures/entropy_and_seg/shapes/reconstruction_entropy.png}
\end{subfigure}
\end{subfigure}
\\
\begin{subfigure}{0.5\textwidth}
\centering
\begin{subfigure}{1.0\columnwidth}
\raisebox{14pt}{\rotatebox[]{90}{$\bm{C}_{\text{map}}$}}\hspace{3pt}
\includegraphics[width=1\columnwidth]{figures/entropy_and_seg/shapes/reconstruction_segmentation.png}
\end{subfigure}
\end{subfigure}
\caption{Reconstructed samples (top), entropy (middle) $\bm{C}_{\text{ent}} = \Hcal ( \bm{C}|\bm{Y}, \bm{X} )$, and map (bottom) $\bm{C}_{\text{map}}= \max_{c_d}  q_{\gammabc}(\bm{C}|\bm{Y}, \bm{X})$ of the posterior distribution over the mixture. }
\label{fig:segmentation}
\end{figure}
\raggedbottom


\section{Introduction}

While many real-world applications lead to data over continuous coordinate systems, such data is often discretized, e.g., by fixing the resolution of images \cite{simonyan2014very} or assuming a fixed sample frequency in time-series \cite{hochreiter1997long}. 
In contrast, recent advances in Implicit Neural Representations (INR) have been shown to  be powerful approaches for directly parameterizing  continuous functions by mapping coordinates into data features. 
To name a few examples, INRs have been successfully applied in diverse fields such as image representation \cite{stanley2007compositional, ha2016generating}, shape  and scene representation \cite{mescheder2019occupancy, genova2019learning, genova2020local, chen2019learning, zeng2022lion, sitzmann2019scene, jiang2020local, mildenhall2021nerf}, audio \cite{sitzmann2020implicit}, graphs \cite{grattarola2022generalised}, and data  manifolds \cite{dupont2021generative, dupontfuncta}.  

A couple of recent works~\cite{dupont2021generative, dupontfuncta}  have relied on INRs 
to generate data at any continuous coordinate (e.g., to generate images of different resolutions). 
\citet{dupontfuncta} disentangle the task of learning functions using an INR from the data generation task; and, \citet{dupont2021generative} 
proposes a \emph{hypergenerator} based on generative adversarial networks (GANs), where the parameters of its generator are the outputs of another network. 
However, they both suffer from  limitations, especially with regard to conditional generation tasks  such as image in/out-painting. The implicit nature of \citet{dupont2021generative} does not provide straightforward ways for conditional generation, and \citet{dupontfuncta} requires solving a computationally expensive numerical optimization problem to generate the modulation vector of each new data point (i.e., unseen during training). 


In this paper, we propose a Variational Mixture of HyperGenerators
for learning distributions over functions, referred to as \ours. Our model relies on a mixture of hyper variational autoencoders (VAEs)
\cite{kingma2013auto, rezende2014stochastic, nguyen2021variational}, where: 
\begin{itemize}[noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt]
\item[i)]  a planar normalizing flow \cite{nf_planar} is used as prior distribution over its latent variables to be able to fit and generate complex data; 
\item[ii)] a hypernetwork  is used to parameterize the mixture of decoders that, in turn,   partition  the function space into meaningful regions (e.g., into visual segmentation maps as shown in~\cref{fig:segmentation}); and, 
\item[iii)]
analogous to \citet{dupont2021generative}, a PointConv network~\cite{wu2019pointconv} is used in the encoder to map any cloud of points, i.e., any set of continuous coordinates and features, into a fixed-sized vector.
\end{itemize}
As demonstrated by our extensive experiments on several benchmark datasets, \ours\ can accurately and efficiently learn distributions over functions and, thus,  generate data over continuous coordinate systems. 
Remarkably,  in contrast to prior work, conditional generation (e.g.,  in/out-painting tasks or generating a higher-resolution version of a given image) is straightforward in \ours\ as it just requires a forward pass on the model (independently of whether the conditioning data were seen during training). 

\subsection{Inference model}
In this subsection, we present the inference model that we propose to approximate the posterior of the latent variables $\zb$ and $\bm{C}$. The model is defined as follows:
\begin{equation}
     q_{\gammab}(\zb, \bm{C}|\bm{Y}, \bm{X}) 
 = q_{\gammabz}(\zb|\bm{Y}, \bm{X}) \prod_{d=1}^{D} q_{\gammabc}(c_{d}|\zb,\bm{y}_{d}, \xb_{d}).
\end{equation}
We illustrate the inference model in Figure~\ref{fig:inference}. Throughout the following, we refer to all the parameters of the inference model as $\gammab=\{ \gammabz, \gammabc \}$.

\myparagraph{Inference for the continues latent variable} We propose to model the posterior distribution of $\zb$  as $q_{\gammabz}(\zb|\bm{Y}, \bm{X}) = \normal{\zb | f_{\gammabz} (\bm{Y}, \bm{X}) }$, parameterized by $\gammabz$.
It's important to note that this distribution is shared among the  complete sample (e.g., image), thus  $\zb$ contains global information.
To parametrize $f_{\gammabz}$, we use a PointConv \cite{wu2019pointconv}. This entails several advantages. Firstly, it generalizes convolutional operations to continuous space coordinate systems, in contrast to the fixed grids used in CNNs \cite{lecun1995convolutional}. 
Secondly, it is independent of data resolution, geometry of the grid, and missingness of the data. Of particular interest is the latter, as grid-based architectures typically require missing dimensions to be filled (e.g., with zeros), which introduces bias to the model \cite{simkus2021variational}.


\myparagraph{Inference of the mixture components}
\label{subsec:inference_model}

We model the posterior over the categorical latent variables $c_{d} $ of each coordinate $d$  as  
\begin{equation}q_{\gammabc}(c_{d}| \cdot) = \cat{c_{d}| f_{\gammabc}(\zb,\bm{y}_{d}, \xb_{d})} =  \prod_{k=1}^{K} \tilde{\pi}_{dk}^{ \indic{c_{d}=k}},
\end{equation}
where we parametrize $\gammabc$ with  a simple MLP. It is important to notice that this posterior depends on the local information of the coordinate, i.e., $(\xb_{d},\bm{y}_d)$, and only on the global information through $\zb$. This choice is made to encourage the model to learn to use different hypergenerators for different parts of the data.


\section{Variational Mixture of HyperGenerators}




In this section, we introduce the \textbf{Va}riational \textbf{M}ixture \textbf{o}f \textbf{H}yperGenerators (\ours) model. \ours\ seamlessly integrates the capabilities of Variational Autoencoders (VAEs), mixtures of generative models, and hypernetworks to handle continuous domain data points effectively. Additionally, by incorporating normalizing flows \cite{nf_planar} as an expressive prior and utilizing Implicit Neural Representations (INRs) and Point Cloud encoders \cite{wu2019pointconv}, \ours\ achieves superior performance and interpretability in a variety of both sample generation and inference tasks, such as in-painting and out-painting. The proposed generative model is depicted in Figure \ref{fig:generative}.





\begin{figure}[t!]
\centering
\begin{subfigure}{.3\textwidth}
    \centering
    \includegraphics[height=\textwidth]{figures/generative}
    \caption{Generative model}\label{fig:generative}
\end{subfigure}%
\begin{subfigure}{.3\textwidth}
    \centering
    \includegraphics[height=\textwidth]{figures/inference}
    \caption{Inference model}\label{fig:inference}
\end{subfigure}%
\caption{The VAMoH generative (a) and inference (b) model.}
\label{fig:vamoh}
\end{figure}
\raggedbottom




\myparagraph{Notation} 
We denote $[D] = \{1 \dots D\} $ to the set of natural numbers.
Let  $(\bm{X}^{(i)}, \bm{Y}^{(i)})$, $i \in [N]$ be a set of $N$ data samples (e.g., images). The $i$-th sample comprises a point cloud of $D_i$ coordinate vectors, $\bm{X}^{(i)} \doteq \{\xb^{(i)}_{j}\}_{j=1}^{D_i}$, and the set of corresponding feature vectors $\bm{Y}^{(i)}\doteq\{\bm{y}^{(i)}_j\}_{j=1}^{D_i}$. Let $\Xcal$ and $\Ycal$ denote the space of coordinate and feature vectors, respectively. As an example, in the context of image analysis, $D_{i}$ represents the number of pixels in image $i$, and $(\bm{X}^{(i)}, \bm{Y}^{(i)})$ correspond to the set of $\Rbb^2$ coordinates and values (e.g. RGB values) of the pixels, respectively. 






\subsection{Mixture of HyperGenerators}
\label{subsec:generative_model}


\ours\; generates a feature set $\bm{Y}$ given a set of corresponding coordinates $\bm{X}$. For simplicity, let us assume that $(\bm{X}, \bm{Y})$ is an image with $D$ pixels.  To generate such an image, first a continuous latent variable $\zb$ is sampled from a prior distribution $p_{\psib_z}(\zb)$ parameterized by $\psib_z$. The resulting vector $\zb$  acts as  the input to $K$ different \emph{hypergenerators}. Here, we refer as a hypergenerator to  both an MLP-based hypernetwork $g_{\phi_k}(\zb)$, with input  $\zb$ that outputs a set of parameters $\bm{\theta}_{k} = g_{\phi_k}(\zb)$; and, a data generator, $f_{\theta_k}$, parametrized by the output of the hypernetwork. 
Thus,  both $\zb$ and $\bm{\theta}_{1}, \ldots,\bm{\theta}_{K}$ encode the information shared among the $D$ coordinates (e.g., pixel location) in the data (e.g., pixel values) generation process. 


In order for the resulting model to be expressive and interpretable, we assume that \emph{each pixel is sampled from a mixture of $K$ hypergenerators}. 
Thus, for each pixel $d \in [D]$, we introduce a latent categorical variable $c_d\in [K]$ in order to select the  hypergenerator responsible of the pixel distribution, such that
\begin{equation}\label{eq:gen}
    \begin{gathered}
    %
    p_{\psi_c}(\bm{C} | \bm{X}, \bm{z}) = 
    \prod_{d=1}^{D} \prod_{k=1}^{K}  \pi_{dk}^{\llbracket c_{d}=k \rrbracket},
    \end{gathered}
\end{equation}
where $\llbracket c_{d}=k \rrbracket$ is the indicator function. 
The probability mass function (pmf) of $c_d$ is also parameterized using a neural network $f_{\psi_c}$ with parameters $\psi_c$ that takes both $\xb_d$ and $\bm{z}$ as input, and outputs $\bm{\pi}_{d}=f_{\psi_c}(\xb_d,\bm{z})$ using a soft-max function, where $\pi_{dk}=P(c_d=k|\xb_d,\bm{z})$.

In summary, the overall generative process is given by:  
\begin{equation}\label{eq:gen_simple}
    %
    p(\bm{Y}, \bm{C}, \bm{z} | \bm{X}) =
    p_{\psi_z}(\zb) \prod_{d=1}^{D} \prod_{k=1}^{K}  
\pi_{dk}^{\llbracket c_{d}=k \rrbracket} \, p_{\bm{\theta_k}}(\bm{y}_{d}|\xb_d),
\end{equation}
where, importantly, we rely on random Fourier Features (RFF) \cite{Tancik20} to encode $\xb_d$ to capture high frequency details with our generative functions $f_{\bm{\theta}}$, as  in \citet{dupont2021generative}. 

 %














    %
    %
    %



    %
    %
    %
    %
    %
    %
    %
    %
    %
    %
    %
    %





\begin{figure}[t]
\vspace{0.5cm}
    \centering
    \centerline{\includegraphics[width=0.58\textwidth]{figures/hole}}
    \caption{Illustration of the \emph{hole problem}. Blue contours are the prior $p(\zb)$, whilst orange contours are approximate posteriors $q(\zb |\bm{X}_i, \bm{Y}_i)$. Left: simple standard prior does not accurately cover the encoder complexity. Decoding samples from the prior that fall far from the aggregated posterior from training data gives unrealistic images. Right: a more flexible prior properly matches the complexity of the encoder, leading to better quality of the images generated from the prior.}
    \label{fig:hole}
\end{figure}
\raggedbottom

\myparagraph{Flow-based prior on $\zb$}

The use of a fixed prior distribution for generating new unconditional samples in VAEs often results in the well-documented \emph{prior hole problem } \cite{rezende2018taming},  illustrated in Figure \ref{fig:hole}. This limitation stems from the poor expressiveness of a fixed prior distribution as compared to the approximate posterior. To tackle this issue, we rely on %
normalizing flows (NF) to learn a prior distribution of the  form $p_{\psi_z}(\zb)$ with parameters $\psi_z$. This improves the prior expressiveness and,  thus, addresses the aforementioned problem. Specifically, \ours\; integrates $T$  %
layers of {a}  planar flow~\cite{nf_planar} with a Gaussian $p_0(\zb)$ as base distribution, resulting in a flexible prior distribution that significantly enhances  the generated samples. Extended empirical analysis is provided in Appendix \ref{app:flow_prior}.






\input{sections/our_model_inference_short.tex}




\subsection{Training}\label{sec:training}
The evidence lower bound (ELBO) of our proposed model is given by
\begin{align}
\label{eq:elbo_ours}
\mathcal{L}(& \bm{X},  \bm{Y})  = \E[q_{\gamma}] { \log \ptheta[ \bm{Y}| \zb, \bm{X}, \bm{C}] }
& - \kld{q_{\gamma_{z}}(\zb | \bm{X}, \bm{Y} )}{p_{\psi_{z}}(\zb)}
 & - \kld{q_{\gamma_{c}}(\bm{C} |\zb, \bm{X}, \bm{Y} )}{p_{\psi_{c}}(\bm{C} | \zb,  \bm{X})}.
\end{align}

We provide the complete derivation in \cref{app_sec:elbo}. We train \ours\ maximizing this ELBO by stochastic gradient descent on randomly selected mini-batches. To stabilize the initialization of the normalizing flow and accelerate convergence, we employ a standard prior $p(\zb) = \mathcal{N}(\bm{0}, \bm{I})$ during the initial epochs of the training to allow the model to focus on organizing the approximate posterior and producing accurate reconstructions. After some iterations, we start training the Planar Flow $p_{\psi_z}(\zb)$.
While KL between discrete distributions in \eqref{eq:elbo_ours} is easily computed in closed form, the continuous KL is approximated by Monte Carlo sampling, just as we do for the reconstruction term. 






\myparagraph{Point dropout}
The design of VAMoH allows for easily handling point clouds $\{\bm{X}, \bm{Y}\}$ of arbitrary sizes. The PointConv encoder is able to convolve the observed points, regardless of their coordinates, and map them into the approximate posterior, which can then be decoded to generate new points at any desired location. To enhance the conditional generation capabilities and robustness of VAMoH in inferring information from partial data, we apply dropout to the points within a set $\{\bm{X}, \bm{Y}\}$ according to a probability $p\sim U(0, \alpha)$, which is sampled independently for each batch. The maximum dropout probability, $\alpha$, is fixed for ensuring that the reduced set contains at least as many points as centroids to be found at the first layer of PointConv. In previous VAE-based models \cite{ma2020vaem, peis2022missing}, this strategy has been successfully employed for masking training batches in order to improve missing data imputation tasks. Nevertheless, since these models deal with grid-type incomplete data, some pre-imputation is required before feeding the encoder, mean imputation or zero filling being typical choices, with the cost of introducing bias in the model \cite{simkus2021variational}. In contrast, in our work, the PointConv encoder easily handles missing data, without requiring any pre-imputation strategy. Additional empirical evaluations can be found in the Appendix \ref{app:point_dropout}. 











\section{Related Work \& Background}
\myparagraph{Variational Autoencoders (VAEs)} 
\citet{kingma2013auto, rezende2014stochastic} 
approximate the intractable posterior over latent variables $p(\zb|\xb)$ by 
performing amortized variational inference \cite{cremer2018inference, zhang2018advances}  
with an auxiliary model that obtains the  approximation $q_\phi(\zb |\xb)$, 
using an encoder-decoder architecture.  
Their objective is the Evidence Lower Bound (ELBO),
\begin{align}
\mathcal{L}(\xb) = \E[\qphi] { \log \ptheta[\xb | \zb] } - \kld{\qphi[\zb | \xb]}{p(\zb)},
\end{align}
which encourages proper data reconstruction via the first term, whilst minimizing mismatch between posterior and prior via the second term.
When complex data spaces are to be encoded in the latent space, flexible priors are required to avoid a significant mismatch between the aggregated posterior and the prior, typically referred to as the \emph{holes problem} \cite{rezende2018taming}. In previous works, this issue has been alleviated by various strategies, including using multimodal priors mimicking the aggregated posterior (VampPrior) \cite{tomczak2018vae}, or training flow-based \cite{nf_planar, kingma2016improved, nf_prob, gatopoulos2021self}, autoregressive \cite{chen2017variational} or hierarchical priors \cite{klushyn2019learning, maaloe2019biva, peis2022missing, zeng2022lion}. However, these methods are all tailored for structured data under grid representations, whereas our proposed model is specifically designed for unstructured data.





\myparagraph{Implicit Neural Representations (INRs)} INRs represent a powerful approach for parameterizing non-linear continuous functions that map coordinates to data using deep neural networks.  This allows for efficient and independent querying of continuous locations, which is useful for various tasks like learning, graphics, vision, and graphs \cite{stanley2007compositional, ha2016generating, mescheder2019occupancy, genova2019learning, genova2020local, chen2019learning, zeng2022lion, sitzmann2019scene, jiang2020local, mildenhall2021nerf,sitzmann2020implicit,grattarola2022generalised,dupont2021generative, dupontfuncta}.
Earlier versions of INRs struggled to capture high-frequency details but advancements have addressed this issue through improved input encoding \cite{Tancik20}, activation functions, and network architectures. Recently, \citet{hao2022loe} proposes a novel INR Levels-of-Experts (LoE) model that generalizes INRs based on MLPs with position-dependent weights, greatly increasing the model capacity. On the downside, the complexity of coordinate-dependent generation functions rules out the use of hypernetwork-based approaches.



\myparagraph{HyperNetworks} HyperNetworks \cite{hypernetworksha} are a powerful class of neural networks that generate the weights for a principal network. 
Recently, \citet{nguyen2021variational} combined hypernetworks with VAEs in order to improve performance and generalization when modeling different tasks concurrently. In this approach, hypernetworks are used to generate the parameters of the approximate posterior ($\phib$) and the likelihood ($\thetab$).
In contrast, our proposed approach draws inspiration from the generator of \gasp\ \cite{dupont2021generative} and utilizes the hypernetwork to output the parameters  of our data generator. 








\myparagraph{Deep Generative Models for INRs}
Recently, several works have proposed to use INRs for learning distributions of functions, rather than distributions of data directly.  
\citet{dupontfuncta} propose to disentangle the task of learning functions by first learning the named \emph{functas}, or modulation vectors that configure an INR for each datapoint. 
SIREN \cite{sitzmann2020implicit} is used as the base INR network. In the second stage, any deep generative model can be trained on the learned \emph{functaset}. Then, for conditional generation tasks, computing the modulation vector for a test point requires solving a numerical optimization problem.


In \cite{rodriguez2022function}, the INR generator is constructed by combining a Bayesian NN with Gaussian weight priors that takes as input both the coordinates and a sample from a latent Gaussian noise distribution. $\alpha$-divergence Variational Inference is developed by jointly approximating the function at different coordinates using a Gaussian Process. Inducing points are introduced to make the model scalable. While the performance of the method is remarkable in a small-to-moderate dimension and it provides posterior inference, scaling it to model high-dimensional objects such as images is certainly not trivial.

In GASP \cite{dupont2021generative}, a generator of functions is built by transforming samples from a standard Gaussian latent variable to 
sets of weights using a hypernetwork. They train this generator using a GAN-style approach jointly with a PointConv-based discriminator that tries to discern fake from real samples. The LoE model in \cite{hao2022loe} is also reformulated in a generative way by taking as input both the coordinate vector and a Gaussian latent noise sample and it is also adversarially trained. 
Therefore, we can state that across \functa, \gasp\ and LoE, inference over test data to perform conditional generation, for instance, image completion or super-resolution, is not trivial as they all require numerical optimization to find the latent codes. 
In \ours,  we use a similar generator approach as in \gasp, but we generalize it by using a mixture of hypergenerators combined with a flexible latent space constructed using normalizing-flows \cite{Rezende2015VariationalIW}. 
Additionally, \ours\ can be robustly trained using stochastic variational inference and do not require any extra optimization to perform inference on unseen data. 
The proposed method is a significant advancement in the field of INRs and deep generative models, and we demonstrate it has the potential to achieve state-of-the-art results in various tasks.









\section*{Acknowledgements}

Batuhan Koyuncu and Isabel Valera acknowledge the support by the German Federal Ministry of Education and Research (BMBF) through the Cell-o project. Pablo S\'anchez Mart\'in thanks the German Research Foundation through the Cluster of Excellence “Machine Learning – New Perspectives for Science”, EXC 2064/1, project number 390727645 for generous funding support. 
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pablo S\'anchez Mart\'in.
Pablo M. Olmos and Ignacio Peis acknowledge the support by Spanish government (AEI/MCI) under grants PID2021-123182OB-I00, PID2021-125159NB-I00 and RTI2018-099655-B-100, by Comunidad de Madrid under grant IND2022/TIC-23550, by the European Union (FEDER) and the European Research Council (ERC) through the European Union's Horizon 2020 research and innovation program under Grant 714161, and by Comunidad de Madrid and FEDER through IntCARE-CM. The work of Ignacio Peis has been also supported by by Spanish government (MIU) under grant FPU18/00516.

\bibliographystyle{unsrtnat}
