\section*{Introduction}

Spectral properties are ubiquitous in materials science, characterizing properties ranging from crystal structure (e.g. x-ray absorption and Raman spectroscopy), the interactions of material with external stimuli (e.g. dielectric function and spectral absorption), and fundamental characteristics of its quasi-particles (e.g. phonon and electronic densities of states).\cite{Martin2004}
Matching the breadth of spectral properties is the breadth of methods to characterize them, traditionally experimental and \textit{ab initio} computational techniques, which are central to materials discovery and fundamental research. Materials discovery efforts are largely defined by searching for materials that exhibit specific properties, and acceleration of materials discovery has been realized with automation of both experiments and computational workflows.\cite{m._l._green_fulfilling_2017,jain_computational_2016} These high throughput techniques have been effectively deployed in a tiered screening strategy wherein high-speed methods that may sacrifice some accuracy and/or precision can effectively down-select materials that merit detailed attention from more resource-intensive methods.\cite{hautier_finding_2019,yan_solar_2017,stein_progress_2019} The recent advent of machine learning prediction of materials properties has introduced the possibility of even higher throughput primary screening due to the minuscule  expense of making a prediction for a candidate material using an already-trained model. \cite{kong2021materials,goodall2020predicting,louis2020graph,ward_general-purpose_2016}
Toward this vision, we introduce the materials to spectrum (Mat2Spec) framework for predicting spectral properties of crystalline materials, demonstrated herein for the prediction of the \textit{ab initio} phonon and electronic densities of state.


Successful development of machine learning models for materials property prediction hinges upon encoding of structure-property relationships to provide predictive models that serve as primary screening tools and/or provide scientific insights via inspection of the trained model.\cite{louis2020graph,xie_crystal_2018,chen2019graph} 
Accelerated screening of candidate materials is especially important when a dilute fraction of materials exhibits the desired properties, as is inherently the case when searching for exemplary materials. Primary screening constitutes a great opportunity for machine learning (ML)-accelerated materials discovery but is challenged by the need for models to generalize, both in predicting the properties of never-before-seen materials and in predicting property values for which there are few training examples.\cite{butler_machine_2018,Tess2021} 

Most of the ML models developed in computational materials science to date have been focused on individual scalar quantities. Common properties are those directly computed  \textit{ab initio}, such as the formation energy \cite{isayev_universal_2017,xie_crystal_2018,banjade2021}, the shear- and bulk-moduli \cite{de_jong_statistical_2016,isayev_universal_2017,xie_crystal_2018,chen_graph_2019}, the band gap energy \cite{xie_crystal_2018,zhuo_predicting_2018,chen_graph_2019,banjade2021}, and the Fermi energy \cite{xie_crystal_2018}. Additional targets include properties calculated from the \textit{ab initio} output, which are typically performance metrics for a target application such as Seebeck coefficient \cite{gaultois_perspective_2016,furmanchuk_prediction_2018}. For a complete review of target properties predicted via ML, see ref.\cite{schmidt_recent_2019}.
Efforts to featurize materials for use in any prediction task started with an engineered featurizer algorithm\cite{Ward2016} and evolved to automatic generation of features via aggregation from multiple sources.\cite{ward_matminer_2018} This evolution mirrors the best practices in machine learning that have evolved from feature engineering to internalizing feature representation in a model trained for a specific prediction task. This is reflected in materials property prediction with models that generate a latent representation of a material from its composition and structure, for which graph neural networks (GNN) are the state of the art due to their high representation learning capabilities.\cite{wu2020comprehensive}. 

CGCNN \cite{xie2018crystal} is among the first graph neural networks proposed for materials property prediction. CGCNN encodes the crystal structures as graphs where the unit cell of the crystal material is represented as a graph such that nodes would represent the atoms and connecting edges would represent the bonds shared among the atoms. MEGNet \cite{chen2019graph}
expanded this concept by introducing a global state input including temperature, pressure, and entropy. The state of the art is well represented by GATGNN,\cite{louis2020graph} which uses local attention layers to capture properties of local atomic environments and then a global attention layer for weighted aggregation of all these atom environment vectors. The expressiveness of this model allows it to outperform previous models in single-property prediction. Lastly, another different approach is demonstrated by AMDNet \cite{banjade2021} which uses structure motifs and their connections as input of a GNN to predict electronic properties in metal oxides.

While multi-property prediction has been addressed only recently, rapid progress has been made by building upon the foundation of the single-target prediction models. De Breuck et al.\cite{DeBreuck2021} presented MODNet and highlighted
the benefit of using feature selection and joint learning in materials multi-property prediction with a small dataset.

Broderick and Rajan\cite{broderick_eigenvalue_2011} used principle component analysis to generate low-dimensional representations of eDOS enabling its prediction for elemental metals. Yeo et al.\cite{Yeo2019} expanded this approach to metal alloys and their surfaces using engineered features to interpolate from simple metals to their alloys. 

Del Rio et al.\cite{DelRio2020} proposed a deep learning architecture to predict the eDOS for carbon allotropes.
Mahmoud et al. \cite{ben_mahmoud_learning_2020} presented a ML framework based on sparse Gaussian process regression, a SOAP-based representation of local environment, and an additive decomposition of the electronic density of states to learn and predict the eDOS for silicon structures. Bang et al.\cite{bang_accelerated_2021} applied CGCNN to eDOS, compressed via principle component analysis, of metal nanoparticles.


These initial demonstrations of eDOS prediction focus on specific classes of materials with a limited structural and chemical diversity, making them ill-suited for the present task of generally-applicable prediction of phDOS and eDOS. 

The state of the art method that is most pertinent to the present work is the E3NN model that  was recently demonstrated for predicting phDOS.\cite{Tess2021} E3NNs are Euclidean neural networks, which by construction include 3D rotations, translations, and inversion and thereby capture full crystal symmetry, and achieve high-quality prediction using a small training set of $\sim 10^3$ examples. E3NN obtains very good performance in predicting the computed phDOS from only the crystal structure of materials. To the best of our knowledge, GNN-based methods have not been reported for phDOS or eDOS prediction of any crystalline material. To create a baseline GNN model in the present work, we adapt GATGNN\cite{louis2020graph} for spectrum prediction.

We recently reported a multi-property prediction model H-CLMP \cite{kong2021materials} for prediction of experimental optical absorption spectra from only materials composition. H-CLMP implements hierarchical correlation learning by coupling multivariate Gaussian representation learning in the encoder with graph attention in the decoder. 

Mat2Spec builds upon the concepts of H-CLMP to address the open challenge of spectral property prediction with a GNN materials encoder coupled to probabilistic embedding generation and contrastive learning. Mat2Spec is demonstrated herein for eDOS and phDOS spectra prediction for a broad set of materials with periodic crystal structures from the Materials Project.\cite{Jain2013}
These densities of states represent the most fundamental spectra from \textit{ab initio} computation and characterize the vibrational and electronic properties of materials. Given the computational expense of generating these spectra, the ability to predict these spectra, even in an approximate way, represents a valuable tool to perform materials screening to guide \textit{ab initio} computation.
We demonstrate such acceleration of materials discovery with a use case based on identification of band gaps below but near the Fermi energy in metallic systems, which have been shown to be pertinent to thermoelectrics and transparent conductors.\cite{Ricci2020,Malyi2019_gapped}


\section*{Results}

\subsection*{Material-to-Spectrum model architecture}


Mat2Spec is a model for predicting a spectrum (output labels) for a given crystalline material (input features) by: (i) encoding the labels and features onto a latent Gaussian mixture space, with alignment of the feature and label embeddings to exploit underlying correlations; and (ii) contrastive learning to maximize agreement between the feature and label representations to learn a label-aware feature representation.
These 2 modules can be considered as a multi-component encoder and decoder, respectively, that collectively form an end-to-end model whose architecture is detailed in Figure~\ref{fig:m2s}.

\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{figures/Mat2Spec.pdf}
\caption{\textbf{The Mat2Spec model architecture}: The prediction task of material (Input Features) to spectrum (Predicted Labels) proceeds with 2 primary modules, a probabilistic embedding generator (Encoder) to learn a suitable representation ($Z_F$) of the material and a Decoder trained via supervised contrastive learning to predict the spectrum from that embedding. The solid arrows show the flow of information for making predictions with the trained model, and the dashed arrows show the additional flow of information during model training, where the ground truth spectrum (Input Labels) is an additional input for which the Encoder produces the embedding $Z_L$. Alignment of the multivariate Gaussian mixture model parameters during training conditions the probabilistic generator of the embeddings.
Both the input material (Feature projection) and input spectrum (Label projection) are reconstructed to train the model via contrastive learning. 
The Representation Translator is shared by the prediction and reconstruction tasks, resulting in parallel latent representations that are transformed into the final outputs by the Predictor and Projector, respectively.}
\label{fig:m2s}
\end{figure}

Conceptually, Mat2Spec commences with feature encoder whose high-level strategy is similar to that of E3NN and GATGNN, where each model aims to learn a materials representation that accurately captures how structure and composition relate to the properties being predicted. The first component of Mat2Spec's encoder is a graph neural network (GNN) that is based on previously-reported approaches to materials property prediction in which the GNN is the entirety of the encoder.\cite{wu2020comprehensive,xie2018crystal,louis2020graph} The remainder of the Mat2Spec framework introduces a new approach to learning from the GNN encodings and comprises our contribution to the design of machine learning models for materials property prediction. 

Compared to single-property prediction, spectrum prediction offers additional structure that may be captured through careful design of the model architecture. We consider discretized spectra where the input features that are important for prediction of the intensity at one point in the spectrum are likely related to those of other points in the spectrum.
While prior models such as E3NN and GATGNN have no mechanism to explicitly encode these relationships, Mat2Spec captures this structure of the task with a probabilistic feature and label embedding generator built with multivariate Gaussians. 
During training, the generator operates on both the material (input features) and its spectrum (input label). In the process of label embedding, each point $i$ in the spectrum with dimension $L$ is embedded as a parameterized multivariate Gaussian $N_i$ with learned mixing coefficient $\alpha_i$, where $\sum_{i} \alpha_i=1$. The spectrum for a material is thus embedded as a multivariate Gaussian mixture $\sum_i \alpha_i N_i$. The mixing coefficients capture relationships among the points in the spectrum where related points tend to have similar weights. To capture the common label structure among different materials, the set of multivariate Gaussians $\{N_i\}_{i=1}^L$ is shared across all materials. 

For feature embedding from the GNN encoding of the material, we learn a set of $K$ multivariate Gaussians $\{M_j\}_{j=1}^K$ as well as a set of $K$ mixing coefficients $\{\beta_j\}_{j=1}^K$, and the features of a material is thus embedded as a multivariate Gaussian mixture $\sum_j \beta_j M_j$. Note that $K$ is a hyperparameter which is not required to be equal to the number of points in the spectrum. While the only input to the multivariate Gaussian for feature embedding is the output of the GNN, we leverage the learned structure of label embedding by promoting alignment of the the two multivariate Gaussian mixtures, which both have dimension $D$, a hyperparameter. Specifically, the training loss includes the Kullback-Leibler (KL) divergence between the two multivariate Gaussian mixtures $\sum_i \alpha_i N_i$ and $\sum_j \beta_j M_j$. 


The probabilistic embedding generator provides the inputs to the contrastive learning decoder. The decoder commences with a shared multi-layer perceptron (MLP) Representation Translator, which during model training is evaluated once with the feature embedding to generate the feature representation and again with the label embedding to generate the label representation. Using these representations, the Predictor and Projector are shallow MLP models that produce the predicted spectrum and feature/label projections, respectively. While only the Predictor is used when making new predictions, the Projector maps representations to the space where the contrastive loss is applied during training. The Representation Translator and Projector are trained to maximize agreement between representations using a contrastive loss, making the feature representation label-aware. We follow the empirical evidence that the contrastive loss is best defined on the projection space rather than the representation space.\cite{chen2020simple}

For spectral data that is akin to a probability distribution for each material, the prediction task has additional structure, for example that an increased intensity in one portion of the spectrum predisposes other portions of the spectrum to have lower intensity. The Mat2Spec strategy of learning relationships among how input features maps to multiple labels supports this aspect of distribution prediction, although deployment of Mat2Spec for distribution learning is ultimately achieved by training using either the Wasserstein distance (WD) or KL loss, which each quantify the difference in 2 probability distributions, as the primary training loss function. Combined with more traditional loss functions,

we compare Mat2Spec with baseline models E3NN and GATGNN with 3 different settings, i.e. different combinations of data scaling and training loss function. Note that our adaptions of E3NN and GATGNN for distribution-based training constitute models beyond those previously reported that are referred to herein by the respective name of the reported model architecture.

\subsection*{Predicting phonon density of states}

The task of phDOS prediction of crystalline materials was recently addressed by E3NN.\cite{Tess2021} To provide the most direct comparison to that work, we commence with their setting wherein each phDOS is normalized to a maximum value of 1 (MaxNorm) and mean squared error (MSE) loss function is used for model training. An important attribute of phDOS is that the integral of the phDOS can be approximated as 3 times the number of sites in the unit cell. As a result, not only are data scaled for model training, but the output of a ML model can be similarly scaled as the final step in spectrum prediction. Since our ground truth knowledge about scaling is with regards to the sum of the phDOS as opposed to the maximum value, we also consider scaling data by its sum (NormSum). This scaling make phDOS mathematically equivalent to a probability distribution, motivating incorporation of ML models for learning distributions, i.e. predicting the distribution of phonon energies in a material with a known number of phonons. Our 2 corresponding settings for phDOS prediction use NormSum scaling paired with each of WD and KL loss functions. The combination of 3 ML models and 3 settings provides 9 distinct models for phDOS prediction whose performance is summarized in Table \ref{tab:phdos} using 4 complementary prediction metrics: the coefficient of determination (R$^2$ score, which is typically between 0 and 1 but can be negative when the sum of squares of the model's prediction residuals are larger than the total sum of squares of difference of the observation values from their mean), mean absolute error (MAE), MSE, and WD. In each of the 3 settings, Mat2Spec outperforms E3NN and GATGNN for all 4 of these metrics. The results are most comparable among the 3 ML models in the MaxNorm-MSE setting, and the best value for each metric is obtained with one of the Mat2Spec models. 


\begin{table*}[hb]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
\hline
ML model & \multicolumn{2}{c|}{Setting} & \multicolumn{4}{c|}{phDOS prediction} & \multicolumn{2}{c|}{Calculated $C_v$}& \multicolumn{2}{c|}{Calculated $\bar{\omega}$}  \\ 

 & Scaling & Loss & R$^2$ & MAE & MSE & WD & MAE & MSE & MAE & MSE \\ \hline
E3NN & MaxNorm & MSE & 0.56 & 0.094 & 0.034 & 39 & 3.58 & 56 & 30.9 & 3161 \\ \hline
GATGNN & MaxNorm & MSE & 0.45 & 0.105 & 0.042 & 44 & 4.66 & 80 & 35.1 & 3392 \\ \hline
Mat2Spec & MaxNorm & MSE & \textbf{0.63} & 0.086 & 0.029 & 33 & 3.30 & 49 & 26.2 & 2284 \\ \hline
E3NN & SumNorm & WD & -0.48 & 0.339 & 1.884 & 132 & 11.7 & 393 & 90.0 & 17343 \\ \hline
GATGNN & SumNorm & WD & -2.78 & 0.185 & 0.065 & 194 & 19.6 & 591 & 183 & 42753 \\ \hline
Mat2Spec & SumNorm & WD & 0.57 & 0.085 & 0.026 & \textbf{21} & \textbf{1.32} & \textbf{10} & \textbf{10.6} & \textbf{348} \\ \hline
E3NN & SumNorm & KL & 0.48 & 0.105 & 0.036 & 50 & 4.88 & 77 & 41.1 & 3718 \\ \hline
GATGNN & SumNorm & KL & -1.05 & 0.177 & 0.057 & 215 & 22.4 & 756 & 205 & 51609 \\ \hline
Mat2Spec & SumNorm & KL & 0.62 & \textbf{0.078} & \textbf{0.023} & 24 & 1.96 & 11 & 17.1 & 625 \\ \hline

\end{tabular}
\caption{\label{tab:phdos} Results of phDOS prediction on the test set. For each combination of 3 ML models and 3 settings, the performance metrics include 4 measures of the prediction of the 51-D phDOS and 2 measures each for the properties $C_v$ (J/K$\cdot$ mol) and $\bar{\omega}$ (cm$^{-1}$) calculated from each material's phDOS. Each loss metric is aggregated over all materials in the test set. Note that for phDOS predictions with MaxNorm scaling, each prediction is re-scaled to a maximum value of 1 prior to evaluating prediction loss, and analogously the predictions with SumNorm scaling are re-scaled to a sum of 1 prior to evaluating prediction loss. MAE and MSE metrics for phDOS prediction inherit the arbitrary units from the respective scaling and should not be directly compared across scaling. R$^2$ (unitless) and WD (units of cm$^{-1}$ from the energy axis) are insensitive to this difference in scaling. The best value in each column is noted in bold. In each setting and for each loss metric, Mat2Spec provides the best predictions. 
}
\end{table*}



\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/phDOSbox.pdf}
\caption{\textbf{Comparison of metrics calculated from phDOS in the test set}: For each of 3 settings (horizontal shaded bars), the results of the $C_v$ (left) and $\bar{\omega}$ (right) calculations from the predicted phDOS in the test set is shown as the relative error with respect to the ground truth value for each material in the test set. For each of the 3 settings, 3 models, and 2 properties, the relative errors are shown with a box plot\cite{}. In the MaxNorm-MSE setting, all 3 ML models have similar median relative losses for both $C_v$ and $\bar{\omega}$, with Mat2Spec providing a smaller interquartile range and less extreme outliers. In the other settings, Mat2Spec outperforms the other ML models with respect to median, interquartile range, and outliers.
}
\label{fig:phDOSbox}
\end{figure}


To highlight the value of phDOS prediction, Chen et al. \cite{Tess2021} noted the importance of calculating properties such as the average phonon frequency $\bar{\omega}$ and the heat capacity $C_V$.
Table \ref{tab:phdos} shows the MAE and MSE loss for these quantities calculated from the test set of each of the 9 phDOS prediction models. Mat2Spec in the SumNorm-WD setting provides the best performance for both metrics and for both physical quantities. Figure \ref{fig:phDOSbox} summarizes the range of prediction errors for these quantities, demonstrating that in each setting, Mat2Spec is not only optimal in the aggregate metrics but also has a lower incidence of extreme outliers, further highlighting how this model facilitates generalization to accurate spectral prediction for all materials.

\subsection*{Predicting electronic density of states}

We consider prediction of the total electronic DOS (eDOS) of nonmagnetic materials, which is more complex than phDOS prediction in a variety of ways. While a full-energy density of occupied states would share the phDOS attribute of having a known sum for each material due to the known electron count of the constituent elements, the eDOS is of greatest interest in a relatively small energy range near the band edges and including both unoccupied and occupied states. In the present work we consider an energy grid of 128 points spanning -4 to 4 eV with respect to band edges with 63 meV intervals. On this energy grid, the Fermi energy, as well as the band edges where applicable, are all set to 0 eV; in what follow, we remove the zero-valued eDOS region between the valence band maximum (VBM) and conduction band minimum (CBM) for materials with a finite band gap. While predicting the band gap energy is also important, this task is the subject of major effort in the community,\cite{xie_crystal_2018,zhuo_predicting_2018,chen_graph_2019} and we focus here on the prediction of eDOS surrounding the band edges of greatest value. 

As a traditional setting for eDOS prediction we use zero-mean, unit-variance (Standard) scaling for each energy point in the 128-D grid with MAE training loss. The SumNorm scaling with both WD and KL loss remain important settings, and when each ML employs one of these distribution-learning settings, the predictions are re-scaled to the native eDOS units of states/eV using the sum of the prediction from the Standard-MAE setting. As a result, each ML model in each setting produces eDOS prediction in the native units such that the metrics for eDOS prediction can be compared across the set of 9 eDOS prediction results. Note that per its definition, computational of the WD metric involves SumNorm scaling of all ground truth and predicted eDOS. 

Figure \ref{fig:quintile} shows representative examples of phDOS and eDOS ground truth and predictions, with additional examples provided in Figure \ref{fig:phDOSquintile} and \ref{fig:eDOSquintile}.
For each ML model the test set materials are split into 5 equal-sized groups based on the quintiles of MAE loss. For each quintile, the intersection over the 3 ML models provides a set of materials with comparable relative prediction quality. Selection from each quintile set enables 
visualization of good (left) to poor (right) predictions. In phDOS prediction, models can take advantage of the broad range of materials with similar phDOS patterns, as evidenced in the lower quintiles of Figures \ref{fig:quintile} and \ref{fig:phDOSquintile}.
The eDOS, particularly with an appreciable energy window, exhibits more high-frequency features with fewer analogous characteristic patterns. For both phDOS and eDOS, Mat2Spec consistently predicts the correct general shape of each pattern with regression errors arising mostly from imperfect prediction of the magnitude of sharp peaks.




\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/combineDOSquintile.pdf}
\caption{\textbf{Example phDOS and eDOS predictions in the test set}: Ground truth and ML predictions are shown for 5 representative materials chosen from the 5 quintiles from low MAE loss ($Q_1$, left) to high MAE loss ($Q_5$, right) for both phDOS (top) and eDOS (bottom). The best overall setting for each model is used, which is SumNorm-KL except for phDOS prediction by E3NN and GATGNN that use MaxNorm-MSE. From $Q_1$ to $Q_5$ the ground truth phDOS is increasingly complex. For $Q_1$ to $Q_3$, Mat2Spec predicts each phDOS feature, where the other models are less consistent. In $Q_4$, E3NN and Mat2Spec are comparable with their prediction of 3 primary peaks in the phDOS. In $Q_5$, the presence of substantial density above 600 cm$^{-1}$ is quite rare, and Mat2Spec is the only model to make the correct qualitative prediction. For eDOS, there is no analogous change in the shape of the eDOS across the quintiles. In $Q_1$ and $Q_2$, Mat2Spec provides the only qualitatively correct predictions. In $Q_3$, each model predicts a smoothed version of the ground truth. In $Q_4$, Mat2Spec prediction is far from perfect but is the only prediction to capture the series of localized states near the Fermi energy. In $Q_5$, each model has qualitatively comparable predictions in the conduction band, but Mat2Spec is the only model to capture the primary structure of the valence band. Mat2Spec’s under-prediction of one localized state makes this one if its highest MAE predictions, which is far lower than the worst predictions from other models. The ability of Mat2Spec to globally capture the qualitative patterns for both phDOS and eDOS leads to its superior performance for each metric in Tables \ref{tab:phdos} and \ref{tab:edos}.
}
\label{fig:quintile}
\end{figure}

\begin{table*}[!hb]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
ML model & \multicolumn{2}{c|}{Setting} & \multicolumn{4}{c|}{eDOS prediction} & \multicolumn{3}{c|}{VB gap identification} & \multicolumn{4}{c|}{VB gap discovery} \\ 
   &   Scaling   &   Loss   &   R$^2$   &   MAE   &   MSE   &   WD   &   F1   &   precision   &   recall   &   TP   &   TN   &   FP   &   FN \\ \hline
E3NN   &   Standard   &   MAE   & 0.39 & 5.24 & 105.1 &   \textit{0.48}   & 0.035 & 0.333 & 0.019 & 0 & 17 & 0 &   15 \\ \hline
GATGNN   &   Standard   &   MAE   & 0.30 & 4.89 & 120.9 & 0.42 & 0.182 & 0.263 & 0.139 & 0 & 17 & 0 &   15 \\ \hline
Mat2Spec   &   Standard   &   MAE   & 0.53 &    \textbf{3.64}   & 80.4 & 0.27 & 0.352 & 0.509 & 0.269 & 4 & 17 & 0 &   11 \\ \hline
E3NN   &   SumNorm   &   WD   &   \textit{-2.1}   &    \textit{9.81}   &   \textit{542.8}   & 0.40 &   \textit{0}   &   \textit{0}   &   \textit{0}   & 0 & 17 & 0 &   15 \\ \hline
GATGNN   &   SumNorm   &   WD   & 0.19 & 6.41 & 140.1 & 0.26 & 0.018 & 0.200 & 0.009 & 0 & 17 & 0 &   15 \\ \hline
Mat2Spec   &   SumNorm   &   WD   & 0.38 & 5.23 & 107.7 & 0.29 & 0.018 & 0.250 & 0.009 & 0 & 17 & 0 &   15 \\ \hline
E3NN   &   SumNorm   &   KL   & 0.41 & 5.01 & 101.9 & 0.47 &   \textit{0}   &   \textit{0}   &   \textit{0}   & 0 & 17 & 0 &   15 \\ \hline
GATGNN   &   SumNorm   &   KL   & 0.32 & 4.89 & 118.2 & 0.35 & 0.243 & 0.450 & 0.167 & 3 & 16 & 1 &   12 \\ \hline
Mat2Spec   &   SumNorm   &   KL   &   \textbf{0.57}   & 3.8 &   \textbf{74.5}   &   \textbf{0.21}   &   \textbf{0.397}   &   \textbf{0.698}   &  \textbf{0.278}  & 15 & 0 & 17 &  0 \\ \hline

\end{tabular}
\caption{\label{tab:edos} Results of eDOS prediction. For each combination of 3 ML models and 3 settings, the performance metrics include 4 measures of the prediction of the full 128-D eDOS (evaluated on the test set), 3 measures of the classification accuracy for identification of gaps in the eDOS in the VB (evaluated on the test set), and classification results for 32 materials predicted to have a VB gap according to the Mat2Spec model with KL loss. Note that the predictions from SumNorm scaling are re-scaled to the original eDOS units using the normalization factor provided by the Standard scaling prediction from the respective ML model. As a result, MAE loss is in native eDOS units of states/eV,
and MSE has the square of these units; WD has units of eV from the energy axis, and the remainder of the metrics are unitless. For the metrics calculated on the test set, the best value is in bold and worst value is in italics, and Mat2Spec exhibits the best value for each metric.
}
\end{table*}

Table \ref{tab:edos} provides the eDOS performance metrics from each combination of 3 ML models and 3 settings using the same 4 regression metrics as shown for phDOS prediction. Here, the WD metric is calculated using SumNorm scaling and thus measures the quality of distribution prediction under an assumption that the sum of each material's eDOS is known. While the best performance against the WD metric in phDOS prediction was achieved using WD as training loss, Mat2Spec in the SumNorm-KL setting provides the best eDOS results with respect to the WD, R$^2$, and MSE metrics. We anticipate that model training with the WD loss function is compromised by the many sharp peaks in eDOS data, although a full diagnosis of the relatively poor performance in eDOS compared to phDOS prediction for the SumNorm-WD setting may be pursued in future work. 

The SumNorm-KL setting improves the eDOS prediction for each ML model compared to the Standard-MAE setting for nearly all of the prediction metrics, with the only exception being a slight increase in MAE for Mat2Spec. While these results highlight the general value of using distribution learning with any ML model, the superior performance of Mat2Spec with respect to the baseline models in the SumNorm-KL setting highlights the particular advantages of distribution learning when using probabilistic embedding generation and contrastive learning to condition the model.

The probabilistic embedding generator as well as the contrastive learning are intended to amplify knowledge extraction from data. One way to assess success toward this goal is to study the prediction performance with decreasing training size, as shown in Figure \ref{fig:eDOSscaling}. For the complementary prediction loss metrics MAE and WD, Mat2Spec has relatively little degradation in performance with decreasing training data, providing better predictions when using 1/8 of the training data than the baseline models that use the full training data.



\begin{figure}[h]
\centering
\includegraphics[width=0.6\linewidth]{figures/eDOStrainsize.pdf}
\caption{\textbf{Data size dependence of eDOS prediction}: Using a static test set, random down-selection of the train set by factors of 2, 4, and 8 enables characterization of how the prediction loss varies with size of the training set. This study was performed for each ML model in the Standard-MAE setting as well as for Mat2Spec in the SumNorm-KL setting. As expected, prediction loss generally increases with decreasing training size. Using only 1/8 of the training data, Mat2Spec (in either setting) provides lower MAE and WD losses compared to the baseline models that use the full training set. Achieving better results with an 8-fold reduction in data size highlights how the structure of Mat2Spec conditions the model to learn more with less data.
}
\label{fig:eDOSscaling}
\end{figure}


\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/eDOSradar.pdf}
\caption{\textbf{Summary of loss metrics for eDOS prediction in the test set}: The 3 panels correspond to 3 eDOS settings, and each shows results for the 3 ML models for a total of 9 radar plots corresponding to the 9 rows in Table \ref{tab:edos}. In the table, the worst and best values are shown in italics and bold, respectively, providing the minimum and maximum values for each axis of the radar plots so that maximum value on each axis corresponds to the best performance. Since each axis uses the same scaling, all 9 radar plots are directly comparable. The axes include 4 metrics for 128-D eDOS prediction (R$^2$, MAE, MSE, WD) and 3 metrics for the classification of gaps in the VB. For each metric, Mat2Spec provides the best value in either the Standard-MAE or SumNorm-KL settings, with Mat2Spec predictions in the SumNorm-KL setting providing the best collective performance as visualized by its maximal area among the radar plots.}
\label{fig:eDOSradar}
\end{figure}

While our focus on phDOS and eDOS prediction is precisely motivated by their transcendence of specific uses of materials, evaluating eDOS prediction for a specific use case provides a complementary mechanism for gauging the quality and value of the predictions. For a use case we choose a classification of materials based on a desirable feature in the eDOS for certain materials. We consider the presence of energy gaps (or near-energy gaps) in occupied states close to the Fermi energy, which corresponds to an intrinsic eDOS that is similar to that of a degenerate semiconductor. Correspondingly, the metals exhibiting an energy gap in occupied states may exhibit transport-related properties such as electronic conductivity and a Seebeck coefficient that mimic a doped semiconductor,\cite{Ricci2020}
broadening the materials search space. 
Indeed, this concept of metallic system having a gap close to the Fermi energy was introduced and studied in the research of potential transparent conductive materials,\cite{Zhang2015,Malyi2019_gapped,Poeppelmeier2016,Rondinelli2019} for applications in low-loss plasmonics,\cite{Gjerding2017} and in catalysis. \cite{xu_red_2012}

Recently, a high-throughput search for "gapped metals" exploited this key feature of the eDOS to find new potential thermoelectric materials.\cite{Ricci2020}
For the present use case we focus on materials with a single energy gap below but near the Fermi energy, hereafter called the ``VB gap'', and we evaluate the ability to recognize this feature in a broad range of materials, considering the entire test set (containing metals and semiconductors). We note that a similar approach can be applied for gaps in the conduction band.
For a VB gap to be sufficiently interesting to merit follow-up study of the material, i) the eDOS must be sufficiently low in intensity throughout the VB gap, for which we use a threshold of 1 state/eV; and ii) the energy gap must be sufficiently wide in energy, for which we use a 1 eV minimum gap width, iii) sufficiently close to the VB edge, for which we require that the high-energy edge of the gap is no smaller than -1 eV, and iv) sufficiently far the Fermi energy so that there are available carriers, for which we require that the high-energy edge of the gap is no larger than -0.25 eV. This stringent set of criteria corresponds to a low positivity rate, in particular only 2.8\% of materials in the test set meet all 4 criteria, highlighting the inherent challenge of discovering such materials. 

To ascertain the ability of each prediction model to identify such materials, the binary classification for the presence of a VB gap was evaluated, as summarized by the precision, recall, and F1 scores in Table \ref{tab:edos}. The requirement for eDOS to remain below the threshold for a wide, contiguous energy range requires the predictions to be simultaneously accurate for many energies, which is aligned with our strategy for distribution-based learning. At the same time, any over-prediction of eDOS in the relevant energy range disrupts the detection of a VB gap, and since KL loss penalizes errors in small eDOS values to a greater extent than WD, the SumNorm-KL setting is intuitively the best approach for this use case, which is reflected in its improved classification scores for both the GATGNN and Mat2Spec models compared to other settings. Each model in the SumNorm-WD setting as well as E3NN in all settings provide poor performance for this classification task, highlighting that this is an aggressive use case that is emblematic of materials discovery efforts wherein one seeks a select set of materials exhibiting unique properties.

Figure \ref{fig:eDOSradar} summarizes, for both eDOS prediction and the VB gap use case, the relative performance across the 3 ML models and 3 settings using radar plots in which each axis is scaled by the minimum and maximum value (or vice versa for loss metrics where lower is better) observed over the 9 prediction models. An eDOS model that is accurate with respect to each regression metric from eDOS prediction as well as the use case classification scores will appear as the largest shaded region in the radar plot. In the Standard-MAE setting, Mat2Spec clearly outperforms the other models, and its performance is further enhanced in the SumNorm-KL setting.

\subsection*{Guiding discovery of materials with tailored electronic properties}

To further demonstrate the importance of eDOS prediction for materials discovery, we extend the VB gap use case to the set of nonmagnetic materials in the Materials Project for which no eDOS has been calculated. The primary computational screening for materials with specific properties includes evaluation of performance-related properties as well as basic properties of the materials. In addition to the VB gap requirements, we consider materials that are relatively near the hull while also considering enough materials to generate meaningful results, prompting an upper limit of 0.5 eV/atom of the free energy hull.
For simplicity, we only consider materials with fewer than 5 elements. 
To make the search specific to gapped metals, we also require the band gap to be zero, which for materials with no available eDOS corresponds to the estimated band gap from the Materials Project structural relaxation calculation.

Querying the Materials Project for materials that lack eDOS and meet the energy above hull, number of elements, and band gap requirements produced 8,106 candidate materials. Using Mat2Spec in the SumNorm-KL setting, the predicted eDOS for each of these materials is evaluated for the VB gap classification, resulting in 32 predicted positives. This down-selection by a factor of 253 corresponds to a proportional savings in the computation time for searching for gapped metals. Based on these findings, we performed density functional theory (DFT) calculations on these 32 materials, demonstrating that the Mat2Spec predictions include 15 True Positives (TP) and 17 False Positives (FP). The corresponding recall of 0.47 is higher than that of the test set, which can be attributed to the underlying biases in the set of materials for which eDOS has not been calculated. 
The classification results for these 32 materials for all 9 prediction models is summarized in Table \ref{tab:edos}. The predictions from other models have a much higher occurrence of False among these 32 materials, making True Negative (TN) counts high at the expense of high False Negative (FN) counts. Of the 15 TP from Mat2Spec in the SumNorm-KL setting, only Mat2Spec in the Standard-MAE setting and GATGNN in the SumNorm-KL setting correctly identify a VB gap in any of the 15 materials where DFT calculations validated its presence.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/eDOSpredictions.pdf}
\caption{\textbf{Discovered materials with VB gaps}: Of the 32 predictions of materials with a VB gap according to the results of the Mat2Spec eDOS predictions in the SumNorm-KL setting, the 15 TP predictions are shown. For each material, the eDOS calculated from DFT is shown with the Mat2Spec prediction, and the energy range of the VB gap is also shown.
MgAl and LiMg have a VB gap but the inherently low electron density of the material and the persistence of a small but finite eDOS to below -3 eV limits the interest in these materials. LiRAu$_2$ (where R=Nd, Pr, Yb) have low eDOS considering the presence of heavy elements, although the eDOS does not reach zero in the energy range of interest. Zero density in the energy range of interest is observed in the 10 other materials that fall within 2 families of candidate thermoelectrics. In the family of fluorides with formula Li$_2$AMF$_6$ (where A=Al,Sc,Ga; M=Hg,Cu), each material exhibits a gap of 3-4 eV starting near 0.5 eV below Fermi energy. The eDOS of each material in this family also exhibits a near-gap above the Fermi energy, which motivates their further study for applications such as transparent conductors. A family of oxides with formula ARNb$_2$O$_6$ (where A=Na,K; R=La,Nd,Pr) share a similar eDOS with a VB gap that is 2-3 eV wide starting near 1 eV below Fermi energy.  
}
\label{fig:eDOSpredictions}
\end{figure}


Figure \ref{fig:eDOSpredictions} shows the DFT calculation and Mat2Spec prediction for each of the 15 TP materials, revealing excellent qualitative agreement and leading to the identification of a family of fluorides Li$_2$AMF$_6$ (A=Al,Sc,Ga; M=Hg,Cu) and a family of oxides ARNb$_2$O$_6$ (A=Na,K;R=La,Nd,Pr) that are of particular interest. To assess these materials as potential thermoelectrics, we note that thermoelectric candidates should maximize the the power factor (PF), $PF=S^2\sigma$, where $S$ is the Seebeck coefficient and $\sigma$ is the electrical conductivity.  According to Mott's formula \cite{Jonson1980} a rapid variation of the eDOS would increase the Seebeck coefficient; since the presence of a band gap close to the Fermi energy makes the DOS sharply decrease, the Seebeck coefficient increases significantly and the PF presents a distinct peak in this case. 


Known high-PF thermoelectric are indeed gapped metals, such as \ce{La3Te4}, \ce{Mo3Sb7}, 
\ce{Yb14MnSb11}, and \ce{NbCoSb}, along with others,\cite{Ricci2020} and present an eDOS similar to those shown in Figure \ref{fig:eDOSpredictions}. 
This similarity allows us to use the presence of the gap combined with the fast rise of the eDOS as a means for identifying systems with large Seebeck coefficients. In particular, looking at Figure \ref{fig:eDOSpredictions}, the flourides possess both characteristics. Also, a high carrier concentration ($\sim10^{23}$ cm$^{-1}$) associated with the partially filled bands between the gap and the Fermi energy may lead to high electrical conductivity.

In contrast, for oxides, since the rise of the eDOS is less rapid than in the flourides, the Seebeck coefficient is expected to be lower. 


Another potential use of materials with large VB gaps would be as transparent conductors. As defined in Refs.\cite{Malyi2019_gapped,Zhang2015} (i) metals with large VB gaps can reduce the interband absorption in the visible range; if they also possess (ii) a high enough carrier density in the CB to provide conductivity,  and (iii) sufficiently low carrier density to limit the interband transition in the CB and plasma frequency ($\omega_p \sim \sqrt{n_e/m}$, where $m$ is an effective carrier mass), free-electron absorption will not interfere with the needed optical transparency.
Inspecting the eDOS of the flourides and oxides suggested by our models, we observe that they firmly meet criterion (i) with eDOS similar to that of the materials investigated in Ref. \cite{Malyi2019_gapped} such as Ba-Nb-O and Ca-A-O systems. 
Regarding criterion (ii), as mentioned before, the fluorides and oxides in Figure \ref{fig:eDOSpredictions} have a carrier density of approximately $10^{23}$ cm$^{-1}$, which is comparable to that of known intrinsic transparent conductors. The assessment of the plasma frequency for criterion (iii) requires an estimation of the carrier effective mass, 
which is beyond of the scope of this work.
Also, the flourides have a eDOS close to those referred to as Type-1 transparent conductors in Ref. \cite{Zhang2015} that are metals with the Fermi energy located in an intermediate band which is energetically isolated from the bands below and above it. 

While these qualitative assessments require a quantitative validation via detailed computation in future work, the fact that the eDOS compare well to established literature for thermoelectric and transparent conductor materials strongly motivates future detailed studies of these systems. Since the oxide family includes transition metals and the fluoride family includes rare earth elements, initial steps could include validation of the position of the d-electron and f-electron states, respectively, as well as 
the absence of a band gap
with techniques more accurate then standard DFT.
Future assessment of the electronic and thermal transport properties can confirm their potential as thermoelectrics, and optical properties including the dielectric function can be used to gauge their suitability as transparent conductors. The most promising materials from these computational assessments would then be prime candidates for assessing synthesizability, where the experimentally realized materials can then be analyzed to further validate the computed properties.

\section*{Discussion}


Mat2Spec brings together several machine learning techniques to exploit prior knowledge and problem structure and to compensate for the relatively small amount of training data compared to the breadth of possible materials and DOS patterns.

Specifically, Mat2Spec's feature encoder was inspired by the GATGNN, \cite{louis2020graph} CGCNN, \cite{xie2018crystal} and MEGNet \cite{chen2019graph} models in  which  GNNs are used to encode the crystal structures. The encoding of the labels and features onto a latent Gaussian mixture space to exploit underlying correlations was inspired by the DMVP,\cite{chen2018end,chen2020deep} DHN,\cite{kong2020deep} and H-CLMP\cite{kong2021materials} models in which the latent Gaussian spaces learn multiple properties' correlations. Integrating correlation learning with neural networks was initially motivated by computational sustainability  applications.\cite{gomes2021computational} The learning of a label-aware feature representation with contrastive learning was inspired by the SimSiam \cite{chen2020simple} model in which contrastive learning is used to maximize mutual information between two latent representations. Generalization of ML models is facilitated by incorporating techniques from disparate domains that share commonalities in the types of relationships that need to be harnessed by the models.

Our design of Mat2Spec focused on encoding label and feature relationships and how composition and structure give rise to intensity in different portions of a given type of spectrum. Interpretability of ML models remains a key challenge for advancing fundamental understanding, and the information-rich embeddings generated by the Mat2Spec encoder motivate future work in analyzing them and their probabilistic generator to reveal the materials features that give rise to specific spectral properties. Such studies are increasingly fruitful with improved prediction capabilities,\cite{huang_leveraging_2021} furthering the importance of Mat2Spec's improved performance compared to baseline models. The Mat2Spec prediction capabilities are further emphasized by the demonstrated identification of relevant and rare features, such as the presence of a VB gap in the eDOS, where an initial down-selection of candidate materials is particularly impactful for accelerating discovery. Our demonstration of discovering gapped metals can be extended in future work by coupling to crystal structure generators
\cite{glass_uspexevolutionary_2006,kim_generative_2020} to predict the eDOS and the phDOS of thousands of hypothetical compounds at a computational time that is orders of magnitude smaller than performing standard DFT computations.

Our focus on predicting fundamental spectral properties of materials is meant to demonstrate the generality of the Mat2Spec framework. Demonstration of the model's learning of appropriate materials embeddings for prediction of both phDOS and eDOS indicates that the model will also be suitable for spectral properties calculated from the phDOS or eDOS, such as phonon scattering and spectral absorption. We hope that the community will participate in the extension of Mat2Spec to these other domains of materials property prediction.


\section*{Methods}

\subsection*{Mat2Spec model}

\begin{algorithm}[tbp]
\caption{Mat2Spec Pseudocode}
\label{alg:algorithm}
\textbf{Input}: $\{(X_i,Y_i)\}_{i=1}^N$, $\lambda_1$, $\lambda_2$, and $\lambda_3$.
\texttt{\\}
\textbf{Output}: Feature encoder $f_e(\cdot)$, decoder $f_d(\cdot)$, and predictor $f_p(\cdot)$.
\begin{adjustwidth}{-0.55em}{}
\begin{algorithmic}[1]
\For{$(X,Y)$ in dataloader}
    \State $Z_F, Z_L = f_e(X), f_l(Y)$. \Comment{latent codes}
    \State $H_F,H_L = f_d(Z_F), f_d(Z_L)$.
    \Comment{representations}
    \State $A_F, A_L = f_{p}(H_F), f_{p}(H_L)$.
    \Comment{predictions}
    \State $B_F, B_L = f_h(H_F), f_h(H_L)$.
    \Comment{projections}
   
   
    \State Compute the negative cosine similarity loss $\mathcal{L}_D$ according to equation~(\ref{eq:cosine1}).
    \State Compute the symmetrical  loss $\mathcal{L}_C$ according to equation~(\ref{eq:bce}).
    \State Compute the KL loss $\mathcal{L}_{KL}$ according to equation~(\ref{eq:kl}).
    \State $\mathcal{L} = \lambda_1 \mathcal{L}_D + \lambda_2 \mathcal{L}_C + \lambda_3 \mathcal{L}_{KL}$.
    \Comment{combine losses}
    \State $\mathcal{L}.\text{backward}()$.
    \Comment{compute gradient}
    \State update$f_e(\cdot), f_l(\cdot), f_d(\cdot), f_p(\cdot)$, and $f_h(\cdot)$.
    \Comment{update parameters}
\EndFor\\
\Return $f_e(\cdot)$, $f_d(\cdot)$, and $f_p(\cdot)$.

\end{algorithmic}
\end{adjustwidth}
\end{algorithm}

The pseudocode of Mat2Spec is in Algorithm~\ref{alg:algorithm}. Given input features $F$ and its corresponding input labels $L$, Mat2Spec first embeds them into two latent embeddings $Z_F$ and $Z_L$ via the feature and label encoders, respectively. 

\subsubsection*{Crystal structure feature encoder and label encoder}
For an input crystal structure, we have not only element composition but also spatial information about the atoms. To leverage the spatial information, each element in the input is represented by an initial unique vector. These initial unique element embeddings capture some prior knowledge about correlations between elements.\cite{tshitoyan2019unsupervised,xie2018crystal} These initial representations are then multiplied by a $N$ by $M$ learnable weight matrix where $N=92$ is the size of the initial vector and $M=128$ is the size of the internal representations of elements used in the model.
The graph attention networks (GATs) \cite{velivckovic2017graph} is then used to update these initial internal representations by propagating contextual information about the different elements present in the material between the nodes in the graph. GATs are constructed by stacking a number of graph attention layers in which nodes are able to attend over their neighborhoods' features via the self-attention mechanism. Specifically for each atom $i$, we first get its set of neighbors $\mathcal{M}_i=\{j|d(i,j) \le \gamma\}$ where $d(i,j)$ denotes the Euclidean distance between $i$ and $j$ in angstroms and $\gamma=8 \in \mathbb{R}^+$ is a predefined threshold. Then for each $j\in \mathcal{M}_i$, the distance $d(i,j)$ is encoded as a vector
$u_{ij}=\langle f(T[0]),\cdots, f(T[m]) \rangle$ where $T\in \mathbb{N}^m$ is any predefined vector, $T[k]$ is the $k$-th element of $T$, and $f$ is the Gaussian probability density function:
\begin{ceqn}
\begin{equation}
    f(x) = \frac{1}{\sigma  \sqrt{2\pi}}\text{exp}\left(-\frac{1}{2}\left(\frac{x-d(i,j)}{\sigma}\right)^2\right),
\end{equation}
\end{ceqn}
where $\sigma=0.2$ is a predefined standard deviation parameter. 

Furthermore, we use the element composition vector as a global context vector characterizing the entire crystal graph and concatenate the global context vector with each node feature vector and its corresponding edge feature vector. Therefore, the attention coefficient between every pair of neighbor nodes is updated as
\begin{ceqn}
\begin{equation}
    e_{ij}=a(\mathbf{W}(h_i\oplus u_{ij}\oplus c),\mathbf{W}(h_j\oplus u_{ij}\oplus c)),
\end{equation}
\end{ceqn}
where $\oplus$ denotes the concatenation operation, $h_i,h_j\in \mathbb{R}^d$ are node features, $u_{ij}\in \mathbb{R}^m$ denotes an edge feature, $c\in \mathbb{R}^n$ denote element composition, $\mathbf{W}\in \mathbb{R}^{d\times (d+m+n)}$ is a weight matrix, and $a$ is single-layer feed-forward neural network. Then the attention weight $\alpha_{ij}$ for nodes $j\in \mathcal{N}_i$ is computed as
\begin{ceqn}
\begin{equation}\label{eq:2}
    \alpha_{ij}=\frac{\text{exp}(e_{ij})}{\sum_{k\in \mathcal{N}_i}\text{exp}(e_{ik})},
\end{equation}
\end{ceqn}
where $\mathcal{N}_i$ is the neighborhood of node $i$ in the graph.
At last, node $i$'s feature $h_i'$ is updated as
\begin{ceqn}
\begin{equation}\label{eq:3}
    h_i'=f(\sum_{j\in\mathcal{N}_i}\alpha_{ij}\mathbf{W}h_j),
\end{equation}
\end{ceqn}
where $f$ is some nonlinear function such as the sigmoid. Multi-head attention \cite{vaswani2017attention} is also used where $K$ independent attention mechanisms are executed and their feature vectors are averaged as
\begin{ceqn}
\begin{equation}\label{eq:4}
    h_i'=f(\frac{1}{K}\sum_{k=1}^{K}\sum_{j\in\mathcal{N}_i}\alpha_{ij}^k\mathbf{W}^k h_j).
\end{equation}
\end{ceqn}

The final feature embedding vectors of the structure feature encoders are obtained by applying global mean pooling operation on node features.

In this work, we use an MLP as the label encoder.


\subsubsection*{Probabilistic feature and label embeddings}

In this work, both features and labels are embedded into a latent Gaussian mixture space and the feature and label embeddings are aligned to exploit feature and label correlations. Therefore, latent variables $Z_F$ and $Z_L$ are assumed to follow some mixture of multivariate Gaussian distributions:
    \begin{ceqn}
    \begin{equation}
    \begin{aligned}
        Z_F&  \thicksim\sum_{i=1}^K \pi_i \mathcal{N}_i(Z_F|\mu_i,diag(\sigma^2_i)) \text{ and} \\
        Z_L& \thicksim\sum_{j=1}^{K'} \pi_j \mathcal{N}_j'(Z_L|\mu_j',diag({\sigma_j'}^2)),
    \end{aligned}
    \end{equation}
    \end{ceqn}
    where $K$ and $K'$ are the presumed number of clusters in the latent space and $\pi_i$ and $\pi_j'$ are the learned prior probabilities of related clusters. In this work, $K$ is 10 and $K'$ equals the dimension of label vectors. 
    Since both $Z_F$ and $Z_L$ are assumed to follow mixture of multivariate Gaussian distributions, we use Kullback–Leibler (KL) divergence to align them, which is not analytically tractable. However, we can optimize the KL divergence between two Gaussian mixtures by optimizing the following upper bound $\mathcal{L}_{KL}$ \cite{hershey2007approximating}:
    \begin{ceqn}
    \begin{equation}\label{eq:kl}
        \text{KL}(Z_F||Z_L) \le \mathcal{L}_{KL}=\sum_{i,j} \pi_i\pi_j' \text{KL}(\mathcal{N}_i, \mathcal{N}_j').
    \end{equation}
    \end{ceqn}

\iffalse
\begin{algorithm}[tbp]
\caption{Mat2Spec}
\label{alg:algorithm}
\textbf{Input}: $\{(X_i,Y_i)\}_{i=1}^N$, $\lambda_1$, $\lambda_2$, and $\lambda_3$.
\texttt{\\}
\textbf{Output}: feature encoder $f_e(\cdot)$, translator $f_d(\cdot)$, and prediction head $f_p(\cdot)$.
\begin{adjustwidth}{-0.55em}{}
\begin{algorithmic}[1]
\For{$(X,Y)$ in dataloader}
    \State $Z_X, Z_Y = f_e(X), f_l(Y)$ \Comment{latent codes}
    \State $H_X,H_Y = f_d(Z_X), f_d(Z_Y)$
    \Comment{representations}
    \State $A_X,A_Y = f_{p}(H_X), f_{p}(H_Y)$
    \Comment{predictions}
    \State $B_X,B_Y = f_h(H_X), f_h(H_Y)$
    \Comment{projections}
   
   
    \State Compute the negative cosine similarity loss $\mathcal{L}_D$ according to equation~(\ref{eq:cosine1}).
    \State Compute the symmetrical loss $\mathcal{L}_C$ according to equation~(\ref{eq:bce}).
    \State Compute the KL loss $\mathcal{L}_{KL}$ according to equation~(\ref{eq:kl}).
    \State $\mathcal{L} = \lambda_1 \mathcal{L}_D + \lambda_2 \mathcal{L}_C + \lambda_3 \mathcal{L}_{KL}$
    \Comment{combine losses}
    \State $\mathcal{L}.\text{backward}()$
    \Comment{compute gradient}
    \State update$(f_e(\cdot), f_l(\cdot), f_d(\cdot), f_p(\cdot), f_h(\cdot))$
    \Comment{update parameters}
\EndFor\\
\Return $f_e(\cdot)$, $f_d(\cdot)$, and $f_p(\cdot)$.

\end{algorithmic}
\end{adjustwidth}
\end{algorithm}
\fi

\subsubsection*{Supervised contrastive learning}
A shared translator $f_d(\cdot)$ extracts representation vectors $H_F$ and $H_L$ from embeddings $Z_F$ and $Z_L$, respectively. In order to further facilitate learning a label-aware feature representation $H_F$, we adopt supervised contrastive learning: 

(i) Maximizing agreement between feature and label representations:
a projector, denoted as $f_h$, transforms the feature and label representations and matches them with each other using a contrastive loss, where the projector is an MLP with one hidden layer. Concretely, let $B_F=f_h(H_F)$ and $B_L=f_h(H_L)$. We train the translator and projector and use the negative cosine similarity loss as the contrastive loss to maximize agreement between feature and label representations:
\begin{ceqn}
\begin{equation}\label{eq:cosine1}
  \mathcal{L}_D = - \frac{B_F}{||B_F||_2} \times \frac{B_L}{||B_L||_2},
\end{equation}
\end{ceqn}
where $||\cdot||$ is the Euclidean norm. We follow the empirical evidence that the contrastive loss is best defined on the projection space rather than the representation space.\cite{chen2020simple}


(ii) Label prediction: a predictor, denoted as $f_p$, transforms the feature and label representations and matches them to the label $L$. Let $A_F=f_p(Z_F)$ and $A_L=f_p(Z_L)$. We define a symmetrical loss to learn a multi-property regressor:
\begin{ceqn}
\begin{equation}\label{eq:bce}
 \mathcal{L}_C = \frac{1}{2} \text{LOSS}(A_F,L) + \frac{1}{2} \text{LOSS}(A_L,L),   
\end{equation}
\end{ceqn}
where $\text{LOSS}(\cdot)$ denotes the loss function for the given setting: KL, MSE, MAE, or WD. 

The final loss function used in Mat2Spec is a combination of $\mathcal{L}_D$, $\mathcal{L}_C$, and $\mathcal{L}_{KL}$, which is defined as follows:
\begin{ceqn}
\begin{equation}
    \mathcal{L} = \lambda_1 \mathcal{L}_D + \lambda_2 \mathcal{L}_C + \lambda_3 \mathcal{L}_{KL}, 
\end{equation}
\end{ceqn}
where $\lambda_1=1$, $\lambda_2=1$, and $\lambda_3=1.1$ control the weights of the three loss terms. 
\subsubsection*{Implementation and hyperparameters}
The crystal feature encoder has two four-head attention layers. Each layer of the crystal feature encoder has 103 nodes where each node corresponds to an element and is represented by a 128 dimensional vector. The label encoder and the translator are all 3-layer MLPs, and the 3 layers have 256, 512, and 128 neurons, respectively. The projector and predictor are both 2-layer MLPs where each has 1 hidden layer, and each hidden layer has 256 neurons. The number of multivariate Gaussians $K$ produced by the feature encoder is $10$ and the dimension $D$ of each multivariate Gaussian is 128. The dimensions of representation and projection vectors are 128 and 1024, respectively. 

We used grid search for hyperparameter optimization, where we set the batch size and the number of training epochs to 128 and 200, respectively, for all experiments. The learning rate was chosen from $0.0005$ to $0.01$ in steps of $0.0005$, dropout ratio from $[0.3, 0.5, 0.7]$, and weight decay ratio from $[0, 0.01, 0.001, 0.0001]$. In this work, we use the same set of hyperparameters for all experiments where learning rate, dropout ratio, and weight decay ratio are set to be $0.001$, $0.5$, and $0.01$, respectively. Our model was implemented with the Pytorch deep learning framework and the whole model was trained with the AdamW optimizer using a 0.01 weight decay ratio in an end-to-end fashion in a machine with NVIDIA RTX 2080 10GB GPUs.


\subsection*{Data generation}
The phDOS dataset was replicated from Ref. \cite{Tess2021} to facilitate direct comparison between the models. The dataset is randomly divided in 1220, 152, and 152 samples for training, validation, and test sets, respectively.
We remind here that the phDOS from this dataset were cut at a frequency of 1000 cm$^{-1}$, smoothed via a Savitzky-Golay filter, and interpolated on a common 51-frequencies grid. 

The calculation of $C_v$ and $\bar{\omega}$ similarly proceeded in the same way as this prior work by using the correspondent functions present in pymatgen \cite{pymatgen}.

The eDOS dataset was acquired from the Materials Project (version 2021-03-22), where the eDOS values are computed according to a DFT recipe reported in the MP documentation \cite{MPwiki}. The dataset contains eDOS and crystal structures of non-magnetic materials, both metallic and semiconductors/insulators, with an energy grid of 2001 points (VASP NEDOS equals to 2001). This dataset has been randomly divided in training, validation, and test sets, containing 80\%,10\%,10\% of the samples, respectively. Another dataset which contains only the structures of materials with no available eDOS in the MP was acquired for prediction. Only materials which are flagged as non-magnetic by MP were considered. These two datasets contains about 30,000 and 24,000 entries, respectively.

The energy range of the eDOS can be very large and varies among the MP entries. To consider a consistent energy grid the eDOS from MP was resampled. 
The energy range taken into account is the first 4 eV below the valence bands maximum and above the conduction band minimum, a range chosen based on common uses of eDOS for studying related material properties.
Also, for reasons mentioned above, we removed the band gap from the eDOS, when present, making the conduction band start just after the valence band. Upon this cutting step, the energy range was divided into 128 bins and the average of the eDOS values in each bin was taken. 

\subsection*{Calculation of eDOS}

The electronic DOS was calculated for materials with no available eDOS in the MP that were predicted to have a band gap in the valence band. The 32 Materials Project entries are as follows: mp-1236246, mp-1236485, mp-1222301, mp-1227246, mp-1227245, mp-1217599, mp-1187874, mp-1222008, mp-1185617, mp-1206725, mp-1184817, mp-1185314, mp-1220591, mp-1184222, mp-1180477, mp-1147636, mp-1094477, mp-1519062, mp-1518752, mp-1518664, mp-1222270, mp-1185435, mp-1185407, mp-1185397, mp-1111899, mp-1111898, mp-1111559, mp-1094298, mp-1039010, mp-1522162, mp-1519126, mp-1520737.
The DFT computations were performed following the MP recipe by means of the atomate package \cite{atomate} so that the eDOS are computed in the same way as those used for model training.


The criteria outlined in the eDOS VB gap use case were applied to the computed DFT eDOS to verify the predictions. We note that the same criteria with different parameters can be applied to find gaps in the conduction band.


\section*{Introduction}

The Introduction section, of referenced text\cite{Figueredo:2009dg} expands on the background of the work (some overlap with the Abstract is acceptable). The introduction should not include subheadings.

\section*{Results}

Up to three levels of \textbf{subheading} are permitted. Subheadings should not be numbered.

\subsection*{Subsection}

Example text under a subsection. Bulleted lists may be used where appropriate, e.g.

\begin{itemize}
\item First item
\item Second item
\end{itemize}

\subsubsection*{Third-level section}
 
Topical subheadings are allowed.

\section*{Discussion}

The Discussion should be succinct and must not contain subheadings.

\section*{Methods}

Topical subheadings are allowed. Authors must ensure that their Methods section includes adequate experimental and characterization data necessary for others in the field to reproduce their work.

