\section{Introduction}
\label{sec:intro}
The visual recognition of materials is essential to understanding nearly every aspect of the world (\Cref{fig:accExamples}). Cooking, chemical research, inspecting materials in construction, agriculture, and industry, and everyday object handling all depend on the ability to identify materials and their state \cite{Sun2022using, Zhang2021computer, patricio2018computer
, Eppel2020computer, eversberg2021generating}. This can involve anything from evaluating whether the ground is wet, a fruit is rotten, water is clean, or metal is rusted to distinguishing between different types of rocks or fabrics. There are several major challenges that make this problem difficult for computer vision methods. First, there is an almost infinite amount of material and textures, and each can look very different under different illumination conditions (unlike object shapes, which are invariant to illumination). The second problem is that transitions between materials and textures tend to be gradual and have a continuous intermediate state, as in the case of cooking food, which makes it hard to use discrete categories to describe the material. Another challenge is that liquids and materials in environments like laboratories, hospitals, and kitchens are usually handled inside transparent vessels that distort the view of the materials. Previous studies on image recognition for materials have mostly focused on distinguishing between a limited set of material classes, like metal, plastic, and wood, or physical phases, like liquids and solids. The limitation of these approaches is that they can only recognize the classes they were trained on \cite{dana1999reflectance, bell2015material, hayman2004significance, sharan2009material,  xiao2018unified, Eppel2020computer, bell2013opensurfaces}. Adding new classes or distinguishing between subclasses requires one to collect a new dataset and retrain the neural networks. Another problem with these approaches is that since they are based on pictures collected in a specific setting, they are usually limited to scenarios in which the given materials appear on specific objects. For example, windows will always be made out of glass, while trees will always be made out of wood. Hence, they are likely to identify materials by associating them with specific objects and are less likely to recognize, for example, a duck made of wood or a tree made of glass\cite{barragan2022adversarially,hendrycks2021many}. In recent years, contrastive learning methods such as Clip, which are based on learning the similarity between images and texts, have been proven to be very powerful one-shot learners that can identify new examples and classes using a single example or a few images  \cite{radford2021learning, schuhmann2022laion}. The main advantage of these approaches is that when they are trained on enough data, they are not limited to a specific set of classes and conditions and can work with classes not seen during training. In addition, such models have proven to be very powerful backbones for solving other problems, like image generation\cite{ramesh2022hierarchical}.
The main limitation of these models is that they require a massive amount of data and computing resources to be trained properly. In this work, we solve this problem by generating a huge material similarity synthetic dataset that utilizes large-scale CGI artist repositories\cite{ambientcg, cgbookcase} used for animation and computer games (\Cref{fig:genExamples}). We use highly diverse physics-based rendering (PBR) materials repositories to simulate realistic materials\cite{Pharr2016physically}. We overlay the PBR materials on 3D objects taken from the ShapeNet dataset with tens of thousands of different objects \cite{chang2015shapenet}(\Cref{fig:genSteps}); these objects are then put into scenes with random illumination conditions and backgrounds by utilizing PolyHaven repository for HDRI images\cite{polyhaven}. The HDRI image is wrapped around the scene and provides a realistic 360-degree background and illumination to the scene (\Cref{fig:genExamples}). 
\begin{figure}[t]
  \centering
   \includegraphics[width=0.9\linewidth]{images/Figure3.jpg}

   \caption{Dataset creation. Random materials are created using the Blender BSDF node or PBR materials downloaded from large-scale artist repositories (AmbientCG, CGBookCases, 3D Textures). The material is UV mapped on the surface of a random object loaded from the ShapeNet dataset. Random background and illumination conditions are loaded from HDRI Haven repositories, ground plane and background objects are added, and the scene is rendered.}
   \label{fig:genSteps}
\end{figure}
Combining these repositories allows us to generate a large-scale, highly diverse dataset (\Cref{fig:genExamples,fig:genSteps}). The large number of materials forces the network to generalize instead of identifying only specific classes. The fact that every material can be used on any object in any environment means that the net has to identify the material everywhere and prevents the net from associating the material with specific objects or environments. Gradual transformations between materials in the dataset allow the net to detect smooth transitions between materials. Additionally, rendering some of the materials inside transparent containers allows the network to learn to recognize materials stored inside glass vessels. In addition to the synthetic dataset, we also supply a new real-world image benchmark for few-shot material recognition (\Cref{fig:accExamples}). The benchmark contains a large number of examples of material states and transitions, ranging from food in different states of cooking, various powders, chemical reactions, metals and rust, wet and dry sidewalks, construction materials, water and beverages in various states (frozen/turbid), fruits in rotten and fresh states, and even different types of candy, pasta, and cereals. The goal of the net is to identify each of these states using one or a few examples.  A second benchmark focus on covering random objects with random materials, making the object and material completely uncorrelated. We use the benchmark to test the network trained on the MatSim dataset, as well as to test a pretrained Clip network. We show that a net trained on the MatSim synthetic data significantly outperformed the Clip model and gave good results despite never being trained on any of the material classes in the benchmark (or real world images in general). 

\subsection{Contribution}

To conclude, the main contributions of our work are as follows:
\begin{itemize}
    \item Demonstrating the first method, benchmark, and dataset for one-shot material recognition that enable the recognition of unfamiliar materials using one or a few examples, as well as identifying material gradual transitions and subclasses.
    \item Showing how large-scale CGI artist repositories can be combined to create highly diverse synthetic training data in a way that can be used for many other image recognition problems.
    \item Proving that a net trained only on simulated data with a limited amount of resources (a single GPU) can outperform large-scale models like Clip on general real-world one-shot learning tasks (Clip H14 was trained on billions of real images and hundreds of GPUs).
     
   
\end{itemize}

\section{Related work}

\subsection{One-shot contrastive learning}
One-shot contrastive learning methods like Clip have been proven to be very powerful for image recognition tasks and have achieved state-of-the-art results on major benchmarks without training on them \cite{radford2021learning, schuhmann2022laion}. In addition, these models are far more robust than nets trained on specific datasets\cite{radford2021learning,hendrycks2021many}. These networks work by predicting a descriptor vector for an image and using the distance between vectors of different images as a similarity metric. This approach allows such nets to recognize new classes using only a single example (one shot). The main limitation of this approach is the need for a very large amount of data. For example, Laion Clip H14 \cite{schuhmann2022laion} was trained on two billion images and their text labels. An alternative approach like SimClr \cite{chen2020simple} used the self-similarity between images and their augmented versions and utilized the same type of descriptor-similarity approach, but so far, it has proven less effective than fully supervised networks or Clip.

\subsection{Computer vision for material recognition}
Computer vision for material recognition has mainly focused on three aspects: 1) classifying material classes, such as wood and metal or liquids and solids; 2) finding visual similarities between materials; and 3) predicting material properties such as the BRDF, reflectance, glossiness, and turbidity \cite{chadwick2015perception, weiss2020correspondence, meka2018lime, vidaurre2019brdf,weiss2020correspondence,lagunas2019similarity, schwartz2019recognizing}.

\subsection{Material classification datasets}
There are a large number of material classification tasks and related datasets. Datasets for general material classes (wood, metal, glass) include CURet\cite{dana1999reflectance}, KTH-TIPS \cite{ hayman2004significance}, FMD\cite{sharan2009material}, MINC\cite{bell2015material}, and OpenSurfaces\cite{bell2015material, bell2013opensurfaces}; all of these datasets use real-world images. Other datasets focus on more specific properties, such as material phases (liquids, solids, powders, foam)\cite{Eppel2020computer}, and more specific topics, such as construction materials, soil type, crystallization, turbidity, etc\cite{mungofa2018chemical, Sun2022using, Zhang2021computer, patricio2018computer,  eversberg2021generating}.

\subsection{Material similarity prediction}
Work on identifying the visual similarity between materials has been done using both synthetic datasets and unsupervised learning\cite{schwartz2019recognizing, perroni2022material, akiva2022self}. Schwartz and Nishino, \cite{schwartz2019recognizing} used 9000 generated images with uniform materials to train a network that measures the similarity of the materials. The visual similarity between the simulated materials was determined by human annotators. Perroni et al.\cite{perroni2022material} used transfer learning to determine the visual similarity by using the inner layer of a network trained on classification tasks with eight classes of materials. Both works are limited to either a small set of classes or uniform materials, and neither work tried to solve the general problem of being able to recognize unseen materials and textures in a different environment or identifying gradual transitions between materials. 

\subsection{Synthetic Datasets}
Simulated data have a significant advantage for material datasets because of the relative accuracy and simplicity with which materials can be simulated and rendered using software like Blender\cite{blender}. Simulated materials datasets mostly focus on homogenous materials, which enable the prediction of numeric properties like surface reflectance (BRDF, BSDF)\cite{lagunas2019similarity, bartell1981theory, asmail1991bidirectional, meka2018lime, vidaurre2019brdf, eppel2022predicting}. Textured PBR materials have also been used mainly to generate datasets for texture map extraction and classification \cite{Pharr2016physically, lichy2021shape, perroni2022material}. One of the main limitations of the current datasets is the use of a limited number of objects, material classes, and environments, which makes networks trained on these datasets limited in their scope. While large CGI repositories have existed for a while, they remained almost unutilized by the computer vision community. 


\section{The MatSim Synthetic dataset generation}

The goal of the MatSim dataset is to train a computer vision system capable of recognizing any material and texture on any surface in any reasonable environment and illumination conditions. The visual appearance of a material depends on its physical properties but also on the object's shape, environment, and illumination. If, during training, any of these properties are restricted in some way (for example, only indoor scenes are used) or correlated with other properties (for example, wood materials only appear on trees), the net is unlikely to achieve true generalization for material recognition \cite{hendrycks2021many, barragan2022adversarially}.   
\begin{figure}[t]
  \centering
   \includegraphics[width=0.9\linewidth]{images/Figure4.jpg}
   \caption{Dataset structure. The dataset is composed of sets. Each set involves two materials and six scenes with a gradual transition between the two materials. Each image column corresponds to a different mixing ratio (R) of the two materials. The ratio of the mixture of the two materials is given in the top column. All images in the same column involve the same material mixture on different objects and in different environments. Each of the six scenes involves one main object. The material on this object gradually transitions from one material to another. The mask of the object is given in the right column. For scenes 1–2, the background remains exactly the same for all images in the scene. For scenes 3–4, the background HDRI is randomly rotated between images, leading to small changes in illumination. For scenes 5–6, the background HDRI is completely replaced between images, leading to large changes in illumination.}
   \label{fig:dataSets}
\end{figure}

\subsection{General dataset structure} \label{sec:genDataset}
The dataset is divided into sets; each set contains two random materials and six scenes involving a gradual transition between these two materials (\Cref{fig:dataSets}). The objects, backgrounds, and environment are randomly selected for each scene separately (\Cref{fig:genSteps}). Each scene involves one static main object and a static camera, which are both positioned randomly, and both remain unchanged for the scene (Appendix 10.1). The object's material is gradually changed between images in the scene (\Cref{fig:dataSets}). For each scene, we render five images with different mixtures of the two materials. The mixture ratios (R) of the two materials are 0\%, 25\%, 50\%, 75\%, and 100\%. 0\% means that the object is only made of material A, while 100\% means the object is made completely of material B, and 50\% indicates an equal mixture of the two materials (\Cref{sec:mixtures}).
For scenes 1–2, all aspects of the scene remain the same. Only the object material is changed gradually between images. In addition, the translation and rotation of the texture UV mapping to the object were randomly changed between each render (Appendix 10.2). For scenes 3–4, the procedure is the same, except that the HDRI background is randomly rotated between images (Appendix 10.3), leading to a small variation in illumination. For scenes 5–6, the HDRI background for each image is randomly replaced, leading to a large variation in illumination. This provides almost any possible variation of each material's appearance. Since all the scenes in a set are composed of the same two materials, it is possible to compare the appearance of the materials between two scenes with different objects and backgrounds, and light. The gradual transition allows the network to learn to distinguish between highly similar materials. Some scenes contain several random objects and a ground plane for variability (Appendix 10.4). Since a scene can contain many background objects, for each scene, we provide the mask (region) of the object on which the material is used (\Cref{fig:dataSets}, right). If the material is uniform (BSDF), the values of the material properties (color, transparency, etc.) are given in the dataset. Altogether, 30k sets, with about 1 million images, were rendered. For more details see the appendix.
\begin{figure}[t]
  \centering
   \includegraphics[width=0.9\linewidth]{images/Figure5.jpg}
   \caption{A material's visual appearance is controlled by several main properties (color, transparency, etc.). For uniform materials (BSDF), each property has a single value across the surface. Textured materials are represented by texture maps for each property. These maps are wrapped around the object (UV mapping) and provide properties for each point on the object's surface.}
   \label{fig:matCreation}
\end{figure}

\subsection{BSDF and PBR materials} \label{sec:BSDFPBR}
The visual appearance of materials is controlled by several physical properties, including color, transparency, roughness, etc. All of these properties can be assigned values and controlled in Blender\cite{blender} by a single system (BSDF node, \Cref{fig:matCreation}a)\cite{bartell1981theory, asmail1991bidirectional}. However, in most real materials, these properties are non-uniformly distributed and cover the object's surface in some unique patterns (textures). Materials are often assigned in the CGI community using physics-based rendering or PBR (\Cref{fig:matCreation}b). PBR materials use 2D texture maps to control the distribution of each visual property. When a material is assigned to an object, the texture maps are wrapped around the object and assign the material properties to each point of the object's surface (\Cref{fig:matCreation}b)\cite{Pharr2016physically}. We downloaded over 4400 textures from open repositories. To further increase the diversity of the materials, we randomly combined texture maps from different materials (by averaging and mixing maps). To generate uniform BSDF materials (\Cref{fig:matCreation}a), we simply select a random value for each material property and set it uniformly on all of the object surfaces using the Blender 3.1 BSDF node. Transparent materials are unique in the sense that their appearance depends on the object's volume (and is not limited to the surface). For these, we use the Blender Volume Glossy node. 

\subsection{Representing mixtures and gradual material transformation} \label{sec:mixtures}
All material properties are continuous values, and two or more texture maps can be combined through the pixel-wise addition (weighted sum). This makes a gradual transition between materials simple to generate. First, the texture maps of the two materials are combined using pixel-wise weighted addition to create a new set of maps. The new maps are then wrapped around the object to create the transition state. The ratios used are 0/1, 0.25/0.75, 0.5/0.5, 0.75/0.25, and 1/0, where 0.25/0.75 indicates the weighted sum of 25\% material A and 75\% material B (\Cref{fig:dataSets}). 
\begin{figure}[t]
  \centering
   \includegraphics[width=1.0\linewidth]{images/Figure6.jpg}
   \caption{Procedurally generating material inside transparent containers. A random 2D curve is generated by combining random polynomial and trigonometric functions. The curve is used as a profile for the symmetric 3D object. The object is assigned random transparent materials and content objects. The content object is assigned a material.}
   \label{fig:transparent}
\end{figure}


\subsection{Materials in transparent vessels} \label{sec:transparentV}
Liquids and other materials in kitchens, hospitals, and labs are usually handled inside transparent containers (glassware, flasks, tubes, etc.). To support these applications, we generated scenes in which we put the material inside a transparent container. The vessel object was procedurally generated. The curvature of the transparent vessel was generated by creating a random 2D curve (by randomly combining linear, polynomial, and trigonometric functions; \Cref{fig:transparent}). This curve was used as the profile of a symmetric vessel by creating a cylindrical (or other symmetrical) shape with the curve as the vertical profile (\Cref{fig:transparent}). In some cases, the shape was also randomly stretched to create more variability. The vessel object was assigned a random transparent material and a random thickness. The content of the vessel was either a random object loaded from ShapeNet or a mesh that filled the bottom part of the vessel (similar to static liquid or powder). As before, the content was assigned a random material. Otherwise, the dataset creation was the same as in \Cref{sec:genDataset}. Examples can be seen in \Cref{fig:dataSets_ves} (Appendix).
\begin{figure}[t]
  \centering
   \includegraphics[width=1.0\linewidth]{images/Figure7b.jpg}
   \caption{Training and loss function. The loss function is based on the cosine similarity with cross-entropy loss. For every three images in the batch, one image is defined as an anchor ($A$). A second image, specifically the image that has material more similar to the anchor (section 4.2), is defined as positive ($P$), and the third image is defined as negative ($N$). All images and the material masks are passed through the neural network to produce descriptors ($D_a$, $D_n$, $D_p$). The cosine similarity is calculated between the descriptor of the anchor and the positive and negative examples. These cosine similarities are the input for the softmax function, which returns the probability of a match between the anchor and the positive image ($P_{p}$). If this probability is below the threshold defined as $0.5+(sim(A, P)-sim(A, N))*0.25$, we calculate the cross-entropy loss ($Loss = (-\log(P_p)$); otherwise, the loss is set to zero. $sim(A, P$) is the material similarity between the anchor ($A$) material and the positive material $P$ (\Cref{sec:materialSim}).}
   \label{fig:training}
\end{figure}


\section{Training and architecture} \label{sec:TrainingAndArch}

We examined several standard image encoder architectures and found that ConvNeXt\cite{liu2022convnet} gave the best results. Two modifications were made in ConvNeXt: First, the input of the network was changed from a standard three-channel (RGB) image to an image plus the mask of the region containing the material. This was done by simply adding the ROI mask as another layer to the RGB image and changing the first layer of the network so that it could receive a four-layer (R, G, B, mask) image instead of a three-layer (RGB) image. In addition, the final linear layer of ConvNeXt was changed to produce a 512-value vector (instead of 1000 ImageNet classes). This vector was then normalized using L2 normalization and used as the output descriptor. The training itself was done using the AdamW optimizer for ten epochs (200000 steps with a batch size of 12) using a single RTX3090. This took around three days per training session. We used the ConvNeXt-base model pretrained on ImageNet.

\subsection{General loss function}
The loss function is described in \Cref{fig:training}.  Similar to recent works, we use the cosine similarity combined with the cross-entropy loss for training \cite{chen2020simple}. An important aspect of our dataset is that the levels of similarity between materials are continuous and can have any value between zero and one (depending on the mixture; \Cref{fig:dataSets}). This was accounted for by using a semi-hard loss with a threshold depending on the similarity level. We found that a semi-hard loss, in which only loss terms that are above some threshold are used, works better than a hard loss (in which all loss terms are used). The threshold, in this case, was controlled by the difference in the similarities between the anchor material and the positive and negative examples.

\subsection{Material similarity} \label{sec:materialSim}
We define the real similarity of the two materials as the difference between their mixture ratios (R, \Cref{fig:dataSets}). As described in \Cref{sec:mixtures}, each material in a set is a linear mixture between two materials $A$ and $B$. We define this ratio as $R(0 < R < 1)$. The similarity between the materials in the two images ($i_1$ and $i_2$) is defined as: \begin{equation*}sim(i_1,i_2)=1-|R(i_1)-R(i_2)|\end{equation*} Where $R(i_1)$ is the mixture ratio in image $i_1$ (\Cref{fig:dataSets}). Assuming both images are from the same set.

\subsection{Predicted material similarity} \label{sec:materialPrediction}
The predicted material similarity between two images, $A$ and $N$, is calculated by first passing the images and their corresponding masks through the neural network and predicting the descriptors $D_a$ and $D_n$, which are then normalized using L2 normalization. The predicted similarity is the cosine similarity between the two descriptors ($S_{an}$, \Cref{fig:training}). 

\subsection{Loss calculation}
The training was performed as shown in \Cref{fig:training}: For each batch, we randomly selected a single set (\Cref{sec:genDataset}) and sampled 12 random images from this set. The loss for every three images in the batch was calculated separately using the procedure shown in \Cref{fig:training}. One of the three images was chosen as an anchor (A). The similarity between the material in the anchor image and the other two images was calculated as described in \Cref{sec:materialSim}. The image that was more similar to the anchor was defined as positive (P), and the other image was defined as negative (N); hence, $sim(A, P) > sim(A, N)$.
If the similarities of the anchor to the two images were equal $(sim(A, P) = sim(A, N))$, the loss for this triplet was set to zero. Next, each of the three images and their material masks were passed through the neural net to predict the material descriptors (\Cref{sec:TrainingAndArch}, \Cref{fig:training}). The cosine similarities between the descriptors of the anchor and the negative and positive images were calculated as described in \Cref{sec:materialPrediction}. These similarities were then passed to a softmax function to calculate the probability of a match between the anchor and the positive image:  \begin{equation}
    P_p = \frac{e^{S_{ap}/t}} {e^{S_{ap}/t} + e^{S_{an}/t}}.
\end{equation}
$S_{ap}$ and $S_{an}$ are the cosine similarity between the anchor ($A$) and the negative($N$) and positive($P$) descriptors, respectively, $P_p$ is the probability for a match between the anchor and the positive image, and $t$ is a constant (0.2 in our case). We then calculate the semi-hard loss using the following condition:
If $P_p > 0.5+(sim(A, P)-sim(A, N))*0.25$, the loss is set to zero; otherwise, we calculate the standard cross-entropy loss $(Loss = -log(P_p))$. 


\section{Testing Clip models}
The Clip neural network has achieved state-of-the-art (SOTA) results on a large number of image recognition benchmarks and is considered the top general one-shot net to date \cite{schuhmann2022laion,radford2021learning}. Similar to our model, Clip predicts image descriptors as a vector, which can then be used to find similarities between the image and other images or texts. Unlike our model, Clip H14 was trained by comparing 2 billion real photos to their corresponding text captions. There are two limitations to using Clip for this task. The first is that the Clip descriptor is not focused on materials and might contain information regarding the object and environment. Second, Clip receives just an image without an attention mask. Hence, it is harder to point to a specific region of the image where the target material is. To solve this issue, we tried a few methods: The first involves cropping the region of the image around the target material and using it as the input. The second involves masking the image region outside the material (by setting it to black), and the third involves combining masking and cropping. All of these approaches gave a significant improvement compared to using the image as is. Cropping without masking gave the best results. We tested all the pretrained Clip versions available and found that ViT-B32 gave the best results for open A.I models (\Cref{tab:results}), but the recently published Open Clip H/14 significantly outperforms all other Clip Models. Nonetheless, MatSim net still outperformed all Clip models on all benchmarks.
\section{Benchmark creation}
Perhaps the most important part of dealing with the task of general material recognition is defining a proper benchmark that will cover the main aspects of this challenge. We define three main capabilities that we want to evaluate:
1) The ability of the network to visually identify unfamiliar materials in new environments using one or a few examples with no restrictions on the material or environment setting. 
2) The ability to identify transitions between material states (e.g., wet/dry, rotten/fresh) and fine-grained subcategories of materials (e.g., types of pasta).
3) The ability to identify materials regardless of the object they appear on (e.g., a cow made out of wood)\cite{barragan2022adversarially,hendrycks2021many}.
 We are not aware of a benchmark that covers any of these three tasks. Therefore, we created two benchmarks by manually taking pictures of materials in a variety of states.

\subsection{Set 1: Material transition states and subclasses}
The first test set is designed to test the hypothesis that the network is able to recognize the similarity between images of the same material, even when they are presented in transparent containers and different surroundings, while distinguishing between subcategories or different states of the same material. To do this, we collected a large number of images of materials from a wide range of fields (\Cref{fig:accExamples}); for example, we collected images of eggs in raw, cooked, and burned states and sidewalks in wet and dry states (\Cref{fig:accExamples}). For each state, we collected at least two images. We divided this test set into superclasses (e.g., eggs) and divided each superclass into subclasses (e.g., cooked/raw/burned; \Cref{fig:accExamples}). We tried to make different examples of the same material subclass appear in different environments as much as possible, making the recognition of the material based on the environment type less likely. In addition, for each image, we created a mask of the material region in the image. Note, however, that in this dataset, there will always be some correlation between materials and objects, i.e., liquids appear in glassware, and rotten textures appear on fruit. This set contained 416 images divided into 116 material types.

\subsection{Set 2: Uncorrelated materials and objects}
The second test set is designed to test the hypothesis that the network is able to recognize the material regardless of the object on which it appears. Materials in the real world are strongly correlated to objects (trees made out of wood). Creating this set meant that we needed to generate our own objects. To achieve this, we collected a number of random objects and materials. The materials were ribbons and sheets that could be wrapped around the object or granular and fiber materials that could be scattered on the object's surface. These materials were used to cover the surfaces of objects using glue for attachment. When creating this test set, we made sure that each material did not appear on the same object or in the same environment in more than one image, forcing the net to use only the material for recognition. This set contains 86 images in 16 material types.


\section{Evaluation methods}
For the evaluation of the network, we consider the standard Top-1 accuracy metric: Given an image of a material, we test the ability of the network to identify another image of the same material from a group of images. An example is correct if the highest cosine similarity calculated by the network is  between the given image to another image of the same material subclass. We calculate two accuracy metrics. The first evaluation method measures the ability of the network to find another example of the same material from the set of all images of material the superclass (\Cref{fig:accExamples}). For example, the task could be to identify an image with spotted bananas from a set of images of bananas in various states (\Cref{fig:accExamples}, \Cref{tab:results} (subclass)). In the second test case, we still expect the network to find another example of the same material. However, we now present it with all the other images in the test set, including those of different superclasses. For example, the image of a spotted banana is compared to all images, including those of rust, rocks, cheese, etc. Note that in this case, it is much easier to use the object type to distinguish between materials (banana vs. not a banana), which gives the Clip network an advantage. For the second test set we used only the second approach. The results given in \Cref{tab:results}, \Cref{fig:accExamples}, are the average accuracy per class (All classes given equal weights).

\section{Results}

\begin{table}
  \centering
  \begin{tabular}{@{}lccc@{}}
    \toprule
    Method & Set1 Subclass & Set1 All & Set2 \\
    \midrule
    Random        & 0.30 & 0.006 & 0.07 \\
    \midrule
    CLIP B32     & 0.51 & 0.32 & 0.44 \\
    CLIP B32+C   & 0.56 & 0.38 & 0.49 \\
    CLIP B32+M   & 0.56 & 0.28 & 0.37 \\
    CLIP B32+C+M & 0.56 & 0.35 & 0.56 \\
    \midrule
    Open CLIP H14     & 0.55 & 0.44 & 0.47 \\
    Open CLIP H14+C   & 0.67 & 0.52 & 0.77 \\
    Open CLIP H14+M   & 0.59 & 0.40 & 0.53 \\
    Open CLIP H14+C+M & 0.66 & 0.52 & 0.67 \\
    \midrule
    MatSim     & 0.71 & 0.56 & 0.73 \\
    MatSim+C   & 0.77 & 0.56 & 0.85 \\
    MatSim+M   & \textbf{0.78} & 0.56 & \textbf{0.91} \\
    MatSim+C+M & 0.72 & \textbf{0.61} & 0.85 \\
    \bottomrule
  \end{tabular}
  \caption{Results. $+C$ indicates cropping, $+M$ indicates masking. Random stands for random matching.}
  \label{tab:results}
\end{table}

As can be seen in \Cref{fig:accExamples} and \Cref{tab:results}, the MatSim network performed well across almost all types of materials and environments. The MatSim network significantly outperforms the best Clip model on all test sets (\Cref{tab:results}). This suggests that despite the fact that the MatSim network was trained only on simulated data, it learned to generalize to unfamiliar real-world materials. Both Clip and MatSim significantly outperformed random matching (\Cref{tab:results}), suggesting both approaches achieved a good understanding of materials. One unexpected aspect that made the benchmark very challenging is the fact that the camera itself often adjusts the color and smooths the image. This image-processing step can make even the same object in the same setting look very different. This variation is not taken to account by the synthetic dataset. To account for this, significant augmentations were used during training and resulted in 10\% improvement (Appendix 10.6). 


\subsection{Comparing masking and cropping for focusing attention}
Focusing the network on the material region of the image by cropping the material region and using it as the input image seems to improve the accuracy of both Clip and the MatSim network (\Cref{tab:results}). Masking the background by blacking out the background pixels seems to improve the results for the MatSim network but proves to be less effective than cropping for Clip. The fact MatSim net is affected by cropping and masking is surprising  because the MatSim network already receives the region of the material as input (\Cref{sec:TrainingAndArch}), so ideally, it should be able to focus on this region without either cropping or masking. It seems that the MatSim network is not yet able to make full use of having the mask image as an extra input in addition to the pure image. Not surprisingly, Clip with no cropping or masking has the worst performance since it has no inherent way to focus on the material region. 

\section{Conclusion}
This work has demonstrated a new benchmark and a synthetic dataset for the problem of one-shot material similarity recognition. A solution to this problem can be beneficial to almost any field in which materials are manipulated; it can be applied in autonomous chemistry labs, cooking and food inspection, or the handling of materials in construction and agriculture. The MatSim network managed to outperform Clip on this general challenge despite being trained only on synthetic data and using 1/1000 of the images and computer resources of Clip. The key to this success is the use of large-scale CGI artist repositories, which allowed us to reach a much higher level of diversity compared to most existing synthetic datasets. Generative AI methods are now capable of generating both 3D models and PBR textures, and they can therefore be used to generate an almost infinite variety of assets for the next generation of synthetic datasets. This is likely to make this approach even more powerful in the future.
\section{Appendix: Building the MatSim Dataset}
\begin{figure}[t]
  \centering
   \includegraphics[width=0.9\linewidth]{images/Figure8.jpg}
   \caption{Set of MatSim dataset with materials inside transparent vessels. The set structure is described in Section 3.1 and Figure 3. The vessel generation is described in section 3.4.}
   \label{fig:dataSets_ves}
\end{figure}
\subsection{Setting objects in the scene}
Objects for the dataset were taken from the ShapeNet core2 dataset, which includes 56000 3D model objects with over 200 categories. During scene creation, a random object was loaded, randomly scaled, and rotated, and its original materials were removed. In addition, overlapping faces on the surface were removed. 
\subsection{UV mapping}
Since the PBR materials are given as 2D texture maps, assigning a material to a 3D object involves wrapping these 2D maps around the object's surface. Wrapping 2D maps around complex 3D objects (UV mapping) is a rather complex problem. Blender 3.1 contains some automatic wrapping techniques, which we use. A major problem that we encounter is that many objects contain overlapping faces (surfaces), which causes the textures to be wrapped around the same areas several times, causing strong unrealistic visual artifacts. We managed to solve this by removing vertexes and faces that were too close to each other using the remove overlapping vertexes tool of Blender. This led to the slight deformation of some objects' shapes. The wrapping itself depends on various parameters, such as the relative scale, orientation, and translation of the texture map relative to the object, as well as the coordinate system. We randomize all of these parameters between each image to achieve maximum variability. 
\subsection{Background illumination and HDRIs}
The environment and illumination greatly affect a material's appearance. The main method used by the CGI community to control the background and illumination involves high dynamic range images (HDRIs). These are panoramic images that wrap around the scene and provide 360 illuminations. The range of intensities of light is from 0–6550, compared to 0–255 in standard images. This allows HDRIs to represent a much broader range of illumination. We downloaded over 500 HDRIs from HDRI Haven repositories. These HDRIs are highly diverse scenes covering both daytime and nighttime conditions in a large number of indoor and outdoor environments. These HDRIs were used to add both illumination and background to the scenes. To increase diversity, the HDRIs were randomly rotated and scaled, and their intensities were randomly increased or decreased for each scene.
\subsection{Adding ground plane and background objects}
Random objects were scattered in the scene to make the environment more diverse and add shadows and back reflections. This was done by loading random objects from the ShapeNet dataset and randomly scaling, rotating, and positioning them in the scenes. In addition, a ground plane was generated by adding a flat plane below the object and assigning a random PBR material to it.
\subsection{Limitation of the dataset}
A major limitation of the dataset is that phase transitions between materials are often not the average between the two states; instead, they are either a completely different state (the transition between wood and coal is fire), or they are not uniformly distributed, like the nucleation of ice in the water. Both cases are not completely addressed by the dataset and require further work.

\subsection{Data augmentation}
A major problem with using only simulated data for learning is the fact that modern cameras significantly modify an image's appearance by smoothing and adjusting colors. To account for this, we use extensive augmentation for the image during training. This includes random Gaussian smoothing, the darkening or brightening of the image, partial or complete decoloring (to greyscale), and noise addition. Each of these augmentations was applied for about 1
\subsection{Data sampling}
Sampling is done by choosing one set for each batch and, from this set, choosing 12 random images (with no repetition). Sets with materials inside and outside transparent vessels were selected with equal probability.


\subsection{CGI assets from artist repositories}
One of the main challenges in creating the MatSim dataset was collecting large and diverse enough examples of backgrounds, objects, and materials. Large-scale repositories of assets that serve the CGI artist community have been available for a while but remain almost unutilized by the machine-learning community. Utilizing these repositories made it possible for us to create a wide range of scenes on a large scale and with sufficient diversity; this would have been very difficult if we had created these data ourselves. The main three repositories used for this work were AmbientCG\cite{ambientcg}, CGBookCase\cite{cgbookcase} and TextureBox\cite{TextureBox} for materials,  HDRI Haven for HDRI environments \cite{polyhaven}, and ShapeNet for objects\cite{chang2015shapenet}.

\subsection{Blender render setting and hardware.}
The dataset was rendered using Blender 3.1, with  CYCLEs rendering engine, 120 rendering cycles per frame. The rendering was done using Nvidia RTX 3090 GPU.
\subsection{Realism vs. generality}
There are two ways to go about generating a synthetic dataset. One approach is to make the dataset as realistic as possible by maximizing the similarity of the generated data to the real world. The other approach is to maximize variability and make the scenes as diverse as possible, even at the cost of realism. In this work, we prefer variability over realism, which means we assign random materials to random objects and set them in random environments. For example, an image in the dataset may contain a car made out of wood on the ground made of metal in a forest surrounded by random objects. This makes the dataset easier to generate and helps the network achieve a much higher level of generalization and identify materials regardless of the environment or the object they appear on.  
{\small
\bibliographystyle{ieee_fullname}
