
\section{Introduction}
\label{sec:introduction}

\begin{comment}
“Motivation of the research”
    What is federated learning
    Pervasive training data flaws in federated learning (over 6
    DNNs are vulnerable to such data flaws (i.e., noisy labels)

“Previous research for noisy labels”
    There have been numerous “sample selection” approaches that identifies clean instances from noisy data -> show promising performance in noisy label scenarios (non-FL setting)
    However, this family of methods does not work well in FL + LNL setting

“Why are they not working well?”  
    Non-iid data distribution over clients
    Varying noise types and noise ratios over clients
    -> significant performance gap between clients -> poor performance after global aggregation
    [put key figures]

“Rationale & key ideas” 
    In FL scenario, there exist expert clients (node) for varying data distributions
    Thus, we consider (i) expertise and (2) distribution similarity between clients
    Key idea is leveraging only K-nearest neighbors to build a clean data detector?
    What are the main challenges?
    What is the definition of K-nearest neighbors?
    How to find K neighbors and how to leverage them to filter out noise inputs
\end{comment}

Federated learning is a popular distributed learning approach that handles a large corpus of decentralized data residing on multiple clients such as mobile or IoT devices \,\cite{bonawitz2019towards, yang2021characterizing}. Since federated learning uses personal devices, data privacy is a critical issue for the learning process\cite{liu2021pfa}. Therefore, federated learning does not directly access raw training data\,\cite{li2020federated}. Instead, each client performs the {local update} phase by training the current model for multiple iterations {only} with its own training data, and then the {model aggregation} phase is executed by a server that merges received local models without sharing {any} of the clients' raw training data\,\cite{mcmahan2017communication}. The final model is obtained by alternating the two aforementioned steps until the global model converges.


\begin{figure}[t!]
\begin{center}
\includegraphics[width=8.4cm]{figures/introduction/motivation_real.pdf}
\end{center}
\vspace*{-0.03cm}
\hspace*{0.65cm} {\small (a) FedAvg.} \hspace*{0.9cm} {\small (b) Co-teaching.} \hspace*{0.8cm} {\small (c) {\algname{}}.}
\vspace*{-0.28cm}
\caption{Performance difference of local models over 100 clients after training on CIFAR-10 with symmetric noise of $0-80\%$ in the Non-IID setting\,\cite{mcmahan2017communication}: (a) shows the performance degradation incurred by label noise in training data; (b) and (c) contrast the improvement by Co-teaching and our proposed \algname{}. Client Ids are sorted in ascending order by their noise rate. }
\label{fig:motivation}
\vspace*{-0.5cm}
\end{figure}


Recently, \textit{robustness} has become an important challenge of federated learning, because the data collection process in each client is naturally accompanied by \textit{noisy labels}, which may be corrupted from the ground-truth labels\,\cite{lee2021machine}. In the presence of noisy labels, deep neural networks\,(DNNs) easily overfit to the noisy labels, thereby leading to poor generalization on unseen data\,\cite{zhang2016understanding}. Extensive efforts have been devoted to overcoming noisy labels in the centralized scenario\,\cite{ xia2019anchor, li2020dividemix}, but have yet to be studied widely in the federated learning\,(decentralized) scenario.

The \textit{small-loss} trick, which updates a DNN with a certain number of small-loss examples every training iteration\,\cite{jiang2017mentornet, yu2019does, song2021robust}, could be a prominent direction to provide robustness.  Small-loss examples are typically regarded as the clean set because DNNs tend to first learn from clean data and then gradually overfit to noisy data\,\cite{arpit2017closer}. However, robust training for federated learning requires more consideration, particularly on the following \textit{two} difficulties: 
\begin{enumerate}
\item \textbf{Data Heterogeneity}: Each client has non-independent and identically distributed\,(Non-IID) data with respect to the number of training examples for each class, which differ between clients.
\item \textbf{Varying Label Noise}: Noise ratios and types of data flaws vary between clients depending on how their data was collected, mainly attributed to the malfunction of data collectors, crowd-sourcing, and adversarial attacks.
\end{enumerate}

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=18.0cm]{figures/introduction/overview.pdf}
\end{center}
\vspace*{-0.03cm}
\hspace*{3.1cm} {\small (a) Local Update Phase.} \hspace*{6.2cm} {\small (b) Model Aggregation Phase.} 
\vspace*{-0.2cm}
\caption{\textbf{Overview of \algname{} ($k=1$)}: (a) $k$-reliable neighbors are retrieved to fit GMMs to the loss values of all training examples in the target client. Their ensembled GMMs are leveraged to identify clean examples, which are used to update the target model in the local update phase; (b) The weights of updated local models are averaged in the model aggregation phase.}
\label{fig:key_idea}
\vspace*{-0.3cm}
\end{figure*}

As illustrated in Figure \ref{fig:motivation}(a), these two difficulties greatly exacerbate the performance discrepancy over clients because local data in each client varies in terms of \textit{data heterogeneity} and \textit{label noise}. 
Such a large performance discrepancy arguably results in poor performance of the global model, because client models are aggregated regardless of whether their models are corrupted from label noise\,\cite{chen2020fedbe}. 
This discrepancy unfortunately remains even with Co-teaching\,\cite{han2018co}, a popular robust method that uses the small-loss trick for sample selection, as can be seen from Figure \ref{fig:motivation}(b); the performance gap between the high- and low-performance model is considerably large, although the overall performance is higher than that of FedAvg. 
Therefore, it is essential to not only \textbf{(i)} identify clean examples from noisy data, but also \textbf{(ii)} {improve the low-performance models to increase the overall robustness for federated learning with noisy labels.} 

In this paper, we propose a robust approach called \textbf{\algname{}}\,({\underline{Fed}-erated learning with \underline{R}eliable \underline{N}eighbors}),
which exploits $k$-reliable neighbors\footnote{A neighbor refers to any client except the given target client.} in the client pool {to help identify clean examples even when the target client is unreliable.}
We introduce the notion of \textit{k-reliable} neighbors, which are $k$ clients that have similar data class distributions to the target client (data similarity), or \textit{minimal} label noise in their data (data expertise). Using $k$-reliable neighbors, \algname{} improves {the performance of every client's local model; the low-performance models benefit more from their $k$-reliable neighbors, as shown in Figure \ref{fig:motivation}(c).} 

In the local update phase shown in Figure \ref{fig:key_idea}(a), \algname{} first retrieves $k$-reliable neighbors from the server. \enquote{Neighbor 2} has a relatively clean dataset and a similar class distribution to the target client's, so the server transmits the model of \enquote{Neighbor 2} to the target client. 
Because the retrieved $k$-reliable models have been updated with training data similar to the target client's, they are used to identify clean examples from the client's dataset by fitting two-component Gaussian mixture models\,(GMMs) to the loss distribution of the target's training data. The GMMs from the reliable neighbors are later ensembled with the target client's to effectively separate clean examples from the noisy ones\footnote{Although we use ``neighbors'' in plural form, comparable performance can be obtained with only one reliable neighbor.} even for datasets heavily corrupted with label noise. 
Based on the small-loss trick, the training examples with a higher probability of belonging to the \textit{small-loss} component in the GMMs are treated as clean examples and used to robustly update the target model.
After the local update phase, the weights of updated models are averaged to build a global model in the model aggregation phase in Figure \ref{fig:key_idea}(b). These two phases are repeated until the global model converges following the standard learning pipeline\,\cite{mcmahan2017communication}. 

Our main contributions are summarized as follows:

\vspace*{-0.05cm}
\begin{itemize}[leftmargin=*]
\item To the best of our knowledge, this is the first work to address the challenge of data heterogeneity and label noise by exploiting reliable neighbors.
\vspace*{0.05cm}
\item We examine the effect of data expertise and similarity used in neighbor search for robust federated learning.
\vspace*{0.05cm}
\item \algname{} remarkably {improves the performance of underperforming client models by mitigating the client-to-client discrepancy, thereby boosting overall robustness.}
\vspace*{0.05cm}
\item \algname{} significantly outperforms state-of-the-art methods on three real-world or synthetic benchmark datasets with \textit{varying levels} of data heterogeneity and label noise.
\end{itemize}


\section{Related Work}
\label{sec:related_work}


\begin{comment}
Federated Learning
Noisy Labels
FL + NLN : Convergence between (1) and (2)
    model aggregation weighting - FOCUS
    Why is sample selection better than weight averaging?
     - [client selection] (데이터 가용성 측면, 어차피 오버피팅 되는 모델을 쓰는거임)
    Relevance with data poisoning with FL 
\end{comment}

Federated learning and robust training have been studied by separate communities, and only a few works have attempted to converge these two topics.

\smallskip
\noindent\textbf{Federated Learning.} 
Federated learning aims to learn a strong global model by exploiting all of the client data without infringing on data privacy. However, due to the heterogeneity in training data, it typically suffers from performance degradation of the global model\,\cite{zhu2021federated} or weight divergence\,\cite{zhao2018federated}.
In this regard, extensive efforts have been made to address these issues.
\citet{duan2019astraea} addressed the problem of long-tailed distribution in training data by performing data augmentation. \citet{zhao2018federated} let all of the clients share the same public data to mitigate the data heterogeneity. Meanwhile, \citet{li2018federated} and \citet{acar2020federated} added a regularization term that prevents the local model from diverging to the global model due to its own skewed Non-IID data. Although this family of methods contributes to improving the effectiveness of federated learning, they simply assume that all the labels in training data are clean, which is not a realistic scenario.

\smallskip
\noindent\textbf{Learning with Noisy Labels.} 
Extensive studies have been conducted to overcome noisy labels in the centralized scenario. 
A prominent direction for handling noisy labels is sample selection, which trains DNNs for a possibly clean subset of noisy training data.
For example, Co-teaching\,\cite{han2018co} trains two DNNs where each DNN selects small-loss examples in a mini-batch and then feeds them to its peer network for further training. MORPH\,\cite{song2021robust} divides the training process into two learning periods and employs two different criteria for sample selection. 
Another possible direction is to modify the loss of training examples based on importance reweighting or relabeling\,\cite{patrini2017making, huang2020self, tanaka2018joint, zheng2021meta}. 

Most recently, to take advantage of all the training data, sample selection methods have been combined with other approaches, such as loss correction and semi-supervised learning. SELFIE\,\cite{song2019selfie} uses relabeled noisy examples in conjunction with the selected clean ones. DivideMix\,\cite{li2020dividemix} treats selected clean examples as labeled data and applies MixMatch\,\cite{berthelot2019mixmatch} for semi-supervised learning. Despite these methods' success, they do not perform well in the federated learning setting due to the client-to-client performance discrepancy, as shown in Figure \ref{fig:motivation}(b).

\smallskip
\noindent\textbf{Robust Federated Learning.} 
To address the problem of noisy labels in federated learning, most studies assume that trustworthy data exists either on the client- or server-side. \citet{chen2020focus} estimated the credibility of each client with a small clean validation dataset, and aggregated models based on the estimated credibilities of clients. \citet{tuor2021overcoming} proposed a data filtering method, which identifies highly relevant examples for the given specific task using a reference model trained on clean benchmark data. In another direction, Robust FL\,\cite{yang2020robust} shared the central representations of clients' local data to maintain the consistent decision boundary over clients, and performed global-guided pseudo labeling assuming the IID scenario. However, existing methods rely on unrealistic assumptions including the existence of clean validation data or the IID scenario. 

\smallskip
\noindent\textbf{Difference from Existing Work.} We clarify \textit{why federated learning with noisy labels is problematic}; The client-to-client performance discrepancy hinders the use of existing robust methods in the label noise community into federated learning. Although a few robust methods have been proposed, this problem has been overlooked.
In this paper, we thus bridge the two different topics to improve overall robustness {by introducing $k$-reliable neighbors such that they deliver reliable information to the clients.} Moreover, compared to all aforementioned methods, \algname{} does not require any unrealistic supervision such as a clean validation set or knowledge of true noise rates per client.

\section{Preliminaries}
\label{sec:preliminaries}

A multi-class classification problem requires training data $\mathcal{D}=\{(x_i, y^{*}_i)\}_{i=1}^{|\mathcal{D}|}$, where $x$ is a training example with its ground-truth label $y^{*}$. However, noisy training data $\tilde{\mathcal{D}}=\{(x_i, \tilde{y}_i)\}_{i=1}^{|\mathcal{D}|}$ may have corrupted labels such that $\tilde{y}_i \neq y^*_i$ for some $i$. We herein briefly summarize the conventional learning pipeline for {(i)} standard federated learning and {(ii)} robust learning with sample selection. \looseness=-1 

\begin{itemize}[leftmargin=*]
\item \textbf{Federated Learning\,(FedAvg):} It is assumed that each client ${\sf i}$ has its own Non-IID dataset $\mathcal{D}_{\sf i}$\,(i.e., $\mathcal{D} = \cup_{\sf i} \{\mathcal{D}_{\sf i}\}$), and cannot directly access another client's data. Therefore, a global model $\Theta_{{\sf global}}$ is trained by alternating the local update and model aggregation phase. 

During the local update phase, a fixed number of target clients $\mathcal{M}$ are selected from the entire client pool.
{For each target client ${\sf c} \! \in \! \mathcal{M}$, all the other clients are defined as its neighbors ${\sf n} \! \in \!\mathcal{N}_{{\sf c}}$.}
Next, the target client's local model $\Theta_{\sf c}$ is trained on its own data $\mathcal{D}_{\sf c}$ for certain local epochs with the standard stochastic gradient decent\,\cite{mcmahan2017communication}. All the trained models for $\mathcal{M}$ are subsequently received and averaged by the server in the model aggregation phase,
\begin{equation}
\label{eq:aggregation}
\Theta_{{\sf global}} \leftarrow \sum_{{\sf c} \in \mathcal{M}} w_{\sf c} \Theta_{{\sf c}}, ~{\rm where}~ w_{\sf c}=\frac{|{\mathcal{D}}_{{\sf c}}|}{\sum_{{\sf c}^{\prime} \in \mathcal{M}}| {\mathcal{D}}_{{\sf c}'}|}
\end{equation}
where $w_{\sf c}$ is the aggregation weight that is based on the data size of each client. Lastly, the updated global model is broadcasted to the clients. This training round is called a transmission round and is repeated until convergence.
\smallskip
\item \textbf{Robust Learning with Sample Selection:} Many recent robust approaches have adopted sample selection, which treats a certain number of small-loss examples as the clean set\,\cite{han2018co, yu2019does, li2020dividemix, song2021robust}. In the centralized scenario, given a noisy mini-batch $\tilde{\mathcal{B}}$\,($\subset \tilde{\mathcal{D}}$), the clean subset $\mathcal{S}$ is identified and used to directly update the global model $\Theta_{\sf global}$,
\begin{equation}
\!\Theta_{\sf global}\! \leftarrow \Theta_{\sf global} - \eta\nabla\frac{1}{|\mathcal{S}|}\! \sum_{x \in \mathcal{S}} \!\ell(x, \tilde{y}; \Theta_{\sf global}),
\end{equation}
where $\eta$ is the learning rate and $\ell$ is a certain loss function. The remaining examples\,(mostly large-loss examples) in the mini-batch are excluded to pursue robust learning.  
\end{itemize}
Note that training data is typically assumed to be \textit{clean} in federated learning, while a global model is assumed to have access to \textit{all} of the training data in robust learning. However, our objective is to robustly train a global model $\Theta_{\sf global}$ even when noisy labels exist in the federated learning scenario.

\begin{comment}
\smallskip
\noindent \textbf{Problem Definition:}
The training data of each client is Non-IID and at the same time corrupted with varying label noise. Let ${\sf type}$ and ${\sf rate}$ be the noise type and noise rate, respectively. 
Our objective is to train a global model $\Theta_{\sf global}$ robustly even when the Non-IID data of clients have different pairs of $\langle {\sf type}, {\sf rate} \rangle$, which exacerbates client-to-client performance discrepancy, in the federated learning scenario.
\end{comment}

\begin{comment}
\subsection{Federated Learning Pipeline} 
Federated learning is a learning technique in which multiple clients and central servers collaboratively learn the global model. With federated learning, the central server can learn the global model without direct access to the client's raw data. FedAvg is the most frequently and widely used federated learning algorithm.
Federated learning is a learning technique in which multiple clients and central servers collaboratively learn the global model. With federated learning, the central server can learn the global model without direct access to the client's raw data. FedAvg is the most frequently and widely used federated learning algorithm.
FedAvg learns global model $\theta_G$ through the following procedure. First, for each communication round $t$, the selected $C$ fraction of clients  $S_t$ among $N$ total clients participates in learning. The central server broadcasts the global model $\theta_{G(t)}$ to the selected clients. Client $i$ participating in training updates the received global model with its own dataset $D_i$ during $E$ local epochs to obtain $\theta_{L_i}^{E}$. After receiving the client local models that have been trained, the central server aggregates them through the following equation: 
\begin{equation}
    \theta_{G(t+1)} \leftarrow \sum_{i\in S_t} \frac{n_i}{n} \theta^E_{L_i}
\end{equation}
where $n_i = |D_i|$ and $n = \sum_{i\in S_t}n_i$. 

In the federated learning setting, all client data labeling is performed by the client. If the wrong label is assigned due to the lack of expertise of the client, it may cause degradation of the performance of the local model during the local update process. In a chain, aggregation of local models overfitted to noisy labels can also lead to degradation of global model performance.


\subsection{Noise Robust Learning Pipeline} 
Training DNNs that perform a classification task, we assume that all labels of a large amount of dataset are clean. However, However, it is not realistic to assume that all labels in real-world data are correct. It spends very expensive cost to manually find data with noisy labels and conduct label correction. In particular, in a federated learning environment, the central server cannot directly access and modify the client's label. However, noise label data cannot be ignored because it can degrade the classification performance for test data.

\textit{Sample selection} is the most widely used method to prevent training data with noisy labels from polluting the DNN model . Intuitively, only the possibly clean subset identified as the learning model will be used as training dataset. To get clean examples, DNN uses small-loss examples. The use of small-loss is based on the fact that deep neural networks learn from the easiest data patterns among the entire dataset\cite{arpit2017closer}. Data with correct labels are easier to learn than noisy labels. The training loss decreases relatively quickly for examples with correct labels. So we can exclude noisy labeled examples with relatively large loss.

Two methods are widely used as a strategy to select a clean subset based on small-loss. The first method is to exclude examples with a large loss in a mini-batch as much as a predetermined noise rate. Co-Teaching\cite{han2018co} and SELFIE\cite{song2019selfie} use this strategy. The second method is to fit a one-dimensional and two-component GMM for all data loss values. DivideMix\cite{li2020dividemix} typically uses this method. In particular, this method has the advantage that it is not necessary to set the noise rate beforehand. To take advantage of not having to set the noise rate, FedRN also performs sample selection through GMM.
\end{comment}
\section{Proposed Method: FedRN}
\label{sec:methodology}

We introduce the notion of $k$-reliable neighbors and describe how to leverage them for robust federated learning with noisy labels.

\subsection{Main Concept: $k$-Reliable Neighbors}

The key idea of \algname{} is to exploit useful information in other clients without violating privacy. Hence, the main challenge is to find the most helpful neighbor clients using limited information about each client. Intuitively speaking, 
these helpful neighbor clients have (1)\,high expertise on their data, or (2)\,data distributions similar to the target client's: 
{
\renewcommand\labelenumi{(\theenumi)}
\begin{itemize}[leftmargin=*]
\item \textbf{Data Expertise:} The training data should be sufficiently \textit{clean}. The neighbor client with heavy noise could hinder identifying clean examples from the client's noisy data.
\smallskip
\item \textbf{Data Similarity:} Data and class distributions should be \textit{similar} to the target client's. The distribution shift problem could lead to model incompatibility between target and neighbor clients.  
\end{itemize}
}

We investigate how these two considerations affect selecting reliable neighbor clients for robust federated learning. Given a target client ${\sf c}$, a reliable neighbor client is defined to be one of the $k$-reliable neighbors $\mathcal{R}_{{\sf c}}(k)$, as described in Definition \ref{def:k_neighbor}. Satisfying the condition, $k$-reliable neighbors can deliver clean and useful information even when the training data of the target client is unreliable due to the heavy label noise.

\smallskip
\begin{definition} 
Let ${\rm Exp(\cdot)}$ and ${\rm Sim(\cdot, \cdot)}$ be the score function for data expertise of a client and data similarity between two clients, respectively. Given a target client ${\sf c}$ and available neighbor clients $\mathcal{N}_{{\sf c}}=\{{\sf n}_{i}\}_{i=1}^{|\mathcal{N}_{{\sf c}}|}$, the $k$-reliable neighbors $\mathcal{R}_{{\sf c}}(k)$ is a subset of $\mathcal{N}_{\sf c}$ with size $k$ satisfying, 
\begin{equation}
\begin{gathered}
\mathcal{R}_{{\sf c}}(k) = {\rm argmax}_{|(\mathcal{R}^{\prime} : {\sf n} \in \mathcal{R}^{\prime})| = k} ~ {\rm R}({\sf c}, {\sf n}),~{\rm where}\\
{\rm R}({\sf c}, {\sf n}) = \alpha \cdot {\rm Exp}({\sf n}) + (1-\alpha) \cdot {\rm Sim}({\sf c}, {\sf n})
\end{gathered}
\label{eq:reliability_score}
\end{equation}
and $\alpha$ is the balancing term that determines the contribution of each score. The data similarity to itself is 1, i.e., ${\rm Sim}({\sf c}, {\sf c})=1$. \qed
\label{def:k_neighbor}
\end{definition}

To compute the reliability score ${\rm R}({\sf c}, {\sf n})$ in Eq.\,\eqref{eq:reliability_score}, we present two indicators of data expertise and similarity using client's local models. Since it is infeasible to access the raw data in neighbor clients, we leverage the server's copies of local models, which are sent to the server during the model aggregation phase.
By doing so, our approach does not compromise the privacy-preserving property in federated learning. 

With the intention of selecting clients with cleaner examples, we define the data expertise score motivated by the memorization effect\,\cite{arpit2017closer, song2021robust}.
In general, DNNs rapidly memorize clean examples in the early stages of training, and start to overfit to noisy examples in the later stages of training. 
In this sense, the early memorization of clean examples leads to higher training accuracy if the client has cleaner data.
Consequently, during the model aggregation phase, the server receives the training accuracy of each client along with their local models, then computes and normalizes the data expertise Exp($\cdot$) per client,\looseness=-1
\begin{equation}
\!{\rm Exp}({\sf c}) = \frac{{\rm Acc}({\sf c}) - {\rm minAcc}(\{{\sf c}\} \cup \mathcal{N}_{\sf c})}{{\rm maxAcc}(\{{\sf c}\} \cup \mathcal{N}_{\sf c}) - {\rm minAcc}(\{{\sf c}\} \cup \mathcal{N}_{\sf c})},\!\!
\label{eq:expertise}
\end{equation}
where ${\rm Acc}$ denotes the training accuracy of the local model for the client ${\sf c}$; ${\rm maxAcc}$ and ${\rm minAcc}$ are the maximum and minimum training accuracy of the given client set.

Regarding the data similarity score, we borrow the idea of {predictive difference}\,\cite{ briggs2020federated,jeong2020federated}, which provides a hint to the similarity between two heterogeneous data distributions \textit{without} the use of raw data. The data similarity between two clients can be approximated by the prediction difference for the \textit{same} random input, where a smaller difference indicates a higher similarity. 
Accordingly, after the local update phase, each client generates the same input $\tilde{x}$ of Gaussian random noise and transmits its softmax outputs $p(\tilde{x};\Theta_{{\sf n}})$ into the server, where $p(x;\Theta_{{\sf n}})$ denotes the softmax output of the client ${\sf n}$ for input $x$. The data similarity Sim($\cdot, \cdot$) between the target client ${\sf c}$ and neighbor client ${\sf n}$ is then computed by the server, 
\begin{equation}
{\rm Sim}({\sf c}, {\sf n}) = {\rm Cosine}\big(p(\tilde{x};\Theta_{{\sf c}}), p(\tilde{x};\Theta_{{\sf n}})\big).
\label{eq:similarity}
\end{equation}
We use the cosine similarity denoted by Cosine to measure the predictive difference, because the \textit{symmetry} of similarity is guaranteed, i.e., ${\rm Sim}({\sf c}, {\sf n})={\rm Sim}({\sf n}, {\sf c})$. The min-max normalization is also applied similarly to data expertise. We provide an in-depth analysis of how the two proposed indicators work precisely in Section \ref{sec:in_depth_metric}.







\subsection{Robust Learning with $k$-Reliable Neighbors}

For every communication round,  each client participating in the upcoming round receives its $k$-reliable neighbors from the server. 
When updating the target client's model, we follow the general philosophy of \textit{sample selection} for robust learning by identifying clean examples from the noisy training data. 

Different to robust methods in centralized scenarios, $k+1$ models, comprising $k$-reliable neighbors and the target client's model, cooperate to mitigate the performance discrepancy among clients and obtain more reliable results.
For each of the $k+1$ models, we fit a two-component GMM to model the loss distributions of clean and noisy examples in view of loss distributions being \textit{bi-modal}\,\cite{li2020dividemix, arazo2019unsupervised}. Based on the ensembled predictions of these GMMs, the target client's model is only updated with examples selected as clean. 

To be specific, our aim is to estimate the probability of each training example being clean with $k+1$ models. At the beginning of the local update phase, the GMM is fitted to the loss of all available training examples in the target client by using the Expectation-Maximization\,(EM) algorithm. Given a noisy example $x$, the probability of being clean is obtained through its posterior probability for the small-loss component,
\begin{equation}
p \big( g|\ell(x, \tilde{y};\Theta) \big) = p(g) p\big(\ell(x, \tilde{y}; \Theta) | g \big) / p\big(\ell(x, \tilde{y};\Theta)\big),
\end{equation}
where g denotes the Gaussian component with a smaller mean\,(i.e., smaller loss) and $\ell(\cdot)$ is the loss function. Subsequently, the probability of each example to be clean is ensembled over $k+1$ models with their reliability scores ${\rm R}(\cdot, \cdot)$ in Eq.\,\eqref{eq:reliability_score}. Given a target client ${\sf c}$, the clean probability can be formulated as:
\begin{equation}
\begin{gathered}
p\big( {\rm clean} | x ; \mathcal{R}_{{\sf c}}(k) \big)= \!\!\!\!\!\!\!\!\!\sum_{{\sf n} \in \{{\sf c}\} \cup \mathcal{R}_{{\sf c}}(k) }\!\!\!\!\!\!\!\! {\rm R}^{\prime}({\sf c}, {\sf n}) \times p \big( g|\ell(x, \tilde{y} ;\Theta_{{\sf n}}) \big), \!\!\! \\
{\rm where} ~~ {\rm R}^{\prime}({\sf c}, {\sf n}) = {\rm R}({\sf c}, {\sf n}) ~~~ / \!\!\!\!\!\!\!\!\sum_{{\sf n}^{\prime} \in \{ {\sf c}\} \cup \mathcal{R}_{{\sf c}}(k)}\!\!\!\!\! {\rm R}({\sf c}, {\sf n}^{\prime}).\!\!\!
\end{gathered}
\end{equation}
\algname{} constructs a clean set $\mathcal{S}_{{\sf c}}$ from these probabilities such that each training example in the set has a probability greater than $0.5$, 
\begin{equation}
\mathcal{S}_{{\sf c}} = \{ x \in \tilde{\mathcal{D}}_{{\sf c}} : p({\rm clean} | x ; \mathcal{R}_{{\sf c}}(k)) > 0.5 \},
\label{eq:selection}
\end{equation}
where $\tilde{\mathcal{D}}_{{\sf c}}$ is the noisy data of the target client ${\sf c}$. The model is finally updated with only the clean set for a specified number of local epochs in the local update phase, while the complement of the set, which are likely to be noisy, are discarded for robust learning. 

This collaborated approach with $k$-reliable neighbors considerably increases the robustness of \textit{low}-performance clients 
with the aid of neighbors with high data expertise or similarity. 
Therefore, \algname{} reduces the performance gap between clients and enhances the overall robustness, thus leading to a stronger global model.


\subsection{Fine-tuning with $k$-Reliable Neighbors}
Due to the data heterogeneity in federated learning, the problem of distribution shift between clients' local models is another challenge when ensembling their predictions for sample selection. 
Hence, \algname{} is integrated with the fine-tuning technique that helps quickly adapt to the local data distribution\,\cite{collins2021exploiting}.

Before constructing the clean set in Eq.\,\eqref{eq:selection}, we identify an auxiliary set of clean examples using \textit{only} the target client's model,
\begin{equation}
\mathcal{S}_{{\sf c}}^{\sf aux} = \{ x \in \tilde{\mathcal{D}}_{{\sf c}} : p({\rm clean} | x ; \{{\sf c}\}) > 0.5 \}
\label{eq:aux_set}
\end{equation}
and fine-tune all the retrieved $k$-reliable neighbors for the auxiliary set. The auxiliary set could, however, be very noisy especially when the target client has heavy noise in its training data. To prevent feature extractors from being corrupted by such client-side noise, we only fine-tune the classification head. In Section \ref{sec:analysis_fine_tune}, we examine the effect of the fine-tuning technique on data expertise and similarity used in neighbor search, and provide insights on their use for robust federated learning with noisy labels. 











 


\subsection{Algorithm Pseudocode}
\vspace*{-0.0cm}


\setlength{\textfloatsep}{8pt
\begin{algorithm}[t!]
\caption{FedRN Algorithm}
\label{alg:fedrn}
\hspace*{-0.8cm}\textsc{Input:} $\tilde{\mathcal{D}}_{{\sf c}}$: noisy data of client ${\sf c}$, $rounds$: $\#$ total rounds,\\ $e$: $\#$ local epochs, $k$: $\#$ reliable neighbors 

\hspace*{-3.9cm}\textsc{Output:} $\Theta_{{\sf global}}^{t}$: a global model 
\begin{algorithmic}[1]
\State $t \leftarrow 1$, ~$\Theta_{{\sf global}}^t \leftarrow \text{Initialize network parameters}$
\For{$round=1$ {\bf to} $rounds$}
\State $\mathcal{M} \leftarrow \text{Randomly select $m$ clients}\,(m > k)$
\State /* Find $k$-reliable neighbors in Definition \ref{def:k_neighbor} */
\State Compute $\forall_{{\sf c} \in \mathcal{M}}\forall_{{\sf n} \in \mathcal{N}_{\sf c}} ~{\rm Sim}({\sf c}, {\sf n})$ by Eq.\,\eqref{eq:similarity}
\State Retrieve $\mathcal{R}_{{\sf c}}(k)$ for ${\sf c} \in \mathcal{M}$
\State Broadcast $\Theta_{{\sf global}}^t$ and $\mathcal{R}_{{\sf c}}(k)$ to clients $\in \mathcal{M}$
\State {\color{blue} /*  {I. Local Update Phase} */} 
\For{${\sf c} \in \mathcal{M}$ in parallel}
\State $\mathcal{R}_{{\sf c}}(k) \leftarrow \text{Receive $k$-reliable neighbors}$
\State $\mathcal{R}_{{\sf c}}(k) \leftarrow $ Fine-tune for $\mathcal{S}_{\sf c}^{\sf aux}$ in Eq.\,\eqref{eq:aux_set}.
\State /* Sample selection with $\mathcal{R}_{{\sf c}}(k)$ by Eq.\,\eqref{eq:selection} */
\State $\mathcal{S}_{{\sf c}} \leftarrow \{ x \in \tilde{\mathcal{D}}_{{\sf c}} : p({\rm clean} | x ; \mathcal{R}_{{\sf c}}(k)) > 0.5 \}$ 
\State $\Theta_{{\sf c}} \leftarrow \Theta_{{\sf global}}^{t}$ /* Initialization */ 
\State $\Theta_{{\sf c}} \leftarrow $ Train for $e$ epochs with $\mathcal{S}_{{\sf c}}$
\State ${\rm Acc}({\sf c}) \leftarrow $ Compute data expertise by Eq.\,\eqref{eq:expertise}
\State $p(\tilde{x}; \Theta_{\sf c}) \leftarrow$ Compute the softmax output for $\tilde{x}$
\State Send $\Theta_{{\sf c}}$, ${\rm Acc}({\sf c})$, and $p(\tilde{x}; \Theta_{\sf c})$ to the server
\EndFor
\State {\color{blue} /*  {II. Model Aggregation Phase} */} 
\State $\Theta_{{\sf global}}^{t+1} \leftarrow \sum_{{\sf c} \in \mathcal{M}} w_{\sf c} \Theta_{{\sf c}}$ s.t. $w_{\sf c}=\frac{|\tilde{\mathcal{D}}_{{\sf c}}|}{\sum_{{\sf c}^{\prime} \in \mathcal{M}}| \tilde{\mathcal{D}}_{{\sf c}'}|}$ 
\State $t \leftarrow t + 1$
\EndFor
\State \Return {$\Theta_{{\sf global}}^t$}
\end{algorithmic}
\end{algorithm} 

Algorithm \ref{alg:fedrn} describes the overall procedure of \algname{}.
We perform standard federated learning, FedAvg, for {warm-up} epochs before applying \algname{}, because robust training methods are generally applied after the warm-up phase in the literature\,\cite{han2018co, song2019selfie}. 
When training with \algname{}, retrieved neighbor models are fine-tuned for 1 epoch before sample selection\,(Line 11). 
{Next, the target client identifies clean examples by leveraging $k$-reliable neighbor models\,(Line 13). The received global model is updated with estimated clean samples\,(Lines 14--15). In the aggregation phase, the central server aggregates the updated local models\,(Line 20).
}

The main additional cost of \algname{} is the communication overhead from sending $k$-reliable models from the server to the selected client\,(Line 7). However, this cost can be drastically reduced by over a factor of 32 times in modern federated learning by sending the difference of parameters\,\cite{jeong2020federated} or a compressed model\,\cite{malekijoo2021fedzip}. 

\begin{comment}













\end{comment}
\section{Evaluation}
\label{sec:evaluation}

\begin{comment}
Baseline
    Pure FL
    Pure FL + Existing selection approach
    FL + LNL methods (X)

Description
    Non-iid, iid setting
    how to inject noise labels
    architecture: simple , complex

Experiments
    Main: synthetic and real world comparison 
    Ablation: (1) label precision graph comparison, (2) neighbor search ablation (K=1, 2,...), (3) components ablation (w. / w.o semi-super?,  alpha - expertise v.s. similarity)
\end{comment}


Our evaluation was conducted to support the following:
\vspace*{-0.05cm}
\begin{itemize}[leftmargin=*]
\item \algname{} is \textbf{more robust} than {five} state-of-the-art methods for federated learning with noisy labels\,(Section \ref{sec:robust_comparison}).
\vspace*{0.05cm}
\item \algname{} consistently identifies clean examples from noisy data with \textbf{high precision} and \textbf{recall}\,(Section \ref{sec:selection_comparison}).
\vspace*{0.05cm}
\item The reliability metric is \textbf{effective} in finding $k$-reliable neighbors for robust learning\,(Section \ref{sec:in_depth_metric}).
\vspace*{0.05cm}
\item The use of fine-tuning is \textbf{necessary} owing to the distribution shift in local data\,(Section \ref{sec:analysis_fine_tune}).
\vspace*{0.05cm}
\item A small number\,($k\leq 2)$ of reliable neighbors is \textbf{sufficient} to achieve high robustness\,(Section \ref{sec:aba_neighbor}).
\end{itemize}
\vspace*{-0.12cm}

\subsection{Experiment Configuration}

We verified the superiority of \algname{} compared with five robust learning methods on three real-world or synthetic benchmark datasets in a Non-IID federated learning setting.

\smallskip
\noindent\textbf{Datasets.} We performed image classification on synthetic and real-world benchmark data: CIFAR-10 and CIFAR-100\footnote{\url{https://www.cs.toronto.edu/~kriz/cifar.html}}, a subset of 80 million categorical images; mini-WebVision\footnote{\url{https://data.vision.ee.ethz.ch/cvl/webvision/dataset2017.html}}, a subset of real-world noisy data consisting of large-scale web images. We used the first 50 classes in WebVision V1 following the literature\,\cite{li2020dividemix}. {The statistics of the three datasets are summarized in Table \ref{tab:summary_data}.} 

For the federated learning setup with noisy labels, we merged the standard setting in the two research communities:
\begin{itemize}[leftmargin=*]
\item \underline{Non-IID Data:} Two popular ways of client data partitioning were applied\,\cite{zhu2021federated}; \textbf{(i)}\,Sharding: training data is sorted by its class labels and divided into {\small \enquote{$S \!  \times \! (\#~{\rm clients})$}} shards, which are randomly assigned to each client with an equal number of $S$. \textbf{(ii)}\,Dirichlet Distribution: each client is assigned with training data drawn from the Dirichlet distribution with a concentration parameter $\beta$. For the experiments, $S$ was set to  $2$, $5$, $10$, or $20$, and $\beta$ was set to $0.5$. \looseness=-1
\vspace*{0.05cm}
\item \underline{Label Noise:} We injected artificial noise by following typical protocols\,\cite{song2020learning}; \textbf{(i)}\,Symmetric Noise: a true label is flipped into all possible labels with equal probability. \textbf{(ii)}\,Asymmetric Noise: a true label is flipped into only a certain label. Note that we injected \textbf{varying levels} of label noise into the clients, so each client had different noise rates. For instance, a noise of $0.0$--$0.8$ means that the noise rate increases linearly from $0.0$ to $0.8$ as the client's index increases; thus, the average noise rate of clients' training data is $0.4$. We further tested for \textbf{mixed noise} where the two different types of label noise are injected into the clients. That is, symmetric noise is applied to half of the clients, while asymmetric noise is applied to the remaining clients.
\end{itemize} 
\vspace*{-0.0cm}
We did not inject any label noise into mini-WebVision because it contains real label noise whose rate is estimated at $20.0\%$\,\cite{song2020learning}. 

\input{tab-summary-data}

\input{tab-cifar10}
\input{tab-cifar100}

\smallskip
\noindent\textbf{Algorithms and Implementation Details.} 
We compared \algname{} with a standard federated learning method, FedAvg\,\cite{mcmahan2017communication}, and {five} state-of-the-art robust methods for handling noisy labels, namely Co-teaching \cite{han2018co}, Joint-optimization \cite{tanaka2018joint}, SELFIE \cite{song2019selfie}, DivideMix \cite{li2020dividemix}, {and Robust FL\,\cite{yang2020robust}}. All the robust methods were combined with FedAvg to support decentralized learning. The total number of clients was set to $100$ with a fixed participation rate of $0.1$. The training\,(transmission) round was set to be $500$ for CIFAR-10 and $1000$ for CIFAR-100 and WebVision, ensuring all the models' training convergence.
The SGD optimizer was used with a learning rate of $0.01$ and a momentum of $0.5$. The number of local epochs was set to be $5$. 

{As for hyperparameters related to our method, \algname{} requires the balancing term $\alpha$ and number of reliable neighbors $k$. We used $\alpha=0.6$, which was obtained via a grid search\,(see Appendix \ref{sec:parameter_search}). We used $k=1$ and $k=2$ since the performance gain of \algname{} was consistently high as long as $k\geq 1$, which is detailed in Section \ref{sec:aba_neighbor}.} 

The hyperparameters of the baseline robust methods were configured favorably in line with the literature. We clarify the hyperparmeter setup of all baseline algorithms, as follows:
\begin{itemize}[leftmargin=*]
\item Joint Optimization\,\cite{tanaka2018joint} requires the two coefficients for its two regularization losses for robust learning. These two coefficients were set to be 1.2 and 0.8, respectively.
\vspace*{0.1cm}
\item SELFIE\,\cite{song2019selfie} requires the uncertainty threshold and the length of label history as its hyperparameters. As suggested by the authors, they were set to be 0.05 and 15, respectively. 
\vspace*{0.1cm}
\item DivideMix\,\cite{li2020dividemix} applies MixMatch for semi-supervised learning, which requires the sharpening temperature, the unsupervised loss coefficient, and the Beta distribution parameter. They were set to be 0.5, 25, and 4, respectively.
\vspace*{0.1cm}
\item Robust FL\,\cite{yang2020robust} requires two coefficients for its two additional regularization losses similar to Joint Optimization. They were set to be 1.0 and 0.8, respectively.
\end{itemize}

\input{tab-webvision}


All of the algorithms were implemented using PyTorch $1.7.0$ and executed using four NVIDIA RTX 2080Ti GPUs. 
In support of reliable evaluation, we repeated every task thrice and reported the average test\,(or validation) accuracy, which is the common measure of robustness to noisy labels\,\cite{han2018co, chen2019understanding}. 


\footnotetext[5]{Oracle trains the network using FedAvg with all the true-labeled examples, i.e., $(1-\text{noise rate}) \times 100 \%$ of all the examples.}



\begin{figure*}[!ht]
\centering
\includegraphics[width=\linewidth]{figures/evaluation/label_cifar10.pdf}
\begin{minipage}[t]{.45\linewidth}
\centering
\label{label-sym80}
\end{minipage}%
\hskip 1cm
\begin{minipage}[t]{.45\linewidth}
\centering
\label{label-asym40}
\end{minipage}
\vspace*{0.05cm}
\hspace*{-0.4cm} {\small (a) Symmetric Noise of $0.0$--$0.8$.} \hspace*{5.6cm} {\small (b) Asymmetric Noise of $0.0$--$0.4$.} 
\vspace*{-0.35cm}
\caption{Label precision and label recall curves on CIFAR-10 with the shard\,($S=2$) setting.}
\label{fig:label-pr}
\vspace*{0.05cm}
\end{figure*}


\begin{figure}
\centering
\begin{subfigure}[b]{0.45\linewidth}
    \centering
    \includegraphics[width=\textwidth]{figures/evaluation/expertise_noniid_sym.pdf}
   
    \label{fig:expertise-sym}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
    \centering
    \includegraphics[width=\textwidth]{figures/evaluation/expertise_noniid_asym.pdf}
   
    \label{fig:expertise-asym}
\end{subfigure}
\\
\vspace*{-0.4cm}
\hspace*{0.55cm} {\small (a) Symmetric Noise.} \hspace*{2.0cm} {\small (b) Asymmetric Noise.}
\vspace*{-0.3cm}
\caption{Correlation between the training accuracy and noise rate on CIFAR-10 with shard\,($S=2$) of symmetric and asymmetric noises $0.0$--$0.4$.}
\vspace*{0.1cm}
\label{fig:expertise-noniid}
\end{figure}

\vspace*{-0.05cm}
\subsection{Robustness Comparison}
\label{sec:robust_comparison}

\subsubsection{Results with Synthetic Noise} 
Tables \ref{tab:cifar10} and \ref{tab:cifar100} show the test accuracy of the global model trained by eight methods for three Non-IID federated learning scenarios with four different noise settings, including symmetric, asymmetric, and mixed label noise. 
Overall, \algname{} achieves the highest test accuracy in every case.
Compared with FedAvg, \algname{}'s accuracy improves relatively more as the noise rate increases from $0.0$--$0.4$ to $0.0$--$0.8$ in symmetric noise, because the performance discrepancy over clients drastically degrades with the increase in noise rate. 
In contrast, the other robust methods show considerably poor performance compared with \algname{}, which is presumably attributed to the client-to-client performance discrepancy, despite the fact that they are incorporated with FedAvg.
Quantitatively, \algname{}\,($k=2$) significantly improves the test accuracy compared with other robust methods by $4.1$--$30.3$, $1.8$--$10.0$, and $0.3$--$6.8$ for shard\,($S\!=\!2$), shard\,($S\!=\!5$), and Dirichlet ($\beta\!=\!0.5$) settings in CIFAR-10, and by $0.2$--$13.9$ for the shard\,($S\!=\!20$) setting in CIFAR-100, respectively.

\subsubsection{Results with Real-world Noise} 
Table \ref{tab:webvision} displays the validation accuracy of seven different learning methods on a real-world noisy mini-WebVision dataset.
We report the top-1 and top-5 classification accuracy on the mini-WebVision validation set.
\algname{} maintains its performance dominance\footnote{Unlike the results regarding synthetic noise, the relatively small improvement is because all the clients have similar levels of label noise, which is more manageable.} over multiple state-of-the-art robust methods for \textit{real-world} label noise as well. \algname{}\,($k=2$) improves the top-1 accuracy by up to $9.2$ compared with other robust methods in mini-WebVision.



\subsection{In-depth Analysis on Selected Examples} 
\label{sec:selection_comparison}



All methods except FedAvg and Joint Optimization follow the pipeline of learning with sample selection. Hence, we evaluate the sample selection performance of them using the \textit{two} metrics, namely label precision\,(LP) and recall\,(LR), which respectively represent the quality and quantity of examples selected as clean\,\cite{han2018co, song2021robust}, 
{
\begin{equation*}
{\rm LP}=\frac{|\{(x, \tilde{y}) \in \mathcal{S}_{\sf c} : \tilde{y} = y^{*} \}|}{|\mathcal{S}_{\sf c}|}, ~ {\rm LR} = \frac{|\{(x, \tilde{y}) \in \mathcal{S}_{\sf c} : \tilde{y} = y^{*} \}|}{|\{(x, \tilde{y}) \in \mathcal{D}_{\sf c} : \tilde{y} = y^{*} \}|},
\end{equation*}
}
where $\mathcal{D}_{\sf c}$ and $\mathcal{S}_{\sf c}$ denote the training data and its selected clean set of the client ${\sf c}$, respectively. 

Figure \ref{fig:label-pr} shows the LP and LR curves averaged across all the clients per epoch on CIFAR-10 with symmetric and asymmetric label noise. The results of Robust FL are omitted due to its too low label precision and recall under the Non-IID scenario. After the warm-up period of $100$ transmission rounds, \algname{} shows the highest LP and LR with a large improvement of up to $0.12$ and $0.11$, respectively. In addition, its dominance over other robust learning methods is consistent regardless of the noise type. Therefore, these results indicate that the superior robustness of \algname{} in Tables \ref{tab:cifar10}--\ref{tab:webvision} is due to its high LP and LR. 

\begin{figure}
\centering
\begin{subfigure}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{figures/evaluation/sim_clean.pdf}
\label{fig:sim-clean}
\vspace*{-0.5cm}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{figures/evaluation/sim_noisy.pdf}
\label{fig:sim-noisy}
\vspace*{-0.5cm}
\end{subfigure}
\\
{\small (a) w.o Label Noise.} \hspace*{2.0cm} {\small (b) w. Label Noise.}
\vspace*{-0.25cm}
\caption{Similarity matrix between the client's softmax output for a Gaussian random noise when trained on CIFAR-10 with shard\,($S=2$) without noise (a) and with symmetric noise $0.0$--$0.4$ (b).}
\vspace*{-0.15cm}
\label{fig:similarity}
\end{figure}

\vspace*{-0.15cm}
\subsection{In-depth Analysis on Reliability Metric}
\label{sec:in_depth_metric}
\vspace*{0.1cm}

We justify the use of two indicators for data expertise and similarity in Eqs.\,\eqref{eq:expertise} and \eqref{eq:similarity}. 



\begin{itemize}[leftmargin=*]
\item \underline{Data Expertise:} Figure \ref{fig:expertise-noniid} shows the training convergence rate (training accuracy) of all clients sorted in ascending order by their noise rate. Owing to the memorization effect of DNNs, the training accuracy of the local model indeed has a strong correlation of over $0.93$ with respect to varying levels of label noise. These results confirm that the client's training accuracy represents its data expertise.
\vspace*{0.05cm}
\item \underline{Data Similarity:} Figure \ref{fig:similarity} shows similarity matrices between client's softmax outputs for a Gaussian random noise when trained on data with or without label noise. As clients with similar Ids\,(indices) have similar data distributions in our Non-IID setting, the similarity around diagonal entries is high, as can be seen from Figure \ref{fig:similarity}(a). Even with label noise shown in Figure \ref{fig:similarity}(b), it turns out that the similarity trend remains. Therefore, it can be concluded that the similarity between the softmax output is a robust metric to find clients with similar data distributions.  
\end{itemize}
These empirical studies provide strong evidence for the use of our two indicators, ${\rm Exp}(\cdot)$ and ${\rm Sim}(\cdot,\cdot)$.


\subsection{Reliable Neighbors with Fine-tuning}
\label{sec:analysis_fine_tune}
\vspace*{0.05cm}

We analyze the effect of fine-tuning on data expertise and similarity used in neighbor search.
Table \ref{tab:ablation-finetune} summarizes the performance of \algname{}\,($k=2$) {with} and {without} fine-tuning, where a small $\alpha$ indicates that data similarity is considered more than data expertise.
When fine-tuning is deactivated, the benefit of retrieving neighbors with high data similarity is remarkably dominant, because lower data similarity implies a larger distribution shift between the target and neighbor models.
On the other hand, an opposite trend is observed when fine-tuning is activated. The test accuracy is high when data expertise is properly considered in conjunction with data similarity. Overall, fine-tuning does not only help mitigate the distribution shift problem but also significantly improves performance of \algname{}. 

\input{tab-fine-tuning}

\vspace*{-0.1cm}
\subsection{Ablation Study on $k$-Reliable Neighbors}
\label{sec:aba_neighbor}
\vspace*{0.05cm}

A larger number of reliable neighbors could increase the communication overhead in federated learning. However, it turns out that a small number\,($k \leq 2)$ of reliable neighbors is sufficient to achieve high robustness without compromising the efficiency of learning. More specifically, we investigated the change in performance of the global model according to the the number of reliable neighbors. Table \ref{tab:ablation-neighbor} and \ref{tab:cifar100-neighbor} show the classification accuracy on CIFAR-10 and CIFAR-100 respectively with varying number of neighbors.

In Tables \ref{tab:ablation-neighbor} and \ref{tab:cifar100-neighbor}, \algname{} with $k=1$ and $k=2$ generally shows comparable performance to when $k>2$ reliable neighbors are used; there is little improvement in performance with the increase in the number of reliable neighbors as long as $k>1$. In other words, using only one or two reliable neighbors is sufficient to improve the robustness because the proposed reliability score successfully retrieves the most helpful neighbor client among all the neighbors. Therefore, in a scenario where the communication burden is significant, practitioners may reduce the communication overhead by only using a single reliable neighbor with \algname{} to handle the problem of noisy labels. 
A more detailed analysis of FedRN can be found in Appendices \ref{sec:random_analysis} and \ref{sec:cost_analysis}.





\begin{comment}

We validated the justification for the use of expertise and similarity used to calculate reliable scores. First of all, we performed the verification of the similarity between client representations. We used 50 clients. And we created ten groups by grouping five clients together. And by assigning two labels to each $i$-th group with the following rule, one label overlapped:
\begin{equation*}
\text{Assign two labels } i \text{ and } (i + 1) \% 10
\end{equation*}
To each group, symmetric noise was applied to clients from 0\% to 40\%, respectively. After 100 rounds of FedAvg algorithm, the similarity between local models was measured. As can be seen from Figure \ref{fig:similarity}, it can be seen that the similarity calculation is performed well in spite of the client's noisy labels.

We also experimentally verified that train accuracy can be used as the client's expertise. We measured the train accuracy of each client immediately after warm-up at a noise rate setting of 40\% for 100 clients. As can be seen in Figure \ref{fig:expertise-noniid}, the ratio of clean data that the client has and the train accuracy measured by the local model show a very high correlation. From this, we can confirm that the client's train accuracy can represent the client's expertise in labeling.
\end{comment}



\section{Conclusion and Future Work}
\label{sec:conclusion}
\vspace*{0.05cm}

\begin{comment}
- 본 연구의 중요성 다시 언급
- 우리의 방법론
- future works
    - expertise를 고려한 client selection과 aggregation에서 성능 부스트가 가능함을 이야기
    - adaptive alpha?
\end{comment}

The scenario of considering both data heterogeneity and label noise is a new important research direction of practical federated learning.
In this paper, we propose a novel federated learning method called FedRN. In the local update phase, $k$-reliable neighbors with high data expertise or similarity are retrieved, and their ensembled GMMs are used to identify clean examples. The models are then trained on the clean set, and the weights are averaged in the model aggregation phase. 

\input{tab-ablation-neighbor}
\input{tab-ablation-neighbor-cifar100}

In particular, the proposed data expertise and similarity scores help retrieve useful reliable neighbors to identify clean examples from noisy data in the client side. As a result, \algname{} achieves high label precision and recall for sample selection, leading to a robust global model even in the presence of label noise. We conducted extensive experiments on three real-world or synthetic noisy datasets. The results verified that \algname{} improves the robustness against label noise consistently and significantly. 

We mainly focused on sample selection in the local update phase with $k$-reliable neighbors, but there could be many possible extensions of the proposed reliability metrics. The expertise score can be exploited when selecting clients in the client selection phase. Instead of selecting target clients uniformly at random from the entire client pool, we can adjust the probability of a client being selected based on the client's expertise score; thus, the higher the expertise score is, the higher the probability of being selected. Likewise, the expertise score can also be used as the model aggregating weight in the model aggregation phase. Through this aggregation approach, higher weights can be assigned to more reliable clients. 

Another extension of \algname{} is the combination with semi-super-vised learning. As \algname{} only uses selected clean examples for robust learning, the remaining examples are simply discarded. To fully use the entire dataset during training, we can treat the noisy samples as the unlabeled set and apply a semi-supervised learning technique, such as MixMatch\,\cite{berthelot2019mixmatch} and UDA\,\cite{xie2019unsupervised}. However, note that the combination with semi-supervised learning is not straightforward considering that the performance of a semi-supervised learning-based approach, DivideMix, was worse than \algname{}'s. We believe that there are two challenges for this extension: (1) the performance of client models is insufficient to perform pseudo labeling for unlabeled data, and (2) an inconsistency exists between client and global models when semi-supervised learning is applied. 



\section{A ~~Experiment Configuration Details}










\section{Grid Search for Balancing Term $\alpha$} 
\label{sec:parameter_search}

We adjusted $\alpha$ to determine the contribution of data expertise and similarity in neighbor search. To ascertain the optimal value, we repeated the experiment on CIFAR-10 with various $\alpha$ values, as summarized in Table \ref{tab:ablation-alpha}. Because the fine-tuning technique is incorporated in \algname{}, merging data expertise and similarity provides a synergistic enhancement in general. In particular, when $\alpha = 0.6$, the performance gain is the highest on average. 
Based on the two studies on our hyperparameters, we set $\alpha$ to $0.6$ for all experiments, respectively.

\input{tab-ablation-alpha}

\input{tab-ablation-random}

\vspace*{-0.1cm}
\section{Comparison with Random Neighbors\!\!\!\!\!}
\label{sec:random_analysis}

We analyzed the usefulness of $k$-reliable neighbors compared with $k$-random neighbors. For $k$-random neighbors, we replaced Lines 6 and 10 of Algorithm \ref{alg:fedrn} with random client selection.
Table \ref{tab:ablation-random} summarizes the test accuracy of \algname{} with $k$-reliable and $k$-random neighbors. 
Since our fine-tuning technique was incorporated with $k$-random neighbors, using $k$-random neighbors improves the robustness against noisy labels, but \algname{} with $k$-reliable neighbors maintains its dominance in every case.
In detail, if the neighbors with low data expertise are selected, they misclassify mislabeled examples as clean examples in sample selection. Likewise, the neighbors with significantly different data distributions harm identifying clean examples from the target client’s noisy data due to the distribution shift problem. It can be confirmed experimentally that this issue becomes more prominent when the noise rate is high. In conclusion, we should leverage the reliability of neighbors to obtain a global model robust to noisy labels.

\vspace*{-0.1cm}


\section{Detailed Cost Analysis}
\label{sec:cost_analysis}

We analyze the communication and computation cost of \algname{} compared with other {six} methods, as summarized in Table \ref{tab:analysis-cost}.
In this comparison, the communication cost is split into two parts, \enquote{server$\rightarrow$client} and \enquote{client$\rightarrow$server,} while the computation cost is split into two other parts, \enquote{forward} and \enquote{backward,} and the \enquote{total cost} aggregates all the costs for each perspective. 

\begin{itemize}[leftmargin=*]
\item \underline{Communication Cost:} \algname{} requires {3M} or {4M} communication cost in total when using one or two reliable neighbors, which is a sufficient number for general use cases. The remaining communication cost for calculating data similarity and expertise\,(i.e., $Acc({\sf c})$ and $p(\tilde{x};\Theta_{\sf c})$) is negligible. By contrast, Co-Teaching and DivideMix both require 4M communication cost in total because they maintain two models for co-training. In summary, \algname{} leads to greater performance with comparable or lower communication cost compared with other baseline methods.
\smallskip

\footnotetext[7]{If the number of clients is large, a large computation cost may be required to search the $k$-reliable neighbor. However, by using the \textit{KD-Tree algorithm}, the computation cost can be greatly reduced\,\cite{jeong2020federated}.}

\item \underline{Computation Cost:} We exclude the cost of fitting GMMs and searching $k$-reliable neighbors\footnotemark[7] because their costs are negligible compared to those of the forward and backward steps. Along the pipeline of our local update phase, \algname{} first requires $k$F and $k$B computation cost to perform fine-tuning with $k$-reliable neighbors for one epoch. Next, to construct the clean set, $(k+1)$F communication cost is needed due to the inference with $k+1$ available models. As the constructed clean set is deterministic over all the local epochs, only $E$F+$E$B computation cost is required to train a single model for the given local epoch $E$ in each communication round. Consequently, \algname{} needs $(E+2k+1)$F + $(k+E)$B computation cost in total. 
In a typical deep learning pipeline, the cost of the backward step is relatively smaller than that of the forward step. Hence, the computation cost for the forward step is the major issue. In light of this fact, the forward cost of \algname{} is comparable to those of FedAvg, Joint-optim, and SELFIE, and less than those of Co-teaching and DivideMix, because the number of local epochs is larger than the number of reliable neighbors, $E > k$. Among the compared methods, DivideMix is the slowest method in that it performs $m$ data augmentations with two models for each local epoch, so its forward cost is $(4m+2)E$F. 
\end{itemize}

Therefore, with one or two reliable neighbors, \algname{} greatly improves the robustness of the global model without adding much communication and computation cost in the scenario of federated learning with noisy labels.

\clearpage


\input{tab-analysis-cost}






\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  











