
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[acmlarge]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[acmsmall]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[acmsmall]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[acmtog]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf,authordraft]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[manuscript,screen]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigchi-a]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{margintable}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{margintable}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{marginfigure}
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{marginfigure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigchi]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigplan,screen]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{The \textsc{Prince}\xspace Algorithm}
\label{subsec:algo}

\begin{figure*}[t]
	\begin{subfigure}[t]{0.3\textwidth}
		\includegraphics[width=\textwidth]{pixie-method-1}
		\caption{\scriptsize{$PPR(n_1, n_4) = 0.160
				\\ PPR(n_1, n_5) = 0.085
				\\ PPR(n_1, n_4) > PPR(n_1, n_5)$}}
		\label{fig-a}
	\end{subfigure}\hfill
	\begin{subfigure}[t]{0.3\textwidth}
		\includegraphics[width=\textwidth]{pixie-method-2}
		\caption{\scriptsize{$A = \{(n_1, n_2), (n_1, n_3)\}
				\\ W(n_1 , n_2) [PPR(n_2, n_4| A) - PPR(n_2, n_5 | A)] = 0.095
				\\ W(n_1 , n_3) [PPR(n_3, n_4 | A) - PPR(n_3, n_5 | A)] = -0.022$}}
		\label{fig-b}
	\end{subfigure}\hfill
	\begin{subfigure}[t]{0.3\textwidth}
		\includegraphics[width=\textwidth]{pixie-method-3}
		\caption{\scriptsize{$ A^*  = \{(n_1, n_2)\}
				\\ PPR(n_1, n_4 | A^*) = 0.078
				\\ PPR(n_1, n_5 | A^*) = 0.110
				\\ PPR(n_1, n_5 | A^*) > PPR(n_1 , n_4| A^*)$}
		}
		\label{fig-b}
	\end{subfigure}
	\caption{Toy Example. (a) A weighted and directed graph where
		the $PPR$ scores are personalized for node $n_1$. Node $n_4$ has
		higher $PPR$ than $n_5$.
		(b) Scores in a graph configuration where outgoing edges
		$(n_1, n_2)$, and $(n_1, n_3)$ are removed (marked in red).
		(c) Removing $(n_1, n_2)$ causes $n_5$ to outrank $n_4$.} 
	\vspace*{-0.3cm}
	\label{fig:pixie-method}

\end{figure*}

\setlength{\textfloatsep}{0.1cm}
\setlength{\floatsep}{0.1cm}
\begin{algorithm} [t]
	\small
	\DontPrintSemicolon
	\SetAlgoLined
	\SetKwInput{Input}{Input}\SetKwInput{Output}{Output}
	\SetKwComment{Comment}{$\triangleright$\ }{}
	\Input{$G=(V, E, \theta)$, $I \subset V$, $u \in V$, $rec \in I$}
	\Output{$A^*$ for $(u, rec)$}
	$A^* \gets A$ \;
	$rec^* \gets rec$\;
	
	\ForEach {$i \in I$} {\label{alg:mainstart}
		$A^i\gets \texttt{SwapOrder}(G, u, rec, i)$ \;
		\tcp*[l]{Actions $A^i$ swap orders of $rec$ and $i$}
		\If { $|A^i| < |A^*|$}
		{
			$A^* \gets A^i$ \;
			$rec^* \gets i$
		}
	
	\ElseIf {$|A^i| = |A^*|$ and $PPR(u, i|A^i) > PPR(u, rec^*|A^i)$}
	{
		\label{alg:repstart}
		$A^* \gets A^i$ \;
		$rec^* \gets i$
		\label{alg:repend}
	}
	}\label{alg:mainend}

	\Return $A^*, rec^*$ \;
	\BlankLine
	\SetKwFunction{FMain}{SwapOrder}
	\SetKwProg{Fn}{Function}{:}{end}
	\Fn{\FMain{$G, u, rec, rec^*$}}{
		$A \gets \{(u, n_i)| n_i \in N_{out}(u), n_i \neq u\}$ \;
		$A^* \gets \emptyset$ \;
		$H\gets \text{MaxHeap}(\phi)$ \;
		$sum \gets 0 $ \;
		\ForEach {$(u, n_i) \in A$} { \label{alg:sortstart}
			$diff \gets W(u, n_i) \cdot (PPR(n_i, rec|A) - PPR(n_i, rec^*|A))$\;
			$H.insert(n_i, diff)$ \;
			$sum \gets sum + diff$
		}\label{alg:sortend}
		\While {$sum > 0 \text{ and } |H| > 0$} {\label{alg:updatestart}
			$(n_i, diff) \gets H.delete\_max()$ \;
			$sum \gets sum - diff$ \;
			$ A^* \gets  A^* \cup (u, n_i)$
		}\label{alg:updateend}
		\lIf {$sum > 0$} {$A^* \gets A$}
		\Return $A^*$
	}


	\caption{\textsc{Prince}\xspace}
\label{alg:poly}	
\end{algorithm}	
\setlength{\textfloatsep}{0.1cm}
\setlength{\floatsep}{0.1cm}
	
In this section, we develop an algorithm for
computing a minimum counterfactual explanation for user $u$
receiving recommended item $rec$, given the PPR-based
recommender framework $RecWalk$~\cite{nikolakopoulos2019recwalk}. 
A na\"{i}ve optimal algorithm enumerates all subsets of
actions $A^* \subseteq A$, 
and checks whether the removal of each of these subsets
replaces $rec$ with
a different item as the top recommendation,
and finally selects the subset with the minimum size.
This approach is exponential in the number of actions of the user. 

To devise a more efficient and practically viable algorithm,
we express the $PPR$ scores as follows~\cite{jeh2003scaling}, with
$PPR(u,rec)$ denoting the PPR of $rec$
personalized for $u$ (i.e., jumping back to $u$):
\begin{equation}
	PPR(u, rec) = (1-\alpha)\sum_{n_i \in N_{out}(u)} W(u, n_i)
	PPR(n_i, rec) + \alpha \delta_{u,rec}   
	\label{eq:invariant}
\end{equation}
where $\alpha$ denotes the teleportation probability (probability 
of jumping back to $u$) and $\delta$ is the Kronecker delta function. The only
required modification, with regard to 
$RecWalk$~\cite{nikolakopoulos2019recwalk}, is the transformation of
the transition probability matrix from $W$ to $W^{\beta}$. 
For simplicity, we will refer to the adjusted probability matrix as $W$.

Eq.~\ref{eq:invariant} shows that the PPR of $rec$ personalized for user $u$,
$PPR(u,rec)$, is a function of the PPR values of $rec$ personalized for
the neighbors of $u$. 
Hence, in order to decrease $PPR(u, rec)$, 
we can remove edges $(u, n_i), n_i \in N_{out}(u)$.
To replace the recommendation $rec$ with 
a different item $rec^*$, a simple heuristic 
would remove edges $(u, n_i)$ in non-increasing order of their contributions
$W(u, n_i) \cdot PPR(n_i, rec)$. 
However, although this would reduce the PPR of $rec$,
it also affects and possibly reduces the PPR of other items, too,
due to the recursive nature of PPR, where all paths matter.

 
Let $A$ be the set of outgoing edges of a user $u$ and let $A^*$
be a subset of $A$, such that $A^* \subseteq A$. The main intuition behind our 
algorithm is that we can express
$PPR(u, rec)$ after the removal of $A^*$,
denoted by $PPR(u, rec|A^*)$, as a function of two 
components: $PPR(u, u|A^*)$ and the values $ PPR(n_i, rec|A)$,
where $n_i \in \{ n_i| (u, n_i) \in A \setminus A^*\}$
and $n_i \neq u$. 
The score $PPR(u, u|A^*)$ does not depend on $rec$, and
the score $PPR(n_i, rec|A)$ is independent of
$A^*$. 

Based on these considerations, we present Algorithm~\ref{alg:poly}, 
proving its correctness in Sec.~\ref{sec:correct}.
Algorithm~\ref{alg:poly} takes as input a graph $G$, a user $u$,
a recommendation $rec$, and a set of items $I$.
In lines~\ref{alg:mainstart}-\ref{alg:mainend}, we iterate through
the items $I$, and find the minimum counterfactual explanation $A^*$.
Here, $A^i$ refers to 
the actions whose removal swaps the orders of items $rec$ and $i$.
In addition, we ensure that after removing $A^*$, we return the item with
the highest PPR score as the replacement item
(lines~\ref{alg:repstart}-\ref{alg:repend}).
Note that in the next section, we propose an equivalent formulation for
the condition $PPR(u, i | A^i) > PPR(u, rec^* | A^i)$, eliminating the need
for recomputing scores in $G \setminus A^*$.   

The core of our algorithm is
the function \texttt{SwapOrder}, which receives as input two items,
$rec$ and $rec^*$, and a user $u$. In
lines \ref{alg:sortstart}-\ref{alg:sortend}, we sort
the interaction edges $(u, n_i) \in A$ in non-increasing
order of their contributions
$W(u, n_i) \cdot (PPR(n_i, rec|A) - PPR(n_i, rec^*|A))$.
In lines \ref{alg:updatestart}-\ref{alg:updateend}, we remove
at each step, the outgoing interaction edge with the highest contribution,
and update $sum$ and $A^*$ correspondingly. The variable $sum$ is strictly positive
if in the current graph configuration ($G \setminus A^*$),
$ PPR(u, rec) > PPR(u, rec^*)$.
This
constitutes the main building block of our approach.
Fig.~\ref{fig:pixie-method} illustrates the execution of
Algorithm~\ref{alg:poly} on a toy example. 		

The time complexity of the algorithm is $O(|I| \times |A| \times \log |A|)$,
plus the cost of computing PPR for these nodes.
The key to avoiding the exponential cost of considering
all subsets of $A$ is the insight that \textit{we need only to compute
PPR values for alternative items with personalization 
based on a graph where all user actions $A$ are removed}.
This is feasible because the action deletions affect only
outgoing edges of the teleportation target $u$,
as elaborated 
in Sec.~\ref{sec:correct}.

The PPR computation could simply re-run a power-iteration
algorithm for the entire graph, or compute the principal
eigenvector for the underlying matrix. 
This could be cubic in the graph size (e.g., if we use
full-fledged SVD), but it keeps us in the regime of
polynomial runtimes. 
In our experiments, we use the much more efficient
reverse local push algorithm~\cite{andersen2007local} 
for PPR calculations.




\section{Correctness Proof}
\label{sec:correct}

We prove two main results:
\squishlist
\item[(i)] $PPR(u, rec|A^*)$  can be computed as a product of two
components where one depends on the modified graph
with the edge set $E \setminus A$ (i.e., removing all user
actions) and the other depends on the choice of
$A^*$ but not on the choice of $rec$.
\item[(ii)] To determine if some $A^*$ replaces
the top node $rec$ with a different node $rec^*$
which is not an out-neighbor of $u$, we need to
compute only the first of the two components in (i).
\squishend
\begin{theorem}
	Given a graph $G = (V, E)$, a node $u$ with outgoing edges
	$A$ such that $(u, u) \notin A$, a set of
	edges $A^* \subset A$, a node $rec \notin N_{out}(u)$, the PPR of $rec$
	personalized for $u$ in 
	the modified graph $G^* = (V, E \setminus A^*)$ can be expressed as follows:
	$$ PPR(u, rec|A^*) = PPR(u, u|A^*) \cdot f\big(\big\{PPR(n_i, rec|A)\big|
	(u, n_i) \in A \setminus A^*\big\}\big) $$
	where $f(\cdot)$ is an aggregation function. 
	\label{thm: pprs-rel}
\end{theorem}
\vspace*{-0.3cm}
\begin{proof}









Assuming that each node has at least one outgoing edge, the PPR can 
be expressed as the sum over the probabilities of 
walks of length $l$ starting at a node $u$~\cite{andersen2006local}:
	\begin{equation}
	PPR(u, \cdot) = \alpha\sum_{l=0}^\infty (1-\alpha)^l e_{u} W^l 
	\label{eq: ppr-walks}
	\end{equation}
	where $e_{u}$ is the one-hot vector for $u$. 





To analyze the effect of deleting $A^*$, we split the walks from $u$ to
	$rec$ into two parts, ($i$) the part representing the sum over
	probabilities of walks that start at $u$ and pass again by $u$,
	which is equivalent to $\alpha^{-1}PPR(u, u|A^*)$ (division by $\alpha$
	is required as the walk does not stop at $u$), and ($ii$) the part
	representing the sum over probabilities of walks starting at node $u$
	and ending at $rec$ without revisiting $u$ again,
	denoted by $p_{-u}(u, rec | A^*)$. 
	Combining these constituent parts, PPR can be stated as follows:
	\begin{equation}
	PPR(u, rec | A^*) = \alpha^{-1} PPR(u, u | A^*) \cdot p_{-u}(u, rec | A^*)
	\label{eq: new-ppr-0}
	\end{equation}
	




As stated previously, $p_{-u}(u, rec | A^*)$ represents the sum over
the probabilities of the walks from $u$ to $rec$ without revisiting $u$.
We can
express these walks using the remaining neighbors of $u$
after removing $A^*$:
\begin{equation}
	p_{-u}(u, rec | A^*) = (1 - \alpha)
	\sum_{(u, n_i) \in A \setminus A^*} W(u, n_i) \cdot p_{-u}(n_i, rec | A^*)
\end{equation}
where $p_{-u}(n_i, rec | A^*)$ refers to the walks starting at $n_i$ ($n_i \neq u$) 
and ending at $rec$ that do not visit $u$.
We replace $p_{-u}(n_i, rec | A^*)$ with its equivalent formulation 
$PPR(n_i, rec | A)$. $PPR(n_i, rec)$ in graph $G\setminus A$ is computed as
the sum over the probabilities of walks that never pass by $u$.
Eq.~\ref{eq: new-ppr-0} can be rewritten as follows:	 
\begin{align}

	& PPR(u, rec|A^*) \notag \\
	& =  PPR(u, u|A^*) \cdot \alpha^{-1} (1-\alpha)
	\sum_{(u, n_i) \in A \setminus A^*}
	W(u, n_i|A^*) PPR(n_i, rec|A)
\end{align}
This equation directly implies:
\begin{equation}
	PPR(u, rec|A^*) = PPR(u, u|A^*) \cdot f\big(\big\{PPR(n_i, rec|A)\big|
	(u, n_i) \in A \setminus A^*\big\}\big)

\end{equation}

\end{proof}
\vspace*{-0.3cm}
\begin{theorem}
The minimum counterfactual explanation for $(u, rec)$ 
can be computed in polynomial time. 
\label{thm: poly}
\end{theorem}
\vspace*{-0.3cm}
\begin{proof}
	

We show that there exists
a polynomial-time algorithm for finding the minimum set $A^* \subset A$
such that $ PPR(u, rec|A^*) < PPR (u, rec^*|A^*)$, if such a set exists. 
Using Theorem~\ref{thm: pprs-rel}, we show that one can compute if some
$rec^*$ can replace the original $rec$ as the top
recommendation, solely based on PPR scores from a single graph
where all user actions $A$ are removed:
\begin{align}
	 & PPR(u, rec|A^*) < PPR (u, rec^*|A^*) \notag \\






	 & \Leftrightarrow \sum_{(u, n_i) \in A \setminus A^*}  W(u, n_i|A^*)
	 \big(PPR(n_i, rec|A)  -  PPR(n_i, rec^*|A) \big) < 0  \notag \\

	 & \Leftrightarrow \sum_{(u, n_i) \in A \setminus A^*} W(u, n_i)\big
	 (PPR(n_i, rec|A)  -  PPR(n_i, rec^*|A) \big) < 0
	 \label{eq:equivalence}
\end{align}
	
The last equivalence is derived from:
\begin{equation}
	W(u, n_i|A^*) = \frac{W(u, n_i)}{1 - \sum_{(u, n_j) \in A^*} W(u, n_j)}
\end{equation}
	
For a fixed choice of $rec^*$, the summands in
expression~\ref{eq:equivalence}
do not depend on $A^*$, and so they are constants for all possible
choices of $A^*$.
Therefore, by sorting the summands in descending order, we can greedily
expand $A^*$ from a single action to many actions until some
$rec^*$ outranks $rec$. This approach is then guaranteed to arrive at
a minimum subset.



	
	
	
\end{proof}




\begin{comment}
\begin{algorithm}
	\small
	\DontPrintSemicolon
	\SetAlgoLined
	\SetKwInOut{Input}{Input}\SetKwInOut{Output}{Output}
	\Input{Interaction Graph $G=(V,E, W)$, user node $u$ and her actions 
		$A$, top recommendation $rec$ and items $\{x\}$}
	\Output{A minimum subset $A' \subset A$ whose removal displaces the 
		top-most recommendation $rec$, the new top-most recommendation $New\_rec$}
	\BlankLine
	\ForEach {$x \in \{x\}$} {
		$A'(x), S(x) = SwapRelativeRank(G, u, rec, x)$ \;
	}
	$A' = \argmin_{A'(x)} |A'(x)|$ \\
	$New\_rec = rec$ \\
	$s = -1$\\
	\If{$A' \neq \phi$}{
		\ForEach {$x \in \{x\}$} {
			\If{$|A'(x)| == |A'|$}{
				\If{$s < S(x)$}{
					$s = S(x)$\;
					$A' = A'(x)$\;
					$New\_rec = x$
				}
			}
		}	
	}
	
	Return $A'$ and $New\_rec$ \\
	\BlankLine


	\SetKwFunction{FMain}{SwapRelativeRank}
	\SetKwProg{Fn}{Function}{:}{}
	\Fn{\FMain{$G$, $u$, $rec$, $x$}}{
		Build $G'$ by removing all the outgoing edges of $u$\\
		$Q = \phi$\\
		\ForEach {OutNeighbor $n_i$ of $u$} {
			$v = W(u, n_i) (PPR(n_i, x|G') - PPR(n_i, rec|G'))$ \;
			Add pair $(n_i, v)$ 
			to $Q$\;
		}
		$A'=\phi$\\
		$S =$ Sum of values in $Q$\\
		Sort $Q$ ascendingly \\
		\While {$S < 0$ \text{and } $|Q| > 0$} {
			$n_i, v = Q.next()$ \;
			$S = S - v$\;
			Add $(u, n_i)$ to $A'$
		}
		\If{$S < 0$}{
			Return $\phi$ 
		}
		Return $A'$, $S$ 
	}
	\textbf{End Function}
	\caption{PPRCounterfactualExplanation}
	\label{alg: poly}
\end{algorithm}

\end{comment}

\section{Introduction}
\label{sec:intro}

\noindent\textbf{Motivation.} Providing user-comprehensible explanations for
machine learning models
has 
gained
prominence in multiple
communities~\cite{zhang2019ears,miller2019xai,rimchala2019kdd,zhang2019cvpr}.
Several 
studies have shown that explanations
increase users' trust in systems that
generate personalized recommendations or other rankings
(in news, entertainment, etc.)
~\cite{ribeiro2016should,kunkel2019let,kouki2019personalized}.
Recommenders have become very sophisticated, exploiting signals from
a complex interplay of factors
like users' activities, interests and social links~\cite{zhang2018explainable}.
Hence the pressing need for explanations.

Explanations for recommenders
can take
several forms, 
depending on the generator (\textit{explanations by whom?}) and
the consumer (\textit{explanations for whom?}). As generators, only
\textit{service providers} can produce 
true explanations for how systems compute
the recommended items~\cite{balog2019transparent,zhang2014explicit,
	wang2018explainable};
\textit{third parties} can merely discover relationships
and create post-hoc rationalizations for black-box models that may look
convincing to 
users~\cite{ghazimatin:19,peake2018explanation,wang2018reinforcement}.
On the consumer side, \textit{end-users} can grasp tangible aspects like
activities, likes/dislikes/ratings or demographic factors.
Unlike {system developers} or accountability engineers, end-users would obtain 
hardly any insight from transparency of internal system workings.
In this work, we deal with explanations
\textit{by the provider and for the end-user}.

\begin{figure} [t]
	\centering
	\includegraphics[width=\columnwidth]{images/pixie.pdf}

	\caption{\textsc{Prince}\xspace generates explanations as a minimal set of actions
		using counterfactual evidence on user-specific HINs.}
	\label{fig:pixie}
	\vspace*{-0.7cm}
\end{figure}

\vspace{0.3cm}
\noindent\textbf{Limitations of state-of-the-art.} At the core of most recommender
systems 
is some variant of matrix or tensor decomposition (e.g., \cite{koren2009matrix})
or spectral graph analysis
(e.g., \cite{DBLP:conf/kdd/JamaliE09}), with various forms of regularization
and often involving gradient-descent methods for parameter learning.
One of the recent and popular paradigms is based on 
\textit{heterogeneous information networks (HIN)}
\cite{yu2014personalized,yu2013recommendation,
	DBLP:journals/tkde/ShiLZSY17,zhang2019shne},
a powerful model
that represents relevant entities and actions
as a directed and weighted graph with multiple
node and edge types. 
Prior efforts towards explanations for HIN-based recommendations have mostly
focused on \textit{paths} that connect the user with the recommended item
~\cite{shi2015semantic,ghazimatin:19,ai2018learning,wang2018ripplenet,
wang2019explainable,xian2019reinforcement,DBLP:conf/icdm/YangLWH18}.
An application of path-based explanations, for an online shop, would
be of the form: 
\squishlist
\item[ ] User $u$ received item $rec$ because $u$ follows user $v$,
who bought item $j$,
which has the same category as $rec$.
\squishend
However, such methods come with critical
privacy concerns arising from nodes in paths that disclose other users'
actions or interests to user $u$, like the purchase of user $v$
above.
Even if user $v$'s id was anonymized, user $u$ would know whom
she is following and could often guess who user $v$ actually is,
that bought item $j$,
assuming that $u$ has a relatively small set
of followees~\cite{machanavajjhala2011personalized}.
If entire paths containing other users are suppressed instead, then
such explanations would no longer be faithful to the true cause.
Another family of
path-based methods~\cite{ghazimatin:19,peake2018explanation,wang2018reinforcement}
presents plausible connections
between users and items as justifications.
However, this is merely post-hoc rationalization,
and not actual causality.


\vspace{0.3cm}
\noindent\textbf{Approach.} This paper presents \textsc{Prince}\xspace, 
a method for
\textbf{Pr}ovider-side \textbf{In}terpretability with \textbf{C}ounterfactual
 \textbf{E}vidence, that
overcomes 
the outlined limitations.
\textsc{Prince}\xspace is a provider-side solution
aimed at detecting the actual cause responsible for the recommendation, in 
a heterogeneous information network with users, items, reviews, and categories.
\textsc{Prince}\xspace's explanations are grounded in the user's own actions, and
thus preclude privacy concerns of path-based models.
 Fig.~\ref{fig:pixie} shows an illustrative example. Here, 
Alice's actions like bought shoes, reviewed a camera, and rated
a power bank are deemed as explanations for her
backpack recommendation.
One way of identifying a user's actions for an explanation would be
to compute scores of actions with regard to the recommended item.
However, this would be an unwieldy distribution over potentially
hundreds of actions -- hardly comprehensible to an end-user. 
Instead, we operate in
a \textit{counterfactual} setup~\cite{martens2014explaining}.
\textsc{Prince}\xspace
identifies a small (and actually minimal) set
of a user's actions such that removing these actions would result in
replacing the recommended item with a different item.
In Fig.~\ref{fig:pixie}, the item \textit{rec =} ``Jack Wolfskin backpack''
would be replaced, as the system's top recommendation,
by $i_3=$``iPad Air'' (the $i$'s represent candidate replacement items).
Note that there may be multiple such minimal sets, but uniqueness is not
a concern here. 

Another perspective here is that the goal of an explanation is often
to show users
\textit{what they can do}
in order to receive more relevant recommendations. Under this claim, the end-user
has no \textit{control} on the network beyond her immediate neighborhood, i.e.,
the network beyond is \textit{not actionable} (shaded zone in
Fig.~\ref{fig:pixie}), motivating \textsc{Prince}\xspace's choice
of grounding explanations in users' own actions.

For true explanations, we need to commit ourselves to a specific family
of recommender models. In this work, we choose a general framework
based on \textit{Personalized PageRank
(PPR)}, as used in the state-of-the-art RecWalk
system~\cite{nikolakopoulos2019recwalk}, and adapt it to
the HIN setup. 
The heart of \textsc{Prince}\xspace is a polynomial-time algorithm for exploring
the (potentially exponential) search space of subsets of user actions --
the candidates for
causing the recommendation.
The algorithm efficiently computes PPR contributions
for groups of actions with regard to an item, by adapting the
reverse
local push algorithm of \cite{andersen2007local} 
to a dynamic graph setting~\cite{zhang2016approximate}.
In summary, the desiderata for the explanations from \textsc{Prince}\xspace (in \textbf{bold})
connect to the technical approaches adopted (in \textit{italics})
in the following ways. Our explanations are:
\squishlist
\item \textbf{Scrutable}, as they are derived in a \textit{counterfactual} setup;
\item \textbf{Actionable}, as they are grounded in the user's
own \textit{actions};
\item \textbf{Concise}, as they are \textit{minimal} sets changing
a recommendation.
\squishend

Extensive experiments with Amazon and Goodreads datasets show 
that \textsc{Prince}\xspace{}'s minimal explanations, achieving the desired
item-replacement effect, cannot be easily obtained by heuristic methods based
on contribution scores and shortest paths.
A crowdsourced user study on Amazon Mechanical Turk (AMT) provides additional
evidence that \textsc{Prince}\xspace{}'s explanations 
are more useful
than ones based on paths~\cite{DBLP:conf/icdm/YangLWH18}.
Our code is public at \url{https://github.com/azinmatin/prince/}.

\vspace{0.3cm}
\noindent\textbf{Contributions.} Our salient contributions in this work are:
\squishlist
	\item \textsc{Prince}\xspace is the first work that explores counterfactual evidence
	for discovering causal explanations in a heterogeneous information network;
	\item \textsc{Prince}\xspace is the first work that defines explanations for recommenders
	in terms of users' own actions; 
	\item We present an optimal algorithm that explores the search space
	of action subsets in polynomial time, for efficient computation
of a minimal subset of user actions;
	\item Experiments with two large datasets and a user study
show that \textsc{Prince}\xspace  can effectively aid a service provider
in generating user-comprehensible causal explanations
for recommended items.
\squishend

\section{Related Work}
\label{sec:related}

Foundational work 
on explainability for collaborative-filtering-based recommenders
was done by Herlocker et al.~\cite{herlocker2000explaining}. Over time, 
generating
explanations (like~\cite{DBLP:conf/icdm/YangLWH18}) has become
tightly coupled with building 
systems that are geared for producing more transparent 
recommendations (like~\cite{balog2019transparent}). 
For broad surveys, see
\cite{zhang2018explainable,tintarev2007survey}.
With methods
using matrix or tensor
factorization~\cite{zhang2014explicit,chen2016learning,wang2018explainable},
the goal has been to make latent factors more tangible.
Recently, interpretable neural models have become popular, especially for 
text~\cite{seo2017interpretable,chen2018neural,chen2018sequential} and
images~\cite{chen2019personalized}, where the attention mechanism
over words, reviews, items, or zones in images
has been
vital for interpretability. Efforts have also been made on generating readable 
explanations using models like LSTMs~\cite{li2017neural}
or GANs~\cite{lu2018like}.

Representing users, items, categories and reviews as
a knowledge graph or a heterogeneous information network (HIN) has 
become popular, where explanations
take the form of paths between the user and an item. This paradigm
comprises a variety of mechanisms:
learning path embeddings~\cite{ai2018learning,wang2018explainable},
propagating user preferences~\cite{wang2018ripplenet},
learning and reasoning with explainable 
rules~\cite{ma2019jointly,xian2019reinforcement},
and ranking user-item connections~\cite{DBLP:conf/icdm/YangLWH18,ghazimatin:19}. 
In this work,
we choose the recent approach in~\cite{DBLP:conf/icdm/YangLWH18} as
a representative for the family of path-based recommenders to compare
\textsc{Prince}\xspace with.
Finally, post-hoc or model-agnostic rationalizations for black-box models have 
attracted interest.
Approaches
include association rule mining~\cite{peake2018explanation},
supervised ranking of user-item relationships~\cite{ghazimatin:19},
and reinforcement learning~\cite{wang2018reinforcement}.

Random walks over HIN's have been pursued by
a suite of works, including \cite{DBLP:reference/rsh/DesrosiersK11,DBLP:conf/www/CooperLRS14,DBLP:conf/recsys/ChristoffelPNB15,DBLP:conf/www/EksombatchaiJLL18,jiang2018recommendation}.
In a nutshell, the Personalized PageRank (PPR) of an item node in the HIN 
is used as a ranking criterion for recommendations.
\cite{nikolakopoulos2019recwalk} introduced 
the RecWalk method, proposing a random walk
with a nearly uncoupled Markov chain.
Our work uses this framework.
As far as we know, we are the 
first to study the problem of computing minimum subsets of edge removals
(user actions) to change the top-ranked node in a counterfactual setup.
Prior research on dynamic graphs, 
such as \cite{csaji2014pagerank,kang2018aurora},
 has addressed related issues, but not this very problem.
A separate line of research focuses on the efficient computation of PPR.
Approximate algorithms include power iteration~\cite{page1999pagerank},
local push~\cite{andersen2006local, andersen2007local, zhang2016approximate}
and Monte Carlo methods~\cite{avrachenkov2007monte, bahmani2010fast}. 

\subsection{Results and Insights}

\begin{table} [t]
	\centering

	\begin{tabular}{l c c c }
		\toprule 
		\textbf{Method}								& \textbf{Mean} & \textbf{Std. Dev.}	& \textbf{\#Samples}	\\ \toprule
		\textsc{Prince}\xspace 									& $1.91$* 		& $0.66$ 				& $200$ 				\\
		CredPaths~\cite{DBLP:conf/icdm/YangLWH18}	& $1.78$ 		& $0.63$ 				& $200$ 				\\ \midrule
		\textsc{Prince}\xspace (Size=$1$) 							& $1.87$		& $0.66$ 				& $154$ 				\\ 
		\textsc{Prince}\xspace	(Size=$2$)							& $1.88$* 		& $0.70$ 				& $28$					\\
		\textsc{Prince}\xspace	(Size=$3$)							& $2.21$*		& $0.52$ 				& $18$					\\ \bottomrule
	\end{tabular
	\caption{Results from the AMT measurement study on usefulness
		conducted on the Amazon data.
	
	
		An asterisk (*) indicates statistical significance of \textsc{Prince}\xspace over
		CredPaths ($1$-tailed paired $t$-test at $p < 0.05$).}
	\label{tab:amt-results}

\end{table}

\begin{table} [t] 
	\centering
	\resizebox*{\columnwidth}{!}{
	
		\begin{tabular}{l p{8 cm}}
			\toprule
			\textbf{Method}		&	\textbf{Explanation for ``Baby stroller''
									with category ``Baby'' [Amazon]} 													\\	\toprule
			\textsc{Prince}\xspace				&	\textbf{Action 1:} You rated highly ``Badger Basket Storage Cubby''
									with category ``Baby''																\\
								&	\textbf{Replacement Item:} "Google Chromecast HDMI Streaming Media Player"
									with category ``Home Entertainment"													\\ \midrule			
			CredPaths			&	You rated highly ``Men's hair paste'' with category ``Beauty'' 						\\
								& 	that was rated by ``Some user'' 													\\
								& 	who also rated highly ``Baby stroller'' with category ``Baby''						\\ \toprule
			\textbf{Method}		&	\textbf{Explanation for ``The Multiversity''
									with categories ``Comics, Historical-fiction, Biography, Mystery'' [Goodreads]} 	\\	\toprule
			\textsc{Prince}\xspace 			&	\textbf{Action 1:} You rated highly ``Blackest Night''
									with categories ``Comics, Fantasy, Mystery, Thriller'' 								\\
								& 	\textbf{Action 2: } You rated highly ``Green Lantern''
									with categories ``Comics, Fantasy, Children''										\\
								& 	\textbf{Replacement Item:} ``True Patriot: Heroes of the Great White North''
									with categories ``Comics, Fiction, Crime, Fiction'' 								\\ \midrule
			CredPaths			&  	You follow ``Some user''															\\
								&	who has rated highly ``The Multiversity''										
									with categories ``Comics, Historical-fiction, Biography, Mystery'' 					\\ \bottomrule
	\end{tabular}}
	\caption{Explanations from \textsc{Prince}\xspace vis-\`{a}-vis CredPaths~\cite{DBLP:conf/icdm/YangLWH18}.}
	\label{tab:anecdotes-amt}

\end{table}

\begin{table} [t]
	\centering
	\resizebox*{\columnwidth}{!}{
	
		\begin{tabular}{p{8.5cm}}
			\toprule
		
			Based on multiple actions explained simply and clearly. [\textsc{Prince}\xspace$\!$] 							\\ \midrule
			The recommendation is for a home plumbing item, but
			the action rated a glue. [\textsc{Prince}\xspace$\!$] 		\\ \midrule
			The explanation is complete as it goes into full details of how
			to use the product,
			which is in alignment of my review and useful to me. [CredPaths] 								\\ \midrule
			It's weird to be given recommendations based on
			other people. [CredPaths] 						\\ \bottomrule
	\end{tabular}}
	\caption{Turkers' comments on their score justifications.}
	\label{tab:comments}

\end{table}


Table~\ref{tab:amt-results} shows the results of
our user study.
It gives average scores and standard deviations,
and it indicates statistical significance of pairwise comparisons
with an asterisk.
\textsc{Prince}\xspace clearly obtains higher usefulness ratings from the AMT judges,
on average. Krippendorff's alpha~\cite{krippendorff2018content}
for \textsc{Prince}\xspace and CredPaths were found to be
$\simeq 0.5$ and $\simeq 0.3$ respectively, showing moderate
to fair inter-annotator agreement.
The superiority of \textsc{Prince}\xspace also holds for slices of samples where 
\textsc{Prince}\xspace generated explanations of size $1, 2$ and $3$. 
We also asked Turkers to provide \textit{succinct justifications}
for their scores
on each data point.
Table~\ref{tab:comments} shows some typical comments, where methods
for generating explanations are in brackets.

\subsection{Setup}

\textbf{Qualitative survey on usefulness.}
To evaluate the usefulness of counterfactual (action-oriented) explanations,
we conducted a survey with
Amazon Mechanical Turk (AMT) Master
workers (\url{www.mturk.com/help#what_are_masters}).
In this survey, we showed $500$ workers three recommendation items
(``Series Camelot'', ``Pregnancy guide book'', ``Nike backpack'') 
and two different explanations for each.
One explanation was
limited to only the user's own actions (action-oriented),
and the other was a path
connecting the user to the item (connection-oriented).

We asked the workers three questions:
(i) \textit{Which method do you find more 
useful?}, where $70\%$ chose the action-oriented method;
(ii) \textit{How do you feel about being exposed through explanations
	to others?},
where $\simeq75\%$ expressed a privacy concern either through
complete disapproval or 
through a demand for anonymization;
(iii) \textit{Personally,
	 which type of explanation matters to you more: ``Action-oriented''
	 or ``connection-oriented''?},
 where $61.2\%$ of the workers chose the action-oriented explanations.
We described action-oriented 
explanations as those allowing users to control their recommendation,
while connection-oriented ones reveal connections between the user and item
via other users and items.


\textbf{Quantitative measurement of usefulness.}
In a separate study (conducted only on Amazon data for resource constraints),
we compared \textsc{Prince}\xspace to
a path-based explanation~\cite{DBLP:conf/icdm/YangLWH18}
(later referred to as CredPaths).
We used the \textit{credibility} measure
from~\cite{DBLP:conf/icdm/YangLWH18}, scoring paths in descending order of
the product of their edge weights.
We computed
the best path for all $500$ user-item pairs (Sec.~\ref{subsec:graph-setup}).
This resulted in paths of a maximum length of three edges
(four nodes including user and $rec$).
For a fair comparison in terms of cognitive load,
we eliminated all data points
where \textsc{Prince}\xspace computed larger counterfactual sets.
This resulted in about $200$ user-item pairs, from where we
sampled exactly $200$. 
As explanations generated by \textsc{Prince}\xspace and CredPaths
have a different format of 
presentation (a list of actions vs. a path),
we evaluated each method separately to avoid presentation bias.
For the 
sake of readability, we broke the paths into edges and showed each edge on
a new line. 
Having three AMT Masters for each task, we collected $600 (200 \times 3)$
annotations for \textsc{Prince}\xspace and the same number for CredPaths. 

A typical data point
looks like a row in Table~\ref{tab:anecdotes-amt},
that shows representative examples (Goodreads shown only for completeness).
We divided 
the samples into ten HITs (Human Intelligence Tasks, a unit of job on AMT)
with $20$ data points in each HIT. 
For each data point, we showed a recommendation item and its explanation, and
asked users about the usefulness of the explanation on a scale of $1 - 3$
(``Not useful at all'', ``Partially useful'', and ``Completely useful''). 
For this, workers had to 
imagine that they were a user of an e-commerce platform who received the 
recommendations as result of doing some actions on the platform.
Only AMT Master workers were
allowed to provide assessments.

To detect \textit{spammers}, we planted one \textit{honeypot}
in each of the $10$ HITs,
that was a completely impertinent explanation. Subsequently, all
annotations of detected spammers (workers who rated such
irrelevant explanations as ``completely useful'')
were removed (~25\% of all annotations). 


  
\section{Conclusions and Future Work}
\label{sec:confut}

This work explored a new paradigm of action-based explanations in
graph
recommenders, with the goal of identifying minimum sets of user actions
with the counterfactual property that their absence would change the
top-ranked recommendation to a different item. 
In contrast to prior works on (largely path-based) recommender explanations, this 
approach offers two advantages:
(i) explanations are \textit{concise, scrutable}, and \textit{actionable},
as they are \textit{minimal} sets derived using
a \textit{counterfactual} setup over
a user's \textit{own} purchases, ratings and reviews; and
(ii) explanations do not expose any information about other users, thus avoiding
privacy breaches by design.


The proposed \textsc{Prince}\xspace method implements these principles using
random walks for Personalized PageRank scores
as a recommender model.
We presented an efficient computation and correctness proof for
computing counterfactual explanations, despite the potentially exponential
search space of user-action subsets.
Extensive experiments
on large real-life data from Amazon and Goodreads showed that 
simpler heuristics  fail to find the best explanations, whereas \textsc{Prince}\xspace
can guarantee optimality.
Studies with AMT Masters showed the superiority of
\textsc{Prince}\xspace over baselines in terms of explanation usefulness.

\vspace*{-0.3cm}
\section*{Acknowledgements}
 
This work was partly supported by the ERC
Synergy Grant 610150 (imPACT) and the DFG Collaborative Research 
Center 1223. We would like to thank Simon Razniewski from
the MPI for Informatics
for his insightful comments on the manuscript.

\subsection{Results and Insights}
\label{subsec:graph-res}

\begin{table} [t]
	\centering

	\begin{tabular}{c c c c c c c}
		\toprule 
		& \multicolumn{3}{c}{\textbf{Amazon}} & \multicolumn{3}{c}{\textbf{Goodreads}}	\\
		\cmidrule(l){2-4} \cmidrule(l){5-7}
		\textbf{\textit{k}}	& \textbf{\textsc{Prince}\xspace} 		& \textbf{\textit{HC}} 	& \textbf{\textit{SP}} 	& \textbf{\textsc{Prince}\xspace} 		& \textbf{\textit{HC}}	& \textbf{\textit{SP}}	\\ \toprule
		$3$ 				& $\boldsymbol{5.09}$* 	& $6.87$ 				& $7.57$				& $\boldsymbol{2.05}$* 	& $2.86$ 				& $5.38$				\\
		$5$ 				& $\boldsymbol{3.41}$* 	& $4.62$ 				& $5.01$				& $\boldsymbol{1.66}$* 	& $2.19$ 				& $4.37$				\\
		$10$ 				& $\boldsymbol{2.66}$* 	& $3.66$ 				& $4.15$	 			& $\boldsymbol{1.43}$ 	& $1.45$ 				& $3.28$				\\
		$15$ 				& $\boldsymbol{2.13}$* 	& $3.00$ 				& $3.68$				& $\boldsymbol{1.11}$ 	& $1.12$ 				& $2.90$				\\
		$20$				& $\boldsymbol{1.80}$* 	& $2.39$ 				& $3.28$				& $\boldsymbol{1.11}$	& $1.12$ 				& $2.90$ 				\\ \midrule
	\end{tabular
	\caption{Average sizes of counterfactual explanations. The best value
		per row in a dataset is in \textbf{bold}. An asterisk (*) indicates
		statistical significance of \textsc{Prince}\xspace over
		the closest baseline, under the $1$-tailed paired $t$-test
		at $p < 0.05$.}
	\label{tab:graph-main}
	\vspace*{-0.3cm}
\end{table}

\begin{table} [t]
	\centering

	\begin{tabular}{c c c c c}
		\toprule 
		\multirow{2}{*}{\textbf{Parameter}} & \multicolumn{2}{c}{\textbf{Amazon}} & \multicolumn{2}{c}{\textbf{Goodreads}} \\
		\cmidrule(l){2-3} \cmidrule(l){4-5}
									& \textbf{Pre-comp} & \textbf{Dynamic} 	& \textbf{Pre-comp}	& \textbf{Dynamic} 	\\ \toprule
		$k = 3$ 					& $0.3ms$ 			& $39.1s$ 			& $0.3ms$ 			& $24.1s$ 			\\
		$k = 5$ 					& $0.6ms$ 			& $60.4s$ 			& $0.4ms$ 			& $34.7s$			\\
		$k = 10$ 					& $1.3ms$ 			& $121.6s$ 			& $0.9ms$ 			& $60.7s$ 			\\
		$k = 15$ 					& $2.0ms$ 			& $169.3s$ 			& $1.5ms$ 			& $91.6s$			\\
		$k = 20$					& $2.6ms$ 			& $224.4s$ 			& $2ms$ 			& $118.8s$			\\ \midrule
		$\beta = 0.01$ 				& $0.4ms$ 			& $1.1s$ 			& $0.3ms$ 			& $2.9s$ 			\\
		$\beta = 0.1$ 				& $0.5ms$ 			& $15.5s$ 			& $0.3ms$ 			& $8.9s$ 			\\
		$\beta = 0.3$ 				& $0.5ms$ 			& $17.0s$ 			& $0.4ms$ 			& $12.5s$ 			\\
		$\beta = 0.5$ 				& $0.6ms$ 			& $60.5s$ 			& $0.4ms$ 			& $34.7s$  		\\ \bottomrule
	\end{tabular
	\caption{Average runtime of \textsc{Prince}\xspace,
	
		when the scores are pre-computed (\textbf{Pre-comp}) and when the scores
		are dynamically computed using
		the reverse push algorithm~\cite{zhang2016approximate}
		(\textbf{Dynamic}).}
	\label{tab:graph-perf}
	\vspace*{-0.3cm}
\end{table}

\begin{table} [!h] 
	\centering
	\resizebox*{\columnwidth}{!}{
	
		\begin{tabular}{l p{7cm}}
			\toprule
			\textbf{Method}		&	\textbf{Explanation for ``Baby stroller''
									with category ``Baby'' [Amazon]} 													\\	\toprule
			\textsc{Prince}\xspace				&	\textbf{Action 1:} You rated highly ``Badger Basket Storage Cubby''
									with category ``Baby'' 																\\
								& 	\textbf{Replacement Item:} ``Google Chromecast HDMI Streaming Media Player''
									with categories ``Home Entertainment''												\\	\midrule
			$HC$				&	\textbf{Action 1:} You rated highly ``Men's hair paste''
									with category ``Beauty'' 															\\
								&	\textbf{Action 2:} You reviewed ``Men's hair paste''
									with category ``Beauty''
									with text ``Good product. Great price.''											\\
								&	\textbf{Action 3:} You rated highly ``Badger Basket Storage Cubby''
									with category ``Baby'' 																\\
								& 	\textbf{Action 4:} You rated highly ``Straw bottle''
									with category ``Baby'' 																\\
								& 	\textbf{Action 5:} You rated highly ``3 Sprouts Storage Caddy''
									with category ``Baby''																\\ 
								&	\textbf{Replacement Item:} ``Bathtub Waste And Overflow Plate''
									with categories ``Home Improvement''												\\	\midrule
			$SP$				&	\textbf{Action 1:} You rated highly ``Men's hair paste''
									with category ``Beauty'' 															\\
								&	\textbf{Action 2:} You rated highly ``Badger Basket Storage Cubby''
									with category ``Baby'' 																\\
								& 	\textbf{Action 3:} You rated highly ``Straw bottle''
								 	with category ``Baby'' 																\\
								& 	\textbf{Action 4:} You rated highly ``3 Sprouts Storage Caddy''
									with category ``Baby'' 																\\
								& 	\textbf{Replacement Item:} ``Google Chromecast HDMI Streaming Media Player''
									with categories ``Home Entertainment''												\\	\toprule \toprule
			\textbf{Method}		&	\textbf{Explanation for ``The Multiversity''
									with categories ``Comics, Historical-fiction, Biography, Mystery'' [Goodreads]} 	\\	\toprule
			\textsc{Prince}\xspace 			&	\textbf{Action 1:} You rated highly ``Blackest Night''
									with categories ``Comics, Fantasy, Mystery, Thriller'' 								\\
								& 	\textbf{Action 2: } You rated highly ``Green Lantern''
									with categories ``Comics, Fantasy, Children''										\\ 			
								& 	\textbf{Replacement item:} ``True Patriot: Heroes of the Great White North''
									with categories ``Comics, Fiction''													\\	\midrule			
			$HC$ 				& 	\textbf{Action 1:} You follow User ID $x$ 											\\
								&	\textbf{Action 2:} You rated highly ``Blackest Night''
									with categories ``Comics, Fantasy, Mystery, Thriller'' 								\\
								& 	\textbf{Action 3: } You rated highly ``Green Lantern''
									with categories ``Comics, Fantasy, Children''										\\ 
								& 	\textbf{Replacement item:} ``The Lovecraft Anthology: Volume 2''
									with categories ``Comics, Crime, Fiction''											\\	\midrule			
			$SP$				& 	\textbf{Action 1:}  You follow User ID $x$ 											\\
								& 	\textbf{Action 2:} You rated highly ``Fahrenheit 451''
									with categories ``Fantasy, Young-adult, Fiction''									\\
								& 	\textbf{Action 3:} You rated highly ``Darkly Dreaming Dexter (Dexter, \#1)''
									with categories ``Mystery, Crime, Fantasy'' 										\\ 
								& 	\textbf{And 6 more actions} 														\\
								& 	\textbf{Replacement item:} ``The Lovecraft Anthology: Volume 2''
									with categories ``Comics, Crime, Fiction'' 											\\ \bottomrule
		\end{tabular}}
		\caption{Anecdotal examples of explanations by \textsc{Prince}\xspace
			and the counterfactual baselines.}
		\label{tab:anecdotes}
		\vspace*{-0.7cm}
\end{table}

We present our main results in Table~\ref{tab:graph-main} and discuss
insights below. 
These comparisons were performed for different values of the parameter $k$. 
Wherever applicable, statistical significance was tested under
the $1$-tailed paired $t$-test at $p < 0.05$. Anecdotal examples of
explanations by \textsc{Prince}\xspace and the baselines
are given in Table~\ref{tab:anecdotes}.
In the Amazon example, we observe that our method produces a topically
coherent explanation, with both the recommendation and the explanation items
in the same category. The $SP$ and $HC$ methods give larger explanations, but with
poorer quality, as the first action in both methods seems unrelated to the recommendation.
In the Goodreads example, both $HC$ and $SP$ yield the same replacement item,
which is different from that of \textsc{Prince}\xspace.

\textbf{Approximating \textsc{Prince}\xspace is difficult.} 
Explanations generated by \textsc{Prince}\xspace are more concise and hence
more user-comprehensible than those by the baselines.
This advantage is quite pronounced; for example, in Amazon, all the 
baselines yield at least one more action in the explanation set on average. 
Note that this translates into unnecessary effort for users who want 
to act upon the explanations.  

\textbf{Explanations shrink with increasing \textit{k}.} The size of
explanations shrinks as the top-$k$ candidate set for choosing
the replacement item is expanded.
For example, the explanation size for \textsc{Prince}\xspace on Amazon drops from
$5.09$ at $k = 3$
to $1.80$ at $k = 20$. This is due to the fact
that with a growing candidate set, it becomes
easier to find an item that can 
outrank $rec$.

\textbf{\textsc{Prince}\xspace is efficient.} To generate a counterfactual explanation,
\textsc{Prince}\xspace only relies on the scores in the graph configuration  $G \setminus A$ 
(where all the outgoing edges of $u$
are deleted). 
Pre-computing $PPR(n_i, rec|A)$ (for all $n_i \in \mathcal{N}_{out}(u)$), 
\textsc{Prince}\xspace could find the explanation for each $(user,rec)$ pair in about 
$1$ millisecond on average (for $k \leq 20$). 
Table~\ref{tab:graph-perf} shows runtimes of \textsc{Prince}\xspace for different 
parameters.  
As we can see, the runtime grows
linearly with $k$
in both datasets. This is justified by
Line~\ref{alg:mainstart} in Algorithm~\ref{alg:poly}.
Computing $PPR(n_i, rec|A)$ on-the-fly slows down the algorithm. 
The second and the fourth columns in Table~\ref{tab:graph-perf}
present the runtimes of \textsc{Prince}\xspace 
when the scores $PPR(n_i, rec|A)$ are computed using the 
reverse push algorithm for dynamic graphs~\cite{zhang2016approximate}.
Increasing $\beta$ makes the computation slower (experimented at $k=5$).
All experiments were performed on an Intel Xeon server with
$8$ cores @ $3.2$ GHz CPU and $512$ GB main memory. 

\section{Graph Experiments}
\label{sec:graph-exp}


We now describe experiments performed with graph-based recommenders built from
real datasets to evaluate \textsc{Prince}\xspace.

\input{sections/graph-setup}
\input{sections/graph-results}



\section{User Study}
\label{sec:user}

\input{sections/user-setup}
\input{sections/user-results}


\section{Problem definition}
\vspace*{-0.2cm}
\section{Computational Model}
\label{sec:model}

\noindent{\bf Heterogeneous Information Networks (HIN).}
A \emph{heterogeneous graph} $G = (V, E, \theta)$ consists of a set of
nodes $V$, a set of edges $E \subseteq V \times V,$ and a mapping $\theta$
from each node and each edge to their types, such that
$\theta_V: V \rightarrow T_V$ and $\theta_E: E \rightarrow T_E$ with
$|T_V| + |T_E| > 2$. In our work, a heterogenous graph contains at least
two node types, users $U \in T_V$ and items $I \in T_V$. For simplicity,
we use the notations $U$ and $I$ to refer both to the type of a node and
the set of all nodes of that type. A  graph is \emph{weighted} if there is
a weight assigned to each edge, $w: E \rightarrow \mathbb{R}$, and a graph is
\emph{directed} if $E $ is a set of ordered pairs of nodes. We denote with
$N_{out}(v)$ and  $N_{in}(v)$ the sets of out-neighbors and  
in-neighbors of node $v$, respectively.  A directed and weighted heterogeneous
graph where 
each node $v \in V$ and each edge $e \in E$  belong to exactly one type,
is  called a \emph{heterogenous information network
	(HIN)}~\cite{DBLP:journals/tkde/ShiLZSY17}.

\vspace{0.2cm}
\noindent{\bf Personalized PageRank (PPR) for recommenders.}
We use Personalized PageRank (PPR) 
for recommendation in
HINs~\cite{haveliwala2003topic,nikolakopoulos2019recwalk}.
PPR is the stationary distribution of a random walk in $G$ in which,
at a given step, with probability $\alpha$, a surfer teleports to a 
set of seed nodes $\{s\}$, and with probability $1 - \alpha$, continues
the walk to a randomly chosen outgoing edge from the current node. 
More precisely, given $G$, teleportation probability $\alpha$,
a single seed $s$, the one-hot vector $e_s$, and the 
transition matrix $W$, the Personalized PageRank vector $PPR(s)$ is defined
recursively as:
\begin{equation}
	PPR(s, \cdot) = \alpha e_s + (1-\alpha)PPR(s, \cdot)W
	\label{eq:ppr}
\end{equation}
Let $PPR(s,v)$ be the PPR score of node $v$ personalized for $s$.
We define the \emph{PPR recommendation} for user $u \in U$, or
the top-$1$ recommendation, as:
\begin{equation}
	rec = \argmax_{i \in I \setminus N_{out}(u)} PPR(u, i)
	\label{eq:rec}
\end{equation}
Given a set of edges $A \subset E$, we use the notation $PPR(u, i|A)$ to define
the PPR 
of an item $i$ personalized for a user $u$ in
the graph $G = (V, E \setminus A, \theta)$.
We refer to this graph as $G \setminus A$.
To improve top-$n$ recommendations,
Nikolakopoulos et al.~\cite{nikolakopoulos2019recwalk} define a random walk
in an HIN $G$ as follows: 
\begin{itemize}
\item With probability $\alpha$, the surfer teleports to $u$
\item With probability $1- \alpha$, the surfer continues the walk in
the following manner:
	\begin{itemize}
	\item[+] With probability $1 - \beta$, the random surfer 

	moves


	to a \textit{node of the same type}, using a 

	similarity-based stochastic 

	transition matrix
	\item[+] With probability $\beta$, the surfer chooses
	any outgoing edge at random.


	\end{itemize}
\end{itemize}

For each node type $t$ in $T_V$, there is an associated stochastic similarity
matrix $S_t$, which encodes the relationship between the nodes of type $t$.
When nodes of the same type are not comparable, the similarity matrix is
the identity matrix, i.e. $S_t = I$. 
Otherwise, an entry $(i,j)$ in $S_t$ corresponds to the similarity between
node $i$ and
node $j$. The stochastic process described by this walk is
a nearly uncoupled Markov chain~\cite{nikolakopoulos2019recwalk}.
The stationary distribution of the random walk is the PPR with
teleportation probability $\alpha$ in a graph $G^{\beta}$
(referred to as $RecWalk$ in~\cite{nikolakopoulos2019recwalk}), where
the  transition probability matrix of $G^{\beta}$ is:
\begin{equation}
	W^{\beta} =  \beta W + (1 - \beta) S
	\label{eq:W}
\end{equation}
The matrix $W$ is the transition probability matrix of
the original graph $G$. Matrix $S = Diag(S_{1}, S_{2}, \cdots, S_{|T_V|})$
is a diagonal matrix of
order $|V|$. 

\vspace{0.2cm}
\noindent{\bf Counterfactual Explanations.}
A user $u$ interacts with items via different types of \emph{actions} $A$,
such as 
clicks, purchases, ratings or reviews, which  are captured as
\emph{interaction edges} in the graph $G$. 
Our goal is to present user $u$ with a set of interaction edges $A^* 
\subseteq
\{(u, n_i)| (u, n_i) \in A\}$
(where $n_i$ is a neighbor of $u$)
responsible for an item recommendation $rec$;
we refer to this as a \emph{counterfactual explanation}. 
An explanation is \emph{counterfactual}, if after removing the edges $A^*$
from the graph, the user receives a different top-ranked recommendation
$rec^*$. 
A counterfactual explanation $A^*$ is \emph{minimal} 
if there is no smaller set $A' \subseteq A$
such that
$|A'| < |A^*|$ and
$A'$ is also a counterfactual explanation for $rec$.

\vspace{0.2cm}
\noindent \textbf{Formal problem statement.} Given 
a heterogenous information network $G = (V, E, \theta)$ and 
the top-ranked recommendation $rec \in I$ for user $u \in U$,
find a minimum counterfactual explanation for $rec$. 
 

\subsection{Setup}
\label{subsec:graph-setup}

\textbf{Datasets.} We used two 
real datasets: 
\squishlist
\item[(i)] The Amazon Customer Review
dataset (released by Amazon:
\url{s3.amazonaws.com/amazon-reviews-pds/readme.html}),
and,
\item[(ii)] The Goodreads review
dataset 
(crawled by the authors of~\cite{wan2018item}:
\url{sites.google.com/eng.ucsd.edu/ucsdbookgraph/home}).
\squishend

Each record in both datasets consists of a user, an item, its categories,
a review, and a rating value (on a $1-5$ scale).
In addition, a Goodreads data record has the book author(s) and
the book description.
We augmented the Goodreads collection with social links
(users following users) that we crawled from 
the Goodreads website.

The high diversity of categories in the Amazon data,
ranging from household equipment to food and toys, allows scope to
examine the interplay of cross-category information
within explanations.
The key reason for additionally choosing Goodreads is to 
include the effect of social connections (absent in the Amazon data). 
The datasets were converted to graphs with
``users'', ``items'', ``categories'', and ``reviews'' as \textit{nodes},
and ``rated'' (user-item),
``reviewed'' (user-item), ``has-review'' (item-review), 
``belongs-to'' (item-category) and
``follows'' (user-user) as \textit{edges}. 
In Goodreads, there is an additional node type ``author'' 
and an edge type ``has-author'' (item-author). All the edges, except 
the ones with type ``follows'', are bidirectional.  
Only ratings with value \textit{higher than three}
were considered, as low-rated items should not influence
further recommendations.

\begin{table} [t] 
	\centering
	\resizebox*{\columnwidth}{!}{

	\begin{tabular}{l c c c c c}
		\toprule
		\textbf{Dataset}	& \textbf{\#Users}	& \textbf{\#Items}	& \textbf{\#Reviews}	& \textbf{\#Categories}	& \textbf{\#Actions}	\\	\toprule
		Amazon				& $2k$				& $54k$				& $58k$					& $43$					& $114k$				\\ 
		Goodreads			& $1k$				& $17k$				& $20k$					& $16$					& $45k$					\\ \bottomrule
	\end{tabular}}
	\caption{Properties of the Amazon and Goodreads samples.}
	\label{tab:data}
	\vspace*{-0.3cm}
\end{table}
	
\textbf{Sampling.} For our experiments, we sampled $500$ seed users who had
between $10$ and $100$ actions, from both Amazon and Goodreads datasets.
The filters served to prune out
under-active and power users (potentially bots). 
Activity graphs were constructed
for the sampled users by taking their four-hop neighborhood from
the sampled data (Table~\ref{tab:data}).
Four is a reasonably small radius to keep the items relevant and personalized
to the seed users. 
On average, this resulted in having about $29k$ items
and $16k$ items for each user in their HIN, for Amazon
and Goodreads, respectively.

The graphs were augmented with weighted edges for
node similarity.
For Amazon, we added review-review edges where weights were computed
using the cosine similarity of the review 
embeddings, generated with Google's
Universal Sentence Encoder~\cite{cer2018universal},
with a cut-off threshold
$\tau = 0.85$ to retain only confident pairs.
This resulted in $194$ review-review edges. 
For Goodreads, we added three types of similarity edges: 
category-category, book-book and review-review, with the same
similarity measure 
($24$ category-category, $113$ book-book, and $1003$ 
review-review edges). Corresponding thresholds were
$0.67, 0.85$ and $0.95$.
We crawled category descriptions 
from the Goodreads' website and used book descriptions and review texts
from the raw data. Table~\ref{tab:data} gives some statistics about the
sampled datasets.

\textbf{Initialization.}
The replacement item for $rec$ is always chosen from the original top-$k$
recommendations generated by the system;
we systematically investigate the effect of $k$ on
the size of explanations in our experiments (with a default $k = 5$).
\textsc{Prince}\xspace 
does not need to be restricted to an explicitly specified candidate set,
and can actually operate over the full space of items $I$.
In practice, however, replacement items need to be guided by some 
measure of relevance to the user, or item-item similarity, so as not
to produce degenerate or trivial explanations if $rec$ is replaced
by some arbitrary item from a pool of thousands.

We use the standard teleportation probability
$\alpha = 0.15$~\cite{brin1998anatomy}. 
The parameter $\beta$ is set to $0.5$. 
To compute PPR scores, we used
the reverse local push method~\cite{zhang2016approximate} with 
$\epsilon=1.7e-08$ for Amazon and $\epsilon=2.7e-08$
for Goodreads. With these settings, \textsc{Prince}\xspace and the baselines 
were executed on all $500$ user-specific HINs to compute
an alternative recommendation (i.e., replacement item) $rec$*
and a counterfactual explanation set $A$*.	

\textbf{Baselines.}
Since
\textsc{Prince}\xspace
is an optimal algorithm with correctness
guarantees, it always finds minimal sets of actions that replace $rec$
(if they exist). We wanted to investigate, to what extent other, 
more heuristic, methods approximate the same effects.
To this end,
we compared \textsc{Prince}\xspace against two natural baselines:
\squishlist
\item[(i)] \textit{Highest Contributions} ($HC$): 
This is analogous to counterfactual evidence
in feature-based classifiers for structured
data~\cite{chen2017enhancing,moeyersoms2016explaining}.
It defines the contribution score of a user action $(u, n_i)$ to
the recommendation
score $PPR(u, rec)$ as $PPR(n_i, rec)$ (Eq.~\ref{eq:invariant}), and 
iteratively deletes edges with highest contributions until
the highest-ranked $rec$
changes to a different item.
\item[(ii)] \textit{Shortest Paths} ($SP$):
$SP$ computes the shortest path from $u$ to $rec$ and deletes
the first edge $(u,n_i)$ on this path.
This step is repeated on the modified graph, until the top-ranked $rec$
changes to a different item.	
\squishend


\textbf{Evaluation Metric.} The metric for assessing the quality of
an explanation is its size, that is, the number of actions in $A^*$
for \textsc{Prince}\xspace, and 
the number of edges deleted in $HC$ and $SP$.
