\section{Introduction}

Scholarship and scholarly writing can be stratified into \textit{primary}, \textit{secondary}, and \textit{tertiary} research. 

\textit{Primary} research involves making observations in the broadest sense of collecting data about objects that are not themselves studies. Computing code metrics, administering questionnaires, interviewing participants, counting bugs, collecting documents, downloading source code, and taking field notes while observing a retrospective meeting are all primary research.

\textit{Secondary} research involves analyzing, synthesizing and critiquing primary studies. Ad hoc reviews, case surveys, critical reviews, meta-analysis, meta-synthesis and scoping reviews are all types of secondary research. 
Secondary research is central to evidence-based practice because (1) practitioners cannot read every study; (2) important decisions typically should be made based on the balance of evidence rather than a single study; (3) in many fields, individual studies are too small to produce accurate estimates of population parameters (e.g. the strength of the relationship between two variables). However, some secondary research degrades into predominately descriptive ``papers about papers'' with limited scope and usefulness.

\textit{Tertiary} scholarship has two related meanings: (1) analyses of groups of secondary studies (meta-reviews); (2) summaries broad areas of scientific knowledge as found in textbooks, encyclopedia entries, etc. 

This paper focuses on secondary research because confusion over different kinds of secondary research, and their divergent purposes, is undermining the effectiveness and usefulness of secondary studies in software engineering (SE). This observation motivates the following objective.

\smallskip
{\narrower \noindent \textit{\textbf{Purpose:} The objective of this article is to compile, describe, compare, and contrast the different kinds of secondary research commonly published in software engineering venues, to pave the way for more mature secondary research.} \par}
\smallskip

\input{tables/types}

\section{Review Types}

\subsection{Ad Hoc Reviews}

An \textit{ad hoc} literature review is simply a discussion of some literature, as contained in most research papers as part of a background or related work section.
Ad hoc reviews may develop theory \cite[e.g.][]{ralph2018toward}, or integrate a new theory into existing literature \cite[cf.][]{stol2016grounded}. Ad hoc reviews can support a position
paper, or tertiary scholarship. 

Ad hoc reviews use purposive sampling \cite{baltes2020sampling}; that is, researchers purposefully select papers or studies that are useful, relevant, or support their arguments. Ad hoc reviews are often appropriate, for example, to support theory development, identify promising research topics, or prepare for a comprehensive exam. However, they suffer from important limitations: their unsystematic nature introduces sampling bias and defies replication.
This may lead to cherry-picking of evidence supporting authors' arguments \cite{gough2017introduction}.
Therefore, \textit{ad hoc reviews are inappropriate for supporting empirical statements} such as ``x causes y'', ``most people/objects have property P'', or ``process P has structure S or follows rules R''.

\subsection{Systematic Reviews}
The term \textit{systematic review} has two meanings, as follows:
\begin{enumerate}
    \item a literature review that employs a systematic (hence the name), replicable process of selecting primary studies for inclusion (below referred to as \textit{systematic review}), including case surveys, critical reviews, meta-analyses,  meta-syntheses, and scoping reviews. 
    \item a systematic review that uses meta-analysis of quantitative studies (especially experiments) to assess the strength of the evidence for specific, usually causal, propositions (below referred to as \textit{meta-analysis}). 
\end{enumerate}

This double meaning is due to the history of systematic reviews. The concept of a meta-analytic systematic review emerged from health and medicine in the late 20\textsuperscript{th} century \cite{purssell2020brief}, when practitioners could not keep current with the acceleration of research results. When Chalmers \cite{chalmers1993cochrane} founded the Cochrane Library in 1993,
the medical community coalesced around using meta-analysis to inform evidence-based practice. Now, \textit{systematic review} is often  conflated with \textit{meta-analysis}.

However, other kinds of systematic reviews have also been around for decades. Scoping reviews were proposed no later than \citeyear{arksey2005scoping} \cite{arksey2005scoping}. Meta-synthesis goes back at least to \citeyear{jensen1996meta} \cite{jensen1996meta}. Case surveys were proposed as far back as \citeyear{lucas1974case} \cite{lucas1974case}. 


Systematic reviews begin by searching databases for primary studies that meet pre-established criteria. Most types of systematic reviews seek to identify \textit{all} the primary studies that meet the selection criteria by applying various techniques to mitigate sampling bias and publication bias including: ``backward and forward snowballing searches; checking profiles of prolific authors in the area;  searching both formal databases (e.g. ACM Digital Library) and indexes (e.g. Google Scholar); searching for relevant dissertations; searching pre-print servers (e.g. arXiv); soliciting unpublished manuscripts through appropriate listservs or social media; contacting known authors in the area'' \cite{ralph2020acm}.


The way that the primary studies are analyzed determines the \textit{type} of systematic review. 

\subsubsection{Meta-analysis}

In an archetypal systematic review, researchers identify a group of randomized controlled experiments with the same hypotheses, and the same independent and dependent variables.

Suppose, for example, we have 10 experiments in which SE undergrads were randomized into a control group (who complete tasks individually) and a treatment group (who complete tasks in pairs); that is, individual vs. pair programming. In all 10 experiments, the dependent variable was the number of tasks completed successfully, and the hypothesis was that the treatment group would complete more tasks successfully. Each primary study reports the results of an independent samples t-test including the mean and standard deviation for each group, the t-statistic, the p-value, Cohen's D (effect size) and the 95\% confidence interval for D. 

Our aim, then, is to combine the results of these ten experiments to estimate the effect size of pair programming on effectiveness. Suppose that four of the studies found a negative effect (hypothesis not supported), three found no significant effect and three found a positive effect (hypothesis supported). We \textbf{DO NOT} proceed by vote counting; that is, assuming that the effect is negative because four studies found a negative effect vs. three finding a positive effect. Vote counting is not valid because primary studies can have wildly different sizes. What if the studies that found positive effects were much larger and more rigorous; while the studies that found negative effects were small and confounded? What if, when the three studies without significant results are aggregated, together their results are significant? 

Instead of vote-counting, we apply meta-analysis \cite{glass1976primary}; that is, we use a statistical model to aggregate the results from the primary studies into a global estimate of effect size. Meta-analysis is often possible with the summary data reported in papers, without the original datasets. 

Meta-analysis is not limited to randomized controlled experiments. Results from different sorts of (quantitative) methods can aggregated through meta-analysis as long as the studies have the same independent and dependent variables. 
Sometimes, instead of exactly the same variables, studies have overlapping sets of variables. The more complicated the overlaps, the more complicated the meta-analytic model. A comprehensive tutorial on statistical procedures for meta-analysis is beyond the scope of this paper, but readily available \cite[cf.][]{borenstein2021introduction}

Meta-analytic reviews essentially have the same research question as the studies being reviewed. The purpose of the meta-analysis is to reach a more reliable, robust conclusion by aggregating all available data, implying two important criteria for meta-analysis: (1) researchers are supposed to go to great lengths to find \textbf{ALL} relevant studies; and (2) research \textit{must} evaluate the quality of each primary study and either exclude low quality studies or include study quality as a covariate in the meta-analytic model.
 
In summary, when scientists equate systematic reviews with evidence-based practice, they usually mean meta-analytic reviews. Meta analysis aggregates quantitative studies that investigate the same or overlapping hypotheses. \textit{They do not simply describe} existing research. 
Meta-analysis is rare in software engineering. While there are several good examples \cite[e.g.][]{shepperd2014researcher,hannay2009effectiveness,rafique2012effects}, quality meta-analysis is dwarfed by superficial scoping reviews \cite{cruzes2011research}.


\subsubsection{Meta-synthesis}

Meta-synthesis---aka thematic analysis, narrative synthesis, meta-ethnography, and interpretive synthesis---refers to a family of methods of aggregating qualitative studies \citep{jensen1996meta}. Meta-synthesis is approximately the constructivist analogue of meta-analysis. After identifying the primary studies, the researcher applies hermeneutical and dialectical analyses to understand each primary study, translate them into each other, and construct an account of the body of research; for example, a theory of the central phenomenon that unites the primary studies. 

Meta-synthesis requires expertise in qualitative methods and some understanding of the underlying philosophical assumptions. If one does not know what ``hermeneutical and dialectical analyses'' means, one should not attempt meta-synthesis. Meta-synthesis is \textbf{NOT} organizing papers into categories (as we see in scoping reviews). Meta-synthesis is synthesizing a credible, nuanced account of a phenomenon, grounded in a body of interconnected qualitative research.  

In principle, meta-synthesis can be applied to both qualitative and quantitative work. In practice, such combinations are philosophically strained.

\subsubsection{Case Survey (aka Case Meta-analysis)}

In a case survey, the primary studies are (typically qualitative) case studies in the broadest sense of a case study (i.e., a scholarly account of some events). Experience reports and grey literature may or may not be included, depending on the study's purposes.  
However, unlike meta-synthesis, a case survey transforms qualitative accounts into a quantitative dataset that supports null-hypothesis testing. Case studies share the philosophy of meta-analysis (positivist), not meta-synthesis (constructivism). 
Case surveys typically begin with a priori hypotheses and an a priori coding scheme. The researcher reads each case and extracts data into the coding scheme, often using simple dichotomous variables like `did the team have retrospective meetings? [yes/no]' or 'does the case mention coordination problems [yes/no]?' This resulting dataset is often too sparse for regression modeling, so researchers use simple bivariate correlations to test hypotheses \cite{bullock1987case}. 
Case surveys were proposed by the Rand Corporation as a ``way to aggregate existing research'' \cite{lucas1974case}, quickly picked up by Yin \cite{yin1975using}, and later elaborated in management \cite{bullock1987case,larsson1993case}. Now case meta-analysis is widely used in management and information systems \cite{jurisch2013using}. While SE-specfic case survey guidelines are available~\cite[e.g.][]{melegati2020case,petersen2020guidelines}, SE case surveys remain rare. However, case surveys have been used to investigate strategic pivots at software start-ups \cite{bajwa2017failures} and how organizations select component sourcing options \cite{petersen2017choosing}. 
We see great potential for case surveys in SE research because case studies are so common. 

\subsubsection{Critical Reviews}

In a \textit{critical review}, researchers analyze a sample of primary studies to support an argument or critique.\footnote{The term \textit{critical review} has been used differently elsewere, but we focus on the meaning in SE.} For example, Stol et al. \citep{stol2016grounded}'s critical review of the use of grounded theory in software engineering \textit{criticizes} method slurring; that is, claiming to have used a research methodology that was not actually used to create the illusion of legitimacy. Similarly, Baltes and Ralph \citep{baltes2020sampling} used a critical review to describe and criticize how SE researchers overstate sample representativeness and often conflate random sampling with representative sampling. Indeed, critical reviews in software engineering often investigate methodological topics, including how ethnography is reported \citep{zhang2019ethnographic} or how qualitative research is synthesized \citep{huang2018synthesizing}. 

Critical reviews differ from case surveys and meta-analysis in two important ways. First, meta-analytic reviews aggregate evidence regarding causal relationships to generate evidence-based recommendations, whereas a critical review critically evaluates issues. Critical reviews are not for supporting evidence-based practice, or summarizing the evidence for a theory. Critical reviews are part of the meta-scientific discourse; i.e., the conversations a scientific community has internally about how it conducts research. 

Second, most systematic reviews aims to collect \textit{all} relevant primary studies on a specific topic \citep{Kitchenham:2007vo}. Including all relevant studies for critical reviews is often impossible. A critical review of adherence to the \textit{Introduction, Method, Results and Discussion framework} (IMRaD) framework~\cite{sollaci2004introduction}, for example, could include all SE papers, ever. However, this is unnecessary. A random sample of papers from a selection of top journals and conferences is sufficient to assess compliance with IMRaD. For meta-analysis, we worry about publication bias. Critical reviews do not assess causal claims so worrying ``what if significant results were published but non-significant results were not?'' is irrelevant. 

Analysis performed within a critical review can be quantitative, qualitative or both. However, critical reviews typically adopt a \textit{critical} stance; that is, they go beyond mere description and offer specific critiques of the work being reviewed. 



\subsubsection{Scoping Reviews (aka Systematic Mapping Studies)}

What is often called a ``systematic mapping study'' in software engineering \cite{petersen2008systematic} is usually called a \textit{scoping review} elsewhere (e.g. in health, medicine, and psychology).
The purpose of a scoping review is to understand the status of research on a particular topic, typically by mapping primary studies into categories (hence, ``mapping study''). 

Scoping reviews are often primarily descriptive. They count the number of studies on a topic. They often organize studies by research method, subtopic, authors, geographical location, publication venue, etc. They often conclude that more research is needed in this or that subtopic. For example, Mohanani et al. \cite{mohanani2018cognitive} mapped primary studies according to which cognitive bias (e.g. confirmation bias) they investigated and in which area of software development (e.g. design, management) they investigated it, then called for more research on \textit{debiasing} (preventing or mitigating cognitive biases). 

The problem with scoping reviews is that they typically fail to fulfill any of the core purposes of systematic reviews:

\emph{Meta-analysis} and \emph{case surveys} synthesize the results of many studies to answer specific empirical (often causal) questions about the world. Scoping reviews include a similar search but typically do not provide sufficient quantitative synthesis to answer important empirical questions. Therefore, scoping reviews do not inform evidence-based practice like meta-analyses and case surveys. 

\emph{Meta-synthesis} involves deep, theory-oriented reinterpretation of related qualitative studies. While scoping reviews often include qualitative analysis (e.g. mapping or categorization), that analysis is often too superficial to generate novel, useful theories.

\emph{Critical reviews} use a sample of papers to demonstrate an important pattern for a scientific community's internal discourse (e.g. widespread construct validity problems). While scoping reviews often give recommendations regarding future research, they focus on an empirical topic (e.g. cognitive biases in SE); not a meta-scientific topic (e.g. construct validity); therefore, they are not configured, from the outset, to deliver useful meta-scientific critique.


In summary, scoping reviews begin like other kinds of systematic reviews, but stop short of \textit{synthesizing} the data into aggregate empirical results, theory, or meta-scientific critique. This is by definition: if a scoping review applies a meta-analytic model to aggregate primary study results, or applies hermeneutics and dialectics to synthesize qualitative accounts, or develops an evidence-based critique of a scientific practice, it is no longer a scoping review; it is a meta-analysis, a case survey, a meta-synthesis or a critical review. Some authors therefore recommend a scoping review ``as a precursor to a systematic review'' \cite{munn2018systematic}. 

\subsubsection{Rapid Reviews}

A \textit{rapid review} is a meta-analysis that makes methodological compromises to reduce completion time~\cite{ganann2010expediting}. Ganann et al. found many such compromises including restricting the literature search, truncating results, omitting techniques for overcoming publication bias (e.g. reference snowballing); streamlining screening and data extraction, and skipping quality assessment \cite{ganann2010expediting}. 


Rapid reviews are justified \textit{if and only if} evidence is needed to support imminent decisions, and waiting for a comprehensive meta-analytic review would be harmful. These conditions occur in health and medicine, for example, when an unprecedented viral pandemic strikes. These conditions do not occur frequently in SE. The term \textit{rapid review} should not be used to legitimize doing a bad job of a systematic review where there is no pressing need for immediate results. Moreover, SE-related topics rarely have so many primary studies that it would take more than a year to complete a comprehensive review.





\section{Discussion}

Secondary research that synthesizes the results of a body of work into robust, reliable, findings and practical ramifications is crucial for evidence-based practice. Secondary research is the funnel that concentrates the overwhelming flow of empirical results into something practitioners can digest and use. 
However, most secondary research in SE contains no synthesis of findings \cite{cruzes2011research}, because most SE secondary studies are scoping reviews. Scoping reviews struggle to get from a broad description of what is happening in a field to specific recommendations for practitioners. Whereas the purpose of meta-analysis is to answer a specific empirical question by combining the results of all of the studies that tested corresponding hypotheses, scoping reviews just put research in buckets. 
Meanwhile, SE researchers perceive inadequate number and quality of primary studies as a major barrier to good secondary research \citep{guzman2014survey}.





\subsection{Recommendations}
\label{section:recommendations}
Our main recommendation is: \textbf{software engineering research needs more meta-analysis, more case surveys, more meta-synthesis, and fewer scoping and rapid reviews.}

If there is a sufficient body of evidence to facilitate a meta-analysis, case survey, or meta-synthesis, by all means, do one. Top venues should welcome competent meta-analyses, case surveys, and meta-syntheses as these approaches are central to evidence-based practice. In contrast, predominately descriptive scoping reviews are essentially works-in-progress. They belong in workshops, poster sessions and short paper tracks. If there is not enough evidence for a meta-analysis, case survey, or meta-synthesis, then a scoping review will not be able to make a substantive contribution to the scholarly discourse. If there is enough evidence, the scoping review is like a pilot study for a more ambitious systematic review.

Rapid reviews similarly have little place in SE. The conditions that justify a rapid review rarely occur in our field. Top venues should expect rapid reviews to justify methodological compromises and explain why results are needed immediately. The term ``rapid review'' should not be used to legitimize a bad meta-analysis.  

The case for critical reviews is more complicated. SE does not need more critical reviews per se; SE needs a more vivid, robust meta-scientific discourse. Meta-science in SE has been hamstrung by the insistence, among many reviewers and top venues, that every paper reports an empirical study, regardless of whether the questions at hand are empirical. As long as this unwise position remains, we need critical reviews because they provide a mechanism to inject meta-science into our publications. We would also like to remind reviewers not to misapply the ``find ALL relevant studies'' or ``exclude low quality studies'' criteria to critical reviews. 

Finally, authors should identify the kind of ``systematic review'' they have done. Avoid using ``systematic review'' as a synonym for meta-analysis or, worse, scoping review. In the broadest sense, any review that uses a systematic process of identifying primary studies is a systematic review. 


\subsection{Limitations}
This is not an empirical paper. This paper discusses a methodological issue. It is meta-science: part of our community's internal discourse about what we are doing, why, and how to do it. One major reason SE has so many methodological challenges is that we have effectively abandoned our meta-scientific discourse by expecting every paper to report an empirical study. This paper is credible because it meets the expectations laid out in the Empirical Standard for Meta-science: it is useful because most SE researchers read or perform literature reviews; makes clear recommendations (Section \ref{section:recommendations}); presents coherent arguments; ``goes beyond summarizing methodological guidance from existing works... [and]  provides insight specifically for software engineering'' \cite{ralph2020acm}. It is limited by excluding tertiary studies and providing relatively brief descriptions of each type of review to respect the venue's page limit.


\section{Conclusion}


\textit{Systematic review} is often conflated with \textit{meta-analysis}. We recommend using \textit{systematic review} as the category of secondary research in which primary studies are selected according to a systematic, replicable process; that is, the opposite of an \textit{ad hoc} review. Meta-analysis, case survey, meta-synthesis, critical review, scoping review and rapid review are all types of systematic review. 

SE needs more meta-analysis, meta-synthesis and case surveys. These methods synthesize bodies of related work into actionable recommendations for practitioners and are therefore crucial for evidence-based practice. Scoping reviews, in contrast, do not include enough synthesis to support evidence-based practice. A scoping review is like a prelude to a more ambitious project. If published at all, scoping reviews belong with other work-in-progress in posters, workshops or short papers. Rapid reviews meanwhile, are a valuable tool but SE phenomena rarely manifest the conditions under which rapid reviews are justified. Finally, critical reviews have different best practices and reviewers should stop misapplying meta-analysis criteria to critical reviews.    


We hope this paper helps researchers distinguish between different types of literature reviews---paving the way for more mature secondary research in software engineering.



\balance
\bibliographystyle{ACM-Reference-Format}


\section{Template}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[manuscript,screen,review]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=0.75\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  