
\section{Introduction}
The rise of knowledge work and contemporaneous information explosion demand the ability to quickly sift through rapidly evolving information. For example, scientific researchers spend a tremendous amount of effort staying up to date with literature in their field.
The process typically involves researchers foraging for a set of potentially relevant work, skimming or reading the selected papers, and organizing the relevant aspects contextually.
AI has been used to assist several aspects of this process: scholarly search engines, such as Google Scholar and Semantic Scholar~\cite{beel_google_2009, ammar_construction_2018},  paper recommendation systems~\cite{beel_research-paper_2016, bai_scientific_2019}, and in-depth reading support tools~\cite{head_augmenting_2021}, but information overload remains a huge challenge for researchers.

Before deciding to read a paper in depth, researchers often skim it selectively~\cite{rayner_so_2016}.
Skimming is a reading technique that involves quickly glancing over text to gain a general idea of its content, extracting only the most important information~\cite{maxwell_skimming_1972}.
Since skimming a paper takes a fraction of the time required for a deep read, researchers have adopted skimming as one strategy to keep pace with the growing literature, a practice intensified by a shift to online scholarly reading~\cite{liu_reading_2005, tenopir_electronic_2009}.

Experienced researchers rely on years of practice skimming papers, accumulating repetitions in key techniques such as scanning for headings, selectively identifying key words that indicate areas of the text to devote more attention, and focusing on visual content.
However, skimming can still be a challenging skill to learn and effectively harness~\cite{yi_qndreview_2014, maxwell_skimming_1972, duggan_text_2009}.
Skimming imposes a cognitive burden on our limited working memory, requiring readers to navigate with deliberate saccades through a text to identify important information and understand the text that is fixated on.
Novice researchers with less developed skimming know-how, and even experienced researchers may find themselves struggling to effectively identify and navigate between important parts of the text.
Effective skimming can then be considered a rapid decision-making process under time pressure, balancing a determination of parts of the text that should be read in more depth and an assimilation of the processed content into their working understanding of the paper.

\makeUserInterfaceFigure

Prior research has explored ways to reduce the cognitive effort required for various aspects of reading and skimming papers~\cite{liu_scientific_2015, head_augmenting_2021, powley_enriching_2009}.
Experimental tools have been developed to specifically help skimming readers, for instance by spotlighting visual content within a paper~\cite{lee_spotlights_2016} and automatically enlarging section headers within paper thumbnails~\cite{buchanan_improving_2008}.
Within the natural language processing community, automated techniques for document understanding and summarization of scientific papers~\cite{xiao_extractive_2019, sefid_scibertsum_2022} have seen a recent growth in research interest and performance, and could be possibly leveraged for developing novel tools to support skimming, but have yet to be meaningfully integrated and studied within user-facing interactive systems. 

We seek to address these issues by exploring the strengths, limitations, and design implications of reading interfaces for skimming augmented by the output of these potentially imprecise AI systems.
We present Scim, a prototype tool designed to support readers in skimming scientific papers (Figure~\ref{fig:user_interface}).
Scim's key features involve automatically identifying and highlighting salient sentences within a paper, classifying these sentences into four rhetorical facets that correspond to common information needs of readers, providing an integrated list view of these highlights organized and linked to its corresponding context in the paper, and offering readers interactive controls to support their skimming.

We used Scim as a design probe to (1) explore the opportunities for an AI-powered reading interface for supporting researchers as they skim scientific papers and (2) characterize the benefits and risks of integrating imprecise AI support into augmented reading interfaces.
In a study with 13 participants, we observed researchers using Scim in various ways.
As might be expected, researchers used the highlights as suggestions of importance from an opaque intelligent agent, treating them as cues of attention to complement linear skimming processes.
We also saw participants making use of Scim in novel and unexpected ways, highlighting important design considerations for future AI-assisted reading interfaces.
For instance, we observed readers using the features to support multiple, complementary skimming passes over a paper. Some readers tended to filter highlights to two of the rhetorical facets—\textsc{Objective} and \textsc{Novelty}—and jumped to read these highlights within the paper first, before continuing to skim the paper from the beginning.
Others used a summary of highlights, provided in an adjacent highlight browser, as a rereading mechanism to efficiently verify their comprehension.

Participants largely expressed interest in using Scim's core features in future augmented reading interfaces, and were excited for the potential of AI assistance in reading interfaces, despite their potential for errors. 
Related to prior studies exploring mental model formation in human-AI systems~\cite{nourani_anchoring_2021}, we also observed researchers forming a first impression for Scim's reliability in identifying important sentences by evaluating the quality of the first highlights they saw.
While they made further adjustments to their impressions of reliability based on subsequent highlights, researchers felt their initial assessment played a vital role in determining their desire to continue using or abandon the system.
Based on qualitative insights from our design probe, we distilled six key design recommendations for the development of AI-infused reading interfaces.
Our recommendations highlight how these systems may support more interactive, need-driven reading experiences, and how their designs may consider errors and biases made by the underlying intelligent agents.   
Finally, we discuss future research opportunities in collaborative highlights and personalization within intelligent reading interfaces.
\section{Related Work}

\subsection{Prior Studies on Skimming}
Skimming is widely considered to be a form of rapid reading in which the goal is to get a general idea of the text or visual content, typically accomplished by focusing on information relevant to one's goals and skipping over irrelevant information~\cite{masson_conceptual_1983, rayner_so_2016}. Skimming is a particularly necessary and useful skill for scholars who read scientific papers. As the number of published papers continue to increase year over year and technology has caused a gradual move from print towards a digital medium, scholars have adapted by reading more papers while spending less time on each~\cite{tenopir_electronic_2009, liu_reading_2005}. 

Prior results from the psychology literature have found that skim readers are not generally very accurate at selecting goal-relevant information for processing within text, and that physical limitations in the oculomotor system responsible for controlling eye movements largely preclude rapid, accurate placements of eye gaze for extended periods such as when skimming a long document~\cite{masson_cognitive_1982, masson_conceptual_1983}.
Beyond limitations in visual acuity, skimming can also be a cognitively demanding task as readers are continually building an ongoing mental model of the text and integrating information across sentences as they read~\cite{rayner_so_2016, rapp_dynamic_2005, tashman_active_2011}.

Other studies suggest that skim readers may be able to effectively direct attention to more important content, for instance by reading in a satisficing manner~\cite{reader_allocating_2007, duggan_text_2009, duggan_skim_2011}.
Satisficing is a skim reading strategy in which readers are inherently sensitive to a proxy for information gain.
Under this strategy, readers set a information threshold, and if while reading a unit of text they determine that the information gain falls below a designated information threshold, they proceed on to the next unit of text.
These studies have found that as a result, people tend to spend more time at the beginning of paragraphs, toward the top of pages, and at the beginning of documents~\cite{duggan_text_2009}.
We use Scim to study how automated assistance may support skimming by cueing readers towards salient sentences, suggested by an AI system, thereby shifting the initial locus of attention for readers under the satisficing strategy.

One study on skimming for scientific document triage found that readers were hasty and incomplete, and documents were scrolled through quickly with attention paid to highly visual content and section headers~\cite{loizides_empirical_2009}.
Since information-dense content may be buried within pages of plain text, we see an opportunity for automated assistance in facilitating the discovery of these relevant information units that may otherwise be skipped.
Scientific documents are also laden with visual content, typographical cues (e.g., italicized, bold, or colored text), and structural information.
Studies have found that readers draw on document features to support rapid comprehension via these macro- and micro-structures~\cite{carrell_facilitating_1985, lacroix_macrostructure_1999, machulla_skimreading_2018} and visual content~\cite{yi_qndreview_2014, how_keshav_2007}.
Scim's design as an AI-augmented reading interface enables readers to leverage AI assistance while still retaining access to a paper's visual and structural information.


\subsection{Tools for Reading and Skimming}
Researchers have long sought to equip readers with tools that support and augment their cognition while reading documents.
The nascent days of human-computer interaction saw the introduction of augmented reading interfaces to support the reading process, including fluid documents that provided contextual access to supplemental information between lines of text~\cite{chang_negotiation_1998}, fluid hypertext~\cite{zellweger_fluid_1998}, visualizations for social annotations within papers~\cite{hill_edit_1992}, and affordances for annotating papers and jumping readers to passages of interest~\cite{graham_readers_1999, schilit_beyond_1999}.
Since then, several classes of approaches have been proposed to support the various aspects of reading, such as document navigation and comprehension.

\subsubsection{Modified Scrolling Interactions}
One line of research sought to facilitate the rapid exploration of long documents by modifying the behavior of reading interfaces during scrolling.
Applications of content-aware scrolling were used to redefine the presentation order of content within a document~\cite{ishak_content-aware_2006}, provide pseudo-haptic feedback when scrolling past relevant information~\cite{kim_content-aware_2014}, and dynamically resize document headings within paper thumbnails in a document viewer~\cite{buchanan_improving_2008}.
Spotlights implemented an attention allocation technique that pinned headings and figures as static overlays to a document as it was continuously scrolled~\cite{lee_spotlights_2016}.

\subsubsection{Typographical Cueing}
Another approach involved augmenting reading interfaces with typographical cues (e.g., highlighting).
The Semantize system used highlights to visualize sentiment within a document, and underlined words with positive or negative sentiment in different colors~\cite{wecker_semantize_2014}.
The ScentHighlights system used highlights to identify conceptually relevant text based on a user's query~\cite{chi_scenthighlights_2005}.
The HiText technique introduced dynamic graded highlighting of sentences within a document in accordance with their salience~\cite{yang_hitext_2017}.
Modern reading interfaces also commonly support readers in marking regions of interest with a document with highlights or free-text annotations.
The pervasiveness of highlighting as a technique for drawing readers' attention can be attributed to the von Restorff isolation effect, which states that an item isolated against a homogenous background will be more likely to be attended to and remembered~\cite{von_restorff_uber_1933}.
Studies have since found evidence of this effect on the visual foraging behavior of readers on highlighted documents, finding that highlights attract about half of the total number of fixations within a document, and are often drawn to by readers' eyes~\cite{chi_visual_2007}.

\subsubsection{Document Augmentations}
Beyond typographical cues, other reading interface augmentations exist to specifically support the reading of scientific papers.
For instance, online paper providers like ScienceDirect, PubMed, and Semantic Scholar provide readers with in-context citation information.
Experimental systems have linked document text to marks within charts~\cite{kong_extracting_2014} and cells within tables~\cite{kim_facilitating_2018}, generated on-demand visualizations based on text within the paper~\cite{badam_elastic_2019}, augmented static visualizations with animated~\cite{grossman_your_2015} or interactive~\cite{masson_chameleon_2020} overlays, and provided in-context definitions for nonce words~\cite{head_augmenting_2021}.
We design Scim with inspiration from many of these prior augmented reading interfaces, augmenting scientific papers with interactive highlights that guide reader attention.
Extending prior systems, Scim not only extracts salient sentences, but also classifies each highlight into common classes of information needs for readers.

\subsubsection{Summarization}
An alternative method to skimming a full paper is to read a shortened representation of the paper's content in the form of a summary.
An author-provided summary is de facto included with each paper as an abstract, which researchers often read before continuing to the rest of the paper.
Automated summarization has garnered significant interest from the natural language processing community, and extractive and abstractive methods for generating summaries from long-form documents have been developed over the years~\cite{aggarwal_survey_2012, aries_automatic_2019, sefid_scibertsum_2022}.
Some methods have even been proposed for generating extreme (single sentence) summaries, called \textsc{TLDR}s, from full papers~\cite{cachola_tldr_2020}.

However, providing only a summary to readers is often unsatisfactory.
Despite recent improvements in the quality of generated summaries, they remain error-prone, susceptible to hallucination~\cite{zhao_reducing_2020}, and are not reliable enough to be used as a standalone replacement for reading the paper itself.
Furthermore, summaries do not provide readers with the ability to interact with the full paper.
For instance, as readers' goals and interests change while reading a paper, they may wish to explore certain sections in further detail.
While traditional summaries cannot support this interaction, augmented reading interfaces naturally retain the context of the paper.
We leverage natural language processing techniques to identify salient sentences and classify sentences into rhetorical facets using a pretrained language model, and present the output within a carefully-designed augmented reading interface to support the interactivity and context lacking in standalone summaries.
\section{Design Motivations}
\subsection{Formative Study}
To better understand how an intelligent reading interface might support readers in skimming scientific papers, we conducted a small formative study with eight researchers.
In the study, researchers were asked to describe their typical approaches and goals for skimming.
We found researchers were largely predisposed to established strategies---what they considered as conventional wisdom---for skimming scientific papers.
Most focused on reading the Abstract and Introduction sections of a paper, before searching for a list of the paper's contributions, a summary of results, or the paper's conclusions.

Strategies for skimming beyond this conventional wisdom tended to diverge between individual researchers, influenced by factors such as their goal for skimming a particular paper and their experience in that paper's research area.
Researchers mentioned using heuristics for identifying regions that were likely to provide the most information gain---typographical cues (e.g., bold or italicized text), structural cues (e.g., section headers), the presence of visual media (e.g., figures and tables), and rhetoric structure (e.g., first and last sentences of a paragraph).
They also expressed a variety of goals that would require different levels of detail in skimming---these included learning about specific techniques introduced in a paper, determining a paper's relationship with prior work or their own research, discovering new research directions, or gaining a general understanding to discuss a paper with colleagues.

\subsection{Design Motivations} \label{sec:design_motivations}
Our formative study suggested that researchers skimming papers may have a diverse set of goals and strategies, but often start by searching for areas indicating a paper's significance before diving into further detail.
We see an opportunity to leverage automated tools to accelerate the discovery of significance-defining cues within the paper.
Based on our formative study and a review of related literature, we identified four design motivations that guide the design of Scim:

\textit{DM1. Scaffold information discovery throughout an entire paper.}
Though conventional wisdom guided readers toward a handful of common sections at the beginning and end of papers, these is little support for the discovery of relevant content within the middle, often text-heavy sections of a paper.
We aimed to explore opportunities in leveraging natural language processing techniques for document-level understanding to provide support for skimming these sections of papers.
By supporting content discovery and classification, we believe intelligent agents may reduce the cognitive processing readers need to perform during skimming.

\textit{DM2. Connect readers to salient content in context.}
One method to support skimming of long texts is through a condensed view of salient text content.
These views aim to provide a representation of the text that requires less effort to process but conveys the same level of understanding.
However, this condensed representation may lack sufficient context if presented statically.
Our design should therefore provide readers with the ability to engage with and gain more context on-demand for any AI-suggested content within these condensed views.

\textit{DM3. Direct reader attention while minimizing distractions.}
Studies in cognitive psychology have found that visual cueing mechanisms can be effective in focusing reader attention~\cite{chi_visual_2007} and improving retention of material~\cite{fowler_effectiveness_1974, rickards_notetaking_1980}.
In developing Scim, we evaluated three common cueing mechanisms: underlining, highlighting, and masking.
User evaluations of these mechanisms within prototypes of Scim found that highlighting was the most familiar due to its use for annotation in existing PDF viewers, underlining was too subtle to consistently attract attention, and masking required additional cognitive effort to shift one's eyes between masked and unmasked text.
Overall, highlighting was recommended by users as the most effective cueing mechanism, which we incorporated into the current design of Scim.

\textit{DM4. Support error recovery.}
Automated approaches for identifying and classifying salient content within papers are susceptible to errors.
In addition to systematic errors, these methods may also make relevance errors when integrated into user-facing systems like an augmented reading interface.
For example, suggested content that is important to one reader may be deemed irrelevant by another reader under a particular context.
Reflecting guidelines in developing imperfect AI-infused user applications~\cite{amershi_guidelines_2019}, we aimed to design Scim to retain the agency of users in efficiently recovering from and dismissing fallible AI assistance.
\section{The Scim System}
\vspace{-.25em}
Scim provides an augmented reading interface that organizes and filters salient content within a scientific paper (Figure~\ref{fig:system_diagram}).
The interface is enabled by an end-to-end document processing pipeline that localizes, classifies, and ranks important sentences in a paper.

\makeSystemDiagramFigure


\subsection{User Interface}
Scim's user interface was developed through an iterative design process, guided by our formative study and informal usability studies of initial prototypes.
We indicate whenever a feature or design choice was informed by one of the four design motivations introduced in Section~\ref{sec:design_motivations}.

We implemented the user interface as a web application, with features built atop the PDF rendering platform \texttt{pdf.js}~\cite{mozilla_pdfjs_2022}. Scim's user interface consists of a suite of features that augment the original text document through overlays and side panels, largely allowing readers to peruse the document in its original structure. One goal was to enable readers to effortlessly switch between standard skimming processes and guided reading through augmented interface features. Scim accomplishes this by presenting readers with the full document and incorporating guidance via faceted highlighting throughout the text. Scim also retains text markup that may aid readers in skimming, such as hyperlinks, interactive citations, bold and italicized text, and other visual cues provided by the authors. To minimize the friction in adopting Scim into readers' existing skimming processes, we used common design patterns found in current PDF viewers and integrated development environments.

\subsubsection{Faceted Highlights}
To support readers in identifying salient sentences within the context of a paper, Scim uses \textit{colored highlighting} as visual cues throughout the document (Figure~\ref{fig:user_interface}.A).
The opacity of highlights ensure visual salience while remaining unobtrusive to readers attending to both the highlighted text and its surrounding context (\textit{DM3}).
In-context highlights also give readers an intuition for the structure of the paper relative to the highlighted content, and enabling them to maintain a sense of how much content is skipped when navigating between consecutive highlights.

Our formative study revealed that despite variation in goals and experience, readers had common high-level information needs while skimming.
To support the information foraging process, we organized highlights into \textit{rhetorical facets} based on the type of information they provided.
Numerous schemes exist for sentence-based classification of scientific literature into rhetorical facets.
Coarse-grained schemes classify sentences according to typical section names found in scientific literature~\cite{hirohata_identifying_2008, cohan_pretrained_2019}, and are composed of a small number of rhetorical facets such as Objective, Method, Result, and Conclusions.
Other schemes are more fine-grained and classify based on argumentative zones and conceptual structure~\cite{teufel_articles_2002, teufel_towards_2009, liakata_corpora_2010, liakata_automatic_2012}.

We aimed to select a minimal set of rhetorical facets for Scim that most closely corresponded to the information needs of readers while skimming.
We combined aspects of a coarse-grained schema for classifying scientific abstracts~\cite{cohan_pretrained_2019} and the \textsc{NOV\_ADV} category (i.e., sentences presenting novelty or advantage) from Argumentative Zoning~\cite{teufel_towards_2009} to create a final taxonomy of four rhetorical facets (hereafter referred to as \textit{facets}).
Scim uses four colors to identify highlights associated with each of the four facets: \textsc{Objective} (green), \textsc{Novelty} (orange), \textsc{Method} (blue), and \textsc{Result} (red).
With repeated usage, we envision these colors could become a simple association device for each facet, enabling readers to quickly scroll through a paper and identify highlights of interest.


\subsubsection{Highlight Browser} 
Rather than scrolling through an entire paper, readers may want an efficient means to view all AI-suggested highlights.
Scim provides a \textit{highlight browser} (or browser for short) that displays an condensed list of all highlights distributed throughout the paper.
The browser is implemented as side panel anchored to the right of the reading interface, and can be hidden away when not needed.
Highlights in the browser are ordered by their location within the paper, and demarcated by which section of the paper they are contained within (Figure~\ref{fig:user_interface}.B). A colored indicator is also displayed to the right of each highlight in the browser, offering a subtle cue its classified facet directly within the browser (\textit{DM3}). Readers desiring more context for a particular highlight can click on it within the browser to scroll to its corresponding position within the paper---we refer to this interaction throughout the paper as \textit{context linking} (\textit{DM2}).

\subsubsection{Scrollbar Marks}
Inspired by edit wear affordances~\cite{hill_edit_1992} and scrollbar maps within integrated development environments~\cite{microsoft_scroll_2022}, Scim provides colored \textit{scrollbar annotations} in the vertical document scrollbar (Figure~\ref{fig:user_interface}.D). These marks intend to convey at a glance the quantity and positional distribution of highlights throughout the document (\textit{DM3}).

\subsubsection{Highlight Controls}
Scim also provides several features that enable readers to tailor the highlighting experience to their own skimming needs. At the top of the browser, a \textit{facet palette} (Figure~\ref{fig:user_interface}.C) contains buttons that can be clicked to select a single facet, displaying only highlights of that facet throughout the document, within the browser, and in the scrollbar annotations. Highlights for all facets are enabled by default, and a single button labeled ``Everything'' allows readers to show all highlights again after filtering to a single facet. For more or less automated guidance while skimming, readers can increase or decrease the overall number of displayed highlights via buttons in the menu bar above the document (Figure~\ref{fig:user_interface}.E). Finally, another button allows readers to toggle between Scim and their default document viewer (Figure~\ref{fig:user_interface}.F) (\textit{DM4}).



\subsection{Document Processing Pipeline}

\subsubsection{PDF Component Extraction}
We used the open-source Multimodal Document Analysis (MMDA) library~\cite{mmda_2022} to process all elements within a PDF document, including textual tokens, mathematical symbols, section headers, and metadata. By default, MMDA only provides PDF bounding box detection for pages, tokens, and rows, so we segmented a paper's text into sentences and merged token and row bounding boxes to form sentence bounding boxes. For each sentence we stored its corresponding section headers, which we later use in heuristics for sentence prioritization (described in Section~\ref{sec:additional_heuristics}).

\subsubsection{Salient Sentence Extraction}
To identify sentences of general importance to a reader, we defined an initial salience score for each sentence within a document. We used the Universal Sentence Encoder~\cite{cer_universal_2018} to retrieve an embedding $v$ for each sentence $s$, and calculated the cosine similarity between embeddings for all pairs of sentences.
$$
    s_c = \sum_{v'} \frac{v \cdot v'}{||v|| ||v'||}
$$
The sum represents the global similarity of each sentence to other sentences in the document. The intuition behind this salience measure is that sentences which are most similar to many other sentences in the document are likely to contain information central to the ideas of the document. We clipped negative similarity scores to 0, and normalized all scores to $[0,1]$.

\subsubsection{Facet Classification}
To classify sentences into the previously described facets, we implemented the BERT-based sentence classifier with SciBERT pretrained weights introduced in ~\cite{cohan_pretrained_2019}. We trained the model on three Nvidia Titan X GPUs, keeping the same dropout rate, optimizer, number of epochs, and learning rate noted in the original paper. To create a larger training set, we merged the three splits of the \textsc{CSAbstruct} dataset~\cite{cohan_pretrained_2019}, a corpora of computer science abstracts with sentences annotated according to their rhetorical role (one of Background, Method, Objective, Result, or Other). We used this trained model to classify sentences within papers for three of the facets within our taxonomy: \textsc{Objective}, \textsc{Method}, and \textsc{Result}. For the remaining \textsc{Novelty} facet, since collecting a separate annotated dataset necessary for training was outside the scope of this work, we instead used a small set of lexical heuristics for classification. Specifically, we use a rule-based approach based on discourse and lexical constraints, similar to~\cite{guo_improved_2013}. We classify sentences as a \textsc{Novelty} statement by matching against a lexical set containing words such as \textit{inconsistent} and \textit{however}, and then verifying the presence of nearby indicators of author intent (i.e., aliases of \{we, our, this study\}) and comparison to previous work (i.e., aliases of \{previous, recent\}). While we found these methods were sufficient for developing Scim as a design probe, future iterations of the system could investigate more sophisticated techniques for salient sentence extraction and classification.

\subsubsection{Additional Heuristics} \label{sec:additional_heuristics}
Several heuristics based on common structural patterns within scientific papers were used to refine the ranking of salient sentences.
These heuristics included favoring sentences that appeared at the beginning and ends of paragraphs, and sentences that appeared within a predefined set of ``expected sections'' (e.g., \textsc{Objective} within an Introduction section and \textsc{Novelty} within a Related Work section).
These heuristics were combined with the probability assigned to each classified sentence by the previous sentence classifier and the computed salience scores to produce a sentence prioritization schema, which was used to select which sentences to highlight within the reading interface.
Pilot studies with earlier Scim prototypes found that readers generally expected AI highlights distributed throughout the document rather than concentrated only in particular sections.
To distribute highlights throughout a paper, a final heuristic was included to prioritize sentences within paragraphs not containing other highlighted sentences.

\subsubsection{Final Output}
For each paper, the pipeline outputs a ranked list of sentence objects.
Each sentence object contains the sentence's original text, its classified facet, its bounding box within the PDF, and its enclosing section headers. These sentences were saved to a single JSON file, which was then stored in a remote server and processed by the previously described user interface. 

\section{Usability Study}
We used Scim as a technological design probe~\cite{boehner_how_2007} to explore the impact of an augmented reading interface with faceted highlights on skimming scientific papers.
We aimed to observe the realistic use of the system within two skimming tasks, and to collect feedback from participants on the current design and potential improvements for the system.
We focused our observations from the user study to answer two research questions:

\noindent \textit{RQ1. How do readers use Scim and its faceted highlights for skimming papers?} \\
\noindent \textit{RQ2. How does AI assistance influence readers while skimming papers?}


\subsection{Participants}
We recruited 13 participants (8 male, 5 female) via purposive and snowball sampling, primarily through university mailing lists. Participants were required to have some prior experience in reading or writing scientific papers. Eleven participants were graduate students with between one and six years of experience in a computer science PhD program, one was an undergraduate computer science student, and one was a senior research programmer. Six participants were between the ages of 18-25, and seven participants were between the ages of 26-40. Nine participants described themselves as somewhat experienced or very experienced with reading papers in computer science, and one participant described themself as somewhat inexperienced. Participants self-reported a median of four scientific papers authored, suggesting most were experienced in skimming and reading scientific papers. None of the participants were affiliated with the organization conducting this research. Participants were compensated with \$20 (USD) for their time.


\subsection{Procedure}
We conducted the study remotely through a video conferencing platform. As a result, we could not account for factors such as the technological setup of each participant (e.g., hardware specifications, screen size) and the presence of external distractions. Rather than a limitation, we see these factors as enabling a more diverse evaluation of the system within realistic reading environments. We recorded each participant's screen and audio, and logged all interactions with Scim during the study session.
Each study was between 45 minutes and one hour long, and consisted of three stages: tutorial, skimming tasks, and exit interview.

\subsubsection{Tutorial}
Participants first completed a self-guided tutorial that introduced the features of Scim within a sample paper (selected to be similar in style and length to those used in the skimming tasks). The tutorial directed participants to explicitly perform actions involving usage of each of Scim's features. Participants shared their screen during this process and the experimenters answered any usability questions that arose.

\subsubsection{Skimming Tasks} \label{sec:skimming_tasks}
To reflect the diversity of scenarios in which readers skim scientific papers, we used three tasks to prompt readers into a skimming mindset. Given the cognitive effort and time required to skim multiple scientific papers in a single study session, participants were split into two groups and assigned separate tasks. All participants were provided with the same three papers~\cite{guo_statelens_2019, richardson_decoding_2020, sun_lip-interact_2018} selected from recent proceedings of the ACM Symposium on User Interface Software and Technology. We prompted participants with the following scenario: \textit{Imagine you are a member of a paper reading group, and you need to select a paper to present. Please skim the selected paper to get a high-level gist of its content.}

In the first group, participants were given six minutes to skim a paper, and another six minutes to answer ten true or false questions on high-level concepts from the paper (Task 1). In the second group, participants were given six minutes to skim a paper and another six minutes to list three strengths and weaknesses of the paper (Task 2). After a short break, they were given another six minutes to skim a second paper and fifteen minutes to create an outline of a short presentation for the paper (Task 3). All participants were allowed to revisit the paper and encouraged to talk aloud while completing their tasks. All reading was done using Scim.

\subsubsection{Exit Interview}
After completing the reading tasks, participants completed a System Usability Survey~\cite{brooke_sus_2013}, a ten-question industry-standard survey assessing a system's perceived usability. Participants then answered several questions regarding the anticipated usefulness of each of Scim's features and quality of the automatically generated highlights, and miscellaneous demographic questions. We concluded with a 10-15 minute semi-structured interview with participants about their experience using Scim, probing into features or interactions they liked, areas for improvement, and opportunities for future AI-assisted reading interfaces.


\subsection{Analysis}
Two authors analyzed the interview data, following a qualitative approach described in~\cite{creswell_qualitative_2016}. Throughout the analysis process, transcripts were created from audio recordings, themes developed and refined, and relevant utterances from participants extracted. In our results, we refer to participants with the pseudonyms P1-P13. The utterances presented below were edited to elide identifying information, while preserving their meaning.
\section{Results}

\makeFeatureUsageFigure

\subsection{RQ1. How do readers use Scim and its faceted highlights for skimming papers?}
Most participants were positive about Scim and saw the potential for AI-infused reading interfaces to assist in facilitating the process of knowledge distillation while skimming.
For instance, P4 mentioned that Scim could help facilitate currently manual processes with its automatically generated faceted highlights:
\begin{displayquote}
\textit{I’m working on a lit review project and I realize that your tool will be super helpful given that the information I need to extract from the papers are mostly method and significance/contribution, which are already highlighted by the tool.} 
\end{displayquote}
Scim was also well-regarded by less seasoned readers. For instance, P10 praised Scim's potential for guidance as they continued to develop their own skimming strategies.
\begin{displayquote}
\textit{I think it was quite amazing, especially as a first-year, just getting into this. So I’m always looking for those keywords, and if I don’t find those I feel really lost. For papers that didn’t have those key words, it was very helpful for something to at least give a clear area of where to look.}
\end{displayquote}

By design, Scim offered participants an augmented skimming experience that resembled closely a traditional document reader. Participants found the system ``easy to use'' (P8) and ``very understandable'' (P9), and skimmed by scrolling linearly through a paper, using the faceted highlights as cues for visual attention.
Many appeared to adopt a satisficing skimming strategy, relying on the faceted highlights and more conventional cues (e.g., lead sentences of paragraphs, presence of jargon, visual content) to approximate the perceived importance of a particular section of a paper.
This dominant usage pattern is to be expected since participants were given a short amount of time to skim a long research paper, participants were relatively unfamiliar with the system (other than interactions from the tutorial), and paper reading processes are deeply ingrained within researcher practices.

Some found that faceted highlights helped them attend to areas of the paper they might have skipped otherwise. For example, one participant noted that the \textsc{Novelty} highlights in the Related Work section---a section they typically skip during skimming---prompted them to slow down and read highlighted sentences with relevant information (P3). Participants also mentioned how AI support could be particularly helpful while skimming under time pressure, since they believed the AI-powered highlights could surface sentences of greater importance than readers could identify manually given limited time (P1, P5, P7, P8, P9, P10).

In addition to the inline faceted highlights, participants also used Scim's highlight browser to support navigation and scaffold comprehension while skimming a paper. Rather than scrolling through a paper itself, participants would instead scroll through the list of faceted highlights in the browser (P1, P3, P7, P9). Figure~\ref{fig:feature_usage} illustrates this pattern, specifically in rows where scrolling interactions through the paper (shown in light blue) are punctuated by scrolling interactions in the highlight browser (red markers) and context linking interactions from highlights in the browser to the paper context (purple markers).
The highlight browser offered readers a sentence-level index into the paper, allowing efficient contextual exploration of a particular highlight of interest.


Participants used Scim's facet palette to filter highlights to reflect specific reading goals.
For instance, they often began a skim by searching for information that conveyed the paper's significance, using the facet palette to filter highlights to two specific facets---\textsc{Objective} and \textsc{Novelty}---before reading through the filtered highlights in the browser.
Some would further click on each highlight to view it in the context of the paper, after which they would resume skimming linearly through the paper.
Other participants opted for a multi-pass approach: they first skimmed linearly through the paper, then used the facet palette to filter the highlights to one or more facets of interest (e.g., \textsc{Method} or \textsc{Result}), and performed one or more additional passes over the paper with the filtered highlights.


Participants also saw Scim as potentially useful in assimilating the knowledge gained while skimming. For instance, P3 mentioned how they used Scim's highlight browser as a reminder of what they had previously skimmed in the paper and to verify their understanding:

\begin{displayquote}
\textit{It was more useful as a post-reading exercise. After I skimmed the entire thing, I'd go into the sidebar, click on the different colored tabs and just skim through the sentences corresponding to that filter. Just to summarize again for myself and get a gist of the paper.}
\end{displayquote}

\textit{Usability.}
Participants gave Scim an average SUS score of 86.8, indicating excellent overall usability (based on the canonical interpretation of SUS scores ~\cite{bangor_determining_2009}). They lauded Scim for its visual and interactive simplicity, and its ability to nest AI-powered features within an intuitive layout which paralleling software for other reading contexts. Those reporting lower usability scores noted the potential for inconsistency within the AI highlights, resulting in an additional cognitive load in occasionally needing to verify the accuracy of the AI highlights.

\textit{Additional Features.}
Participants suggested additional features they believed would improve the usability of Scim.
Some mentioned annotating papers during their usual skimming sessions, and asked for analogous features to engage with the faceted highlights within Scim, such as the ability to add, update, and annotate highlights.
The increased interactivity may also help Scim better support both skimming and deep reading processes.
To persist the augmented reading experience, some suggested Scim should support exporting automated highlights and any user-generated annotations as a post-reading summary.

\makeFeaturePreferencesFigure

\subsection{RQ2. How does AI assistance influence readers while skimming papers?}
Numerous guidelines have emerged over the past two decades from the human-computer interaction literature outlining design paradigms for effective human interaction with AI-infused systems~\cite{amershi_guidelines_2019, horvitz_principles_1999, norman_how_1994}.
We present our findings from Scim to expand upon these guidelines within the specific context of AI-infused reading interfaces.

\subsubsection{Errors in AI-Powered Highlighting}
Participants identified several types of errors that they encountered while using Scim: (i) salience detection errors---highlighting of an unimportant sentence (False Positive) or lack of highlighting of an important sentence (False Negative), (ii) facet classification errors, and (iii) PDF processing errors.
Most participants were not affected by unimportant highlights while scrolling through a paper since they could be easily ignored, and mentioned simply spending more time during a skim in sections of interest when Scim provided less highlights than expected.
While individual errors did not appear to strongly influence the usability of Scim, participants appeared to develop a judgment on the perceived quality of the AI highlights while skimming a paper.
They were quick to form initial impressions, focusing on the precision of the first few highlights in the paper or highlight browser.
Those who gained a positive impression continued to use Scim for skimming and completing the tasks:

\begin{displayquote}
\textit{This was the first sentence that I clicked. This actually pointed out a very important sentence in this paper. At that time, I felt like the tool was very powerful. (P7)}
\end{displayquote}

Others who deemed the quality of the highlights too low based on their first impressions disabled the features of Scim to complete their skim of subsequent papers (P11, P12).
Our results echo prior studies on trust formation and automation reliance that found trust declines rapidly and slowly increases over periods of appropriate behavior~\cite{muir_trust_1994, dzindolet_role_2003}.

While some participants felt their experience with the system was strongly affected by errors made by the AI models (P5, P11, P12), others were more tolerant to potential errors, since they felt the benefits of having automated reading guidance outweighed the potential for errors (P10).
Our results do not indicate any tolerable level of error within AI-infused reading interfaces, but rather that expectations of individual researchers will drive the eventual decision to adopt these intelligent tools.
That said, we found that errors in Scim served to violate an assumption of the system's trustworthiness, and were therefore better remembered than ``correct'' highlights made by the system.

To reduce the impact of errors, some participants suggested Scim could include affordances that offered readers better transparency into the AI model. For instance, the system could convey the AI's confidence in highlighting particular sentences (e.g., by varying the opacity of highlights or with a confidence score), or how the AI selected the prioritization of sentences to be shown when readers modified the density of highlights with the highlight controls.


\subsubsection{Trade-offs in Precision And Recall for AI-Powered Highlights}
An alternative approach to reduce the impact of errors in AI-infused systems is to focus on calibrating the system to a high precision, and hence avoid false positive errors at the expense of recall.
We found that participants' opinions varied regarding the importance of recall and precision within the AI-powered highlights.
Pilot studies with Scim suggested that readers typically wanted to see a greater number of highlights distributed throughout the paper, a reassurance of the system's ability to recall all important information.
While in our usability study some participants echoed this preference for high recall, others preferred fewer highlights with greater precision:
\begin{displayquote}
\textit{If it began with a place where the highlights were restricted to where the AI has very high confidence, and it’s up to me to populate with more noise, that might be more effective for doing a first pass through deciding if I want to read. (P12)}
\end{displayquote}

Within AI-infused reading interfaces, a lack of AI-powered highlights requires a reader to revert to a typical skimming and reading flow (appearing to provide insufficient support).
In contrast, an overabundance of irrelevant AI-powered highlights requires a reader to recover from errors by ignoring those highlighted sentences.
While the cognitive effort required to ignore individual highlights may not be significant, these loads may accumulate over time, resulting in lowering impressions of AI competence or abandonment of the reading interface entirely.

\subsubsection{Potentially Biased Narratives from AI-Powered Highlighting}
While automated highlights may identify important content for readers, the resultant story that is crafted by the selected highlights could unintentionally offer a biased skim of a paper.
For instance, P8 noted how they relied heavily on Scim's highlights:

\begin{displayquote}
\textit{Having the highlights in the paper and then having the different colors was a really helpful system as I’m skimming through. My eyes definitely focused on the highlights, and maybe read 10\% of what wasn’t highlighted..which is good and bad.}
\end{displayquote}

They further described their typical skimming technique involved a penchant for critique, but felt that skimming only highlights provided by the AI caused them to feel unnaturally positive about a paper:

\begin{displayquote}
\textit{I felt that I got a very positive impression of the work based on the highlights...which for me is surprising because I tend to pick out tiny details...and really complain about the small details like a wizard of oz method. But when I'm hopping from highlight to highlight there was a bit of skew towards the good, important things are highlighted.}
\end{displayquote}

Though we had designed Scim's facets based on common information needs of skimming readers, facets such as \textsc{Objective} and \textsc{Novelty} may by nature tend to favor spotlighting sentences that reinforce the strengths of the paper rather than its limitations.
The other facets---\textsc{Method} and {Result}---also tend to prescribe objective sentences to readers, and therefore do not typically expose the limitations of a paper that may be surfaced in a longer, more critical read.


\subsubsection{Toward Adopting AI-Infused Reading Interfaces}
Overall, participants appeared optimistic about the potential for intelligent reading interfaces to support the paper skimming experience despite their potential for errors, but suggested improvements to enable future adoption. Participants desired more control and personalization over highlights suggested by the system, as many believed there should be some fine-tuning into what was highlighted for each individual reader. One participant remarked how an interpretation of a paper represented by AI-powered highlights may resonate across different readers, though perhaps highlights pertaining to more technical aspects may be more universally useful.

\begin{displayquote}
\textit{Reading these research articles is like art, music. It's based on interpretation, it's like a very abstract concept. Different people reading papers will have different points...Though I think there are definitely some common concepts that's similar across different papers and very technical things that can be extrapolated by the AI, like methods. (P11)}
\end{displayquote}

Participants also believed that continued usage of Scim would be useful to familiarize themselves with the various features, facet color associations, and reliability of the AI-powered highlights. One participant mentioned how their ``main issue is getting used to seeing highlights that aren't my own'' (P13), suggesting that readers may need some time to acclimate to AI assistance in traditionally manual processes.

\vspace{5em}
\section{Discussion}

\balance

\subsection{Design Recommendations}
\textit{R1. Support details-on-demand.}
Reading interfaces should offer readers the ability to view details-on-demand.
Scim provided a context linking feature which connected readers from highlights in the browser to its context in the paper.
Participants also suggested alternative designs for details-on-demand.
One suggested providing a summary of a paper that could be iteratively ``zoomed in'' on, perhaps visualized as a tree-like structure.
Readers could then interact with sentences in the tree, expanding it on-demand to display additional detail from the paper, akin to Wikum, a tool for recursive summarization of discussion forums~\cite{zhang_wikum_2017}.

\textit{R2. Prefer a consistent distribution of generated highlights.}
Through initial usability testing in the iterative design process of Scim, we found that readers generally preferred highlights distributed throughout the entire paper, since they expected salient content within most sections of the paper.
When skimming past extended regions of text without highlights, readers believed the AI system was missing relevant content, and devolved their skim into a detailed reading of those sections.
Developers of future AI-infused reading interfaces should carefully consider the impact of probabilistic features (e.g., the distribution of generated highlights) on reader perceptions, accounting for potentially adverse effects.

\textit{R3.Support passive and active reading processes.}
While some participants preferred to passively skim papers, others engaged in more active reading such as highlighting and note-taking in an external tool while skimming.
Reading interfaces should consider providing affordances that support both passive and active reading processes.
For instance, they could present nested features---colored highlights support readers in passive skimming, and clicking on a highlight could reveal tools for on-demand annotation.
Their implementation should be uncluttered and intuitive, supporting readers while minimizing any introduced distractions from reading.

\textit{R4. Persist augmented reading beyond a single reading session.}
Participants in our study wanted to export paper annotations after an initial skim.
Augmented reading interfaces provide an additional layer of information atop the paper itself that readers spend cognitive effort in consuming.
These systems should therefore give users the ability to retain the reading process beyond the lifespan of a single reading session.
The exported information represents a snapshot of the human-AI collaborative reading process, including both AI-provided augmentations such as faceted highlights and annotations created by the reader.
Persistence can also manifest as saving user preferences or interactions with the paper (e.g., likes or dislikes).

\textit{R5. Support readers in recovering from errors in AI features.}
Instantiating guidelines for designing interactions with imperfect AI-infused systems~\cite{amershi_guidelines_2019}, AI-infused reading interfaces should offer readers paths to recovery when the AI models err.
In Scim, visualizing AI suggestions as highlights atop a paper naturally lends itself to a simple recovery mechanism when the highlights are irrelevant or inadequate---ignore the highlight, or read the surrounding text to gain additional context.
However, AI errors may not always be easily recoverable from by design, for instance if paper context is inaccessible in a particular reading interface.
Reading interfaces should also enable readers to stay in control of their reading experience---provide readers with the ability to efficiently disable and reactivate the AI augmentations within the reading interface.
They should also assist readers in developing an intuition for the system's reliability---offer transparency into the system's reasoning or confidence in highlighting a particular sentence.
This recommendation hearkens back to our fourth design motivation for Scim (Section~\ref{sec:design_motivations}).

\textit{R6. Reduce biases induced by the AI-infused reading interface.}
Some participants in our study relied on skimming mostly content highlighted by the AI, reading less of the non-highlighted text than they may have in a traditional skim.
In our results, we discussed potential consequences of this behavior---one participant believed Scim inadvertently induced a positive bias on their skim of a paper through its selective highlighting.
Future AI-infused reading interfaces should carefully consider how the design and visualization of their AI-powered features may impact readers, particularly in offering a biased or selectively incomplete narrative of a paper.


\subsection{Limitations}
One limitation of our study is its focus on only one category of papers within the human-computer interaction field of computer science. Papers across computer science (let alone other scholarly disciplines) can vary greatly in structure, clarity, or length. Despite grounding the design of Scim in established natural language processing techniques and user interface paradigms, future studies should evaluate the system's generalizability on a variety of papers.

Participants had limited time to interact with Scim during the user study, and interacted with a small set of predetermined papers. Some participants expected to benefit more from Scim with continued usage, particularly as they gained familiarity with the faceted highlights, facets' color associations, and types of reading tasks the system could support. Though, given longitudinal access to Scim, participants may also discover limitations that could preclude them from further adoption. While observations from our study suggest readers would like to use Scim within numerous reading scenarios, we believe a longitudinal deployment with in-situ feedback would be valuable in assessing how readers might truly use the tool ``in the wild''.


\subsection{Future Work}
\subsubsection{Highlights via collaborative reading.}
One reason readers may be hesitant to adopt an augmented reading interface like Scim is distrust of the system's ability to provide the most relevant highlights.
Some were concerned whether AI-powered highlights could ever instill enough confidence to allow readers to navigate between only the automated highlights.
Instead, some mentioned potentially greater trust in highlights created by other people (e.g., fellow researchers).
We note that platforms such as Medium show ``popular highlights,'' which appear popular.
Combining social and AI-powered highlights raises interesting design challenges.

\subsubsection{Eliciting user interactions for improved highlights.}
As readers continue to interact with augmented papers, we envision an opportunity for AI models to adaptively learn from these user interactions.
Future systems could elicit and learn from passive (e.g., dwell patterns on certain parts of a paper) or active (e.g., highlighting a sentence or removing an AI-generated highlight) user interaction signals.
They could also explore user interactions with features designed for conveying the uncertainty of individual highlights and recovery from AI errors as trainable instances for improving the underlying models.

\subsubsection{Tailoring the reading experience through personalization.}
In addition to improving the overall performance of the models, future reading interfaces could be tailored to individual readers' characteristics.
One could explore approaches for modeling long-term user attributes---experience generally reading scientific papers, experience with the research area of the current paper, types of information a reader typically looks for in a paper, and their expectations for AI assistance within a reading interface---or short-term states for a particular paper, such as their goals for reading the paper or the number of times the paper had been read previously.
\section{Conclusion}
We presented Scim, an augmented reading interface that provides faceted highlights to support researchers when skimming scientific papers.
Using Scim as a design probe with 13 participants, we found that colored highlights placed within the paper and organized into rhetorical facets within a highlight browser complemented participants' existing skimming processes by facilitating behaviors such as rereading and guided reading.
We found that participants were eager to use Scim's contextually-linked faceted highlights and highlight browser in their future reading interfaces, and were more broadly enthusiastic about the potential of AI-augmented reading interfaces.
Based on qualitative insights from our study, we identified several design recommendations that we hope can inform and inspire designers of future augmented reading interfaces.



