Question: A generative design method for creating layouts of rooted tree diagrams.

This paper delves into the discrepancies between the theoretical underpinnings and practical application of the thinking-aloud method in usability testing, a technique widely employed for gathering user feedback. Initially conceptualized by Ericsson and Simon in their influential work, "Protocol Analysis: Verbal Reports as Data," the thinking-aloud protocol is grounded in the theory that verbalizing thoughts during task execution can provide valuable insights into cognitive processes. However, the authors of this paper observe that the current practice of usability experts often deviates significantly from the original theoretical framework. They note that the pragmatic approach to thinking aloud does not always adhere to the stringent guidelines proposed by Ericsson and Simon, which emphasize the importance of eliciting undirected, undisturbed, and consistent verbal reports.

In response to these discrepancies, the authors propose the adoption of the concept of "speech genre" as an alternative theoretical framework for the thinking-aloud method. They argue that this new perspective could potentially reconcile the theoretical and practical aspects of the technique. The concept of speech genre allows for a more nuanced understanding of the verbal reports generated during usability testing, acknowledging the influence of context and situational factors on the participants' speech. This approach not only aligns with the goal of obtaining undirected and undisturbed verbal reports but also addresses the practical challenges encountered in usability testing that require some level of intervention, a practice not explicitly supported in the original model proposed by Ericsson and Simon. By integrating the concept of speech genre, the thinking-aloud method can be enhanced to better capture the complexities of user cognition while remaining adaptable to the dynamic nature of usability testing environments.

In many contexts, humans often represent their own “neighborhood” in great detail, yet only major landmarks further away. This suggests that such views (“fisheye views”) might be useful for the computer display of large information structures like programs, data bases, online text, etc. This paper explores fisheye views presenting, in turn, naturalistic studies, a general formalism, a specific instantiation, a resulting computer program, example displays and an evaluation.

Many methods have been developed to recognize those progresses of technologies, and one of them is to analyze patent information. And visualization methods are considered to be proper for representing patent information and its analysis results. However, current visualization methods for patent analysis patent maps have some drawbacks. Therefore, we propose an alternative visualization method in this paper. With colleted keywords from patent documents of a target technology field, we cluster patent documents by the k-Means algorithm. With the clustering results, we form a semantic network of keywords without respect of filing dates. And then we build up a patent map by rearranging each keyword node of the semantic network according to its earliest filing date and frequency in patent documents. Our approach contributes to establishing a patent map which considers both structured and unstructured items of a patent document. Besides, differently from previous visualization methods for patent analysis, ours is based on forming a semantic network of keywords from patent documents. And thereby it visualizes a clear overview of patent information in a more comprehensible way. And as a result of those contributions, it enables us to understand advances of emerging technologies and forecast its trend in the future. 2007 Elsevier Ltd. All rights reserved.

Radial visualization, or the practice of displaying data in a circular or elliptical pattern, is an increasingly common technique in information visualization research. In spite of its prevalence, little work has been done to study this visualization paradigm as a methodology in its own right. We provide a historical review of radial visualization, tracing it to its roots in centuries-old statistical graphics. We then identify the types of problem domains to which modern radial visualization techniques have been applied. A taxonomy for radial visualization is proposed in the form of seven design patterns encompassing nearly all recent works in this area. From an analysis of these patterns, we distill a series of design considerations that system builders can use to create new visualizations that address aspects of the design space that have not yet been explored. It is hoped that our taxonomy will provide a framework for facilitating discourse among researchers and stimulate the development of additional theories and systems involving radial visualization as a distinct design metaphor.

Effectively showing the relationships between objects in a dataset is one of the main tasks in information visualization. Typically there is a well-defined notion of distance between pairs of objects, and traditional approaches such as principal component analysis or multi-dimensional scaling are used to place the objects as points in 2D space, so that similar objects are close to each other. In another typical setting, the dataset is visualized as a network graph, where related nodes are connected by links. More recently, datasets are also visualized as maps, where in addition to nodes and links, there is an explicit representation of groups and clusters. We consider these three Techniques, characterized by a progressive increase of the amount of encoded information: node diagrams, node-link diagrams and node-link-group diagrams. We assess these three types of diagrams with a controlled experiment that covers nine different tasks falling broadly in three categories: node-based tasks, network-based tasks and group-based tasks. Our findings indicate that adding links, or links and group representations, does not negatively impact performance (time and accuracy) of node-based tasks. Similarly, adding group representations does not negatively impact the performance of network-based tasks. Node-link-group diagrams outperform the others on group-based tasks. These conclusions contradict results in other studies, in similar but subtly different settings. Taken together, however, such results can have significant implications for the design of standard and domain snecific visualizations tools.

Given a real world graph, how should we lay-out its edges? How can we compress it? These questions are closely related, and the typical approach so far is to find clique-like communities, like the `cavemen graph', and compress them. We show that the block-diagonal mental image of the `cavemen graph' is the wrong paradigm, in full agreement with earlier results that real world graphs have no good cuts. Instead, we propose to envision graphs as a collection of hubs connecting spokes, with super-hubs connecting the hubs, and so on, recursively. Based on the idea, we propose the Slash Burn method (burn the hubs, and slash the remaining graph into smaller connected components). Our view point has several advantages: (a) it avoids the `no good cuts' problem, (b) it gives better compression, and (c) it leads to faster execution times for matrix-vector operations, which are the back-bone of most graph processing tools. Experimental results show that our Slash Burn method consistently outperforms other methods on all datasets, giving good compression and faster running time.

A tutorial survey is presented of the quadtree and related hierarchical data structures. They are based on the principle of recursive decomposition. The emphasis is on the representation of data used in applications in image processing, computer graphics, geographic information systems, and robotics. There is a greater emphasis on region data (i.e., two-dimensional shapes) and to a lesser extent on point, curvilinear, and threedimensional data. A number of operations in which such data structures find use are examined in greater detail.

In February 2008, the New York Times published an unusual chart of box office revenues for 7500 movies over 21 years. The chart was based on a similar visualization, developed by the first author, that displayed trends in music listening. This paper describes the design decisions and algorithms behind these graphics, and discusses the reaction on the Web. We suggest that this type of complex layered graph is effective for displaying large data sets to a mass audience. We provide a mathematical analysis of how this layered graph relates to traditional stacked graphs and to techniques such as ThemeRiver, showing how each method is optimizing a different ldquoenergy functionrdquo. Finally, we discuss techniques for coloring and ordering the layers of such graphs. Throughout the paper, we emphasize the interplay between considerations of aesthetics and legibility.

A document image is composed of a variety of physical entities or regions such as text blocks, lines, words, figures, tables, and background. We could also assign functional or logical labels such as sentences, titles, captions, author names, and addresses to some of these regions. The process of document structure and layout analysis tries to decompose a given document image into its component regions and understand their functional roles and relationships. The processing is carried out in multiple steps, such as preprocessing, page decomposition, structure understanding, etc. We will look into each of these steps in detail in the following sections. Document images are often generated from physical documents by digitization using scanners or digital cameras. Many documents, such as newspapers, magazines and brochures, contain very complex layout due to the placement of figures, titles, and captions, complex backgrounds, artistic text formatting, etc. (see Figure 1). A human reader uses a variety of additional cues such as context, conventions and information about language/script, along with a complex reasoning process to decipher the contents of a document. Automatic analysis of an arbitrary document with complex layout is an extremely difficult task and is beyond the capabilities of the state-of-the-art document structure and layout analysis systems. This is interesting since documents are designed to be effective and clear to human interpretation unlike natural images.

We consider the problem of human-assisted graph search: given a directed acyclic graph with some (unknown) target node(s), we consider the problem of finding the target node(s) by asking an omniscient human questions of the form “Is there a target node that is reachable from the current node?”. This general problem has applications in many domains that can utilize human intelligence, including curation of hierarchies, debugging workflows, image segmentation and categorization, interactive search and filter synthesis. To our knowledge, this work provides the first formal algorithmic study of the optimization of human computation for this problem. We study various dimensions of the problem space, providing algorithms and complexity results. We also compare the performance of our algorithm against other algorithms, for the problem of webpage categorization on a real taxonomy. Our framework and algorithms can be used in the design of an optimizer for crowdsourcing platforms such as Mechanical Turk.

We present DreamSketch, a novel 3D design interface that combines the free-form and expressive qualities of sketching with the computational power of generative design algorithms. In DreamSketch, a user coarsely defines the problem by sketching the design context. Then, a generative design algorithm produces multiple solutions that are augmented as 3D objects in the sketched context. The user can interact with the scene to navigate through the generated solutions. The combination of sketching and generative algorithms enables designers to explore multiple ideas and make better informed design decisions during the early stages of design. Design study sessions with designers and mechanical engineers demonstrate the expressive nature and creative possibilities of DreamSketch.


Question: A creative design method for illustrating rooted tree structures.

This paper introduces the Cone Tree, a novel 3D visualization and interactive animation technique designed to manage and access vast information spaces, particularly addressing challenges in large scale cognition. When the information's structure can be visualized, emerging technologies for 3D visualization offer promising solutions. The Cone Tree specifically targets the visualization of hierarchical information structures, presenting them in a 3D format to optimize the use of screen space and facilitate a comprehensive view of the entire hierarchy. Interactive animation is employed to alleviate cognitive load by leveraging the human perceptual system, thereby enhancing the user's ability to comprehend and navigate complex information architectures.

Traffic analysis, in the context of Telecommunications or Internet and Web data, is crucial for large network operations. Data in such networks is often provided as large graphs with hundreds of millions of vertices and edges. We propose efficient techniques for managing such graphs at the storage level in order to facilitate its processing at the interface level(visualization). The methods are based on a hierarchical decomposition of the graph edge set that is inherited from a hierarchical decomposition of the vertex set. Real time navigation is provided by an efficient two level indexing schema called the gkd*-tree. The first level is a variation of a kd-tree index that partitions the edge set in a way that conforms to the hierarchical decomposition and the data distribution (the gkd-tree). The second level is a redundant R-tree that indexes the leaf pages of the gkd-tree. We provide computational results that illustrate the superiority of the gkd-tree against conventional indexes like the kd-tree and the R*-tree both in creation as well as query response times.

There is increasing evidence that the effectiveness of information visualization techniques can be impacted by the particular needs and abilities of each user. This suggests that it is important to investigate information visualization systems that can dynamically adapt to each user. In this paper, we address the question of how to adapt. In particular, we present a study to evaluate a variety of visual prompts, called "interventions", that can be performed on a visualization to help users process it. Our results show that some of the tested interventions perform better than a condition in which no intervention is provided, both in terms of task performance as well as subjective user ratings. We also discuss findings on how intervention effectiveness is influenced by individual differences and task complexity.

Network data frequently arises in a wide variety of fields, and node-link diagrams are a very natural and intuitive representation of such data. In order for a node-link diagram to be effective, the nodes must be arranged well on the screen. While many graph layout algorithms exist for this purpose, they often have limitations such as high computational complexity or node colocation. This paper proposes a new approach to graph layout through the use of space filling curves which is very fast and guarantees that there will be no nodes that are colocated. The resulting layout is also aesthetic and satisfies several criteria for graph layout effectiveness.

B-trees have been ubiquitous in database management systems for several decades, and they are used in other storage systems as well. Their basic structure and basic operations are well and widely understood including search, insertion, and deletion. Concurrency control of operations in B-trees, however, is perceived as a difficult subject with many subtleties and special cases. The purpose of this survey is to clarify, simplify, and structure the topic of concurrency control in B-trees by dividing it into two subtopics and exploring each of them in depth.

Structure and Algorithms Jakob Puchinger NICTA Victoria Laboratory Department of Computer Science & Software Engineering University of Melbourne, Australia jakobp@csse.unimelb.edu.au Günther R. Raidl Institute of Computer Graphics and Algorithms Vienna University of Technology, Austria raidl@ads.tuwien.ac.at Ulrich Pferschy Department of Statistics and Operations Research University of Graz, Austria pferschy@uni-graz.at

We describe the two most commonly used systems for induction of decision trees for classi cation: C4.5 and CART. We highlight the methods and di erent decisions made in each system with respect to splitting criteria, pruning, noise handling, and other di erentiating features. We describe how rules can be derived from decision trees and point to some di erence in the induction of regression trees. We conclude with some pointers to advanced techniques, including ensemble methods, oblique splits, grafting, and coping with large data.

Designations used by companies to distinguish their products are often claimed as trademarks or registered trademarks. In all instances in which Morgan Kaufmann Publishers is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and registration. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopying, scanning, or otherwise—without prior written permission of the publisher. may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting " Support & Contact " then " Copyright and Permission " and then " Obtaining Permissions. " Page 197: " Make a new plan Stan. .. and get yourself free. Physical database design : the database professional's guide to exploiting indexes, views, storage, and more / Sam Lightstone, Toby Teorey, and Tom Nadeau. p. cm.-(The Morgan Kaufmann series in database management systems) Includes bibliographical references and index.

The objective of security data visualizations is to help cyber analysts to perceive trends and patterns, and gain insights into security data. Sound and systematic evaluations of security data visualizations are rarely performed, in part due to the lack of proper quantitative and qualitative measures. In this paper, we present a novel evaluation approach for security visualization based on Christopher Alexander's fifteen properties of living structures. Alexander's fifteen properties are derived from various visual patterns that appear in nature. Each property represents the guidelines for good design. We believe that using these fundamental properties have the potential for building a more robust evaluation. Each property offers essential qualities that enable better analytical reasoning. We demonstrate how to use Alexander's properties to evaluate security related visualizations. We derive a set of visualization-specific properties for evaluation, developed based on Alexander's original properties.

On the issue of how to explore the complex relationships of entities in a story, Storyline visualization has proven to be a useful approach. The traditional Storyline visualization technology only applies to a linear continuous story and has the limitation of reflecting the development of the plots in a story. In this paper, we propose a hierarchical plot visualization method called StoryCake to improve this problem. First, according to the story elements, inter-sessions are divided into groups, whereby the layout of the entities will be optimized in individual groups. Then, the hierarchical relationships of entities in polar coordinates can be confirmed. Finally, we use several skills, e.g., interaction, labels and a fan-shaped visualization view, to enable people better understand how story evolves and track the hierarchical relationship more conveniently. Experimental results show that the proposed hierarchical plot visualization method in polar coordinates can reveal discontinuous events and nonlinear stories. The results also show that our algorithm presents the narrative structure and development of the story.

In 2 experiments, highand low-spatial ability students viewed a computer-generated animation and listened simultaneously (concurrent group) or successively (successive group) to a narration that explained the workings either of a bicycle tire pump (Experiment 1) or of the human respiratory system (Experiment 2). The concurrent group generated more creative solutions to subsequent transfer problems than did the successive group; this contiguity effect was strong for highbut not for low-spatial ability students. Consistent with a dual-coding theory, spatial ability allows high-spatial learners to devote more cognitive resources to building referential connections between visual and verbal representations of the presented material, whereas low-spatial ability learners must devote more cognitive resources to building representation connections between visually presented material and its visual representation.


Question: A creative design method for illustrating rooted tree diagrams.

This paper delves into the design choices and algorithms that underpin a unique type of complex layered graph, first showcased in a New York Times chart from February 2008. This chart, which depicted box office revenues for over 7500 movies spanning 21 years, was inspired by a similar visualization created by the paper's lead author to illustrate trends in music listening habits. We provide an in-depth exploration of the rationale behind the graphical representation and the public response it garnered online. We argue that this layered graph is particularly effective for presenting vast datasets to a broad audience, balancing aesthetic appeal and comprehensibility. We also conduct a mathematical examination of the relationship between this layered graph, conventional stacked graphs, and methodologies like ThemeRiver, demonstrating how each approach optimizes distinct "energy functions." Additionally, we explore strategies for enhancing the visual appeal and clarity of these layered graphs through coloration and layer organization. Throughout the paper, we underscore the synergy between aesthetic considerations and readability in the design of such graphics.

We propose a novel graph-based approach for constructing concept hierarchy from a large text corpus. Our algorithm, GraBTax, incorporates both statistical co-occurrences and lexical similarity in optimizing the structure of the taxonomy. To automatically generate topic-dependent taxonomies from a large text corpus, GraBTax first extracts topical terms and their relationships from the corpus. The algorithm then constructs a weighted graph representing topics and their associations. A graph partitioning algorithm is then used to re-cursively partition the topic graph into a tax-onomy. For evaluation, we apply GraBTax to articles, primarily computer science, in the CiteSeerX digital library and search engine. The quality of the resulting concept hierarchy is assessed by both human judges and comparison with Wikipedia categories.

The ability to restructure a decision tree efficiently enables a variety of approaches to decision tree induction that would otherwise be prohibitively expensive. Two such approaches are described here, one being incremental tree induction (ITI), and the other being non-incremental tree induction using a measure of tree quality instead of test quality (DMTI). These approaches and several variants offer new computational and classifier characteristics that lend themselves to particular applications.

A well-told story conveys great quantities of information in relatively few words in a format that is easily assimilated by the listener or viewer. People usually find it easier to understand information integrated into stories than information spelled out in serial lists (such as bulleted items in an overhead slide). Stories are also just more compelling. For example, despite its sketchiness, the story fragment in Figure 1 is loaded with information, following an analysis similar to that of John Thomas of IBM Research [5]. We find that Jim uses technology (a pager and the Internet) and is dedicated to his job. Many other pieces of information can be deduced about Jim and his work, as well as about his relationships with his coworkers, as noted in the right side of the figure. The story does not express all this information explicitly; some is only implied; for example, we can surmise that Jim is probably not at the gym and his attendance at the meeting is important to his boss and coworkers, as well as to his company’s business performance. As in most stories, this one involves uncerFor as long as people have been around, they have used stories to convey information, cultural values, and experiences. Since the invention of writing and the printing press until today, technology and culture have constantly provided new and increasingly sophisticated means to tell stories. More recently, technology, entertainment, and art have converged in the computer. The ancient art of storytelling and its adaptation in film and video can now be used to efficiently convey information in our increasingly computerized world. What Storytelling Can Do for Information Visualization

In this work, we advance research efforts in combining the casual sketching approach of whiteboards with the machine's computing power. We present SketchInsight, a system that applies the familiar and collaborative features of a whiteboard interface to the accurate data exploration capabilities of interactive visualizations. SketchInsight enables data analysis with more fluid interaction, allowing people to visually explore their data by drawing simple charts and directly manipulating them. In addition, we report results from a qualitative study conducted to evaluate user experience in exploring data with SketchInsight, expanding our understanding on how people use a pen- and touch-enabled digital whiteboard for data exploration. We also discuss the challenges in building a working system that supports data analytic capabilities with pen and touch interaction and freeform annotation.

New systems are designed with improved quality or features. However, not every new system lives to its pre-development expectations when tested with end-users. The evaluation of the first version of our LOCO-Analyst tool for providing teachers with feedback on students learning activities and performance led to the enhancing the visualization features of the tool. The second evaluation of the improved tool allowed us to see how the improvements affect the user perceptions of the tool. Here, we present the qualitative results of our two evaluations. The results show that educators find the kinds of feedback implemented in the tool informative and they value the mix of textual and graphical representations of different kinds of feedback provided by the tool.

Inspired by the recent advances of deep learning and computer graphics techniques in generating sophisticated artistic styles and architectural designs, we envision a general design generation methodology that uses various specification and generative methods and tools in an integral design cycle.

The analysis of large graphs plays a prominent role in various fields of research and is relevant in many important application areas. Effective visual analysis of graphs requires appropriate visual presentations in combination with respective user interaction facilities and algorithmic graph analysis methods. How to design appropriate graph analysis systems depends on many factors, including the type of graph describing the data, the analytical task at hand, and the applicability of graph analysis methods. The most recent surveys of graph visualization and navigation techniques were presented by Herman et al. [HMM00] and Diaz [DPS02]. The first work surveyed the main techniques for visualization of hierarchies and graphs in general that had been introduced until 2000. The second work concentrated on graph layouts introduced until 2002. Recently, new techniques have been developed covering a broader range of graph types, such as time-varying graphs. Also, in accordance with ever growing amounts of graph-structured data becoming available, the inclusion of algorithmic graph analysis and interaction techniques becomes increasingly important. In this State-of-the-Art Report, we survey available techniques for the visual analysis of large graphs. Our review firstly considers graph visualization techniques according to the type of graphs supported. The visualization techniques form the basis for the presentation of interaction approaches suitable for visual graph exploration. As an important component of visual graph analysis, we discuss various graph algorithmic aspects useful for the different stages of the visual graph analysis process.

Cooking recipes exist in abundance; but due to their unstructured text format, they are hard to study quantitatively beyond treating them as simple bags of words. In this paper, we propose an ingredientinstruction dependency tree data structure to represent recipes. The proposed representation allows for more refined comparison of recipes and recipe-parts, and is a step towards semantic representation of recipes. Furthermore, we build a parser that maps recipes into the proposed representation. The parser’s edge prediction accuracy of 93.5% improves over a strong baseline of 85.7% (54.5% error reduction).

Abstract. In the article, an additional visualization of self-organizing maps (SOM) has been investigated. The main objective of self-organizing maps is data clustering and their graphical presentation. Opportunities of SOM visualization in four systems (NeNet, SOM-Toolbox, Databionic ESOM and Viscovery SOMine) have been investigated. Each system has its additional tools for visualizing SOM. A comparative analysis has been made for two data sets: Fisher’s iris data set and the economic indices of the European Union countries. A new SOM system is also introduced and researched. The system has a specific visualization tool. It is missing in other SOM systems. It helps to see the proportion of neurons, corresponding to the data items, belonging to the different classes, and fallen in the same SOM cell.

Structured Abstract Purpose – Designing a model and tools that are (a) capable of representing and handling personal knowledge in different degrees of structuredness and formalisation and (b) usable and extensible by end-users. Design/methodology/approach – This paper presents the result of analysing literature and various data-models and formalism used to structure information on the desktop. Findings – The unified data model (CDS) is capable of representing structures from various information tools, e. g. documents, file system, hypertext, tagging, and mind maps. The five knowledge axes of CDS are: identity, order, hierarchy, annotation and linking. Research limitations / implications – The CDS model is based on text. Extensions for multimedia annotations have not been investigated. Practical implications – Future PKM tools should take the mentioned shortcoming of existing PKM tools into account. Implementing the CDS model can be a way to make PKM tools interoperable. Originality/value – This paper presents research combining cognitive psychology, personal knowledge management and semantic web technologies. The CDS model shows a way to let end-users work on different levels of granularity and different levels of formality in one environment. Introduction " The most important contribution of management in the 20th century was to increase manual worker productivity fifty-fold. The most important contribution of management in the 21st century will be to increase knowledge worker productivity – hopefully by the same percentage. [...] The methods, however, are totally different from those that increased the productivity of manual workers. " Drucker (1999a, p. 79) What might these methods be? The field of knowledge management investigates since 1995 (Stankosky, 2005) how people and knowledge work together. North (2007) defines knowledge work as work based on knowledge with an immaterial result; value creation is based on processing, generating and communicating knowledge. Polanyi (1966) makes a distinction between explicit knowledge encoded in artefacts such as books or web pages, and tacit knowledge which resides in the individual. The SECI-model of Nonaka (1994) describes knowledge transfers between humans and artefacts. The field of knowledge management has focused on knowledge transfer between people, either via socialisation or via externalised artefacts. The high expectations of central enterprise knowledge repositories have often not been met (Braganza and Mollenkramer, 2002). The following wave of expert finders and corporate white pages focused mostly on connecting the right people and let them communicate. Today, knowledge workers are flooded with information (Alvarado et al., 2003). The field of Personal …


Question: A creative design method for illustrating rooted tree structures.

In "Visual Thinking: Using Graphic Displays to Amplify Understanding," the authors delve into the concept of information visualization, emphasizing its role in enhancing cognitive processes through computer-supported, interactive visual representations of data. This definition underscores the dual focus on both the method of visualization and its ultimate goal—augmenting cognition. As Hamming (1973) eloquently stated, "the aim of computation is insight, not just numbers," paralleling the sentiment that visualization is geared towards generating insight, not merely producing images. Central to this insight are three key objectives: discovery, decision-making, and explanation. Through visual thinking, users can uncover hidden patterns, make informed decisions, and effectively communicate complex ideas, all of which are critical components in the realm of data analysis and interpretation.

**Question**: A generative layout approach for rooted tree drawings

**Documents**: A survey of graph drawing algorithms for rooted trees
This paper provides a comprehensive survey of algorithms for drawing rooted trees, focusing on techniques that ensure readability and aesthetics. A rooted tree is a tree structure in which a particular node, called the root, is distinguished. The drawing of rooted trees is an essential task in many areas such as software engineering, biology, and information visualization. We classify the existing algorithms into three main categories: radial layout, orthogonal layout, and hierarchical layout. Radial layout algorithms arrange the nodes of the tree in concentric circles around the root, while orthogonal layout algorithms draw the tree with horizontal and vertical edges to create a grid-like structure. Hierarchical layout algorithms, on the other hand, position nodes in a top-down manner, with the root at the top and subsequent levels below. Each layout type has its own set of advantages and is suitable for different scenarios. We discuss the characteristics, advantages, and limitations of each layout technique, and compare their performance in terms of visual clarity, space utilization, and drawing time. The survey concludes with a discussion of open problems and future directions in the field of rooted tree drawing.

**Similar Documents**: This comprehensive review explores the landscape of graph drawing algorithms specifically designed for rooted trees, with an emphasis on methods that optimize both readability and aesthetic appeal. Rooted trees, characterized by a designated root node, play a pivotal role in diverse fields including software development, biological studies, and data visualization. The review categorizes the plethora of existing algorithms into three primary groups: radial, orthogonal, and hierarchical layouts. Radial layouts employ a circular arrangement of nodes, with the root at the center, facilitating a clear visual hierarchy. Orthogonal layouts,

Node-link diagrams provide an intuitive way to explore networks and have inspired a large number of automated graph layout strategies that optimize aesthetic criteria. However, any particular drawing approach cannot fully satisfy all these criteria simultaneously, producing drawings with visual ambiguities that can impede the understanding of network structure. To bring attention to these potentially problematic areas present in the drawing, this paper presents a technique that highlights common types of visual ambiguities: ambiguous spatial relationships between nodes and edges, visual overlap between community structures, and ambiguity in edge bundling and metanodes. Metrics, including newly proposed metrics for abnormal edge lengths, visual overlap in community structures and node/edge aggregation, are proposed to quantify areas of ambiguity in the drawing. These metrics and others are then displayed using a heatmap-based visualization that provides visual feedback to developers of graph drawing and visualization approaches, allowing them to quickly identify misleading areas. The novel metrics and the heatmap-based visualization allow a user to explore ambiguities in graph layouts from multiple perspectives in order to make reasonable graph layout choices. The effectiveness of the technique is demonstrated through case studies and expert reviews.

We propose a novel approach to enhance the discriminability of Convolutional Neural Networks (CNN). The key idea is to build a tree structure that could progressively learn fine-grained features to distinguish a subset of classes, by learning features only among these classes. Such features are expected to be more discriminative, compared to features learned for all the classes. We develop a new algorithm to effectively learn the tree structure among a large number of classes. Experiments on large-scale image classification tasks demonstrate that our method could boost the performance of a given basic CNN model. Our method is quite general, hence it can potentially be used in combination with many other deep learning models.

This paper presents a technique for visualizing the interactions between developers in software project evolution. The goal is to produce a visualization that shows more detail than animated software histories, like code_swarm [15], but keeps the same focus on aesthetics and presentation. Our software evolution storylines technique draws inspiration from XKCD's "Movie Narrative Charts" and the aesthetic design of metro maps. We provide the algorithm, design choices, and examine the results of using the storylines technique. Our conclusion is that the it is able to show more details when compared to animated software project history videos. However, it does not scale to the largest projects, such as Eclipse and Mozilla.

We propose a method to precisely segment books on bookshelves in images taken from general viewpoints. The proposed segmentation algorithm overcomes difficulties due to text and texture on book spines, various book orientations under perspective projection, and book proximity. A shape dependent active contour is used as a first step to establish a set of book spine candidates. A subset of these candidates are selected using spatial constraints on the assembly of spine candidates by formulating the selection problem as the maximal weighted independent set (MWIS) of a graph. The segmented book spines may be used by recognition systems (e.g., library automation), or rendered in computer graphics applications. We also propose a novel application that uses the segmented book spines to assist users in bookshelf reorganization or to modify the image to create a bookshelf with a tidier look. Our method was successfully tested on challenging sets of images.

B-trees have been ubiquitous in database management systems for several decades, and they are used in other storage systems as well. Their basic structure and basic operations are well and widely understood including search, insertion, and deletion. Concurrency control of operations in B-trees, however, is perceived as a difficult subject with many subtleties and special cases. The purpose of this survey is to clarify, simplify, and structure the topic of concurrency control in B-trees by dividing it into two subtopics and exploring each of them in depth.

There are many interface schemes that allow users to work at, and move between, focused and contextual views of a dataset. We review and categorize these schemes according to the interface mechanisms used to separate and blend views. The four approaches are overview+detail, which uses a spatial separation between focused and contextual views; zooming, which uses a temporal separation; focus+context, which minimizes the seam between views by displaying the focus within the context; and cue-based techniques which selectively highlight or suppress items within the information space. Critical features of these categories, and empirical evidence of their success, are discussed. The aim is to provide a succinct summary of the state-of-the-art, to illuminate both successful and unsuccessful interface strategies, and to identify potentially fruitful areas for further work.

As a typical type of structured documents, table drawings are widely used in engineering fields. Knowledge extraction of such structured documents plays an important role in automatic interpretation systems. In this paper, we propose a new knowledge extraction method based on automatically analyzing drawing layout and extracting physical or logical structures from the given engineering table drawings. Then based on the automatic interpretation results, we further propose normalization method to integrate varied types of engineering tables with other engineering drawings and extract implied domain knowledge.

A s computers grow more powerful, less expensive , and more widely available, people are expecting them not only to perform obvious computational tasks, but also to assist in people-oriented tasks, such as writing , drawing, and designing. This shift is causing some user-interface (UI) researchers to rethink the traditional reliance on methods that are more machine-oriented and to look at ways to support properties like ambiguity, creativity, and informal communication. The idea is to bend computers to people's way of interacting , not the other way around. This flexibility is particularly important in the early stages of UI design itself, when designers need the freedom to sketch rough design ideas quickly, the ability to test designs by interacting with them, and the flexibility to fill in the design details as they make choices. 1 Tools at this stage must support conceptual design, which is characterized by ambiguity and the need to create several design variations quickly, as the " Why Sketching Is Important " sidebar describes. Unfortunately , with current UI tools, designers tend to focus on issues such as colors, fonts, and alignment, which are more appropriate later in the design. Thus, most UI designers resort to sketching ideas on paper, but these are hard to edit and inconvenient for user evaluations. (CMU) have designed, implemented, and evaluated SILK (Sketching Interfaces Like Krazy), an informal sketching tool that combines many of the benefits of paper-based sketching with the merits of current electronic tools. With SILK, designers can quickly sketch an interface using an electronic pad and stylus, and SILK recognizes widgets and other interface elements as the designer draws them. Unlike paper-based sketching, however, designers can exercise these elements in their sketchy state. For example, a sketched scrollbar is likely to contain an elevator or thumbnail, the small rectangle a user drags with a mouse. In a paper sketch, the elevator would just sit there, but in a SILK sketch, designers can drag it up and down, which lets them test the components' or widgets' behavior. SILK also supports the creation of storyboards—the arrangement of sketches to show how design elements behave, such as how a dialog box appears when the user activates a button. Storyboards are important because they give designers a way to show colleagues, customers, or end users early on how an interface will behave. Designers can test the interface at any point, not just …

This paper presents a framework for knowledge discovery and concept exploration. In order to enhance the concept exploration capability of knowledge-based systems and to alleviate the limitations of the manual browsing approach, we have developed two spreading activation-based algorithms for concept exploration in large, heterogeneous networks of concepts (e.g., multiple thesauri). One algorithm, which is based on the symbolic Al paradigm, performs a conventional branch-and-bound search on a semantic net representation to identify other highly relevant concepts (a serial, optimal search process). The second algorithm, which is based on the neural network approach, executes the Hopfield net parallel relaxation and convergence process to identify “convergent” concepts for some initial queries (a parallel, heuristic search process). Both algorithms can be adopted for automatic, multiple-thesauri consultation. We tested these two algorithms on a large text-based knowledge network of about 13,000 nodes (terms) and 80,000 directed links in the area of computing technologies. This knowledge network was created from two external thesauri and one automatically generated thesaurus. We conducted experiments to compare the behaviors and performances of the two algorithms with the hypertext-like browsing process. Our experiment revealed that manual browsing achieved higher-term recall but lower-term precision in comparison to the algorithmic systems. However, it was also a much more laborious and cognitively demanding process. In document retrieval, there were no statistically significant differences in document recall and precision between the algorithms and the manual browsing process. In light of the effort required by the manual browsing process, our proposed algorithmic approach presents a viable option for efficiently traversing largescale, multiple thesauri (knowledge network).

In optimizing the area of Very Large Scale Integrated (VLSI) layouts, circuit interconnections should generally be realized with minimum total interconnect. This chapter addresses several variations of the corresponding fundamental Steiner minimal tree (SMT) problem, where a given set of pins is to be connected using minimum total wirelength. Steiner trees are important in global routing and wirelength estimation [15], as well as in various non-VLSI applications such as phylogenetic tree reconstruction in biology [48], network routing [61], and civil engineering, among many other areas [21, 25, 26, 29, 51, 74].


Question: Analyzing audio adversarial examples through the lens of temporal dependency.

This paper investigates the application of conditional generative adversarial networks (cGANs) to speech enhancement (SE) and noise-robust speaker verification, aiming to enhance speech system performance in noisy environments. Inspired by the success of GANs in image processing, we adapt the framework introduced by Isola et al. [1] to learn a mapping from noisy speech spectrograms to enhanced versions. The proposed SE cGAN system comprises two components: a generator, tasked with enhancing the input noisy spectrogram, and a discriminator, which distinguishes between enhanced spectrograms produced by the generator and clean spectrograms from a database, conditioned on the noisy input. Performance evaluation is conducted using perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and equal error rate (EER) for speaker verification. Our experimental findings reveal that the cGAN approach outperforms the traditional short-time spectral amplitude minimum mean square error (STSA-MMSE) SE algorithm and matches the performance of a deep neural network-based SE method (DNN-SE), demonstrating its effectiveness in speech enhancement and noise-robust speaker verification tasks.

Many machine learning image classifiers are vulnerable to adversarial attacks, inputs with perturbations designed to intentionally trigger misclassification. Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls: pixel perturbations smaller than a specified magnitude, according to a measurement norm. This evaluation, however, has limited practical utility since perturbations in the pixel space do not correspond to underlying real-world phenomena of image formation that lead to them and has no security motivation attached. Pixels in natural images are measurements of light that has interacted with the geometry of a physical scene. As such, we propose a novel evaluation measure, parametric normballs, by directly perturbing physical parameters that underly image formation. One enabling contribution we present is a physically-based differentiable renderer that allows us to propagate pixel gradients to the parametric space of lighting and geometry. Our approach enables physically-based adversarial attacks, and our differentiable renderer leverages models from the interactive rendering literature to balance the performance and accuracy trade-offs necessary for a memory-efficient and scalable adversarial data augmentation workflow.

Temporal features is important for face anti-spoofing. Unfortunately existing methods have limitations to explore such temporal features. In this work, we propose a deep neural network architecture combining Long Short-Term Memory (LSTM) units with Convolutional Neural Networks (CNN). Our architecture works well for face anti-spoofing by utilizing the LSTM units' ability of finding long relation from its input sequences as well as extracting local and dense features through convolution operations. Our best model shows significant performance improvement over general CNN architecture (5.93% vs. 7.34%), and hand-crafted features (5.93% vs. 10.00%) on CASIA dataset.

We study the problem of analyzing falsifying traces of cyber-physical systems. Specifically, given a system model and an input which is a counterexample to a property of interest, we wish to understand which parts of the inputs are "responsible" for the counterexample as a whole. Whereas this problem is well known to be hard to solve precisely, we provide an approach based on learning from repeated simulations of the system under test.
 Our approach generalizes the classic concept of "one-at-a-time" sensitivity analysis used in the risk and decision analysis community to understand how inputs to a system influence a property in question. Specifically, we pose the problem as one of finding a neighborhood of inputs that contains the falsifying counterexample in question, such that each point in this neighborhood corresponds to a falsifying input with a high probability. We use ideas from statistical hypothesis testing to infer and validate such neighborhoods from repeated simulations of the system under test. This approach not only helps to understand the sensitivity of these counterexamples to various parts of the inputs, but also generalizes or widens the given counterexample by returning a neighborhood of counterexamples around it.
 We demonstrate our approach on a series of increasingly complex examples from automotive and closed loop medical device domains. We also compare our approach against related techniques based on regression and machine learning.

The emanations of electronic and mechanical devices have raised serious privacy concerns. It proves possible for an attacker to recover the keystrokes by acoustic signal emanations. Most existing malicious applications adopt context-based approaches, which assume that the typed texts are potentially correlated. Those approaches often incur a high cost during the context learning stage, and can be limited by randomly typed contents (e.g., passwords). Also, context correlations can increase the risk of successive false recognition. We present a context-free and geometry-based approach to recover keystrokes. Using off-the-shelf smartphones to record acoustic emanations from keystrokes, this design estimates keystrokes' physical positions based on the Time Difference of Arrival (TDoA) method. We conduct extensive experiments and the results show that more than 72.2\% of keystrokes can be successfully recovered.

Singing pitch estimation and singing voice separation are challenging due to the presence of music accompaniments that are often nonstationary and harmonic. Inspired by computational auditory scene analysis (CASA), this paper investigates a tandem algorithm that estimates the singing pitch and separates the singing voice jointly and iteratively. Rough pitches are first estimated and then used to separate the target singer by considering harmonicity and temporal continuity. The separated singing voice and estimated pitches are used to improve each other iteratively. To enhance the performance of the tandem algorithm for dealing with musical recordings, we propose a trend estimation algorithm to detect the pitch ranges of a singing voice in each time frame. The detected trend substantially reduces the difficulty of singing pitch detection by removing a large number of wrong pitch candidates either produced by musical instruments or the overtones of the singing voice. Systematic evaluation shows that the tandem algorithm outperforms previous systems for pitch extraction and singing voice separation.

We present a new stochastic learning machine for speech separation based on the variational recurrent neural network (VRNN). This VRNN is constructed from the perspectives of generative stochastic network and variational auto-encoder. The idea is to faithfully characterize the randomness of hidden state of a recurrent neural network through variational learning. The neural parameters under this latent variable model are estimated by maximizing the variational lower bound of log marginal likelihood. An inference network driven by the variational distribution is trained from a set of mixed signals and the associated source targets. A novel supervised VRNN is developed for speech separation. The proposed VRNN provides a stochastic point of view which accommodates the uncertainty in hidden states and facilitates the analysis of model construction. The masking function is further employed in network outputs for speech separation. The benefit of using VRNN is demonstrated by the experiments on monaural speech separation.

When considering the problem of audio-to-audio matching, determining musical similarity using low-level features such as Fourier transforms and MFCCs is an extremely difficult task, as there is little semantic information available. Full semantic transcription of audio is an unreliable and imperfect task in the best case, an unsolved problem in the worst. To this end we propose a robust mid-level representation that incorporates both harmonic and rhythmic information, without attempting full transcription. We describe a process for creating this representation automatically, directly from multi-timbral and polyphonic music signals, with an emphasis on popular music. We also offer various evaluations of our techniques. Moreso than most approaches working from raw audio, we incorporate musical knowledge into our assumptions, our models, and our processes. Our hope is that by utilizing this notion of a musically-motivated mid-level representation we may help bridge the gap between symbolic and audio research.

This work describes a system for acoustic scene classification using large-scale audio feature extraction. It is our contribution to the Scene Classification track of the IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (D-CASE). The system classifies 30 second long recordings of 10 different acoustic scenes. From the highly variable recordings, a large number of spectral, cepstral, energy and voicing-related audio features are extracted. Using a sliding window approach, classification is performed on short windows. SVM are used to classify these short segments, and a majority voting scheme is employed to get a decision for longer recordings. On the official development set of the challenge, an accuracy of 73 % is achieved. SVM are compared with a nearest neighbour classifier and an approach called Latent Perceptual Indexing, whereby SVM achieve the best results. A feature analysis using the t-statistic shows that mainly Mel spectra are the most relevant features.

Behavior of Machine Learning Algorithms in Adversarial Environments

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on image manipulations, such as scratches or changes to camera angle or lighting conditions, and define safety for an image classification decision in terms of invariance of the classification with respect to manipulations of the original image within a region of images that are close to it. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.


Question: Analyzing audio adversarial instances by utilizing temporal dependence.

This paper introduces an innovative solution to the unsupervised domain adaptation challenge in robust speech recognition, a critical issue when training and testing data exhibit significant domain disparities. Such mismatches often result in performance degradation, a common obstacle in real-world speech recognition applications. The focus here is on scenarios where both source and target domain speech data are accessible, yet word transcripts are exclusively available for the source domain. The proposed methodology hinges on data augmentation techniques that manipulate speech signals without altering their transcripts. Initially, a variational autoencoder is trained on a combination of source and target domain data (without any supervision) to capture the latent speech representation. Subsequently, the latent representations are modified to alter the speech's nuisance attributes—those not pertinent to recognition—thus expanding the labeled training data with synthetic samples more closely aligned with the target domain distribution. This novel approach is validated on the CHiME-4 dataset, demonstrating a substantial reduction in word error rate (WER) of up to 35% when compared to a baseline model without adaptation.

Machine learning (ML) models, e.g., state-of-the-art deep neural networks (DNNs), are vulnerable to adversarial examples: malicious inputs modified to yield erroneous model outputs, while appearing unmodified to human observers. Potential attacks include having malicious content like malware identified as legitimate or controlling vehicle behavior. Yet, all existing adversarial example attacks require knowledge of either the model internals or its training data. We introduce the first practical demonstration of an attacker controlling a remotely hosted DNN with no such knowledge. Indeed, the only capability of our black-box adversary is to observe labels given by the DNN to chosen inputs. Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN. We then use the local substitute to craft adversarial examples, and find that they are misclassified by the targeted DNN. To perform a real-world and properly-blinded evaluation, we attack a DNN hosted by MetaMind, an online deep learning API. After labeling 6,400 synthetic inputs to train our substitute (an order of magnitude smaller than the training set used by MetaMind), we find that their DNN misclassifies adversarial examples crafted with our substitute at a rate of 84.24%. We demonstrate that our strategy generalizes to many ML techniques like logistic regression or SVMs, regardless of the ML model chosen for the substitute. We instantiate the same attack against models hosted by Amazon and Google, using logistic regression substitutes trained with only 800 label queries. They yield adversarial examples misclassified by Amazon and Google at rates of 96.19% and 88.94%. We also find that this black-box attack strategy is capable of evading defense strategies previously found to make adversarial example crafting harder. We conclude with experiments exploring why adversarial samples transfer between models.

In this paper, we review recent results on audiovisual (AV) fusion. We also discuss some of the challenges and report on approaches to address them. One important issue in AV fusion is how the modalities interact and influence each other. This review will address this question in the context of AV speech processing, and especially speech recognition, where one of the issues is that the modalities both interact but also sometimes appear to desynchronize from each other. An additional issue that sometimes arises is that one of the modalities may be missing at test time, although it is available at training time; for example, it may be possible to collect AV training data while only having access to audio at test time. We will review approaches to address this issue from the area of multiview learning, where the goal is to learn a model or representation for each of the modalities separately while taking advantage of the rich multimodal training data. In addition to multiview learning, we also discuss the recent application of deep learning (DL) toward AV fusion. We finally draw conclusions and offer our assessment of the future in the area of AV fusion.

In contrast to the wealth of saliency models in the vision literature, there is a relative paucity of models exploring auditory saliency. In this work, we integrate the approaches of (Kayser, Petkov, Lippert, & Logothetis, 2005) and (Zhang, Tong, Marks, Shan, & Cottrell, 2008) and propose a model of auditory saliency. The model combines the statistics of natural soundscapes and the recent past of the input signal to predict the saliency of an auditory stimulus in the frequency domain. To evaluate the model output, a simple behavioral experiment was performed. Results show the auditory saliency maps calculated by the model to be in excellent accord with human judgments of saliency.

Audio recordings of gunshots can provide information about the gun location with respect to the microphone(s), the speed and trajectory of the projectile, and in some cases the type of firearm and ammunition. Recordings obtained under carefully controlled conditions can be well-modeled by geometrical acoustics. Special acoustic processing systems for real time gunshot detection and localization are used by the military and law enforcement agencies for sniper detection. Forensic analysis of audio recordings is also used to provide evidence in criminal and civil cases. This paper reviews the distinctive features and limitations of acoustic gunshot analysis using DSP techniques

At its peak, the Internet-of-Things will largely be composed of low-power devices with wireless radios attached. Yet, secure authentication of these devices amidst adversaries with much higher power and computational capability remains a challenge, even for advanced cryptographic and wireless security protocols. For instance, a high-power software radio could simply replay chunks of signals from a low-power device to emulate it. This paper presents a deep-learning classifier that learns hardware imperfections of low-power radios that are challenging to emulate, even for high- power adversaries. We build an LSTM framework, specifically sensitive to signal imperfections that persist over long durations. Experimental results from a testbed of 30 low-power nodes demonstrate high resilience to advanced software radio adversaries.

A novel, data-driven approach to voice activity detection is presented. The approach is based on Long Short-Term Memory Recurrent Neural Networks trained on standard RASTA-PLP frontend features. To approximate real-life scenarios, large amounts of noisy speech instances are mixed by using both read and spontaneous speech from the TIMIT and Buckeye corpora, and adding real long term recordings of diverse noise types. The approach is evaluated on unseen synthetically mixed test data as well as a real-life test set consisting of four full-length Hollywood movies. A frame-wise Equal Error Rate (EER) of 33.2% is obtained for the four movies and an EER of 9.6% is obtained for the synthetic test data at a peak SNR of 0 dB, clearly outperforming three state-of-the-art reference algorithms under the same conditions.

This paper evaluates the potential of convolutional neural networks in classifying short audio clips of environmental sounds. A deep model consisting of 2 convolutional layers with max-pooling and 2 fully connected layers is trained on a low level representation of audio data (segmented spectrograms) with deltas. The accuracy of the network is evaluated on 3 public datasets of environmental and urban recordings. The model outperforms baseline implementations relying on mel-frequency cepstral coefficients and achieves results comparable to other state-of-the-art approaches.

We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster. We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on a single GPU server.

Deep learning is at the heart of the current rise of artificial intelligence. In the field of computer vision, it has become the workhorse for applications ranging from self-driving cars to surveillance and security. Whereas, deep neural networks have demonstrated phenomenal success (often beyond human capabilities) in solving complex problems, recent studies show that they are vulnerable to adversarial attacks in the form of subtle perturbations to inputs that lead a model to predict incorrect outputs. For images, such perturbations are often too small to be perceptible, yet they completely fool the deep learning models. Adversarial attacks pose a serious threat to the success of deep learning in practice. This fact has recently led to a large influx of contributions in this direction. This paper presents the first comprehensive survey on adversarial attacks on deep learning in computer vision. We review the works that design adversarial attacks, analyze the existence of such attacks and propose defenses against them. To emphasize that adversarial attacks are possible in practical conditions, we separately review the contributions that evaluate adversarial attacks in the real-world scenarios. Finally, drawing on the reviewed literature, we provide a broader outlook of this research direction.

Neural networks have been found vulnerable to adversarial examples. Carefully chosen perturbations to real images, while imperceptible to humans, induce misclassification and threaten the reliability of deep learning systems in the wild. To guard against adversarial examples, we take inspiration from game theory. We cast the problem as a minimax zero-sum game between the adversary and the model. For such games, it is in general optimal for players to use a stochastic policy, also known as a mixed strategy. In light of this, we propose a mixed strategy for the model: stochastic activation pruning. SAP prunes a random subset of activations (preferentially pruning activations with smaller magnitude) and scales up the survivors to compensate. SAP can be applied to pre-trained neural networks—even adversarially trained models—without fine-tuning, providing robustness against adversarial examples. Experiments demonstrate that SAP confers robustness in the adversarial setting, increasing accuracy whilst preserving calibration.


Question: Analyzing audio adversarial examples through the lens of temporal dependency.

Introducing MagNet: A Robust Defense Mechanism against Adversarial Examples
Deep learning models have achieved remarkable success in solving complex perceptual tasks. Nonetheless, they have been shown to be susceptible to small, meticulously designed perturbations that are imperceptible to the human eye. These perturbations can lead deep learning models to misclassify adversarial examples, posing significant risks in safety-critical or security-sensitive applications. Previous defense strategies against adversarial examples either focused on specific types of attacks or proved to be ineffective.
In this paper, we present MagNet, a novel framework designed to protect neural network classifiers from adversarial examples. Unlike existing solutions, MagNet does not alter the original classifier and operates without prior knowledge of the adversarial example generation process. It consists of detector networks and a reformer network. The detector networks learn to distinguish between benign and adversarial examples by approximating the manifold of normal examples. Their effectiveness lies in their ability to generalize without assuming a specific adversarial generation process. The reformer network, on the other hand, shifts adversarial examples back towards the manifold of normal examples, which is crucial for accurately classifying adversarial examples with minimal perturbation. We acknowledge the challenges in defending against whitebox attacks and propose a strategy to combat graybox attacks. Drawing inspiration from cryptographic principles, we incorporate diversity to enhance MagNet's robustness. Through empirical evaluations, we demonstrate that MagNet is highly effective against state-of-the-art attacks in both blackbox and graybox settings, all while maintaining a low false positive rate on legitimate examples.

This work proposes a method to exploit both audio and visual speech information to extract a target speaker from a mixture of competing speakers. The work begins by taking an effective audio-only method of speaker separation, namely the soft mask method, and modifying its operation to allow visual speech information to improve the separation process. The audio input is taken from a single channel and includes the mixture of speakers, where as a separate set of visual features are extracted from each speaker. This allows modification of the separation process to include not only the audio speech but also visual speech from each speaker in the mixture. Experimental results are presented that compare the proposed audio-visual speaker separation with audio-only and visual-only methods using both speech quality and speech intelligibility metrics.

A computational model of auditory attention to environmental sound, inspired by the structure of the human auditory system, is presented. The model simulates how listeners switch their attention over time between different auditory streams, based on bottom-up and top-down cues. The bottom-up cues are determined by the time-dependent saliency of each stream. The latter is calculated on the basis of an auditory saliency map, which encodes the intensity and the amount of spectral and temporal irregularities of the sound, and binary spectro-temporal masks for the different streams. The top-down cues are determined by the amount of volitional focusing on particular auditory streams. A competitive winner-takes-all mechanism, which balances bottom-up and top-down attention for each stream, determines which stream is selected for entry into the working memory. Consequently, the model is able to delimit the time periods during which particular streams are paid attention to. Although the main ideas could be applied to all types of sound, the implementation of the model was targeted at environmental sound in particular. As an illustration, the model is used to reproduce results from a detailed field experiment on the perception of transportation noise. Finally, it is shown how this model could be a valuable tool, complementing auralization, in the design of outdoor soundscapes.

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets SomethingSomething, Jester, and Charades which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.

Nowadays most audio content identification systems are based on watermarking technology. In this paper we present a different technology, referred to as robust audio hashing. By extracting robust features and translating them into a bit string, we get an object called a robust hash. Content can then be identified by comparing hash values of a received audio clip with the hash values of previously stored original audio clips. A distinguishing feature of the proposed hash scheme is its ability to extract a bit string for every so many milliseconds. More precisely, for every windowed time interval a hash value of 32 bits is computed by thresholding energy differences of several frequency bands. A sequence of 256 hash values, corresponding to approximately 3 seconds of audio, can uniquely identify a song. Experimental results show that the proposed scheme is robust against severe compression, but bit errors do occur. This implies that searching and matching is a non-trivial task for large databases. For instance, a brute force search approach is already prohibitive for databases containing hash values of more than 100 songs. Therefore we propose a scheme that exploits a structured search algorithm that allows searching databases containing over 100,000 songs.

The effect of smearing the temporal envelope on the speech-reception threshold (SRT) for sentences in noise and on phoneme identification was investigated for normal-hearing listeners. For this purpose, the speech signal was split up into a series of frequency bands (width of 1/4, 1/2, or 1 oct) and the amplitude envelope for each band was low-pass filtered at cutoff frequencies of 0, 1/2, 1, 2, 4, 8, 16, 32, or 64 Hz. Results for 36 subjects show (1) a severe reduction in sentence intelligibility for narrow processing bands at low cutoff frequencies (0-2 Hz); and (2) a marginal contribution of modulation frequencies above 16 Hz to the intelligibility of sentences (provided that lower modulation frequencies are completely present). For cutoff frequencies above 4 Hz, the SRT appears to be independent of the frequency bandwidth upon which envelope filtering takes place. Vowel and consonant identification with nonsense syllables were studied for cutoff frequencies of 0, 2, 4, 8, or 16 Hz in 1/4-oct bands. Results for 24 subjects indicate that consonants are more affected than vowels. Errors in vowel identification mainly consist of reduced recognition of diphthongs and of confusions between long and short vowels. In case of consonant recognition, stops appear to suffer most, with confusion patterns depending on the position in the syllable (initial, medial, or final).

Despite the enormous interest in emotion classification from speech, the impact of noise on emotion classification is not well understood. This is important because, due to the tremendous advancement of the smartphone technology, it can be a powerful medium for speech emotion recognition in the outside laboratory natural environment, which is likely to incorporate background noise in the speech. We capitalize on the current breakthrough of Recurrent Neural Network (RNN) and seek to investigate its performance for emotion classification from noisy speech. We particularly focus on the recently proposed Gated Recurrent Unit (GRU), which is yet to be explored for emotion recognition from speech. Experiments conducted with speech compounded with eight different types of noises reveal that GRU incurs an 18.16% smaller run-time while performing quite comparably to the Long Short-Term Memory (LSTM), which is the most popular Recurrent Neural Network proposed to date. This result is promising for any embedded platform in general and will initiate further studies to utilize GRU to its full potential for emotion recognition on smartphones.

Whispered speech is a natural mode of speech in which voicing is absent – its acoustics differ significantly from normally spoken speech or so-called neutral speech, such that it is challenging to use only neutral speech to build speech processing and automatic recognition systems that can deal effectively with whisper. At the same time, humans can naturally produce and perceive whispered speech without explicit training. Tonal languages such as Mandarin present an interesting dilemma – tone is primarily encoded by pitch tracks which are absent during whispered speech, but humans can still tell tones apart. How humans manage to process whispered speech well without explicit training on it, whereas machine algorithms fail, is presently an unresolved question which could prove fruitful with study. This, however, is hindered by the lack of suitable, systematically collected corpora. We present iWhisper-Mandarin, a 25-hour parallel corpus of neutral and whispered Mandarin, designed to support research in linguistics and speech technology. We demonstrate and verify that earlier techniques applied to whispered speech from non-tonal languages also work with Mandarin, and present some preliminary studies on voice activity detection and whispered Mandarin speech recognition.

A set of simple new procedures has been developed to enable the real-time manipulation of speech parameters. The proposed method uses pitch-adaptive spectral analysis combined with a surface reconstruction method in the time± frequency region. The method also consists of a fundamental frequency (F0) extraction using instantaneous frequency calculation based on a new concept called `fundamentalness'. The proposed procedures preserve the details of time± frequency surfaces while almost perfectly removing ®ne structures due to signal periodicity. This close-to-perfect elimination of interferences and smooth F0 trajectory allow for over 600% manipulation of such speech parameters as pitch, vocal tract length, and speaking rate, while maintaining high reproductive quality. Ó 1999 Elsevier Science B.V. All rights reserved.

We provide a speech coding scheme employing a generative model based on SampleRNN that, while operating at significantly lower bitrates, matches or surpasses the perceptual quality of state-of-theart classic wide-band codecs. Moreover, it is demonstrated that the proposed scheme can provide a meaningful rate-distortion trade-off without retraining. We evaluate the proposed scheme in a series of listening tests and discuss limitations of the approach.

This paper presents a method for recognizing musical instruments in user-generated videos. Musical instrument recognition from music signals is a well-known task in the music information retrieval (MIR) field, where current approaches rely on the analysis of the good-quality audio material. This work addresses a real-world scenario with several research challenges, i.e. the analysis of user-generated videos that are varied in terms of recording conditions and quality and may contain multiple instruments sounding simultaneously and background noise. Our approach does not only focus on the analysis of audio information, but we exploit the multimodal information embedded in the audio and visual domains. In order to do so, we develop a Convolutional Neural Network (CNN) architecture which combines learned representations from both modalities at a late fusion stage. Our approach is trained and evaluated on two large-scale video datasets: YouTube-8M and FCVID. The proposed architectures demonstrate state-of-the-art results in audio and video object recognition, provide additional robustness to missing modalities, and remains computationally cheap to train.


Question: Analyzing audio adversarial examples through the lens of temporal dependency.

Exploring the Phenomenon of Transferable Adversarial Examples and its Implications for Black-box Attacks

A fascinating characteristic of deep neural networks is their susceptibility to adversarial examples, which exhibit a peculiar ability to transfer across various model architectures. This transferability poses a significant threat to the robustness of deep learning applications, particularly those operating in black-box environments. Prior research has predominantly focused on analyzing this phenomenon using relatively small datasets. In contrast, our comprehensive investigation is the first to extensively explore the transferability of adversarial examples across large-scale models and datasets, and notably, it is also the first to delve into the transferability of targeted adversarial examples in conjunction with their intended labels.

Our study encompasses both non-targeted and targeted adversarial examples, revealing that while non-targeted adversarial examples are easily transferable, targeted examples generated through conventional methods rarely transfer with their designated labels. To address this limitation, we introduce innovative ensemble-based strategies for generating targeted adversarial examples that maintain their labels during transfer. With these methods, we observe, for the first time, a substantial proportion of targeted adversarial examples successfully transferring with their labels.

We also provide insightful geometric analyses to facilitate a deeper understanding of the nature of transferable adversarial examples. Our findings culminate in demonstrating the effectiveness of adversarial examples produced via ensemble-based approaches in successfully attacking Clarifai.com, a prominent black-box image classification system, thereby underscoring the critical need for enhanced security measures against adversarial attacks in real-world applications.

We present a method to create universal, robust, targeted adversarial image patches in the real world. The patches are universal because they can be used to attack any scene, robust because they work under a wide variety of transformations, and targeted because they can cause a classifier to output any target class. These adversarial patches can be printed, added to any scene, photographed, and presented to image classifiers; even when the patches are small, they cause the classifiers to ignore the other items in the scene and report a chosen target class.

Whispered speech is a natural mode of speech in which voicing is absent – its acoustics differ significantly from normally spoken speech or so-called neutral speech, such that it is challenging to use only neutral speech to build speech processing and automatic recognition systems that can deal effectively with whisper. At the same time, humans can naturally produce and perceive whispered speech without explicit training. Tonal languages such as Mandarin present an interesting dilemma – tone is primarily encoded by pitch tracks which are absent during whispered speech, but humans can still tell tones apart. How humans manage to process whispered speech well without explicit training on it, whereas machine algorithms fail, is presently an unresolved question which could prove fruitful with study. This, however, is hindered by the lack of suitable, systematically collected corpora. We present iWhisper-Mandarin, a 25-hour parallel corpus of neutral and whispered Mandarin, designed to support research in linguistics and speech technology. We demonstrate and verify that earlier techniques applied to whispered speech from non-tonal languages also work with Mandarin, and present some preliminary studies on voice activity detection and whispered Mandarin speech recognition.

Videos serve to convey complex semantic information and ease the understanding of new knowledge. However, when mixed semantic meanings from different modalities (i.e., image, video, text) are involved, it is more difficult for a computer model to detect and classify the concepts (such as flood, storm, and animals). This paper presents a multimodal deep learning framework to improve video concept classification by leveraging recent advances in transfer learning and sequential deep learning models. Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNN) models are then used to obtain the sequential semantics for both audio and textual models. The proposed framework is applied to a disaster-related video dataset that includes not only disaster scenes, but also the activities that took place during the disaster event. The experimental results show the effectiveness of the proposed framework.

This paper explores the limits of audio ranging on mobile devices in the context of a keystroke snooping scenario. Acoustic keystroke snooping is challenging because it requires distinguishing and labeling sounds generated by tens of keys in very close proximity. Existing work on acoustic keystroke recognition relies on training with labeled data, linguistic context, or multiple phones placed around a keyboard --- requirements that limit usefulness in an adversarial context. In this work, we show that mobile audio hardware advances can be exploited to discriminate mm-level position differences and that this makes it feasible to locate the origin of keystrokes from only a single phone behind the keyboard. The technique clusters keystrokes using time-difference of arrival measurements as well as acoustic features to identify multiple strokes of the same key. It then computes the origin of these sounds precise enough to identify and label each key. By locating keystrokes this technique avoids the need for labeled training data or linguistic context. Experiments with three types of keyboards and off-the-shelf smartphones demonstrate scenarios where our system can recover $94\%$ of keystrokes, which to our knowledge, is the first single-device technique that enables acoustic snooping of passwords.

Mel Frequency Cepstral Coefficients (MFCC) and Perceptual Linear Prediction (PLP) are the most popular acoustic features used in speech recognition. Often it depends on the task, which of the two methods leads to a better performance. In this work we develop acoustic features that combine the advantages of MFCC and PLP. Based on the observation that the techniques have many similarities, we revise the processing steps of PLP. In particular, the filter-bank, the equal-loudness pre-emphasis and the input for the linear prediction are improved. It is shown for a broadcast news transcription task and a corpus of children’s speech that the new variant of PLP performs better than both MFCC and conventional PLP for a wide range of clean and noisy acoustic conditions.

When considering the problem of audio-to-audio matching, determining musical similarity using low-level features such as Fourier transforms and MFCCs is an extremely difficult task, as there is little semantic information available. Full semantic transcription of audio is an unreliable and imperfect task in the best case, an unsolved problem in the worst. To this end we propose a robust mid-level representation that incorporates both harmonic and rhythmic information, without attempting full transcription. We describe a process for creating this representation automatically, directly from multi-timbral and polyphonic music signals, with an emphasis on popular music. We also offer various evaluations of our techniques. Moreso than most approaches working from raw audio, we incorporate musical knowledge into our assumptions, our models, and our processes. Our hope is that by utilizing this notion of a musically-motivated mid-level representation we may help bridge the gap between symbolic and audio research.

The emanations of electronic and mechanical devices have raised serious privacy concerns. It proves possible for an attacker to recover the keystrokes by acoustic signal emanations. Most existing malicious applications adopt context-based approaches, which assume that the typed texts are potentially correlated. Those approaches often incur a high cost during the context learning stage, and can be limited by randomly typed contents (e.g., passwords). Also, context correlations can increase the risk of successive false recognition. We present a context-free and geometry-based approach to recover keystrokes. Using off-the-shelf smartphones to record acoustic emanations from keystrokes, this design estimates keystrokes' physical positions based on the Time Difference of Arrival (TDoA) method. We conduct extensive experiments and the results show that more than 72.2\% of keystrokes can be successfully recovered.

Detection and rejection of adversarial examples in security sensitive and safety-critical systems using deep CNNs is essential. In this paper, we propose an approach to augment CNNs with out-distribution learning in order to reduce misclassification rate by rejecting adversarial examples. We empirically show that our augmented CNNs can either reject or classify correctly most adversarial examples generated using well-known methods (> 95% for MNIST and > 75% for CIFAR-10 on average). Furthermore, we achieve this without requiring to train using any specific type of adversarial examples and without sacrificing the accuracy of models on clean samples significantly (< 4%).

Many machine learning image classifiers are vulnerable to adversarial attacks, inputs with perturbations designed to intentionally trigger misclassification. Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls: pixel perturbations smaller than a specified magnitude, according to a measurement norm. This evaluation, however, has limited practical utility since perturbations in the pixel space do not correspond to underlying real-world phenomena of image formation that lead to them and has no security motivation attached. Pixels in natural images are measurements of light that has interacted with the geometry of a physical scene. As such, we propose a novel evaluation measure, parametric normballs, by directly perturbing physical parameters that underly image formation. One enabling contribution we present is a physically-based differentiable renderer that allows us to propagate pixel gradients to the parametric space of lighting and geometry. Our approach enables physically-based adversarial attacks, and our differentiable renderer leverages models from the interactive rendering literature to balance the performance and accuracy trade-offs necessary for a memory-efficient and scalable adversarial data augmentation workflow.

In 2010, the outbreak of Stuxnet sounded a warning in the field of industrial control.security. As the major attack form of Stuxnet, data injection attack is characterized by high concealment and great destructiveness. This paper proposes a new method to detect data injection attack in Industrial Control Systems (ICS), in which Long Short Term Memory Recurrent Neural Network (LSTM-RNN) is a temporal sequences predictor. We then use the Euclidean detector to identify attacks in a model of a chemical plant. With simulation and evaluation in Tennessee Eastman (TE) process, we show that this method is able to detect various types of data injection attacks.