Question: Incorporating information visualization into visual depiction methods.

This paper explores the concept of visualization criticism as a bridge connecting information visualization with artistic expression, highlighting the often overlooked artistic dimensions in the field. Traditional classifications of visualization methods frequently focus on technical aspects, neglecting the aesthetic and creative potential of visualizing information. By acknowledging and understanding the distinctions between information visualization and broader forms of visual communication, we can uncover new avenues for innovation and creativity. We propose a novel classification scheme that incorporates aesthetic considerations into the categorization of information visualization techniques. Here, we differentiate between artistic and pragmatic visualizations, delving into their unique characteristics and applications. The introduction of visualization criticism offers a framework for evaluating visualizations beyond mere technical execution or functionality. It encourages a shift in perspective, moving from a narrow focus on implementation details or immediate interactions to a deeper exploration of the meaning and impact of visual representations. Visualization criticism fosters a more holistic appreciation of the visual medium, guiding viewers to appreciate the artistry and intent behind the design. This approach not only enriches the viewer's experience but also inspires creators to push the boundaries of what is possible in information visualization, merging functionality with artistry.

In a visualization task, every nonlinear projection method needs to make a compromise between trustworthiness and continuity. In a trustworthy projection the visualized proximities hold in the original data as well, whereas a continuous projection visualizes all proximities of the original data. We show experimentally that one of the multidimensional scaling methods, curvilinear components analysis, is good at maximizing trustworthiness. We then extend it to focus on local proximities both in the input and output space, and to explicitly make a user-tunable parameterized compromise between trustworthiness and continuity. The new method compares favorably to alternative nonlinear projection methods.

As part of the information fusion task we wish to automatically fuse information derived from the text extraction process with data from a structured knowledge base. This process will involve resolving, aggregating, integrating and abstracting information via the methodologies of Knowledge Representation and Reasoning into a single comprehensive description of an individual or event. This report surveys the key principles underlying research in the field of Knowledge Representation and Reasoning. It represents an initial step in deciding upon a Knowledge Representation and Reasoning system for our information fusion task. APPROVED FOR PUBLIC RELEASE

Pythagoras Trees are fractals that can be used to depict binary hierarchies. But this binary encoding is an obstacle for visualizing hierarchical data such as file systems or phylogenetic trees, which branch into n sub-hierarchies. Although any hierarchy can be modeled as a binary one by subsequently dividing n-ary branches into a sequence of n − 1 binary branches, we follow a different strategy. In our approach extending Pythagoras Trees to arbitrarily branching trees, we only need a single visual element for an n-ary branch instead of spreading the binary branches along a strand. Each vertex in the hierarchy is visualized as a rectangle sized according to a metric. We analyze several visual parameters such as length, width, order, and color of the nodes against the use of different metrics. The usefulness of our technique is illustrated by two case studies visualizing directory structures and a large phylogenetic tree. We compare our approach with existing tree diagrams and discuss questions of geometry, perception, readability, and aesthetics.

The scatterplot matrix (SPLOM) is a well-established technique to visually explore high-dimensional data sets. It is characterized by the number of scatterplots (plots) of which it consists of. Unfortunately, this number quadratically grows with the number of the data set’s dimensions. Thus, a SPLOM scales very poorly. Consequently, the usefulness of SPLOMs is restricted to a small number of dimensions. For this, several approaches already exist to explore such “small” SPLOMs. Those approaches address the scalability problem just indirectly and without solving it. Therefore, we introduce a new greedy approach to manage “large” SPLOMs with more than hundred dimensions. We establish a combined visualization and interaction scheme that produces intuitively interpretable SPLOMs by combining known quality measures, a pre-process reordering, and a perception-based abstraction. With this scheme, the user can interactively find large amounts of relevant plots in large SPLOMs.

Issues about city utility services reported by citizens can provide unprecedented insights into the various aspects of such services. Analysis of these issues can improve living quality through evidence-based decision making. However, these issues are complex, because of the involvement of spatial and temporal components, in addition to having multi-dimensional and multivariate natures. Consequently, exploring utility service problems and creating visual representations are difficult. To analyze these issues, we propose a visual analytics process based on the main tasks of utility service management. We also propose an aggregate method that transforms numerous issues into legible events and provide visualizations for events. In addition, we provide a set of tools and interaction techniques to explore such issues. Our approach enables administrators to make more informed decisions.

Parallel coordinate plots (PCPs) are a well-known visualization technique for viewing multivariate data. In the past, various visual modifications to PCPs have been proposed to facilitate tasks such as correlation and cluster identification, to reduce visual clutter, and to increase their information throughput. Most modifications pertain to the use of color and opacity, smooth curves, or the use of animation. Although many of these seem valid improvements, only few user studies have been performed to investigate this, especially with respect to cluster identification. We performed a user study to evaluate cluster identification performance – with respect to response time and correctness – of nine PCP variations, including standard PCPs. To generate the variations, we focused on covering existing techniques as well as possible while keeping testing feasible. This was done by adapting and merging techniques, which led to the following novel variations. The first is an effective way of embedding scatter plots into PCPs. The second is a technique for highlighting fuzzy clusters based on neighborhood density. The third is a spline-based drawing technique to reduce ambiguity. The last is a pair of animation schemes for PCP rotation. We present an overview of the tested PCP variations and the results of our study. The most important result is that a fair number of the seemingly valid improvements, with the exception of scatter plots embedded into PCPs, do not result in significant performance gains.

Moments before the launch of every space vehicle, engineering discipline specialists must make a critical go/no-go decision. The cost of a false positive, allowing a launch in spite of a fault, or a false negative, stopping a potentially successful launch, can be measured in the tens of millions of dollars, not including the cost in morale and other more intangible detriments. The Aerospace Corporation is responsible for providing engineering assessments critical to the go/no-go decision for every Department of Defense (DoD) launch vehicle. These assessments are made by constantly monitoring streaming telemetry data in the hours before launch. For this demonstration, we will introduce VizTree, a novel time-series visualization tool to aid the Aerospace analysts who must make these engineering assessments. VizTree was developed at the University of California, Riverside and is unique in that the same tool is used for mining archival data and monitoring incoming live telemetry. Unlike other time series visualization tools, VizTree can scale to very large databases, giving it the potential to be a generally useful data mining and database tool.

Biological studies involve a growing number of distinct high-throughput experiments to characterize samples of interest. There is a lack of methods to visualize these different genomic datasets in a versatile manner. In addition, genomic data analysis requires integrated visualization of experimental data along with constantly changing genomic annotation and statistical analyses. We developed GenomeGraphs, as an add-on software package for the statistical programming environment R, to facilitate integrated visualization of genomic datasets. GenomeGraphs uses the biomaRt package to perform on-line annotation queries to Ensembl and translates these to gene/transcript structures in viewports of the grid graphics package. This allows genomic annotation to be plotted together with experimental data. GenomeGraphs can also be used to plot custom annotation tracks in combination with different experimental data types together in one plot using the same genomic coordinate system. GenomeGraphs is a flexible and extensible software package which can be used to visualize a multitude of genomic datasets within the statistical programming environment R.

Tree visualization is one of the best-studied areas of information visualization; researchers have developed more than 200 visualization and layout techniques for trees. The treevis.net project aims to provide a hand-curated bibliographical reference to this ever-growing wealth of techniques. It offers a visual overview that users can filter to a desired subset along the design criteria of dimensionality, edge representation, and node alignment. Details, including links to the original publications, can be brought up on demand. Treevis.net has become a community effort, with researchers sending in preprints of their tree visualization techniques to be published or pointing out additional information.

In applied visualization research, artifacts are shaped by a series of small design decisions, many of which are evaluated quickly and informally via methods that often go unreported and unverified. Such design decisions are influenced not only by visualization theory, but also by the people and context of the research. While existing applied visualization models support a level of reliability throughout the design process, they fail to explicitly account for the influence of the research context in shaping the resulting design artifacts. In this work, we look to action design research (ADR) for insight into addressing this issue. In particular, ADR offers a framework along with a set of guiding principles for navigating and capitalizing on the disruptive, subjective, human-centered nature of applied design work, while aiming to ensure reliability of the process and design, and emphasizing opportunities for conducting research. We explore the utility of ADR in increasing the reliability of applied visualization design research by: describing ADR in the language and constructs developed within the visualization community; comparing ADR to existing visualization methodologies; and analyzing a recent design study retrospectively through the lens of ADR's framework and principles.


Question: Incorporating information visualization into visual depiction methods.

This paper presents a comprehensive taxonomy for information visualizations, categorizing them by task and data type, to provide a structured approach to understanding and designing visual interfaces. The taxonomy identifies seven distinct data types: one-dimensional, two-dimensional, three-dimensional, temporal, multi-dimensional, tree, and network data. Corresponding to these data types are seven primary tasks that users perform when interacting with visualizations: obtaining an overview, zooming, filtering, accessing details on demand, relating data points, tracking history, and extracting information. This classification system serves as a useful framework for designers and developers aiming to create effective and intuitive visual representations, guiding them through the selection of appropriate visualization techniques for specific data and user tasks. By grounding the design process in the taxonomy, information visualizations can be tailored to enhance user understanding and interaction, ensuring that the visual representations are not only aesthetically pleasing but also functionally effective in conveying the underlying data.

Radial visualization, or the practice of displaying data in a circular or elliptical pattern, is an increasingly common technique in information visualization research. In spite of its prevalence, little work has been done to study this visualization paradigm as a methodology in its own right. We provide a historical review of radial visualization, tracing it to its roots in centuries-old statistical graphics. We then identify the types of problem domains to which modern radial visualization techniques have been applied. A taxonomy for radial visualization is proposed in the form of seven design patterns encompassing nearly all recent works in this area. From an analysis of these patterns, we distill a series of design considerations that system builders can use to create new visualizations that address aspects of the design space that have not yet been explored. It is hoped that our taxonomy will provide a framework for facilitating discourse among researchers and stimulate the development of additional theories and systems involving radial visualization as a distinct design metaphor.

Parallel coordinates is a visualization technique that provides an unbiased representation of high-dimensional data. The parallel configuration of axes treats data dimensions uniformly and is well suited for exploratory visualization. However, first-time users of parallel coordinate visualizations can find the representation confusing and difficult to understand.We used eye tracking to study how parallel coordinate visualizations are perceived, and compared the results to the optimal visual scan path required to complete the tasks. The results indicate that even first-time users quickly learn how to use parallel coordinate visualizations, pay attention to the correct task-specific areas in the visualization, and become rapidly proficient with it.

Early hands-on experiences with the Microsoft Hololens augmented/mixed reality device are reported and discussed, with a general aim of exploring basic 3D visualization. A range of usage cases are tested, including data visualization and immersive data spaces, in-situ visualization of 3D models and full scale architectural form visualization. Ultimately, the Hololens is found to provide a remarkable tool for moving from traditional visualization of 3D objects on a 2D screen, to fully experiential 3D visualizations embedded in the real world.

There are many interface schemes that allow users to work at, and move between, focused and contextual views of a dataset. We review and categorize these schemes according to the interface mechanisms used to separate and blend views. The four approaches are overview+detail, which uses a spatial separation between focused and contextual views; zooming, which uses a temporal separation; focus+context, which minimizes the seam between views by displaying the focus within the context; and cue-based techniques which selectively highlight or suppress items within the information space. Critical features of these categories, and empirical evidence of their success, are discussed. The aim is to provide a succinct summary of the state-of-the-art, to illuminate both successful and unsuccessful interface strategies, and to identify potentially fruitful areas for further work.

Order picking is one of the most important process steps in logistics. Because of their flexibility human beings cannot be replaced by machines. But if workers in order picking systems are equipped with a head-mounted display, Augmented Reality can improve the information visualization.
 In this paper the development of such a system -- called Pick-by-Vision - is presented. The system is evaluated in a user study performed in a real storage environment. Important logistics figures as well as subjective figures were measured. The results show that a Pick-by-Vision system can improve considerably industrial order picking processes.

This paper aims to tackle on Linked Open Data (LOD) customised visualisations. The work is part of an ongoing research on interfaces and tools for helping non-programmers to easily present, analyse and support sense making over large semantic dataset. The customisation is a key aspect of our work. Producing effective customised visualisations is still difficult due to the complexity of the existing tools and the limited set of options they offer, especially to those with little knowledge in LOD and semantic data models. How can we give users full control on the primitives of the visualisation and their properties, without requiring them to master Semantic Web technologies or programming languages? The paper presents a conceptual model that can be used as a reference to build tools for generating customisable infoviews. We used it to conduct a survey on existing tools in terms of customisation capabilities. Starting from the feedback collected in this phase, we will implement these customisation features into some prototypes we are currently experimenting on.

We outline the design of a visualizer, named Vishit, for texts in the Hindi language. The Hindi language is lingua franca in many states of India where people speak different languages. The visualized text serves as a universal language where seamless communication is needed by many people who speak different languages and have different cultures. Vishit consists of the following three major processing steps: language processing, knowledge base creation and scene generation. Initial results from the Vishit prototype are encouraging.

During data preprocessing, analysts spend a significant part of their time and effort profiling the quality of the data along with cleansing and transforming the data for further analysis. While quality metrics—ranging from general to domain-specific measures—support assessment of the quality of a dataset, there are hardly any approaches to visually support the analyst in customizing and applying such metrics. Yet, visual approaches could facilitate users’ involvement in data quality assessment. We present MetricDoc, an interactive environment for assessing data quality that provides customizable, reusable quality metrics in combination with immediate visual feedback. Moreover, we provide an overview visualization of these quality metrics along with error visualizations that facilitate interactive navigation of the data to determine the causes of quality issues present in the data. In this article, we describe the architecture, design, and evaluation of MetricDoc, which underwent several design cycles, including heuristic evaluation and expert reviews as well as a focus group with data quality, human-computer interaction, and visual analytics experts.

Parallel coordinates can visualize multidimensional data efficiently, during to its shortage in displaying time-varying data, we present a new method to add time dimension, which can extend parallel coordinates into 3D space(3D-Parallel Coordinates) consisted of attribute, range and time dimension. So that time-varying multidimensional data can be displayed as polygonal line cluster for recording and analyzing. A technique called clipping shade is taken to highlight datasets around the current time, and results reveal that 3D-Parallel Coordinates can effectively analyze attribute's time-varying character.

Animated transition has been a popular design choice for smoothly switching between different visualization views or layouts, in which movement trajectories are created as cues for tracking objects during location shifting. Tracking moving objects, however, becomes difficult when their movement paths overlap or the number of tracking targets increases. We propose a novel design to facilitate tracking moving objects in animated transitions. Instead of simply animating an object along a straight line, we create "bundled" movement trajectories for a group of objects that have spatial proximity and share similar moving directions. To study the effect of bundled trajectories, we untangle variations due to different aspects of tracking complexity in a comprehensive controlled user study. The results indicate that using bundled trajectories is particularly effective when tracking more targets (six vs. three targets) or when the object movement involves a high degree of occlusion or deformation. Based on the study, we discuss the advantages and limitations of the new technique, as well as provide design implications.


Question: Incorporating information visualization techniques into visual representation methods.

This paper outlines a novel strategy for creating an application-agnostic presentation tool capable of automatically generating high-quality graphical representations, such as bar charts, scatter plots, and interconnected graphs, for relational data. Two main challenges arise: Firstly, the formulation of graphic design principles in a manner that can be utilized by the presentation tool, and secondly, the capability to produce a diverse array of designs to accommodate a broad spectrum of information types. The proposed methodology views graphical presentations as sentences within graphical languages. Graphic design concerns are encapsulated as expressiveness and effectiveness criteria for these languages. Expressiveness criteria ascertain whether a graphical language can adequately convey the intended information, while effectiveness criteria evaluate whether the language optimally utilizes the output medium's capabilities and the human visual system's strengths. To ensure a wide variety of designs, a composition algebra is employed, which systematically combines a limited set of basic graphical languages. Artificial intelligence techniques are integrated into a prototype presentation tool, named APT (A Presentation Tool), which is grounded in the composition algebra and adheres to the outlined graphic design criteria, aiming to automate the creation of visually impactful relational data presentations.

Node-link diagrams provide an intuitive way to explore networks and have inspired a large number of automated graph layout strategies that optimize aesthetic criteria. However, any particular drawing approach cannot fully satisfy all these criteria simultaneously, producing drawings with visual ambiguities that can impede the understanding of network structure. To bring attention to these potentially problematic areas present in the drawing, this paper presents a technique that highlights common types of visual ambiguities: ambiguous spatial relationships between nodes and edges, visual overlap between community structures, and ambiguity in edge bundling and metanodes. Metrics, including newly proposed metrics for abnormal edge lengths, visual overlap in community structures and node/edge aggregation, are proposed to quantify areas of ambiguity in the drawing. These metrics and others are then displayed using a heatmap-based visualization that provides visual feedback to developers of graph drawing and visualization approaches, allowing them to quickly identify misleading areas. The novel metrics and the heatmap-based visualization allow a user to explore ambiguities in graph layouts from multiple perspectives in order to make reasonable graph layout choices. The effectiveness of the technique is demonstrated through case studies and expert reviews.

Natural language processing systems have the ability to analyse not only the sentiment of human language, but also the stance of the speaker. Representing this information visually from unevenly distributed and potentially sparse datasets is challenging, in particular when trying to facilitate exploration and knowledge discovery. We present work on a novel visualisation approach for scalable visualisation of sentiment and stance and provide a language resource of e-government public engagement of 9,278 user comments with stance explicitly declared by the author.

The growing volume and variety of data presents both opportunities and challenges for visual analytics. Addressing these challenges is needed for big data to provide valuable insights and novel solutions for business, security, social media, and healthcare. In the case of temporal event sequence analytics it is the number of events in the data and variety of temporal sequence patterns that challenges users of visual analytic tools. This paper describes 15 strategies for sharpening analytic focus that analysts can use to reduce the data volume and pattern variety. Four groups of strategies are proposed: (1) extraction strategies, (2) temporal folding, (3) pattern simplification strategies, and (4) iterative strategies. For each strategy, we provide examples of the use and impact of this strategy on volume and/or variety. Examples are selected from 20 case studies gathered from either our own work, the literature, or based on email interviews with individuals who conducted the analyses and developers who observed analysts using the tools. Finally, we discuss how these strategies might be combined and report on the feedback from 10 senior event sequence analysts.

Parallel coordinate plots (PCPs) are a well-known visualization technique for viewing multivariate data. In the past, various visual modifications to PCPs have been proposed to facilitate tasks such as correlation and cluster identification, to reduce visual clutter, and to increase their information throughput. Most modifications pertain to the use of color and opacity, smooth curves, or the use of animation. Although many of these seem valid improvements, only few user studies have been performed to investigate this, especially with respect to cluster identification. We performed a user study to evaluate cluster identification performance – with respect to response time and correctness – of nine PCP variations, including standard PCPs. To generate the variations, we focused on covering existing techniques as well as possible while keeping testing feasible. This was done by adapting and merging techniques, which led to the following novel variations. The first is an effective way of embedding scatter plots into PCPs. The second is a technique for highlighting fuzzy clusters based on neighborhood density. The third is a spline-based drawing technique to reduce ambiguity. The last is a pair of animation schemes for PCP rotation. We present an overview of the tested PCP variations and the results of our study. The most important result is that a fair number of the seemingly valid improvements, with the exception of scatter plots embedded into PCPs, do not result in significant performance gains.

Time series data is an important data type in many different application scenarios. Consequently, there are a great variety of approaches for analyzing time series data. Within these approaches different strategies for cleaning, segmenting, representing, normalizing, comparing, and aggregating time series data can be found. When combining these operations, the time series analysis preprocessing workflow has many degrees of freedom. To define an appropriate preprocessing pipeline, the knowledge of experts coming from the application domain has to be included into the design process. Unfortunately, these experts often cannot estimate the effects of the chosen preprocessing algorithms and their parameterizations on the time series. We introduce a system for the visual-interactive exploitation of the preprocessing parameter space. In contrast to ‘black box’-driven approaches designed by computer scientists based on the requirements of domain experts, our system allows these experts to visual-interactively compose time series preprocessing pipelines by themselves. Visual support is provided to choose the right order and parameterization of the preprocessing steps. We demonstrate the usability of our approach with a case study from the digital library domain, in which time-oriented scientific research data has to be preprocessed to realize a visual search and analysis application.

We present the results of three web-based field experiments, in which we evaluate the impact of using initial narrative visualization techniques and storytelling on user-engagement with exploratory information visualizations. We conducted these experiments on a popular news and opinion outlet, and on a popular visualization gallery website. While data-journalism exposes visualizations to a large public, we do not know how effectively this public makes sense of interactive graphics, and in particular if people explore them to gain additional insight to that provided by the journalists. In contrast to our hypotheses, our results indicate that augmenting exploratory visualizations with introductory 'stories' does not seem to increase user-engagement in exploration.

We outline the design of a visualizer, named Vishit, for texts in the Hindi language. The Hindi language is lingua franca in many states of India where people speak different languages. The visualized text serves as a universal language where seamless communication is needed by many people who speak different languages and have different cultures. Vishit consists of the following three major processing steps: language processing, knowledge base creation and scene generation. Initial results from the Vishit prototype are encouraging.

As a typical type of structured documents, table drawings are widely used in engineering fields. Knowledge extraction of such structured documents plays an important role in automatic interpretation systems. In this paper, we propose a new knowledge extraction method based on automatically analyzing drawing layout and extracting physical or logical structures from the given engineering table drawings. Then based on the automatic interpretation results, we further propose normalization method to integrate varied types of engineering tables with other engineering drawings and extract implied domain knowledge.

In 2 experiments, students who lacked prior knowledge about car mechanics read a passage about vehicle braking systems that either contained labeled illustrations of the systems, illustrations without labels, labels without illustrations, or no labeled illustrations. Students who received passages that contained labeled illustrations of braking systems recalled more explanative than nonexplanative information as compared to control groups, and performed better on problem solving transfer but not on verbatim recognition as compared to control groups. Results support a model of meaningful learning in which illustrations can help readers to focus their attention on explanative information in text and to reorganize the information into useful mental models.

We review some applications of human-computer interaction that alleviate the complexity of visual recognition by partitioning it into human and machine tasks to exploit the differences between human and machine capabilities. Human involvement offers advantages, both in the design of automated pattern classification systems, and at the operational level of some image retrieval and classification tasks. Recent development of interactive systems has benefited from the convergence of computer vision and psychophysics in formulating visual tasks as computational processes. Computer-aided classifier design and exploratory data analysis are already well established in pattern recognition and machine learning, but interfaces and functionality are improving. On the operational side, earlier recognition systems made use of human talent only in preprocessing and in coping with rejects. Most current content-based image retrieval systems make use of relevance feedback without direct image interaction. In contrast, some visual object classification systems can exploit such interaction. They require, however, a domain-specific visible model that makes sense to both human and computer.


Question: Incorporating information visualization techniques into visual representation methods.

This paper introduces a technique for visualizing multi-dimensional geometry using parallel coordinates, offering a novel approach to represent analytic and synthetic geometry in N-dimensional space (RN). The method employs a system of parallel coordinates, which establishes a non-projective mapping between N-dimensional and 2-dimensional sets, allowing for the visualization of hypersurfaces through their planar images. These images exhibit geometrical properties that mirror those of the hypersurfaces they represent. When N equals 2, a point ← → line duality is observed, which extends to lines and hyperplanes for higher dimensions, facilitating the representation of polyhedra in RN. The paper delves into the representation of both convex and non-convex hypersurfaces, presenting an algorithm for constructing and displaying any interior point. The visualization technique reveals local properties of the hypersurface and provides insights into the proximity of a point to the boundary. The utility of this method is explored across various domains, including Air Traffic Control, Robotics, Computer Vision, Computational Geometry, Statistics, and Instrumentation, highlighting its broad applicability and potential impact on these fields.

In applied visualization research, artifacts are shaped by a series of small design decisions, many of which are evaluated quickly and informally via methods that often go unreported and unverified. Such design decisions are influenced not only by visualization theory, but also by the people and context of the research. While existing applied visualization models support a level of reliability throughout the design process, they fail to explicitly account for the influence of the research context in shaping the resulting design artifacts. In this work, we look to action design research (ADR) for insight into addressing this issue. In particular, ADR offers a framework along with a set of guiding principles for navigating and capitalizing on the disruptive, subjective, human-centered nature of applied design work, while aiming to ensure reliability of the process and design, and emphasizing opportunities for conducting research. We explore the utility of ADR in increasing the reliability of applied visualization design research by: describing ADR in the language and constructs developed within the visualization community; comparing ADR to existing visualization methodologies; and analyzing a recent design study retrospectively through the lens of ADR's framework and principles.

Many methods have been developed to recognize those progresses of technologies, and one of them is to analyze patent information. And visualization methods are considered to be proper for representing patent information and its analysis results. However, current visualization methods for patent analysis patent maps have some drawbacks. Therefore, we propose an alternative visualization method in this paper. With colleted keywords from patent documents of a target technology field, we cluster patent documents by the k-Means algorithm. With the clustering results, we form a semantic network of keywords without respect of filing dates. And then we build up a patent map by rearranging each keyword node of the semantic network according to its earliest filing date and frequency in patent documents. Our approach contributes to establishing a patent map which considers both structured and unstructured items of a patent document. Besides, differently from previous visualization methods for patent analysis, ours is based on forming a semantic network of keywords from patent documents. And thereby it visualizes a clear overview of patent information in a more comprehensible way. And as a result of those contributions, it enables us to understand advances of emerging technologies and forecast its trend in the future. 2007 Elsevier Ltd. All rights reserved.

Public health (PH) data can generally be characterized as big data. The efficient and effective use of this data determines the extent to which PH stakeholders can sufficiently address societal health concerns as they engage in a variety of work activities. As stakeholders interact with data, they engage in various cognitive activities such as analytical reasoning, decision-making, interpreting, and problem solving. Performing these activities with big data is a challenge for the unaided mind as stakeholders encounter obstacles relating to the data's volume, variety, velocity, and veracity. Such being the case, computer-based information tools are needed to support PH stakeholders. Unfortunately, while existing computational tools are beneficial in addressing certain work activities, they fall short in supporting cognitive activities that involve working with large, heterogeneous, and complex bodies of data. This paper presents visual analytics (VA) tools, a nascent category of computational tools that integrate data analytics with interactive visualizations, to facilitate the performance of cognitive activities involving big data. Historically, PH has lagged behind other sectors in embracing new computational technology. In this paper, we discuss the role that VA tools can play in addressing the challenges presented by big data. In doing so, we demonstrate the potential benefit of incorporating VA tools into PH practice, in addition to highlighting the need for further systematic and focused research.

The analysis of brain connectivity is a vast field in neuroscience with a frequent use of visual representations and an increasing need for visual analysis tools. Based on an in-depth literature review and interviews with neuroscientists, we explore high-level brain connectivity analysis tasks that need to be supported by dedicated visual analysis tools. A significant example of such a task is the comparison of different connectivity data in the form of weighted graphs. Several approaches have been suggested for graph comparison within information visualization, but the comparison of weighted graphs has not been addressed. We explored the design space of applicable visual representations and present augmented adjacency matrix and node-link visualizations. To assess which representation best support weighted graph comparison tasks, we performed a controlled experiment. Our findings suggest that matrices support these tasks well, outperforming node-link diagrams. These results have significant implications for the design of brain connectivity analysis tools that require weighted graph comparisons. They can also inform the design of visual analysis tools in other domains, e.g. comparison of weighted social networks or biological pathways.

Parallel coordinate plots (PCPs) are commonly used in information visualization to provide insight into multi-variate data. These plots help to spot correlations between variables. PCPs have been successfully applied to unstructured datasets up to a few millions of points. In this paper, we present techniques to enhance the usability of PCPs for the exploration of large, multi-timepoint volumetric data sets, containing tens of millions of points per timestep. The main difficulties that arise when applying PCPs to large numbers of data points are visual clutter and slow performance, making interactive exploration infeasible. Moreover, the spatial context of the volumetric data is usually lost. We describe techniques for preprocessing using data quantization and compression, and for fast GPU-based rendering of PCPs using joint density distributions for each pair of consecutive variables, resulting in a smooth, continuous visualization. Also, fast brushing techniques are proposed for interactive data selection in multiple linked views, including a 3D spatial volume view. These techniques have been successfully applied to three large data sets: Hurricane Isabel (Vis'04 contest), the ionization front instability data set (Vis'08 design contest), and data from a large-eddy simulation of cumulus clouds. With these data, we show how PCPs can be extended to successfully visualize and interactively explore multi-timepoint volumetric datasets with an order of magnitude more data points.

In a visualization task, every nonlinear projection method needs to make a compromise between trustworthiness and continuity. In a trustworthy projection the visualized proximities hold in the original data as well, whereas a continuous projection visualizes all proximities of the original data. We show experimentally that one of the multidimensional scaling methods, curvilinear components analysis, is good at maximizing trustworthiness. We then extend it to focus on local proximities both in the input and output space, and to explicitly make a user-tunable parameterized compromise between trustworthiness and continuity. The new method compares favorably to alternative nonlinear projection methods.

The integration of information of different kinds, such as spatial and alphanumeric at different levels of detail, is a challenge. While a solution is not reached, it is widely recognized that the need to integrate information is so pressing that it does not matter if detail is lost, as long as integration is achieved. This paper shows the potential for information retrieval at different levels of granularity inside the framework of information systems based on ontologies. Ontologies are theories that use a specific vocabulary to describe entities, classes, properties and functions related to a certain view of the world. The use of an ontology, translated into an active information system component, leads to ontology-driven information systems and, in the specific case of GIS, leads to what we call ontology-driven geographic information systems.

In today’s applications data is produced at unprecedented rates. While the capacity to collect and store new data rapidly grows, the ability to analyze these data volumes increases at much lower rates. This gap leads to new challenges in the analysis process, since analysts, decision makers, engineers, or emergency response teams depend on information hidden in the data. The emerging field of visual analytics focuses on handling these massive, heterogenous, and dynamic volumes of information by integrating human judgement by means of visual representations and interaction techniques in the analysis process. Furthermore, it is the combination of related research areas including visualization, data mining, and statistics that turns visual analytics into a promising field of research. This paper aims at providing an overview of visual analytics, its scope and concepts, addresses the most important research challenges and presents use cases from a wide variety of application scenarios.

We introduce several new methods for visualizing map projections and their associated distortions. These methods are embodied in the Interactive Map Projections system which allows users to view a representation of the Earth simultaneously as a sphere and as a projection with the ability to interact with both images. The relationship between the globe and the projection is enhanced by the use of explicit visualization of the intermediate developable geometric shapes used in the projection. A tool is built on top of the Interactive Map Projections system that provides a new method of visualizing map projection distortion. The central idea is one or more floating rings on the globe that can be interactively positioned and scaled. As the rings are manipulated on the globe, the corresponding projection of the rings are distorted using the same map projection parameters. This method is applied to study areal and angular distortion and is particularly useful when analyzing large geographical extents (such as in global climate studies) where distortions are significant, as well as visualizations for which information is geo-referenced and perhaps scaled to the underlying map. The floating ring tool is further enhanced to study 3D data sets placed over or under map projections. Examples include atmospheric and oceanographic data, respectively. Here, the ring is extended into a cone with apex at the center of the sphere and emanating beyond the surface into the atmosphere. It serves as a reminder that distortion exists in maps and data overlayed over maps, and provides information about the degree, location, and type of distortion. # 2001 Elsevier Science Ltd. All rights reserved.

In order to facilitate product design and realization processes, presently, research is actively carried out for developing methodologies and technologies of collaborative computer-aided design systems to support design teams geographically dispersed based on the quickly evolving information technologies. In this paper, the developed collaborative systems, methodologies and technologies, which are organized as a horizontal or a hierarchical manner, are reviewed. Meanwhile, a 3D streaming technology, which can effectively transmit visualization information across networks for Web applications, is highlighted and the algorithms behind it are disclosed.


Question: Incorporating information visualization into visual depiction methods.

This paper outlines a method for integrating information visualization (IV) techniques into the visual representation of text document data, enabling spatial analysis and interaction. The focus is on extensive collections of text documents, including digital libraries, regulations, procedures, and archived reports. The core concept involves translating the textual content into a spatial format that retains key informational aspects from the documents. This spatial visualization facilitates an intuitive browsing and analysis process, sidestepping the complexities of language processing and alleviating the cognitive load on the user. By doing so, the interaction with textual information becomes more akin to perceiving and acting within the real world, rather than dealing with the abstract constructs of written language. This approach offers a novel way to engage with large volumes of textual data, making it more accessible and understandable through visual and spatial means.

Managing large collections of documents is an important problem for many areas of science, industry, and culture. Probabilistic topic modeling offers a promising solution. Topic modeling is an unsupervised machine learning method that learns the underlying themes in a large collection of otherwise unorganized documents. This discovered structure summarizes and organizes the documents. However, topic models are high-level statistical tools—a user must scrutinize numerical distributions to understand and explore their results. In this paper, we present a method for visualizing topic models. Our method creates a navigator of the documents, allowing users to explore the hidden structure that a topic model discovers. These browsing interfaces reveal meaningful patterns in a collection, helping end-users explore and understand its contents in new ways. We provide open source software of our method. Understanding and navigating large collections of documents has become an important activity in many spheres. However, many document collections are not coherently organized and organizing them by hand is impractical. We need automated ways to discover and visualize the structure of a collection in order to more easily explore its contents. Probabilistic topic modeling is a set of machine learning tools that may provide a solution (Blei and Lafferty 2009). Topic modeling algorithms discover a hidden thematic structure in a collection of documents; they find salient themes and represent each document as a combination of themes. However, topic models are high-level statistical tools. A user must scrutinize numerical distributions to understand and explore their results; the raw output of the model is not enough to create an easily explored corpus. We propose a method for using a fitted topic model to organize, summarize, visualize, and interact with a corpus. With our method, users can explore the corpus, moving between high level discovered summaries (the “topics”) and the documents themselves, as Figure 1 illustrates. Our design is centered around the idea that the model both summarizes and organizes the collection. Our method translates these representations into a visual system for exploring a collection, but visualizing this structure is not enough. The discovered structure induces relationships—between topics and articles, and between articles and articles—which lead to interactions in the visualization. Copyright c © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Thus, we have three main goals in designing the visualization: summarize the corpus for the user; reveal the relationships between the content and summaries; and, reveal the relationships across content. We aim to present these in a ways that are accessible and useful to a spectrum of users, not just machine learning experts.

On the issue of how to explore the complex relationships of entities in a story, Storyline visualization has proven to be a useful approach. The traditional Storyline visualization technology only applies to a linear continuous story and has the limitation of reflecting the development of the plots in a story. In this paper, we propose a hierarchical plot visualization method called StoryCake to improve this problem. First, according to the story elements, inter-sessions are divided into groups, whereby the layout of the entities will be optimized in individual groups. Then, the hierarchical relationships of entities in polar coordinates can be confirmed. Finally, we use several skills, e.g., interaction, labels and a fan-shaped visualization view, to enable people better understand how story evolves and track the hierarchical relationship more conveniently. Experimental results show that the proposed hierarchical plot visualization method in polar coordinates can reveal discontinuous events and nonlinear stories. The results also show that our algorithm presents the narrative structure and development of the story.

In today’s applications data is produced at unprecedented rates. While the capacity to collect and store new data rapidly grows, the ability to analyze these data volumes increases at much lower rates. This gap leads to new challenges in the analysis process, since analysts, decision makers, engineers, or emergency response teams depend on information hidden in the data. The emerging field of visual analytics focuses on handling these massive, heterogenous, and dynamic volumes of information by integrating human judgement by means of visual representations and interaction techniques in the analysis process. Furthermore, it is the combination of related research areas including visualization, data mining, and statistics that turns visual analytics into a promising field of research. This paper aims at providing an overview of visual analytics, its scope and concepts, addresses the most important research challenges and presents use cases from a wide variety of application scenarios.

During data preprocessing, analysts spend a significant part of their time and effort profiling the quality of the data along with cleansing and transforming the data for further analysis. While quality metrics—ranging from general to domain-specific measures—support assessment of the quality of a dataset, there are hardly any approaches to visually support the analyst in customizing and applying such metrics. Yet, visual approaches could facilitate users’ involvement in data quality assessment. We present MetricDoc, an interactive environment for assessing data quality that provides customizable, reusable quality metrics in combination with immediate visual feedback. Moreover, we provide an overview visualization of these quality metrics along with error visualizations that facilitate interactive navigation of the data to determine the causes of quality issues present in the data. In this article, we describe the architecture, design, and evaluation of MetricDoc, which underwent several design cycles, including heuristic evaluation and expert reviews as well as a focus group with data quality, human-computer interaction, and visual analytics experts.

In recent years, main cities in China have been suffering from hazy weather, which is gaining great attention among the public, government managers and researchers in different areas. Many studies have been conducted on the topic of urban air quality to reveal different aspects of the air quality problem in China. This paper focuses on the visualization problem of the big air quality monitoring data of all main cities on a nationwide scale. To achieve the intuitive visualization of this dataset, this study develops two novel visualization tools for multi-granularity time series visualization (timezoom.js) and a dynamic symbol declutter map mashup layer for thematic mapping (symadpative.js). With the two invented tools, we develops an interactive web map visualization application of urban air quality data of all main cities in China. This application shows us significant air pollution findings at the nationwide scale. These results give us clues for further studies on air pollutant characteristics, forecasting and control in China. As the tools are invented for general visualization purposes of geo-referenced time series data, they can be applied to other environmental monitoring data (temperature, precipitation, etc.) through some configurations.

The application of eye tracking for the evaluation of humans' viewing behavior is a common approach in psychological research. So far, the use of this technique for the evaluation of visual analytics and visualization is less prominent. We investigate recent scientific publications from the main visualization and visual analytics conferences and journals that include an evaluation by eye tracking. Furthermore, we provide an overview of evaluation goals that can be achieved by eye tracking and state-of-the-art analysis techniques for eye tracking data. Ideally, visual analytics leads to a mixed-initiative cognitive system where the mechanism of distribution is the interaction of the user with visualization environments. Therefore, we also include a discussion of cognitive approaches and models to include the user in the evaluation process. Based on our review of the current use of eye tracking evaluation in our field and the cognitive theory, we propose directions of future research on evaluation methodology, leading to the grand challenge of developing an evaluation approach to the mixed-initiative cognitive system of visual analytics.

Conveying data uncertainty in visualizations is crucial for preventing viewers from drawing conclusions based on untrustworthy data points. This paper proposes a methodology for efficiently generating density plots of uncertain multivariate data sets that draws viewers to preattentively identify values of high certainty while not calling attention to uncertain values. We demonstrate how to augment scatter plots and parallel coordinates plots to incorporate statistically modeled uncertainty and show how to integrate them with existing multivariate analysis techniques, including outlier detection and interactive brushing. Computing high quality density plots can be expensive for large data sets, so we also describe a probabilistic plotting technique that summarizes the data without requiring explicit density plot computation. These techniques have been useful for identifying brain tumors in multivariate magnetic resonance spectroscopy data and we describe how to extend them to visualize ensemble data sets.

The scatterplot matrix (SPLOM) is a well-established technique to visually explore high-dimensional data sets. It is characterized by the number of scatterplots (plots) of which it consists of. Unfortunately, this number quadratically grows with the number of the data set’s dimensions. Thus, a SPLOM scales very poorly. Consequently, the usefulness of SPLOMs is restricted to a small number of dimensions. For this, several approaches already exist to explore such “small” SPLOMs. Those approaches address the scalability problem just indirectly and without solving it. Therefore, we introduce a new greedy approach to manage “large” SPLOMs with more than hundred dimensions. We establish a combined visualization and interaction scheme that produces intuitively interpretable SPLOMs by combining known quality measures, a pre-process reordering, and a perception-based abstraction. With this scheme, the user can interactively find large amounts of relevant plots in large SPLOMs.

Animated transition has been a popular design choice for smoothly switching between different visualization views or layouts, in which movement trajectories are created as cues for tracking objects during location shifting. Tracking moving objects, however, becomes difficult when their movement paths overlap or the number of tracking targets increases. We propose a novel design to facilitate tracking moving objects in animated transitions. Instead of simply animating an object along a straight line, we create "bundled" movement trajectories for a group of objects that have spatial proximity and share similar moving directions. To study the effect of bundled trajectories, we untangle variations due to different aspects of tracking complexity in a comprehensive controlled user study. The results indicate that using bundled trajectories is particularly effective when tracking more targets (six vs. three targets) or when the object movement involves a high degree of occlusion or deformation. Based on the study, we discuss the advantages and limitations of the new technique, as well as provide design implications.

Early hands-on experiences with the Microsoft Hololens augmented/mixed reality device are reported and discussed, with a general aim of exploring basic 3D visualization. A range of usage cases are tested, including data visualization and immersive data spaces, in-situ visualization of 3D models and full scale architectural form visualization. Ultimately, the Hololens is found to provide a remarkable tool for moving from traditional visualization of 3D objects on a 2D screen, to fully experiential 3D visualizations embedded in the real world.


Question: Identifying perceptual artifacts in compressed video data streams.

In recent years, there has been a significant push towards developing algorithms capable of objectively assessing the quality of video signals, given their widespread use in various applications and products that ultimately target human viewers. To ensure the quality of video content delivered to this discerning audience, automatic methods for video quality assessment (VQA) are crucial. However, the accurate representation of motion in videos is a critical factor in overall video quality perception, an aspect that many existing VQA algorithms overlook, thereby compromising their efficacy. To address this gap, we propose a comprehensive, spatio-spectrally localized multiscale framework that evaluates video fidelity by integrating both spatial and temporal, as well as spatio-temporal, aspects of distortion. This framework uniquely assesses video quality in space, time, and space-time by evaluating the quality of motion along computed motion trajectories. We introduce a full-reference VQA algorithm, which we term the MOtion-based Video Integrity Evaluation index, or MOVIE index, within this framework. Our findings reveal that the MOVIE index produces VQA scores that closely align with human subjective evaluations, as validated using the Video Quality Expert Group (VQEG) FRTV Phase 1 database. In fact, the MOVIE index not only competes effectively with, but in some cases even surpasses, algorithms that were developed and submitted for the VQEG FRTV Phase 1 study, as well as newer VQA algorithms tested on this database.

Battery lifetime is critical for wireless video sensors. To enable battery-powered wireless video sensors, low-power design is required. In this paper, we consider applying multi-view summarization to wireless video sensors to remove redundant contents such that the compression and transmission power can be reduced. A low-complexity online multi-view video summarization scheme is proposed. Experiments show that the proposed summarization method successfully reduces the video content while keeping important events. A power analysis of the system also shows that a significant amount of energy can be saved.

The adoption of the latest OLED (organic light emitting diode) technology does not change the fact that screen is still one of the most energy-consuming modules in modern smartphones. In this work, we found that video streams from the same video category share many common power consumption features on OLED screens. Therefore, we are able to build a Hidden Markov Model (HMM) classifier to categorize videos based on OLED screen power characteristics. Using this HMM classifier, we propose a video classification based dynamic tone mapping (DTM) scheme, namely, DaTuM, to remap output color range and minimize the power-hungry color compositions on OLED screens for power saving. Experiment shows that DaTuM scheme averagely reduces OLED screen power by 17.8% with minimum display quality degradation. Compared to DTM scheme based on official category info provided by the video sources and one state-of-the-art scheme, DaTuM substantially enhances OLED screens' power efficiency and display quality controllability.

This paper presents a classified review of soccer video analysis works. The existing approaches in the aspects of highlight event detection, video summarization and retrieval based on video stream, ball and player tracking for provision of match statistics, technical and tactical analysis and application of different sources in soccer video analysis have been surveyed. In addition, some major existing commercial softwares developed for video analysis are introduced and compared. With regard to the existing challenge for automatic and realtime provision of video analysis, different computer vision approaches are discussed and compared. Audio, video and text feature extraction methods have been investigated and the future trends for improvement of the reviewed systems have been introduced in terms of response time optimization, increase of precision and eliminating the need of human intervention for video analysis.

As consumer-level cameras are becoming cheaper and cheaper, the scene of interest can now be captured by multiple cameras simultaneously from different viewpoints. By transmitting texture and depth maps of the views captured using two adjacent cameras, users can synthesize any view in between using depth image based rendering (DIBR) technique such as 3D-warping. This provides users with continuous view angles and is key technique for Free Viewpoint TV (FTV), etc. In this scenario, multiple views must be delivered through bandwidth-limited wireless networks, how to adaptive reduce the source coding rate to fit the bandwidth becomes an important issue. The standard multi-view video codec H.264 MVC explores the inter-view and intra-view correlation to reduce source encoding rate. But all views are required at the client side for decoding any single view, which is apparently not transmission efficient. A better solution is using H.264 to differentially encode the two adjacent views separately, but still this requires double-folder bandwidth comparing to the traditional video streaming. Multiple description coding (MDC) encodes one view into multiple descriptions for robust transmission. This inspires another option: sending only part of each view. This can greatly reduce the source encoding rate, but interpolating skipped frames before synthesizing the middle view consumes mobile devices' precious battery life. Whether to send at a lower source encoding rate but with extra energy consumption or send at a high source encoding rate is nontrivial, and has not been studied in any formal way. In this paper, we investigated both source encoding rate reduction and energy consumption with the bandwidth constraint. We formulated the problem into an optimization problem, i.e., maximizing the system utility, which is defined as the received video quality minus the weighted energy consumption. Simulation results showed that our proposed scheme provides a significant advantage over competing schemes in typical network conditions.

Image-based rendering techniques include those with geometry and those without. Geometric information in the form of a depth map aligned with the image holds a lot of promise for IBR due to the several methods available to capture it. It can improve the quality of generated views using a limited number of views. Compression of light fields or multiple images has attracted a lot of attention in the past. Compression of multiple depth maps of the same scene has not been explored much in the literature. We propose a method for compressing multiple depth maps in this paper using geometric proxy. Different quality of rendering and compression ratio can be achieved by varying different parameters. Experiments show the effectiveness of the compression technique on several model data.

We consider the problem of single image object motion deblurring from a static camera. It is well-known that deblurring of moving objects using a traditional camera is ill-posed, due to the loss of high spatial frequencies in the captured blurred image. A coded exposure camera modulates the integration pattern of light by opening and closing the shutter within the exposure time using a binary code. The code is chosen to make the resulting point spread function (PSF) invertible, for best deconvolution performance. However, for a successful deconvolution algorithm, PSF estimation is as important as PSF invertibility. We show that PSF estimation is easier if the resulting motion blur is smooth and the optimal code for PSF invertibility could worsen PSF estimation, since it leads to non-smooth blur. We show that both criterions of PSF invertibility and PSF estimation can be simultaneously met, albeit with a slight increase in the deconvolution noise. We propose design rules for a code to have good PSF estimation capability and outline two search criteria for finding the optimal code for a given length. We present theoretical analysis comparing the performance of the proposed code with the code optimized solely for PSF invertibility. We also show how to easily implement coded exposure on a consumer grade machine vision camera with no additional hardware. Real experimental results demonstrate the effectiveness of the proposed codes for motion deblurring.

Currently, we witness dramatically increasing interest in immersive media technologies like Virtual Reality (VR), particularly in omnidirectional video (OV) streaming. Omnidirectional (also called 360-degree) videos are panoramic spherical videos in which the user can look around during playback and which therefore can be understood as hybrids between traditional movie streaming and interactive VR worlds. Unfortunately, streaming this kind of content is extremely bandwidth intensive (compared to traditional 2D video) and therefore, Quality of Experience (QoE) tends to deteriorate significantly in absence of continuous optimal bandwidth conditions. In this paper, we present a first approach towards subjective QoE assessment for omnidirectional video (OV) streaming. We present the results of a lab study on the QoE impact of stalling in the context of OV streaming using head-mounted displays (HMDs). Our findings show that subjective testing for immersive media like OV is not trivial, with even simple cases like stalling leading to unexpected results. After a discussion of characteristic pitfalls and lessons learned, we provide a a set of recommendations for upcoming OV assessment studies.

In this work we describe a Convolutional Neural Network (CNN) to accurately predict image quality without a reference image. Taking image patches as input, the CNN works in the spatial domain without using hand-crafted features that are employed by most previous methods. The network consists of one convolutional layer with max and min pooling, two fully connected layers and an output node. Within the network structure, feature learning and regression are integrated into one optimization process, which leads to a more effective model for estimating image quality. This approach achieves state of the art performance on the LIVE dataset and shows excellent generalization ability in cross dataset experiments. Further experiments on images with local distortions demonstrate the local quality estimation ability of our CNN, which is rarely reported in previous literature.

We present a soccer video abstraction method based on the analysis of the audio and video streams. This method could be applied to other sports as rugby or american football. The main contribution of this paper is the design of an unsupervised summarization method, and more specifically, the introduction of an efficient detector of excited speech segments. An excited commentary is supposed to correspond to an interesting moment of the game. It is simultaneously characterized by an increase of the pitch (or fundamental frequency) within the voiced segments and an increase of the energy supported by the harmonics of the pitch. The pitch is estimated from the autocorrelation function and its local increases are detected from a multiresolution technique. We introduce a specific energy measure for the voiced segments. A statistical analysis of the energy measures is performed to detect the most excited parts of the speech. A deterministic combination of excited speech detection, dominant color identification and camera motion analysis is then performed in order to discriminate between excited speech sequences of the game and excited speech sequences in commercials or in studio shots included in the processed TV programs.
 The method presented here does not need any learning stage. It has been tested on seven soccer videos for a total duration of almost 20 hours.

In this paper, a neural network approach for color classification in HSV color space based on color JND (just noticeable difference) is presented. The HSV color samples are generated using an interactive tool based on JND concept and these samples are used for supervised training. A four layer feed forward neural network is trained for classifying a given color pair as similar or dissimilar pair. An interactive tool for color pixel comparison is developed for testing the color classifier on real life images. Research shows that neural network classifier in HSV color space works better than RGB classifier in terms of efficiency for segmentation purposes. Thus the experimentation results can be used for segmenting real life images.


Question: Identifying perceptual artifacts in compressed video data streams.

This document investigates the visibility of quantization noise in the discrete wavelet transform (DWT) for image compression. The DWT decomposes images into bands characterized by varying spatial frequencies and orientations, making it a popular choice for compression. To optimize compression while maintaining visual quality, it's essential to understand how quantization errors are perceived. The study focuses on DWT uniform quantization noise, which emerges when a single band of coefficients is uniformly quantized. This noise is visualized as a lattice of randomly amplitude-modulated basis functions corresponding to the synthesis filter of the DWT. The authors conducted experiments to determine visual detection thresholds for this noise in the Y, Cb, and Cr color channels. They found that detection thresholds rise significantly with increasing wavelet spatial frequency, and also vary across color channels, with thresholds being lowest in Y and highest in Cb. Additionally, orientation affects visibility, with lowpass orientations being less noticeable than horizontal/vertical or diagonal orientations. A mathematical model is proposed to predict DWT noise detection thresholds based on level, orientation, and display visual resolution. This model enables the calculation of a "perceptually lossless" quantization matrix, ensuring that all errors are theoretically undetectable by the human eye. Furthermore, the model can serve as a foundation for adaptive quantization methods that balance compression efficiency with perceptual quality.

The paper gives a snapshot of the state of the art in affine covariant region detectors, and compares their performance on a set of test images under varying imaging conditions. Six types of detectors are included: detectors based on affine normalization around Harris  (Mikolajczyk and  Schmid, 2002; Schaffalitzky and  Zisserman, 2002) and Hessian points  (Mikolajczyk and  Schmid, 2002), a detector of ‘maximally stable extremal regions', proposed by Matas et al. (2002); an edge-based region detector  (Tuytelaars and Van Gool, 1999) and a detector based on intensity extrema (Tuytelaars and Van Gool, 2000), and a detector of ‘salient regions', proposed by Kadir, Zisserman and Brady (2004). The performance is measured against changes in viewpoint, scale, illumination, defocus and image compression. The objective of this paper is also to establish a reference test set of images and performance software, so that future detectors can be evaluated in the same framework.

This paper presents a novel technique for camera tampering detection. It is implemented in real-time and was developed for use in surveillance and security applications. This method identifies camera tampering by detecting large differences between older frames of video and more recent frames. A buffer of incoming video frames is kept and three different measures of image dissimilarity are used to compare the frames. After normalization, a set of conditions is tested to decide if camera tampering has occurred. The effects of adjusting the internal parameters of the algorithm are examined. The performance of this method is shown to be extremely favorable in real-world settings.

In the wide-angle (WA) images embedded in an around-view monitor system, the subject(s) in the peripheral region is normally small and has little information. Furthermore, since the outer region suffers from the non-uniform blur phenomenon and artifact caused by the inherent optical characteristic of WA lenses, its visual quality tends to deteriorate. According to our experiments, conventional image enhancement techniques rarely improve the degraded visual quality of the outer region of WA images. In order to solve the above-mentioned problem, this paper proposes a joint sharpness enhancement (SE) and super-resolution (SR) algorithm which can improve the sharpness and resolution of WA images together. The proposed SE algorithm improves the sharpness of the deteriorated WA images by exploiting self-similarity. Also, the proposed SR algorithm generates super-resolved images by using high-resolution information which is classified according to the extended local binary pattern-based classifier and learned on a pattern basis. Experimental results show that the proposed scheme effectively improves the sharpness and resolution of the input deteriorated WA images. Even in terms of quantitative metrics such as just noticeable blur, structural similarity, and peak signal-to-noise ratio. Finally, the proposed scheme guarantees real-time processing such that it achieves 720p video at 29 Hz on a low cost GPU platform.

Recently, there has been a substantial increase in efforts to develop 3-D visualization technologies that can provide the viewers with a realistic 3-D visual experience. Various terms such as “reality communication” have been used to categorize these efforts. In order to provide the viewers with a complete and realistic visual sensation, the display or visualization system and the displayed content need to match the physiological 3-D information sensing capabilities of the human visual system which can be quite complex. These may include spatial and temporal resolutions, depth perception, dynamic range, spectral contents, nonlinear effects, and vergence accommodation effects. In this paper, first we present an overview of some of the 3-D display research efforts which have been extensively pursued in Asia, Europe, and North America among other areas. Based on the limitations and comfort-based requirements of the human visual system when viewing a nonnatural visual input from 3-D displays, we present an analytical framework that combines main perception and human visual requirements with analytical tools and principles used in related disciplines such as optics, computer graphics, computational imaging, and signal processing. Building on the widely used notion of light fields, we define a notion of perceivable light fields to account for the human visual system physiological requirements, and propagate it back to the display device to determine the display device specifications. This helps us clarify the fundamental and practical requirements of the 3-D display devices for reality viewing communication. In view of the proposed analytical framework, we overview various methods that can be applied to overcome the extensive information needed to be displayed in order to meet the requirements imposed by the human visual system.

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

Although image compression has been actively studied for decades, there has been relatively little research on learning to compress images with modern neural networks. Standard approaches, such as those employing patch-based autoencoders, have shown a great deal of promise but cannot compete with popular image codecs because they fail to address three questions: 1) how to effectively binarize activations: in the absence of binarization, a bottleneck layer alone tends not to lead to efficient compression; 2) how to achieve variable-rate encoding: a standard autoencoder generates a fixed-length code for each fixed-resolution input patch, resulting in the same cost for lowand high-entropy patches, and requiring the network to be completely retrained to achieve different compression rates; and 3) how to avoid block artifacts: patch-based approaches are prone to block discontinuities. We propose a general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional recurrent networks, including LSTMs, that address these issues and report promising results compared to existing baseline codecs. We evaluate the proposed methods on a large-scale benchmark consisting of tiny images (32× 32), which proves to be very challenging for all the methods.

Steganalysis of the quantization index modulation QIM steganography in a low-bit-rate encoded speech stream is conducted in this research. According to the speech generation theory and the phoneme distribution properties in language, we first point out that the correlation characteristics of split vector quantization VQ codewords of linear predictive coding filter coefficients are changed after the QIM steganography. Based on this observation, we construct a model called the Quantization codeword correlation network QCCN based on split VQ codeword from adjacent speech frames. Furthermore, the QCCN model is pruned to yield a stronger correlation network. After quantifying the correlation characteristics of vertices in the pruned correlation network, we obtain feature vectors that are sensitive to steganalysis. Finally, we build a high-performance detector using the support vector machine SVM classifier. It is shown by experimental results that the proposed QCCN steganalysis method can effectively detect the QIM steganography in encoded speech stream when it is applied to low-bit-rate speech codec such as G.723.1 and G.729.

We present an algorithm to remove wobble artifacts from a video captured with a rolling shutter camera undergoing large accelerations or jitter. We show how estimating the rapid motion of the camera can be posed as a temporal super-resolution problem. The low-frequency measurements are the motions of pixels from one frame to the next. These measurements are modeled as temporal integrals of the underlying high-frequency jitter of the camera. The estimated high-frequency motion of the camera is then used to re-render the sequence as though all the pixels in each frame were imaged at the same time. We also present an auto-calibration algorithm that can estimate the time between the capture of subsequent rows in the camera.

A method is described which quantifies the speed and direction of several moving objects in a sequence of digital images. A relationship between the time variation of intensity, the spatial gradient, and velocity has been developed which allows the determination of motion using clustering techniques. This paper describes these relationships, the clustering technique, and provides examples of the technique on real images containing several moving objects.

We introduce a new compression scheme for high-dimensional vectors that approximates the vectors using sums of M codewords coming from M different codebooks. We show that the proposed scheme permits efficient distance and scalar product computations between compressed and uncompressed vectors. We further suggest vector encoding and codebook learning algorithms that can minimize the coding error within the proposed scheme. In the experiments, we demonstrate that the proposed compression can be used instead of or together with product quantization. Compared to product quantization and its optimized versions, the proposed compression approach leads to lower coding approximation errors, higher accuracy of approximate nearest neighbor search in the datasets of visual descriptors, and lower image classification error, whenever the classifiers are learned on or applied to compressed vectors.


Question: Identifying perceptual distortions in compressed video sequences.

This paper outlines a novel technique for identifying and characterizing 'visually intriguing' regions within images, which holds significant potential in the realm of automated machine vision systems. Our approach is centered on integrating eye tracking methodologies with principal component analysis to pinpoint low-level image attributes that naturally draw human attention. Through extensive analysis of image patches captured at the exact locations where observers fixate, we have uncovered a set of features that closely mirror the derivatives of a two-dimensional Gaussian operator. To further probe the uniqueness of human visual preferences, we contrast these fixation-driven features with the general image structure obtained through random sampling in Monte Carlo simulations. This comparison highlights the distinct patterns that differentiate human fixations from random occurrences. Additionally, we showcase a straightforward application where these identified features are utilized to predict eye fixations, demonstrating the practical implications of our findings in enhancing the efficiency and accuracy of active machine vision systems.

The image gradient is a commonly computed image feature and a potentially predictive factor for image quality assessment (IQA). Indeed, it has been successfully used for both fulland noreference image quality prediction. However, the gradient orientation has not been deeply explored as a predictive source of information for image quality assessment. Here we seek to amend this by studying the quality relevance of the relative gradient orientation, viz., the gradient orientation relative to the surround. We also deploy a relative gradient magnitude feature which accounts for perceptual masking and utilize an AdaBoosting back-propagation (BP) neural network to map the image features to image quality. The generalization of the AdaBoosting BP neural network results in an effective and robust quality prediction model. The new model, called Oriented Gradients Image Quality Assessment (OG-IQA), is shown to deliver highly competitive image quality prediction performance as compared with the most popular IQA approaches. Furthermore, we show that OG-IQA has good database independence properties and a low complexity. & 2015 Elsevier B.V. All rights reserved.

One of the main problems in the post-harvest processing of citrus is the detection of visual defects in order to classify the fruit depending on their appearance. Species and cultivars of citrus present a high rate of unpredictability in texture and colour that makes it difficult to develop a general, unsupervised method able of perform this task. In this paper we study the use of a general approach that was originally developed for the detection of defects in random colour textures. It is based on a Multivariate Image Analysis strategy and uses Principal Component Analysis to extract a reference eigenspace from a matrix built by unfolding colour and spatial data from samples of defect-free peel. Test images are also unfolded and projected onto the reference eigenspace and the result is a score matrix which is used to compute defective maps based on the T2 statistic. In addition, a multiresolution scheme is introduced nsupervised Methods in the original method to speed up the process. Unlike the techniques commonly used for the detection of defects in fruits, this is an unsupervised method that only needs a few samples to be trained. It is also a simple approach that is suitable for real-time compliance. Experimental work was performed on 120 samples of oranges and mandarins from four different cultivars: Clemenules, Marisol, Fortune, and Valencia. The success ratio for the detection of individual defects was 91.5%, while the classification ratio of damaged/sound samples was 94.2%. These results show that the studied method can be suitable for on. the task of citrus inspecti

Human vision takes time to adapt to large changes in scene intensity, and these transient adjustments have a profound effect on visual appearance. This paper offers a new operator to include these appearance changes in animations or interactive real-time simulations, and to match a user's visual responses to those the user would experience in a real-world scene.
Large, abrupt changes in scene intensities can cause dramatic compression of visual responses, followed by a gradual recovery of normal vision. Asymmetric mechanisms govern these time-dependent adjustments, and offer adaptation to increased light that is much more rapid than adjustment to darkness. We derive a new tone reproduction operator that simulates these mechanisms. The operator accepts a stream of scene intensity frames and creates a stream of color display images.
All operator components are derived from published quantitative measurements from physiology, psychophysics, color science, and photography. ept intentionally simple to allow fast computation, the operator is meant for use with real-time walk-through renderings, high dynamic range video cameras, and other interactive applications. We demonstrate its performance on both synthetically generated and acquired “real-world” scenes with large dynamic variations of illumination and contrast.

Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algorithms called deep neural networks (DNNs) that achieve human-level classification performance on object recognition tasks. Furthermore, a growing number of studies report similarities in the way DNNs and the human visual system process objects, suggesting that current DNNs may be good models of human visual object recognition. Yet there clearly exist important architectural and processing differences between stateof-the-art DNNs and the primate visual system. The potential behavioural consequences of these differences are not well understood. We aim to address this issue by comparing human and DNN generalisation abilities towards image degradations. We find the human visual system to be more robust to image manipulations like contrast reduction, additive noise or novel eidolon-distortions. In addition, we find progressively diverging classification error-patterns between man and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition. We envision that our findings as well as our carefully measured and freely available behavioural datasets1 provide a new useful benchmark for the computer vision community to improve the robustness of DNNs and a motivation for neuroscientists to search for mechanisms in the brain that could facilitate this robustness. Data and materials available at https://github.com/rgeirhos/object-recognition 1 ar X iv :1 70 6. 06 96 9v 1 [ cs .C V ] 2 1 Ju n 20 17

We explore the potential of image-dependent gamut mapping as a constrained optimization problem. The performance of our new approach is compared to standard reference gamut mapping algorithms in psycho-visual tests.

Video, like images, should support content aware resizing. We present video retargeting using an improved seam carving operator. Instead of removing 1D seams from 2D images we remove 2D seam manifolds from 3D space-time volumes. To achieve this we replace the dynamic programming method of seam carving with graph cuts that are suitable for 3D volumes. In the new formulation, a seam is given by a minimal cut in the graph and we show how to construct a graph such that the resulting cut is a valid seam. That is, the cut is monotonic and connected. In addition, we present a novel energy criterion that improves the visual quality of the retargeted images and videos. The original seam carving operator is focused on removing seams with the least amount of energy, ignoring energy that is introduced into the images and video by applying the operator. To counter this, the new criterion is looking forward in time - removing seams that introduce the least amount of energy into the retargeted result. We show how to encode the improved criterion into graph cuts (for images and video) as well as dynamic programming (for images). We apply our technique to images and videos and present results of various applications.

In this paper, a neural network approach for color classification in HSV color space based on color JND (just noticeable difference) is presented. The HSV color samples are generated using an interactive tool based on JND concept and these samples are used for supervised training. A four layer feed forward neural network is trained for classifying a given color pair as similar or dissimilar pair. An interactive tool for color pixel comparison is developed for testing the color classifier on real life images. Research shows that neural network classifier in HSV color space works better than RGB classifier in terms of efficiency for segmentation purposes. Thus the experimentation results can be used for segmenting real life images.

High efficiency video coding (HEVC) standard achieves half bit-rate reduction while keeping the same quality compared with AVC. However, it still cannot satisfy the demand of higher quality in real applications, especially at low bit rates. To further improve the quality of reconstructed frame while reducing the bitrates, a residual highway convolutional neural network (RHCNN) is proposed in this paper for in-loop filtering in HEVC. The RHCNN is composed of several residual highway units and convolutional layers. In the highway units, there are some paths that could allow unimpeded information across several layers. Moreover, there also exists one identity skip connection (shortcut) from the beginning to the end, which is followed by one small convolutional layer. Without conflicting with deblocking filter (DF) and sample adaptive offset (SAO) filter in HEVC, RHCNN is employed as a high-dimension filter following DF and SAO to enhance the quality of reconstructed frames. To facilitate the real application, we apply the proposed method to I frame, P frame, and B frame, respectively. For obtaining better performance, the entire quantization parameter (QP) range is divided into several QP bands, where a dedicated RHCNN is trained for each QP band. Furthermore, we adopt a progressive training scheme for the RHCNN where the QP band with lower value is used for early training and their weights are used as initial weights for QP band of higher values in a progressive manner. Experimental results demonstrate that the proposed method is able to not only raise the PSNR of reconstructed frame but also prominently reduce the bit-rate compared with HEVC reference software.

The ability to identify objects or region correspondences between consecutive frames of a given hand-drawn animation sequence is an indispensable tool for automating animation modification tasks such as sequence-wide recoloring or shape-editing of a specific animated character. Existing correspondence identification methods heavily rely on appearance features, but these features alone are insufficient to reliably identify region correspondences when there exist occlusions or when two or more objects share similar appearances. To resolve the above problems, manual assistance is often required. In this paper, we propose a new correspondence identification method which considers both appearance features and motions of regions in a global manner. We formulate correspondence likelihoods between temporal region pairs as a network flow graph problem which can be solved by a well-established optimization algorithm. We have evaluated our method with various animation sequences and results show that our method consistently outperforms the state-of-the-art methods without any user guidance.

We introduce the challenge problem for generic video indexing to gain insight in intermediate steps that affect performance of multimedia analysis methods, while at the same time fostering repeatability of experiments. To arrive at a challenge problem, we provide a general scheme for the systematic examination of automated concept detection methods, by decomposing the generic video indexing problem into 2 unimodal analysis experiments, 2 multimodal analysis experiments, and 1 combined analysis experiment. For each experiment, we evaluate generic video indexing performance on 85 hours of international broadcast news data, from the TRECVID 2005/2006 benchmark, using a lexicon of 101 semantic concepts. By establishing a minimum performance on each experiment, the challenge problem allows for component-based optimization of the generic indexing issue, while simultaneously offering other researchers a reference for comparison during indexing methodology development. To stimulate further investigations in intermediate analysis steps that inuence video indexing performance, the challenge offers to the research community a manually annotated concept lexicon, pre-computed low-level multimedia features, trained classifier models, and five experiments together with baseline performance, which are all available at http://www.mediamill.nl/challenge/.