Question: CENTRIST: A visual descriptor utilized in scene categorization.

This paper introduces an enhanced Monte Carlo Localization (MCL) technique for mobile robots, specifically designed to address the shortcomings of conventional particle filters in challenging environments. Monte Carlo Localization, a subset of algorithms based on particle filters, is renowned for its success in accurately estimating a robot's pose using sensor data. However, standard particle filters can falter in complex scenarios. The proposed Mixture-MCL algorithm tackles these issues by employing a dual sampling strategy, combining two complementary methods for generating samples in the posterior estimation process. To implement this advanced algorithm for mobile robot localization, a kd-tree is constructed from empirical data, enabling efficient dual sampling. Extensive experimental results, gathered in bustling public areas, underscore the algorithm's superior performance, resilience, and computational efficiency when juxtaposed with leading localization techniques.

We present a 3-D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion. Recognition is based on matching surfaces by matching points using the spin-image representation. The spin-image is a data level shape descriptor that is used to match surfaces represented as surface meshes. We present a compression scheme for spin-images that results in efficient multiple object recognition which we verify with results showing the simultaneous recognition of multiple objects from a library of 20 models. Furthermore, we demonstrate the robust performance of recognition in the presence of clutter and occlusion through analysis of recognition trials on 100 scenes. This research was performed at Carnegie Mellon University and was supported by the US Department of Energy under contract DE-AC21-92MC29104. 1

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

A computer vision system has been implemented that can recognize threedimensional objects from unknown viewpoints in single gray-scale images. Unlike most other approaches, the recognition is accomplished without any attempt to reconstruct depth information bottom-up from the visual input. Instead, three other mechanisms are used that can bridge the gap between the two-dimensional image and knowledge of three-dimensional objects. First, a process of perceptual organization is used to form groupings and structures in the image that are likely to be invariant over a wide range of viewpoints. Second, a probabilistic ranking method is used to reduce the size of the search space during model based matching. Finally, a process of spatial correspondence brings the projections of three-dimensional models into direct correspondence with the image by solving for unknown viewpoint and model parameters. A high level of robustness in the presence of occlusion and missing data can be achieved through full application of a viewpoint consistency constraint. It is argued that similar mechanisms and constraints form the basis for recognition in human vision. This paper has been published in Artificial Intelligence, 31, 3 (March 1987), pp. 355–395.

For a machine to be able to ‘see’, it must know something about the object it is ‘looking’ at, A common method in machine vision is to provide the machine with general rather than specific knowledge about the object. An alternative technique, and the one used in thti paper, b a model-based approach in which particulars aboat the object are given and this drives the analysis. The computer program described here, the WALKER model, maps images into a description in which a person is represented by the series of hierarchical levels, i.e. a person has an arm which has a lower-arm which has a hand. l7te performance of the program ti illustrated by superimposing the machinegenerated picture over the original photographic images.

Human eyes can recognize person identities based on some small salient regions. However, such valuable salient information is often hidden when computing similarities of images with existing approaches. Moreover, many existing approaches learn discriminative features and handle drastic viewpoint change in a supervised way and require labeling new training data for a different pair of camera views. In this paper, we propose a novel perspective for person re-identification based on unsupervised salience learning. Distinctive features are extracted without requiring identity labels in the training procedure. First, we apply adjacency constrained patch matching to build dense correspondence between image pairs, which shows effectiveness in handling misalignment caused by large viewpoint and pose variations. Second, we learn human salience in an unsupervised manner. To improve the performance of person re-identification, human salience is incorporated in patch matching to find reliable and discriminative matched patches. The effectiveness of our approach is validated on the widely used VIPeR dataset and ETHZ dataset.

It is very attractive to formulate vision in terms of pattern theory [26], where patterns are defined hierarchically by compositions of elementary building blocks. But applying pattern theory to real world images is currently less successful than discriminative methods such as deep networks. Deep networks, however, are black-boxes which are hard to interpret and can easily be fooled by adding occluding objects. It is natural to wonder whether by better understanding deep networks we can extract building blocks which can be used to develop pattern theoretic models. This motivates us to study the internal representations of a deep network using vehicle images from the PASCAL3D+ dataset. We use clustering algorithms to study the population activities of the features and extract a set of visual concepts which we show are visually tight and correspond to semantic parts of vehicles. To analyze this we annotate these vehicles by their semantic parts to create a new dataset, VehicleSemanticParts, and evaluate visual concepts as unsupervised part detectors. We show that visual concepts perform fairly well but are outperformed by supervised discriminative methods such as Support Vector Machines (SVM). We next give a more detailed analysis of visual concepts and how they relate to semantic parts. Following this, we use the visual concepts as building blocks for a simple pattern theoretical model, which we call compositional voting. In this model several visual concepts combine to detect semantic parts. We show that this approach is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection. Finally, we return to studying occlusion by creating an annotated dataset with occlusion, called VehicleOcclusion, and show that compositional voting outperforms even deep networks when the amount of occlusion becomes large.

This paper describes a novel multi-view matching framework based on a new type of invariant feature. Our features are located at Harris corners in discrete scale-space and oriented using a blurred local gradient. This defines a rotationally invariant frame in which we sample a feature descriptor, which consists of an 8 /spl times/ 8 patch of bias/gain normalised intensity values. The density of features in the image is controlled using a novel adaptive non-maximal suppression algorithm, which gives a better spatial distribution of features than previous approaches. Matching is achieved using a fast nearest neighbour algorithm that indexes features based on their low frequency Haar wavelet coefficients. We also introduce a novel outlier rejection procedure that verifies a pairwise feature match based on a background distribution of incorrect feature matches. Feature matches are refined using RANSAC and used in an automatic 2D panorama stitcher that has been extensively tested on hundreds of sample inputs.

This paper describes a patch-based approach for rapid image correlation or template matching. By representing a template image with an ensemble of patches, the method is robust with respect to variations such as local appearance variation, partial occlusion, and scale changes. Rectangle filters are applied to each image patch for fast filtering based on the integral image representation. A new method is developed for feature dimension reduction by detecting the "salient" image structures given a single image. Experiments on a variety images show the success of the method in dealing with different variations in the test images. In terms of computation time, the approach is faster than traditional methods by up to two orders of magnitude and is at least three times faster than a fast implementation of normalized cross correlation.

This paper proposes an automated system for recognizing plant species based on leaf images. Plant leaf images corresponding to three plant types, are analyzed using two different shape modeling techniques, the first based on the Moments-Invariant (M-I) model and the second on the CentroidRadii (C-R) model. For the M-I model the first four normalized central moments have been considered and studied in various combinations viz. individually, in joint 2-D and 3-D feature spaces for producing optimum results. For the C-R model an edge detector has been used to identify the boundary of the leaf shape and 36 radii at 10 degree angular separation have been used to build the feature vector. To further improve the accuracy, a hybrid set of features involving both the M-I and C-R models has been generated and explored to find whether the combination feature vector can lead to better performance. Neural networks are used as classifiers for discrimination. The data set consists of 180 images divided into three classes with 60 images each. Accuracies ranging from 90%-100% are obtained which are comparable to the best figures reported in extant literature. Keywords-plant recognition; moment invariants; centroid-radii model; neural network; computer vision.

This work introduces a model that can recognize objects in images even if no training data is available for the objects. The only necessary knowledge about the unseen categories comes from unsupervised large text corpora. In our zero-shot framework distributional information in language can be seen as spanning a semantic basis for understanding what objects look like. Most previous zero-shot learning models can only differentiate between unseen classes. In contrast, our model can both obtain state of the art performance on classes that have thousands of training images and obtain reasonable performance on unseen classes. This is achieved by first using outlier detection in the semantic space and then two separate recognition models. Furthermore, our model does not require any manually defined semantic features for either words or images.


Question: CENTRIST: A visual descriptor utilized for scene categorization.

This article introduces an innovative approach for categorizing scene types based on the detection of approximate global geometric patterns. The method involves dividing the image into progressively finer segments and compiling histograms of local features found within these segments. The outcome is a "spatial pyramid," a straightforward and efficient enhancement of the conventional unordered bag-of-features image representation. This technique showcases a notable enhancement in performance for complex scene categorization tasks. Notably, the proposed method surpasses existing state-of-the-art techniques on the Caltech-101 dataset and attains high precision on a comprehensive database encompassing fifteen natural scene categories. Additionally, the spatial pyramid framework provides a new perspective on the efficacy of several recently introduced image descriptors, such as Torralba’s "gist" and Lowe’s SIFT descriptors. This approach not only improves upon traditional methods but also offers a deeper understanding of how these advanced descriptors operate within the context of scene categorization.

Visual attributes are great means of describing images or scenes, in a way both humans and computers understand. In order to establish a correspondence between images and to be able to compare the strength of each property between images, relative attributes were introduced. However, since their introduction, hand-crafted and engineered features were used to learn increasingly complex models for the problem of relative attributes. This limits the applicability of those methods for more realistic cases. We introduce a deep neural network architecture for the task of relative attribute prediction. A convolutional neural network (ConvNet) is adopted to learn the features by including an additional layer (ranking layer) that learns to rank the images based on these features. We adopt an appropriate ranking loss to train the whole network in an end-to-end fashion. Our proposed method outperforms the baseline and state-of-the-art methods in relative attribute prediction on various coarse and fine-grained datasets. Our qualitative results along with the visualization of the saliency maps show that the network is able to learn effective features for each specific attribute. Source code of the proposed method is available at https://github.com/yassersouri/ghiaseddin.

We propose a two-class classification model for grouping. Human segmented natural images are used as positive examples. Negative examples of grouping are constructed by randomly matching human segmentations and images. In a preprocessing stage an image is oversegmented into superpixels. We define a variety of features derived from the classical Gestalt cues, including contour, texture, brightness and good continuation. Information-theoretic analysis is applied to evaluate the power of these grouping cues. We train a linear classifier to combine these features. To demonstrate the power of the classification model, a simple algorithm is used to randomly search for good segmentations. Results are shown on a wide range of images.

Recent approaches on visual scene understanding attempt to build a scene graph – a computational representation of objects and their pairwise relationships. Such rich semantic representation is very appealing, yet difficult to obtain from a single image, especially when considering complex spatial arrangements in the scene. Differently, an image sequence conveys useful information using the multi-view geometric relations arising from camera motions. Indeed, object relationships are naturally related to the 3D scene structure. To this end, this paper proposes a system that first computes the geometrical location of objects in a generic scene and then efficiently constructs scene graphs from video by embedding such geometrical reasoning. Such compelling representation is obtained using a new model where geometric and visual features are merged using an RNN framework. We report results on a dataset we created for the task of 3D scene graph generation in multiple views.

In recent years we have seen a tremendous growth in the volume of text documents available on the Internet, digital libraries, news sources, and company-wide intrane ts. Automatic text categorization, which is the task of assigning text documents to pre-specified classes (topics o r themes) of documents, is an important task that can help both in organizing as well as in finding information on these h uge resources. Text categorization presents unique challenges due to the large number of attributes present in t he data set, large number of training samples, and attribute dependencies. In this paper we focus on a simple linear-time centroid-based document classification algorithm, that despite its simplicity and robust performance, has not been extensively studied and analyzed. Our extensive experiments show that this centroid-based classifier consi stently and substantially outperforms other algorithms su ch as Naive Bayesian, k-nearest-neighbors, and C4.5, on a wide range of datasets. O ur analysis shows that the similarity measure used by the centroid-based scheme allows it to class ify new document based on how closely its behavior matches the behavior of the documents belonging to differen t classes, as measured by the average similarity between the documents. This matching allows it to dynamically adjus t for classes with different densities. Furthermore, our analysis shows that the similarity measure of the centroidbased scheme accounts for dependencies between the terms in the different classes. We believe that this feature is the reason why it consistently outperforms other classifiers th at cannot take these dependencies into account.

In the past decade, SIFT descriptor has been witnessed as one of the most robust local invariant feature descriptors and widely used in various vision tasks. Most traditional image-classification systems depend on the gray-based SIFT descriptors, which only analyze the gray level variations of the images. Misclassification may happen since their color contents are ignored. In this article, we concentrate on improving the performance of existing image-classification algorithms by adding color information. To achieve this purpose, different kinds of colored SIFT descriptors are introduced and implemented. locality-constrained linear coding (LLC), a state-of-the-art sparse coding technology, is employed to construct the image-classification system for the evaluation. Moreover, we propose a simple $$\ell _2$$ ℓ 2 -norm regularized local distance to improve the traditional LLC method. The real experiments are carried out on several benchmarks. With the enhancements to color SIFT and $$\ell _2$$ ℓ 2 -norm regularization, the proposed image-classification system obtains approximately $$2\,\%$$ 2 % improvement of classification accuracy on the Caltech-101 dataset and approximately $$5\,\%$$ 5 % improvement of classification accuracy on the Caltech-256 dataset.

This thesis focuses on the area of face processing and aims at designing a reliable framework to facilitate face, age, and gender recognition. A Bag-of-Words framework has been optimized for the task of face recognition by evaluating different feature descriptors and different bag-of-words configurations. More specifically, we choose a compact set of features (e.g., descriptors, window locations, window sizes, dictionary sizes, etc.) in order to produce the highest possible rate of accuracy. Experiments on a challenging dataset shows that our framework achieves a better level of accuracy when compared to other popular approaches such as dimension reduction techniques, edge detection operators, and texture and shape feature extractors. The second contribution of this thesis is the proposition of a general framework for age and gender classification. Although the vast majority of the existing solutions focus on a single visual descriptor that often only encodes a certain characteristic of the image regions, this thesis aims at integrating multiple feature types. For this purpose, feature selection is employed to obtain more accurate and robust facial descriptors. Once descriptors have been computed, a compact set of features is chosen, which facilitates facial image processing for age and gender analysis. In addition to this, a new color descriptor (CLR-LBP) is proposed and the results obtained is shown to be comparable to those of other pre-existing color descriptors. The experimental results indicates that our age and gender framework outperforms other proposed methods when examined on two challenging databases, where face objects are present with different expressions and levels of illumination. This achievement demonstrates the effectiveness of our proposed solution and allows us to achieve a higher accuracy over the existing state-of-the-art methods.

This paper addresses the problem of generating possible object locations for use in object recognition. We introduce selective search which combines the strength of both an exhaustive search and segmentation. Like segmentation, we use the image structure to guide our sampling process. Like exhaustive search, we aim to capture all possible object locations. Instead of a single technique to generate possible object locations, we diversify our search and use a variety of complementary image partitionings to deal with as many image conditions as possible. Our selective search results in a small set of data-driven, class-independent, high quality locations, yielding 99 % recall and a Mean Average Best Overlap of 0.879 at 10,097 locations. The reduced number of locations compared to an exhaustive search enables the use of stronger machine learning techniques and stronger appearance models for object recognition. In this paper we show that our selective search enables the use of the powerful Bag-of-Words model for recognition. The selective search software is made publicly available (Software: http://disi.unitn.it/~uijlings/SelectiveSearch.html ).

Image retrieval is an important problem for researchers in computer vision and content-based image retrieval (CBIR) fields. Over the last decades, many image retrieval systems were based on image representation as a set of extracted low-level features such as color, texture and shape. Then, systems calculate similarity metrics between features in order to find similar images to a query image. The disadvantage of this approach is that images visually and semantically different may be similar in the low level feature space. So, it is necessary to develop tools to optimize retrieval of information. Integration of vector space models is one solution to improve the performance of image retrieval. In this paper, we present an efficient and effective retrieval framework which includes a vectorization technique combined with a pseudo relevance model. The idea is to transform any similarity matching model (between images) to a vector space model providing a score. A study on several methodologies to obtain the vectorization is presented. Some experiments have been undertaken on Wang, Oxford5k and Inria Holidays datasets to show the performance of our proposed framework.

Learning powerful discriminative features for remote sensing image scene classification is a challenging computer vision problem. In the past, most classification approaches were based on handcrafted features. However, most recent approaches to remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is only to use original RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show class activation map (CAM) encoded CNN models, codenamed DDRL-AM, trained using original RGB patches and attention map based class information provide complementary information to the standard RGB deep models. To the best of our knowledge, we are the first to investigate attention information encoded CNNs. Additionally, to enhance the discriminability, we further employ a recently developed object function called "center loss," which has proved to be very useful in face recognition. Finally, our framework provides attention guidance to the model in an end-to-end fashion. Extensive experiments on two benchmark datasets show that our approach matches or exceeds the performance of other methods.

This work introduces a model that can recognize objects in images even if no training data is available for the objects. The only necessary knowledge about the unseen categories comes from unsupervised large text corpora. In our zero-shot framework distributional information in language can be seen as spanning a semantic basis for understanding what objects look like. Most previous zero-shot learning models can only differentiate between unseen classes. In contrast, our model can both obtain state of the art performance on classes that have thousands of training images and obtain reasonable performance on unseen classes. This is achieved by first using outlier detection in the semantic space and then two separate recognition models. Furthermore, our model does not require any manually defined semantic features for either words or images.


Question: CENTRIST: A visual descriptor utilized for scene categorization.

This paper presents an innovative approach to classifying events in static images by merging scene and object recognition techniques. The concept of an event in a static image is defined as a human activity unfolding within a distinct setting. To illustrate the classification of such events, the paper uses a selection of sports games, including snowboarding, rock climbing, and badminton. The objective is not only to identify the event depicted in the image but also to attach semantic labels to the objects and the scene environment. For instance, in the case of a rowing scene, the system recognizes the event as rowing by identifying the scene as a lake and detecting key objects such as athletes, rowing boats, and water. This comprehensive and integrated recognition is achieved through the use of a generative graphical model. The researchers have compiled a database of 8 diverse sports events, demonstrating the system's ability to classify these event categories with a high accuracy of 73.4%. The results indicate that while both scene and object recognition contribute to the final classification, relying solely on either scene or object recognition does not yield the same level of performance.

In this paper we revisit local feature detectors/descriptors developed for 2D images and extend them to the more general framework of scalar fields defined on 2D manifolds. We provide methods and tools to detect and describe features on surfaces equiped with scalar functions, such as photometric information. This is motivated by the growing need for matching and tracking photometric surfaces over temporal sequences, due to recent advancements in multiple camera 3D reconstruction. We propose a 3D feature detector (MeshDOG) and a 3D feature descriptor (MeshHOG) for uniformly triangulated meshes, invariant to changes in rotation, translation, and scale. The descriptor is able to capture the local geometric and/or photometric properties in a succinct fashion. Moreover, the method is defined generically for any scalar function, e.g., local curvature. Results with matching rigid and non-rigid meshes demonstrate the interest of the proposed framework.

This thesis focuses on the area of face processing and aims at designing a reliable framework to facilitate face, age, and gender recognition. A Bag-of-Words framework has been optimized for the task of face recognition by evaluating different feature descriptors and different bag-of-words configurations. More specifically, we choose a compact set of features (e.g., descriptors, window locations, window sizes, dictionary sizes, etc.) in order to produce the highest possible rate of accuracy. Experiments on a challenging dataset shows that our framework achieves a better level of accuracy when compared to other popular approaches such as dimension reduction techniques, edge detection operators, and texture and shape feature extractors. The second contribution of this thesis is the proposition of a general framework for age and gender classification. Although the vast majority of the existing solutions focus on a single visual descriptor that often only encodes a certain characteristic of the image regions, this thesis aims at integrating multiple feature types. For this purpose, feature selection is employed to obtain more accurate and robust facial descriptors. Once descriptors have been computed, a compact set of features is chosen, which facilitates facial image processing for age and gender analysis. In addition to this, a new color descriptor (CLR-LBP) is proposed and the results obtained is shown to be comparable to those of other pre-existing color descriptors. The experimental results indicates that our age and gender framework outperforms other proposed methods when examined on two challenging databases, where face objects are present with different expressions and levels of illumination. This achievement demonstrates the effectiveness of our proposed solution and allows us to achieve a higher accuracy over the existing state-of-the-art methods.

Photo aesthetic quality evaluation is a challenging task in multimedia and computer vision fields. Conventional approaches suffer from the following three drawbacks: 1) the deemphasized role of semantic content that is many times more important than low-level visual features in photo aesthetics; 2) the difficulty to optimally fuse low-level and high-level visual cues in photo aesthetics evaluation; and 3) the absence of a sequential viewing path in the existing models, as humans perceive visually salient regions sequentially when viewing a photo.
 To solve these problems, we propose a new aesthetic descriptor that mimics humans sequentially perceiving visually/semantically salient regions in a photo. In particular, a weakly supervised learning paradigm is developed to project the local aesthetic descriptors (graphlets in this work) into a low-dimensional semantic space. Thereafter, each graphlet can be described by multiple types of visual features, both at low-level and in high-level. Since humans usually perceive only a few salient regions in a photo, a sparsity-constrained graphlet ranking algorithm is proposed that seamlessly integrates both the low-level and the high-level visual cues. Top-ranked graphlets are those visually/semantically prominent graphlets in a photo. They are sequentially linked into a path that simulates the process of humans actively viewing. Finally, we learn a probabilistic aesthetic measure based on such actively viewing paths (AVPs) from the training photos that are marked as aesthetically pleasing by multiple users. Experimental results show that: 1) the AVPs are 87.65% consistent with real human gaze shifting paths, as verified by the eye-tracking data; and 2) our photo aesthetic measure outperforms many of its competitors.

Semantically-rich photos contain a rich variety of semantic objects (e.g., pedestrians and bicycles). Retargeting these photos is a challenging task since each semantic object has fixed geometric characteristics. Shrinking these objects simultaneously during retargeting is prone to distortion. In this paper, we propose to retarget semantically-rich photos by detecting photo semantics from image tags, which are predicted by a multi-label SVM. The key technique is a generative model termed latent stability discovery (LSD). It can robustly localize various semantic objects in a photo by making use of the predicted noisy image tags. Based on LSD, a feature fusion algorithm is proposed to detect salient regions at both the low-level and high-level. These salient regions are linked into a path sequentially to simulate human visual perception . Finally, we learn the prior distribution of such paths from aesthetically pleasing training photos. The prior enforces the path of a retargeted photo to be maximally similar to those from the training photos. In the experiment, we collect 217 1600 ×1200 photos, each containing over seven salient objects. Comprehensive user studies demonstrate the competitiveness of our method.

While image alignment has been studied in different areas of computer vision for decades, aligning images depicting different scenes remains a challenging problem. Analogous to optical flow, where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its nearest neighbors in a large image corpus containing a variety of scenes. The SIFT flow algorithm consists of matching densely sampled, pixelwise SIFT features between two images while preserving spatial discontinuities. The SIFT features allow robust matching across different scene/object appearances, whereas the discontinuity-preserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach robustly aligns complex scene pairs containing significant spatial differences. Based on SIFT flow, we propose an alignment-based large database framework for image analysis and synthesis, where image information is transferred from the nearest neighbors to a query image according to the dense scene correspondence. This framework is demonstrated through concrete applications such as motion field prediction from a single image, motion synthesis via object transfer, satellite image registration, and face recognition.

We address the problem of image search on a very large scale, where three constraints have to be considered jointly: the accuracy of the search, its efficiency, and the memory usage of the representation. We first propose a simple yet efficient way of aggregating local image descriptors into a vector of limited dimension, which can be viewed as a simplification of the Fisher kernel representation. We then show how to jointly optimize the dimension reduction and the indexing algorithm, so that it best preserves the quality of vector comparison. The evaluation shows that our approach significantly outperforms the state of the art: the search accuracy is comparable to the bag-of-features approach for an image representation that fits in 20 bytes. Searching a 10 million image dataset takes about 50ms.

Human eyes can recognize person identities based on some small salient regions. However, such valuable salient information is often hidden when computing similarities of images with existing approaches. Moreover, many existing approaches learn discriminative features and handle drastic viewpoint change in a supervised way and require labeling new training data for a different pair of camera views. In this paper, we propose a novel perspective for person re-identification based on unsupervised salience learning. Distinctive features are extracted without requiring identity labels in the training procedure. First, we apply adjacency constrained patch matching to build dense correspondence between image pairs, which shows effectiveness in handling misalignment caused by large viewpoint and pose variations. Second, we learn human salience in an unsupervised manner. To improve the performance of person re-identification, human salience is incorporated in patch matching to find reliable and discriminative matched patches. The effectiveness of our approach is validated on the widely used VIPeR dataset and ETHZ dataset.

Establishing dense correspondences reliably between a pair of images is an important vision task with many applications. Though significant advance has been made towards estimating dense stereo and optical flow fields for two images adjacent in viewpoint or in time, building reliable dense correspondence fields for two general images still remains largely unsolved. For instance, two given images sharing some content exhibit dramatic photometric and geometric variations, or they depict different 3D scenes of similar scene characteristics. Fundamental challenges to such an image or scene alignment task are often multifold, which render many existing techniques fall short of producing dense correspondences robustly and efficiently. This paper presents a novel approach called DAISY filter flow (DFF) to address this challenging task. Inspired by the recent PatchMatch Filter technique, we leverage and extend a few established methods: DAISY descriptors, filter-based efficient flow inference, and the PatchMatch fast search. Coupling and optimizing these modules seamlessly with image segments as the bridge, the proposed DFF approach enables efficiently performing dense descriptor-based correspondence field estimation in a generalized high-dimensional label space, which is augmented by scales and rotations. Experiments on a variety of challenging scenes show that our DFF approach estimates spatially coherent yet discontinuity-preserving image alignment results both robustly and efficiently.

We consider visual category recognition in the framework of measuring similarities, or equivalently perceptual distances, to prototype examples of categories. This approach is quite flexible, and permits recognition based on color, texture, and particularly shape, in a homogeneous framework. While nearest neighbor classifiers are natural in this setting, they suffer from the problem of high variance (in bias-variance decomposition) in the case of limited sampling. Alternatively, one could use support vector machines but they involve time-consuming optimization and computation of pairwise distances. We propose a hybrid of these two methods which deals naturally with the multiclass setting, has reasonable computational complexity both in training and at run time, and yields excellent results in practice. The basic idea is to find close neighbors to a query sample and train a local support vector machine that preserves the distance function on the collection of neighbors. Our method can be applied to large, multiclass data sets for which it outperforms nearest neighbor and support vector machines, and remains efficient when the problem becomes intractable for support vector machines. A wide variety of distance functions can be used and our experiments show state-of-the-art performance on a number of benchmark data sets for shape and texture classification (MNIST, USPS, CUReT) and object recognition (Caltech- 101). On Caltech-101 we achieved a correct classification rate of 59.05%(±0.56%) at 15 training images per class, and 66.23%(±0.48%) at 30 training images.

In recent years we have seen a tremendous growth in the volume of text documents available on the Internet, digital libraries, news sources, and company-wide intrane ts. Automatic text categorization, which is the task of assigning text documents to pre-specified classes (topics o r themes) of documents, is an important task that can help both in organizing as well as in finding information on these h uge resources. Text categorization presents unique challenges due to the large number of attributes present in t he data set, large number of training samples, and attribute dependencies. In this paper we focus on a simple linear-time centroid-based document classification algorithm, that despite its simplicity and robust performance, has not been extensively studied and analyzed. Our extensive experiments show that this centroid-based classifier consi stently and substantially outperforms other algorithms su ch as Naive Bayesian, k-nearest-neighbors, and C4.5, on a wide range of datasets. O ur analysis shows that the similarity measure used by the centroid-based scheme allows it to class ify new document based on how closely its behavior matches the behavior of the documents belonging to differen t classes, as measured by the average similarity between the documents. This matching allows it to dynamically adjus t for classes with different densities. Furthermore, our analysis shows that the similarity measure of the centroidbased scheme accounts for dependencies between the terms in the different classes. We believe that this feature is the reason why it consistently outperforms other classifiers th at cannot take these dependencies into account.


Question: CENTRIST: A visual descriptor utilized for scene categorization.

This paper introduces a robust and efficient method for multiresolution gray-scale and rotation invariant texture classification, utilizing local binary patterns (LBP) and nonparametric discrimination of sample and prototype distributions. The approach is centered around the recognition that certain local binary patterns, termed "uniform," are fundamental descriptors of local image texture. The occurrence histogram of these patterns is found to be an exceptionally powerful texture feature. A generalized gray-scale and rotation invariant operator presentation is developed to detect these "uniform" patterns, applicable at any quantization of the angular space and any spatial resolution, and provides a method for combining multiple operators for multiresolution analysis. The operator is inherently robust against gray-scale variations due to its invariance against any monotonic transformation of the gray scale, and its computational simplicity is a significant advantage, as it can be realized with minimal operations in a small neighborhood and a lookup table. The method's effectiveness is substantiated through excellent experimental results in real-world problems of rotation invariance, where the classifier is trained at one specific rotation angle and tested with samples from various other angles. These results demonstrate that the occurrence statistics of simple rotation invariant local binary patterns can achieve high discrimination. The performance can be further enhanced by combining these operators with rotation invariant variance measures that characterize the contrast of local image texture. The joint distributions of these orthogonal measures are shown to be exceptionally powerful tools for rotation invariant texture analysis. Keywords: Nonparametric, texture analysis, Outex, Brodatz, distribution, histogram, contrast.

Robust and accurate visual localization across large appearance variations due to changes in time of day, seasons, or changes of the environment is a challenging problem which is of importance to application areas such as navigation of autonomous robots. Traditional featurebased methods often struggle in these conditions due to the significant number of erroneous matches between the image and the 3D model. In this paper, we present a method for scoring the individual correspondences by exploiting semantic information about the query image and the scene. In this way, erroneous correspondences tend to get a low semantic consistency score, whereas correct correspondences tend to get a high score. By incorporating this information in a standard localization pipeline, we show that the localization performance can be significantly improved compared to the state-of-the-art, as evaluated on two challenging long-term localization benchmarks.

In this paper, we propose to learn a discriminative and shareable feature transformation filter bank to transform local image patches (represented as raw pixel values) into features for scene image classification. The learned filters are expected to: (1) encode common visual patterns of a flexible number of categories; (2) encode discriminative and class-specific information. For each category, a subset of the filters are activated in a data-adaptive manner, meanwhile sharing of filters among different categories is also allowed. Discriminative power of the filter bank is further enhanced by enforcing the features from the same category to be close to each other in the feature space, while features from different categories to be far away from each other. The experimental results on three challenging scene image classification datasets indicate that our features can achieve very promising performance. Furthermore, our features also show great complementary effect to the state-of-the-art ConvNets feature.

Visual place recognition is challenging in the urban environment and is usually viewed as a large scale image retrieval task. The intrinsic challenges in place recognition exist that the confusing objects such as cars and trees frequently occur in the complex urban scene, and buildings with repetitive structures may cause over-counting and the burstiness problem degrading the image representations. To address these problems, we present an Attention-based Pyramid Aggregation Network (APANet), which is trained in an end-to-end manner for place recognition. One main component of APANet, the spatial pyramid pooling, can effectively encode the multi-size buildings containing geo-information. The other one, the attention block, is adopted as a region evaluator for suppressing the confusing regional features while highlighting the discriminative ones. When testing, we further propose a simple yet effective PCA power whitening strategy, which significantly improves the widely used PCA whitening by reasonably limiting the impact of over-counting. Experimental evaluations demonstrate that the proposed APANet outperforms the state-of-the-art methods on two place recognition benchmarks, and generalizes well on standard image retrieval datasets.

In this paper, we propose a Scale and Rotation Invariant Implicit Shape Model (SRIISM), and develop a local feature matching based system using the model to accurately locate and identify large numbers of object instances in an image. Due to repeated instances and cluttered background, conventional methods for multiple object instance identification suffer from poor identification results. In the proposed SRIISM, we model the joint distribution of object centers, scale, and orientation computed from local feature matches in Hough voting, which is not only invariant to scale changes and rotation of objects, but also robust to false feature matches. In the multiple object instance identification system using SRIISM, we apply a fast 4D bin search method in Hough space with complexity O(n), where n is the number of feature matches, in order to segment and locate each instance. Futhermore, we apply maximum likelihood estimation (MLE) for accurate object pose detection. In the evaluation, we created datasets simulating various industrial applications such as pick-and-place and inventory management. Experiment results on the datasets show that our method outperforms conventional methods in both accuracy (5%-30% gain) and speed (2x speed up).

We present a learning-based, sliding window-style approach for the problem of detecting humans in still images. Instead of traditional concatenation-style image location-based feature encoding, a global descriptor more invariant to pose variation is introduced. Specifically, we propose a principled approach to learning and classifying human/non-human image patterns by simultaneously segmenting human shapes and poses, and extracting articulation-insensitive features. The shapes and poses are segmented by an efficient, probabilistic hierarchical part-template matching algorithm, and the features are collected in the context of poses by tracing around the estimated shape boundaries. Histograms of oriented gradients are used as a source of low-level features from which our pose-invariant descriptors are computed, and kernel SVMs are adopted as the test classifiers. We evaluate our detection and segmentation approach on two public pedestrian datasets.

Image retrieval is an important problem for researchers in computer vision and content-based image retrieval (CBIR) fields. Over the last decades, many image retrieval systems were based on image representation as a set of extracted low-level features such as color, texture and shape. Then, systems calculate similarity metrics between features in order to find similar images to a query image. The disadvantage of this approach is that images visually and semantically different may be similar in the low level feature space. So, it is necessary to develop tools to optimize retrieval of information. Integration of vector space models is one solution to improve the performance of image retrieval. In this paper, we present an efficient and effective retrieval framework which includes a vectorization technique combined with a pseudo relevance model. The idea is to transform any similarity matching model (between images) to a vector space model providing a score. A study on several methodologies to obtain the vectorization is presented. Some experiments have been undertaken on Wang, Oxford5k and Inria Holidays datasets to show the performance of our proposed framework.

We propose to shift the goal of recognition from naming to describing. Doing so allows us not only to name familiar objects, but also: to report unusual aspects of a familiar object (“spotty dog”, not just “dog”); to say something about unfamiliar objects (“hairy and four-legged”, not just “unknown”); and to learn how to recognize new objects with few or no visual examples. Rather than focusing on identity assignment, we make inferring attributes the core problem of recognition. These attributes can be semantic (“spotty”) or discriminative (“dogs have it but sheep do not”). Learning attributes presents a major new challenge: generalization across object categories, not just across instances within a category. In this paper, we also introduce a novel feature selection method for learning attributes that generalize well across categories. We support our claims by thorough evaluation that provides insights into the limitations of the standard recognition paradigm of naming and demonstrates the new abilities provided by our attribute-based framework.

The paper gives a snapshot of the state of the art in affine covariant region detectors, and compares their performance on a set of test images under varying imaging conditions. Six types of detectors are included: detectors based on affine normalization around Harris  (Mikolajczyk and  Schmid, 2002; Schaffalitzky and  Zisserman, 2002) and Hessian points  (Mikolajczyk and  Schmid, 2002), a detector of ‘maximally stable extremal regions', proposed by Matas et al. (2002); an edge-based region detector  (Tuytelaars and Van Gool, 1999) and a detector based on intensity extrema (Tuytelaars and Van Gool, 2000), and a detector of ‘salient regions', proposed by Kadir, Zisserman and Brady (2004). The performance is measured against changes in viewpoint, scale, illumination, defocus and image compression. The objective of this paper is also to establish a reference test set of images and performance software, so that future detectors can be evaluated in the same framework.

The ability to automatically detect ‘visually interesting’ regions in an image has many practical applications especially in the design of active machine vision systems. This paper describes a data-driven approach that uses eye tracking in tandem with principal component analysis to extract lowlevel image features that attract human gaze. Data analysis on an ensemble of image patches extracted at the observer’s point of gaze revealed features that resemble derivatives of the 2D Gaussian operator. Dissimilarities between human and random fixations are investigated by comparing the features extracted at the point of gaze to the general image structure obtained by random sampling in monte-carlo simulations. Finally, a simple application where these features are used to predict fixations is illustrated.

Recursive structure is commonly found in the inputs of different modalities such as natural scene images or natural language sentences. Discovering this recursive structure helps us to not only identify the units that an image or sentence contains but also how they interact to form a whole. We introduce a max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences. The same algorithm can be used both to provide a competitive syntactic parser for natural language sentences from the Penn Treebank and to outperform alternative approaches for semantic scene segmentation, annotation and classification. For segmentation and annotation our algorithm obtains a new level of state-of-theart performance on the Stanford background dataset (78.1%). The features from the image parse tree outperform Gist descriptors for scene classification by 4%.


Question: CENTRIST: A visual descriptor utilized for scene categorization.

This paper presents a cutting-edge method for robust real-time face detection in various environmental conditions. Unlike previous approaches that often struggle with lighting changes, pose variations, and occlusions, this novel technique employs a multi-scale sliding window approach combined with a cascaded classifier to achieve high detection rates while maintaining real-time performance. The system first uses a Haar-like feature set for fast initial screening, significantly reducing the number of false positives. Subsequently, a more sophisticated feature set, such as Local Binary Patterns (LBP) or Histogram of Oriented Gradients (HOG), is applied to refine the detection, ensuring accuracy even in challenging scenarios. To further enhance robustness, the method incorporates adaptive thresholding, which adjusts the detection parameters based on the local image statistics, effectively handling different lighting conditions. Moreover, the algorithm is optimized for real-time processing by utilizing efficient data structures and parallel computing techniques, making it suitable for applications in surveillance, security, and human-computer interaction where instantaneous response is crucial. Extensive testing on diverse datasets demonstrates the method's superior performance compared to existing face detection algorithms, making it a reliable choice for real-world deployment.

Template matching is a technique widely used for finding patterns in digital images. An efficient template matching algorithm should be able to detect template instances that have undergone geometric transformations. Similarly, a color template matching should be able to deal with color constancy problem. Recently we have proposed a new grayscale template matching algorithm named Ciratefi, invariant to rotation, scale, translation, brightness and contrast. In this paper we introduce the Color-Ciratefi that takes into account the color information. We use a new similarity metric in the CIELAB space to obtain invariance to brightness and contrast changes. Experiments show that Color-Ciratefi is more accurate than C-color-SIFT, the well-known SIFT algorithm that uses a set of color invariants. In conventional computers, Color-Ciratefi is slower than C-color-SIFT. However Color-Ciratefi is more suitable than C-color-SIFT to be implemented in highly parallel architectures like FPGA, because it repeats exactly the same set of operations for each pixel.

While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry. Large labeled training datasets, expensive and tedious to produce, are required to optimize millions of parameters in deep network models. Lagging behind the growth in model capacity, the available datasets are quickly becoming outdated in terms of size and density. To circumvent this bottleneck, we propose to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop. Starting from a large set of candidate images for each category, we iteratively sample a subset, ask people to label them, classify the others with a trained model, split the set into positives, negatives, and unlabeled based on the classification confidence, and then iterate with the unlabeled set. To assess the effectiveness of this cascading procedure and enable further progress in visual recognition research, we construct a new image dataset, LSUN. It contains around one million labeled images for each of 10 scene categories and 20 object categories. We experiment with training popular convolutional networks and find that they achieve substantial performance gains when trained on this dataset.

This paper presents our research in image content based indexing and retrieval, a key technology in digital image libraries. In most of the existing image content-based techniques, image features used for indexing and retrieval are global, features are computed over the entire image. The major problem with the global image feature based retrieval methods is that background features can be easily mistakenly taken as object features. When a user attempts to retrieve images using color features, he/she usually means the color feature of an object or objects of interests contained in the image. The approach we describe in this paper utilizes color clusters for image background analysis. Once the background regions are identified, they are removed from the image indexing procedure; therefore, no longer interfering with the meaningful image content during the retrieval process. The algorithm consists of three major steps of computation, fuzzy clustering, color image segmentation, and background analysis.

We propose a multi-resolution framework inspired by human visual search for general object detection. Different resolutions are represented using a coarse-to-fine feature hierarchy. During detection, the lower resolution features are initially used to reject the majority of negative windows at relatively low cost, leaving a relatively small number of windows to be processed in higher resolutions. This enables the use of computationally more expensive higher resolution features to achieve high detection accuracy. We applied this framework on Histograms of Oriented Gradient (HOG) features for object detection. Our multi-resolution detector produced better performance for pedestrian detection than state-of-the-art methods (Dalal and Triggs, 2005), and was faster during both training and testing. Testing our method on motorbikes and cars from the VOC database revealed similar improvements in both speed and accuracy, suggesting that our approach is suitable for realtime general object detection applications.

We posit that visually descriptive language offers computer vision researchers both information about the world, and information about how people describe the world. The potential benefit from this source is made more significant due to the enormous amount of language data easily available today. We present a system to automatically generate natural language descriptions from images that exploits both statistics gleaned from parsing large quantities of text data and recognition algorithms from computer vision. The system is very effective at producing relevant sentences for images. It also generates descriptions that are notably more true to the specific image content than previous work.

We present Local Naive Bayes Nearest Neighbor, an improvement to the NBNN image classification algorithm that increases classification accuracy and improves its ability to scale to large numbers of object classes. The key observation is that only the classes represented in the local neighborhood of a descriptor contribute significantly and reliably to their posterior probability estimates. Instead of maintaining a separate search structure for each class's training descriptors, we merge all of the reference data together into one search structure, allowing quick identification of a descriptor's local neighborhood. We show an increase in classification accuracy when we ignore adjustments to the more distant classes and show that the run time grows with the log of the number of classes rather than linearly in the number of classes as did the original. Local NBNN gives a 100 times speed-up over the original NBNN on the Caltech 256 dataset. We also provide the first head-to-head comparison of NBNN against spatial pyramid methods using a common set of input features. We show that local NBNN outperforms all previous NBNN based methods and the original spatial pyramid model. However, we find that local NBNN, while competitive with, does not beat state-of-the-art spatial pyramid methods that use local soft assignment and max-pooling.

We aim at combining color and shape invariants for indexing and retrieving images. To this end, color models are proposed independent of the object geometry, object pose, and illumination. From these color models, color invariant edges are derived from which shape invariant features are computed. Computational methods are described to combine the color and shape invariants into a unified high-dimensional invariant feature set for discriminatory object retrieval. Experiments have been conducted on a database consisting of 500 images taken from multicolored man-made objects in real world scenes. From the theoretical and experimental results it is concluded that object retrieval based on composite color and shape invariant features provides excellent retrieval accuracy. Object retrieval based on color invariants provides very high retrieval accuracy whereas object retrieval based entirely on shape invariants yields poor discriminative power. Furthermore, the image retrieval scheme is highly robust to partial occlusion, object clutter and a change in the object's pose. Finally, the image retrieval scheme is integrated into the PicToSeek system on-line at http://www.wins.uva.nl/research/isis/PicToSeek/ for searching images on the World Wide Web.

Recursive structure is commonly found in the inputs of different modalities such as natural scene images or natural language sentences. Discovering this recursive structure helps us to not only identify the units that an image or sentence contains but also how they interact to form a whole. We introduce a max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences. The same algorithm can be used both to provide a competitive syntactic parser for natural language sentences from the Penn Treebank and to outperform alternative approaches for semantic scene segmentation, annotation and classification. For segmentation and annotation our algorithm obtains a new level of state-of-theart performance on the Stanford background dataset (78.1%). The features from the image parse tree outperform Gist descriptors for scene classification by 4%.

Human Vision system is the most complex and accurate system. In order to extract better features about Synthetic Aperture Radar (SAR) targets, a SAR automatic target recognition (ATR) algorithm based on human visual cortical system is proposed. This algorithm contains three stages: (1) Image preprocessing (we use a Kuan filter to do the enhancement and an adaptive Intersecting Cortical Model (ICM) to do the segmentation) (2) Feature extraction using a sparse autoencoder. (3) Classification using a softmax regression classifier. Experiment result of MSTAR public data shows a better performance of recognition.

The ability to automatically detect ‘visually interesting’ regions in an image has many practical applications especially in the design of active machine vision systems. This paper describes a data-driven approach that uses eye tracking in tandem with principal component analysis to extract lowlevel image features that attract human gaze. Data analysis on an ensemble of image patches extracted at the observer’s point of gaze revealed features that resemble derivatives of the 2D Gaussian operator. Dissimilarities between human and random fixations are investigated by comparing the features extracted at the point of gaze to the general image structure obtained by random sampling in monte-carlo simulations. Finally, a simple application where these features are used to predict fixations is illustrated.


Question: The argument for a visual discovery aide: a comprehensive resolution to expedite visual data exploration.

This paper introduces (s|qu)eries, a visual query interface designed to facilitate the exploration and querying of event sequence data. Event sequence data is prevalent across various domains, and the ability to identify and analyze patterns within this data is crucial for deriving valuable insights. However, existing systems often fall short in providing comprehensive query capabilities, hindering exploratory workflows and offering suboptimal visualization and interaction models for large, complex datasets. (s|qu)eries, pronounced "Squeries," addresses these limitations by offering a touch-based interface that leverages the full power of regular expressions for querying series of data. This system not only simplifies the query specification process but also integrates real-time result visualizations, enabling users to visually inspect the outcomes of different query components. This feature not only aids in debugging but also fosters an iterative and exploratory approach to query building. To ensure the effectiveness and usability of (s|qu)eries, we conducted informal interviews with data scientists who regularly analyze event sequences. Their feedback validated the design and implementation of our system, highlighting its potential to significantly enhance the efficiency and ease of visual data exploration.

The Case for a Visual Discovery Assistant: A Comprehensive Solution for Accelerating Visual Data Exploration

In the era of big data, the ability to efficiently explore and analyze visual data has become increasingly critical. Traditional data exploration tools often fall short in providing intuitive and interactive interfaces, particularly for non-technical users. This paper argues for the development of a Visual Discovery Assistant, a holistic solution designed to expedite and simplify the process of visual data exploration. The Visual Discovery Assistant is envisioned as a comprehensive platform that integrates advanced visual analytics, machine learning, and natural language processing to offer an intuitive and interactive experience. It is touch-based and designed to accommodate both structured and unstructured data, providing users with the ability to query, visualize, and analyze data in real-time. Key features include dynamic visualizations that adjust based on user interactions, real-time feedback mechanisms to guide query refinement, and a user-friendly interface that supports exploratory workflows. The paper discusses the design principles, implementation details, and potential applications of the Visual Discovery Assistant. To validate its effectiveness, we conducted a series of informal interviews with data analysts from various fields, whose feedback confirmed the system's potential to significantly enhance data exploration capabilities.

Anyone who has gazed through the eyepiece of an astronomical telescope knows that, with the exception of the Moon and the planets, extra-solar astronomical objects are disappointing to observe visually. This is mainly due to their low surface brightness, but also depends on the visibility, sky brightness and telescope aperture. We propose a system which projects images of astronomical objects (with focus on nebulae and galaxies), animations and additional information directly into the eyepiece view of an astronomical telescope. As the telescope orientation is queried continuously, the projected image is adapted in real-time to the currently visible field of view. For projection, a custom-built video projection module with high contrast and low maximum luminance value was developed. With this technology visitors to public observatories have the option to experience the richness of faint astronomical objects while directly looking at them through a telescope.

The ability to automatically detect ‘visually interesting’ regions in an image has many practical applications especially in the design of active machine vision systems. This paper describes a data-driven approach that uses eye tracking in tandem with principal component analysis to extract lowlevel image features that attract human gaze. Data analysis on an ensemble of image patches extracted at the observer’s point of gaze revealed features that resemble derivatives of the 2D Gaussian operator. Dissimilarities between human and random fixations are investigated by comparing the features extracted at the point of gaze to the general image structure obtained by random sampling in monte-carlo simulations. Finally, a simple application where these features are used to predict fixations is illustrated.

Early hands-on experiences with the Microsoft Hololens augmented/mixed reality device are reported and discussed, with a general aim of exploring basic 3D visualization. A range of usage cases are tested, including data visualization and immersive data spaces, in-situ visualization of 3D models and full scale architectural form visualization. Ultimately, the Hololens is found to provide a remarkable tool for moving from traditional visualization of 3D objects on a 2D screen, to fully experiential 3D visualizations embedded in the real world.

In this paper, we propose the attention extraction method on a textbook of physics by using eye tracking glasses. We prepare a document including text and tasks in physics, and record reading behavior on the document from 6-grade students. The result confirms that students pay attention to a different region depending on situations (using the text as a material to learn the content, or using it as hints to solve tasks) and their comprehension.

Virtual Reality (VR) has generated much excitement but little formal proof that it is useful. Because VR interfaces are difficult and expensive to build, the computer graphics community needs to be able to predict which applications will benefit from VR. In this paper, we show that users with a VR interface complete a search task faster than users with a stationary monitor and a hand-based input device. We placed users in the center of the virtual room shown in Figure 1 and told them to look for camouflaged targets. VR users did not do significantly better than desktop users. However, when asked to search the room and conclude if a target existed, VR users were substantially better at determining when they had searched the entire room. Desktop users took 41% more time, re-examining areas they had already searched. We also found a positive transfer of training from VR to stationary displays and a negative transfer of training from stationary displays to VR.

Examining large-scale, long-term application use is critical to understanding how an application meets the needs of its user community. However, there have been few published analyses of long-term use of desktop applications, and none that have examined applications that support creating and modifying content using direct manipulation. In this paper, we present an analysis of 2 years of usage data from an instrumented version of the GNU Image Manipulation Program, including data from over 200 users. In the course of our analysis, we show that previous findings concerning the sparseness of command use and idiosyncrasy of users’ command vocabularies extend to a new domain and interaction style. These findings motivate continued research in adaptive and mixed-initiative interfaces. We also describe the novel application of a clustering technique to characterize a user community’s higher-level tasks from low-level logging data.

Multidisciplinary dataflow computing is a powerful approach to scientific computing that has led to orders-of-magnitude performance improvements for a wide range of applications.

We explore the concept of abstraction as it is used in visualization, with the ultimate goal of understanding and formally defining it. Researchers so far have used the concept of abstraction largely by intuition without a precise meaning. This lack of specificity left questions on the characteristics of abstraction, its variants, its control, or its ultimate potential for visualization and, in particular, illustrative visualization mostly unanswered. In this paper we thus provide a first formalization of the abstraction concept and discuss how this formalization affects the application of abstraction in a variety of visualization scenarios. Based on this discussion, we derive a number of open questions still waiting to be answered, thus formulating a research agenda for the use of abstraction for the visual representation and exploration of data. This paper, therefore, is intended to provide a contribution to the discussion of the theoretical foundations of our field, rather than attempting to provide a completed and final theory.

When people search for a target in a novel image they often make use of eye movements to bring the relatively high acuity fovea to bear on areas of interest. The strategies that control these eye movements for visual search have been of substantial scientific interest. In the current article we report a new computational model that shows how strategies for visual search are an emergent consequence of perceptual/motor constraints and approximately optimal strategies. The model solves a Partially Observable Markov Decision Process (POMDP) using deep Q-learning to acquire strategies that optimise the tradeoff between speed and accuracy. Results are reported for the Distractor-ratio task.

In recent years a new category of data analysis applications have evolved, known as data pipelining tools, which enable even nonexperts to perform complex analysis tasks on potentially huge amounts of data. Due to the complex and computing intensive analysis processes and methods used, it is often neither sufficient nor possible to simply rely on the increase of performance of single processors. Promising solutions to this problem are parallel and distributed approaches that can accelerate the analysis process. In this paper we discuss the parallel and distribution potential of pipelining tools by demonstrating several parallel and distributed implementations in the open source pipelining platform KNIME. We verify the practical applicability in a number of real world experiments.


Question: The argument for a visual discovery aide: a comprehensive resolution to expedite the exploration of visual data.

Introducing QueryGuide: Contextual Assistance for SQL Query Construction

This paper unveils QueryGuide, a system designed to offer real-time, context-sensitive support during the SQL query creation process. QueryGuide is tailored to assist the growing number of non-expert database users who require the ability to conduct intricate analyses on their extensive datasets but struggle with formulating SQL queries. As the user drafts a query, QueryGuide dynamically suggests potential additions to various clauses by drawing from a repository of previously executed queries. Its current features encompass recommending tables, views, and table-valued functions for the FROM clause, columns for the SELECT clause, predicates for the WHERE clause, columns for the GROUP BY clause, aggregate functions, and basic support for sub-queries. QueryGuide's recommendations adapt to the evolving context of the query: as more of the query is written, the system refines its suggestions for greater accuracy. To assess QueryGuide's efficacy, we conduct evaluations using two distinct datasets: one sourced from an undergraduate database course and another from the Sloan Digital Sky Survey database. Our findings reveal that QueryGuide achieves an impressive average precision of up to 93.7% for its suggestions, all delivered at interactive speeds. Additionally, we demonstrate that QueryGuide significantly outperforms simplistic approaches, such as recommending the most frequently used snippets, underscoring its superiority in enhancing the SQL query composition experience.

Rapid advancements in consumer technologies enable people to collect a wide range of personal data. With a proper means for people to ask questions and explore their data, longitudinal data feeds from multiple self-tracking tools pose great opportunities to foster deep self-reflection. However, most self-tracking tools lack support for self-reflection beyond providing simple feedback. Our overarching goal is to support self-trackers in reflecting on their data and gaining rich insights through visual data exploration. As a first step toward the goal, we built a web-based application called Visualized Self, and conducted an in-lab think-aloud study (N = 11) to examine how people reflect on their personal data and what types of insights they gain throughout the reflection. We discuss lessons learned from studying with Visualized Self, and suggest directions for designing visual data exploration tools for fostering self-reflection.

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).

We propose a multi-resolution framework inspired by human visual search for general object detection. Different resolutions are represented using a coarse-to-fine feature hierarchy. During detection, the lower resolution features are initially used to reject the majority of negative windows at relatively low cost, leaving a relatively small number of windows to be processed in higher resolutions. This enables the use of computationally more expensive higher resolution features to achieve high detection accuracy. We applied this framework on Histograms of Oriented Gradient (HOG) features for object detection. Our multi-resolution detector produced better performance for pedestrian detection than state-of-the-art methods (Dalal and Triggs, 2005), and was faster during both training and testing. Testing our method on motorbikes and cars from the VOC database revealed similar improvements in both speed and accuracy, suggesting that our approach is suitable for realtime general object detection applications.

Networked systems still suffer from poor firewall configuration and monitoring. VisualFirewall seeks to aid in the configuration of firewalls and monitoring of networks by providing four simultaneous views that display varying levels of detail and time-scales as well as correctly visualizing firewall reactions to individual packets. The four implemented views, real-time traffic, visual signature, statistics, and IDS alarm, provide the levels of detail and temporality that system administrators need to properly monitor their systems in a passive or an active manner. We have visualized several attacks, and we feel that even individuals unfamiliar with networking concepts can quickly distinguish between benign and malignant traffic patterns with a minimal amount of introduction.

Effectively showing the relationships between objects in a dataset is one of the main tasks in information visualization. Typically there is a well-defined notion of distance between pairs of objects, and traditional approaches such as principal component analysis or multi-dimensional scaling are used to place the objects as points in 2D space, so that similar objects are close to each other. In another typical setting, the dataset is visualized as a network graph, where related nodes are connected by links. More recently, datasets are also visualized as maps, where in addition to nodes and links, there is an explicit representation of groups and clusters. We consider these three Techniques, characterized by a progressive increase of the amount of encoded information: node diagrams, node-link diagrams and node-link-group diagrams. We assess these three types of diagrams with a controlled experiment that covers nine different tasks falling broadly in three categories: node-based tasks, network-based tasks and group-based tasks. Our findings indicate that adding links, or links and group representations, does not negatively impact performance (time and accuracy) of node-based tasks. Similarly, adding group representations does not negatively impact the performance of network-based tasks. Node-link-group diagrams outperform the others on group-based tasks. These conclusions contradict results in other studies, in similar but subtly different settings. Taken together, however, such results can have significant implications for the design of standard and domain snecific visualizations tools.

In this paper, we propose and implement a multi-layered visualization technique in 3D space called Multi-Layered Visualization for Multi-Dimensional Data (MLMD) with its corresponding interaction techniques, for visualizing multi-dimensional data. Layers of point based plots are stacked and connected in a virtual visualization cube for comparison between different dimension settings. Viewed from the side, the plot layers intrinsically form parallel coordinates plots. MLMD integrates point based plots and parallel coordinates compactly so as to present more information at a time to help data investigation. Pertinent user interactions for MLMD method to enable convenient manipulation are designed. By using MLMD and its matching interaction techniques, proper dimension settings and in-depth data perception can be achieved.

This paper describes a visual object detection framework that is capable of processing images extremely rapidly while achieving high detection rates. There are three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features and yields extremely efficient classifiers [6]. The third contribution is a method for combining classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. A set of experiments in the domain of face detection are presented. The system yields face detection performace comparable to the best previous systems [18, 13, 16, 12, 1]. Implemented on a conventional desktop, face detection proceeds at 15 frames per second.

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail.

Statistical topic models can help analysts discover patterns in large text corpora by identifying recurring sets of words and enabling exploration by topical concepts. However, understanding and validating the output of these models can itself be a challenging analysis task. In this paper, we offer two design considerations - interpretation and trust - for designing visualizations based on data-driven models. Interpretation refers to the facility with which an analyst makes inferences about the data through the lens of a model abstraction. Trust refers to the actual and perceived accuracy of an analyst's inferences. These considerations derive from our experiences developing the Stanford Dissertation Browser, a tool for exploring over 9,000 Ph.D. theses by topical similarity, and a subsequent review of existing literature. We contribute a novel similarity measure for text collections based on a notion of "word-borrowing" that arose from an iterative design process. Based on our experiences and a literature review, we distill a set of design recommendations and describe how they promote interpretable and trustworthy visual analysis tools.

When people search for a target in a novel image they often make use of eye movements to bring the relatively high acuity fovea to bear on areas of interest. The strategies that control these eye movements for visual search have been of substantial scientific interest. In the current article we report a new computational model that shows how strategies for visual search are an emergent consequence of perceptual/motor constraints and approximately optimal strategies. The model solves a Partially Observable Markov Decision Process (POMDP) using deep Q-learning to acquire strategies that optimise the tradeoff between speed and accuracy. Results are reported for the Distractor-ratio task.


Question: The argument for a visual discovery aide: a comprehensive resolution to expedite the exploration of visual data.

Introducing VisualInsight: A Comprehensive Visual Discovery Assistant for Streamlined Data Exploration

Data quality challenges, encompassing missing, inaccurate, extreme, and duplicate values, significantly hinder analytical processes and consume substantial time for rectification. Automated techniques can assist in pinpointing anomalies; however, defining what constitutes an error is contingent upon the specific context and necessitates human discernment. Visualization tools, although beneficial, often necessitate manual creation of the requisite views, demanding considerable expertise from analysts. This paper unveils VisualInsight, a visual analysis solution designed to streamline the evaluation of quality issues in structured data. VisualInsight harnesses data mining methodologies to automatically highlight questionable data entries and proposes coordinated summary visualizations to facilitate contextual assessment. The system introduces innovative approaches for integrated statistical and visual analysis, automated visualization recommendations, and scalable visual summaries that enable real-time interaction with voluminous datasets, encompassing millions of data points. We delineate the architectural design of VisualInsight, encompassing modular components for custom data types, anomaly detection algorithms, and summary visualizations, and illustrate its practical application through case studies involving motion picture, natural disaster, and water quality datasets. Through these applications, we demonstrate how VisualInsight can significantly expedite and enhance the process of visual data exploration.

In this paper we propose ValueCharts, a set of visualizations and interactive techniques intended to support decision-makers in inspecting linear models of preferences and evaluation. Linear models are popular decision-making tools for individuals, groups and organizations. In Decision Analysis, they help the decision-maker analyze preferential choices under conflicting objectives. In Economics and the Social Sciences, similar models are devised to rank entities according to an evaluative index of interest. The fundamental goal of building models expressing preferences and evaluations is to help the decision-maker organize all the information relevant to a decision into a structure that can be effectively analyzed. However, as models and their domain of application grow in complexity, model analysis can become a very challenging task. We claim that ValueCharts will make the inspection and application of these models more natural and effective. We support our claim by showing how ValueCharts effectively enable a set of basic tasks that we argue are at the core of analyzing and understanding linear models of preferences and evaluation.

The increasing availability of affordable color raster graphics displays has made it important to develop a better understanding of how color can be used effectively in an interactive environment. Most contemporary graphics displays offer a choice of some 16 million colors; the user's problem is to find the right color.
Folklore has it that the RGB color space arising naturally from color display hardware is user-hostile and that other color models such as the HSV scheme are preferable. Until now there has been virtually no experimental evidence addressing this point.
We describe a color matching experiment in which subjects used one of two tablet-based input techniques, interfaced through one of five color models, to interactively match target colors displayed on a CRT.
The data collected show small but significant differences between models in the ability of subjects to match the five target colors used in this experiment. Subjects using the RGB color model matched quickly but inaccurately compared with those using the other models. The largest speed difference occurred during the early convergence phase of matching. Users of the HSV color model were the slowest in this experiment, both during the convergence phase and in total time to match, but were relatively accurate. There was less variation in performance during the second refinement phase of a match than during the convergence phase.
Two-dimensional use of the tablet resulted in faster but less accurate performance than did strictly one-dimensional usage.
Significant learning occurred for users of the Opponent, YIQ, LAB, and HSV color models, and not for users of the RGB color model.

During data preprocessing, analysts spend a significant part of their time and effort profiling the quality of the data along with cleansing and transforming the data for further analysis. While quality metrics—ranging from general to domain-specific measures—support assessment of the quality of a dataset, there are hardly any approaches to visually support the analyst in customizing and applying such metrics. Yet, visual approaches could facilitate users’ involvement in data quality assessment. We present MetricDoc, an interactive environment for assessing data quality that provides customizable, reusable quality metrics in combination with immediate visual feedback. Moreover, we provide an overview visualization of these quality metrics along with error visualizations that facilitate interactive navigation of the data to determine the causes of quality issues present in the data. In this article, we describe the architecture, design, and evaluation of MetricDoc, which underwent several design cycles, including heuristic evaluation and expert reviews as well as a focus group with data quality, human-computer interaction, and visual analytics experts.

Visualization designers regularly use color to encode quantitative or categorical data. However, visualizations “in the wild” often violate perceptual color design principles and may only be available as bitmap images. In this work, we contribute a method to semi-automatically extract color encodings from a bitmap visualization image. Given an image and a legend location, we classify the legend as describing either a discrete or continuous color encoding, identify the colors used, and extract legend text using OCR methods. We then combine this information to recover the specific color mapping. Users can also correct interpretation errors using an annotation interface. We evaluate our techniques using a corpus of images extracted from scientific papers and demonstrate accurate automatic inference of color mappings across a variety of chart types. In addition, we present two applications of our method: automatic recoloring to improve perceptual effectiveness, and interactive overlays to enable improved reading of static visualizations.

We explore the concept of abstraction as it is used in visualization, with the ultimate goal of understanding and formally defining it. Researchers so far have used the concept of abstraction largely by intuition without a precise meaning. This lack of specificity left questions on the characteristics of abstraction, its variants, its control, or its ultimate potential for visualization and, in particular, illustrative visualization mostly unanswered. In this paper we thus provide a first formalization of the abstraction concept and discuss how this formalization affects the application of abstraction in a variety of visualization scenarios. Based on this discussion, we derive a number of open questions still waiting to be answered, thus formulating a research agenda for the use of abstraction for the visual representation and exploration of data. This paper, therefore, is intended to provide a contribution to the discussion of the theoretical foundations of our field, rather than attempting to provide a completed and final theory.

The volume of available data has been growing exponentially, increasing data problem's complexity and obscurity. In response, visual analytics (VA) has gained attention, yet its solutions haven't scaled well for big data. Computational methods can improve VA's scalability by giving users compact, meaningful information about the input data. However, the significant computation time these methods require hinders real-time interactive visualization of big data. By addressing crucial discrepancies between these methods and VA regarding precision and convergence, researchers have proposed ways to customize them for VA. These approaches, which include low-precision computation and iteration-level interactive visualization, ensure real-time interactive VA for big data.

Publications play a central role in presenting the outcome of scientific research but are typically presented as textual lists, whereas related work in visualization of publication focuses on exploration – not presentation. To bridge this gap, we conducted a design study of an interactive visual representation of publication data in a BibTeX file. This paper reports our domain and problem characterization as well as our visualization design decisions in light of our user-centered design process including interviews, two user studies with a paper prototype and a d3.js prototype, and practical application at our group’s website.

We present the novel high-level visualization taxonomy. Our taxonomy classifies visualization algorithms rather than data. Algorithms are categorized based on the assumptions they make about the data being visualized; we call this set of assumptions the design model. Because our taxonomy is based on design models, it is more flexible than existing taxonomies and considers the user's conceptual model, emphasizing the human aspect of visualization. Design models are classified according to whether they are discrete or continuous and by how much the algorithm designer chooses display attributes such as spatialization, timing, colour, and transparency. This novel approach provides an alternative view of the visualization field that helps explain how traditional divisions (e.g., information and scientific visualization) relates and overlap, and that may inspire research ideas in hybrid visualization areas

Space time cube representation is an information visualization technique where spatiotemporal data points are mapped into a cube. Information visualization researchers have previously argued that space time cube representation is beneficial in revealing complex spatiotemporal patterns in a data set to users. The argument is based on the fact that both time and spatial information are displayed simultaneously to users, an effect difficult to achieve in other representations. However, to our knowledge the actual usefulness of space time cube representation in conveying complex spatiotemporal patterns to users has not been empirically validated. To fill this gap, we report on a between-subjects experiment comparing novice users' error rates and response times when answering a set of questions using either space time cube or a baseline 2D representation. For some simple questions, the error rates were lower when using the baseline representation. For complex questions where the participants needed an overall understanding of the spatiotemporal structure of the data set, the space time cube representation resulted in on average twice as fast response times with no difference in error rates compared to the baseline. These results provide an empirical foundation for the hypothesis that space time cube representation benefits users analyzing complex spatiotemporal patterns.

Introduction A rapid development of eye tracking technology has been observed in recent years. Today’s eye trackers can determine the current focus point of the eye precisely while being relatively unobtrusive in their application. Also, a variety of research and commercial groups has been working on this technology, and there is a growing interest for such devices on the market. Eye tracking has great potential and it can be assumed that it will advance further and might become a widespread technology used at a large number of personal or office computer workplaces. Approaches using simple webcams for eye tracking already exist, for example webcams integrated into laptop computers by default. Thus, they allow for new kinds of applications using eye gaze data. However, not only eye tracking technology is advancing rapidly to an easily usable state. Additionally, during the past 100 years researchers gathered a considerable amount of knowledge on eye movements, why and how they occur, and what they might mean. So, today, we have the technology and knowledge for tracking and analyzing eye movements, making an excellent starting point for sophisticated interactive gaze-based applications. Naive approaches where gaze data is directly employed for interacting with the system, e. g., pressing buttons on the screen with the “blink of an eye” generally have serious problems. Because the eyes are organs used for perceiving the world and not for manipulating the world, it is hard and against human nature to control eye movements deliberately. However, a highly promising approach is just to observe eye movements of the user during his or her daily work in front of the computer, to infer user intentions based on eye movement behavior, and to provide assistance where helpful. Gaze can be seen as a proxy for the user’s attention, and eye movements are known to be usually tightly coupled with cognitive processes in the brain, so that a great deal about those processes can be observed by eye tracking. For example, by interpreting eye movements, reading behavior of the user can be detected, which most likely entails cognitive processes of understanding with regard to the currently read text. In this paper we are focusing particularly on reading behavior since reading is probably the most common activity of knowledge workers sitting in front of a computer screen. We present an algorithm for online reading detection based on eye tracking data and introduce an application for assisted and augmented reading called the eyeBook. The idea behind the eyeBook is to create an interactive and entertaining reading experience. The system observes, which text parts are currently being read by the user on the screen and generates appropriate effects such as playing sounds, presenting