Question: Is it possible to surpass Hadamard multiplexing? A data-driven methodology for designing and evaluating computational imaging systems.

In this work, we introduce a novel methodology that integrates two prominent strategies for image restoration: basis set (dictionary) learning for sparse signal representation and the exploitation of non-local self-similarities in natural images. Dictionary learning has been widely recognized for its efficacy in enhancing image reconstruction and classification by promoting sparse signal descriptions. Concurrently, the non-local means technique, which capitalizes on the inherent repetitiveness within images, has proven successful in restoring image quality. Our proposed framework, simultaneous sparse coding, seamlessly merges these two paradigms. This is accomplished by decomposing clusters of alike signals onto shared subsets of the learned dictionary. We validate the superiority of our method through extensive experimentation in image denoising and demosaicking scenarios, involving both synthetic and real-world noise. The outcomes demonstrate that our approach surpasses current state-of-the-art techniques, facilitating the efficient restoration of raw digital camera images with acceptable computational resources.

**Question**: The role of network topology in community detection

**Documents**: An Information-Theoretic Framework for Resolving Network Community Structure
We present an information-theoretic framework for resolving community structure in complex networks. This framework is based on a statistical mechanics approach to the problem of partitioning networks into communities. We define the community structure of a network in terms of the maximum likelihood of a model that is constructed to reproduce the observed network, given a partition into communities. The model parameters are estimated using maximum likelihood, and the partition that maximizes the likelihood of the model is taken to be the best partition of the network into communities. We show that this framework can be used to resolve community structure in both static and dynamic networks, and that it can be applied to networks with overlapping communities. We also show that this framework can be used to resolve community structure in networks with multiple types of edges, and that it can be applied to networks with directed edges. We demonstrate the effectiveness of our framework by applying it to a variety of real-world networks, including social networks, biological networks, and technological networks.

**Similar Documents**: This paper introduces an innovative information-theoretic framework designed to elucidate community structure within complex networks. Our approach is grounded in statistical mechanics, focusing on network partitioning to identify communities. We define a network's community structure through the maximization of a model's likelihood, which is constructed to mimic the observed network given a specific community partition. The model parameters are optimized via maximum likelihood estimation, and the partition that yields the highest likelihood is deemed the optimal community structure. We illustrate the versatility of our framework, demonstrating its

Technologies now exist for implementing dense surface-normal optical interconnections for silicon CMOS VLSI using hybrid integration techniques. The critical factors in determining the performance of the resulting photonic chip are the yield on the transceiver device arrays, the sensitivity and power dissipation of the receiver and transmitter circuits, and the total optical power budget available. The use of GaAs–AlGaAs multiple-quantum-well p-i-n diodes for on-chip detection and modulation is one effective means of implementing the optoelectronic transceivers. We discuss a potential roadmap for the scaling of this hybrid optoelectronic VLSI technology as CMOS linewidths shrink and the characteristics of the hybrid optoelectronic tranceiver technology improve. An important general conclusion is that, unlike electrical interconnects, such dense optical interconnections directly to an electronic circuit will likely be able to scale in capacity to match the improved performance of future CMOS technology.

Underlying principles of stereoscopic direct-view displays, binocular head-mounted displays, and autostereoscopic direct-view displays are explained and some early work as well as the state of the art in those technologies are reviewed. Stereoscopic displays require eyewear and can be categorized based on the multiplexing scheme as: 1) color multiplexed (old technology but there are some recent developments; low-quality due to color reproduction and crosstalk issues; simple and does not require additional electronics hardware); 2) polarization multiplexed (requires polarized light output and polarization-based passive eyewear; high-resolution and high-quality displays available); and 3) time multiplexed (requires faster display hardware and active glasses synchronized with the display; high-resolution commercial products available). Binocular head-mounted displays can readily provide 3-D, virtual images, immersive experience, and more possibilities for interactive displays. However, the bulk of the optics, matching of the left and right ocular images and obtaining a large field of view make the designs quite challenging. Some of the recent developments using unconventional optical relays allow for thin form factors and open up new possibilities. Autostereoscopic displays are very attractive as they do not require any eyewear. There are many possibilities in this category including: two-view (the simplest implementations are with a parallax barrier or a lenticular screen), multiview, head tracked (requires active optics to redirect the rays to a moving viewer), and super multiview (potentially can solve the accommodation-convergence mismatch problem). Earlier 3-D booms did not last long mainly due to the unavailability of enabling technologies and the content. Current developments in the hardware technologies provide a renewed interest in 3-D displays both from the consumers and the display manufacturers, which is evidenced by the recent commercial products and new research results in this field.

Photoplethysmography relies on characteristic changes in the optical absorption of tissue due to pulsatile (arterial) blood flow in peripheral vasculature. Sensors for observing the photoplethysmographic effect have traditionally required contact with the skin surface. Recent advances in non-contact imaging photoplethysmography have demonstrated that measures of cardiopulmonary system state, such as pulse rate, pulse rate variability, and respiration rate, can be obtained from a participant by imaging their face under relatively motionless conditions. A critical limitation in this method that must be resolved is the inability to recover these measures under conditions of head motion artifact. To investigate the adequacy of channel space dimensionality for the use of blind source separation in this context, nine synchronized, visible spectrum imagers positioned in a semicircular array centered on the imaged participant were used for data acquisition in a controlled lighting environment. Three-lead electrocardiogram and finger-tip reflectance photoplethysmogram were also recorded as ground truth signals. Controlled head motion artifact trial conditions were compared to trials in which the participant remained stationary, with and without the aid of a chinrest. Bootstrapped means of one-minute, non-overlapping trial segments show that, for situations involving little to no head motion, a single imager is sufficient for recovering pulse rate with an average absolute error of less than two beats per minute. However, error in the recovered pulse rate measurement for the single imager can be as high as twenty-two beats per minute when head motion artifact is severe. This increase in measurement error during motion artifact was mitigated by increasing the dimensionality of the imager channel space with multiple imagers in the array prior to applying blind source separation. In contrast to single-imager results, the multi-imager channel space resulted in an absolute error in the recovered pulse rate measurement that is comparable with pulse rate measured via fingertip reflectance photoplethysmography. These results demonstrate that non-contact, imaging photoplethysmography can be accurate in the presence of head motion artifact when a multi-imager array is implemented to increase the dimensionality of the decomposed channel space.

Hyperspectral imaging is an important visual modality with growing interest and range of applications. The latter, however, is hindered by the fact that existing devices are limited in either spatial, spectral, and/or temporal resolution, while yet being both complicated and expensive. We present a low cost and fast method to recover high quality hyperspectral images directly from RGB. Our approach first leverages hyperspectral prior in order to create a sparse dictionary of hyperspectral signatures and their corresponding RGB projections. Describing novel RGB images via the latter then facilitates reconstruction of the hyperspectral image via the former. A novel, larger-than-ever database of hyperspectral images serves as a hyperspectral prior. This database further allows for evaluation of our methodology at an unprecedented scale, and is provided for the benefit of the research community. Our approach is fast, accurate, and provides high resolution hyperspectral cubes despite using RGB-only input.

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

The mathematical properties of high-dimensional spaces seem remarkably suited for describing behaviors produces by brains. Brain-inspired hyperdimensional computing (HDC) explores the emulation of cognition by computing with hypervectors as an alternative to computing with numbers. Hypervectors are high-dimensional, holographic, and (pseudo)random with independent and identically distributed (i.i.d.) components. These features provide an opportunity for energy-efficient computing applied to cyberbiological and cybernetic systems. We describe the use of HDC in a smart prosthetic application, namely hand gesture recognition from a stream of Electromyography (EMG) signals. Our algorithm encodes a stream of analog EMG signals that are simultaneously generated from four channels to a single hypervector. The proposed encoding effectively captures spatial and temporal relations across and within the channels to represent a gesture. This HDC encoder achieves a high level of classification accuracy (97.8%) with only 1/3 the training data required by state-of-the-art SVM on the same task. HDC exhibits fast and accurate learning explicitly allowing online and continuous learning. We further enhance the encoder to adaptively mitigate the effect of gesture-timing uncertainties across different subjects endogenously; further, the encoder inherently maintains the same accuracy when there is up to 30% overlapping between two consecutive gestures in a classification window.

A variety of methods for modeling scattered data are discussed, with an emphasis on two types of data: volumetric and spherical. To demonstrate the performance of the various methods, results from an empirical study using trivariate scattered data are presented. The author's objective is to provide information to aid in selecting a type of method or to form the basis for customizing a method for a particular application.<<ETX>>

Current methodologies for the generation of 3D point cloud from real world scenes rely upon a set of 2D images capturing the scene from several points of view. Novel plenoptic cameras sample the light field crossing the main camera lens creating a light field image. The information available in a plenoptic image must be processed in order to render a view or create the depth map of the scene. This paper analyses a method for the reconstruction of 3D models. The reconstruction of the model is obtained from a single image shot. Exploiting the properties of plenoptic images, a point cloud is generated and compared with a point cloud of the same object but generated with a different plenoptic camera.

Multiple resolution analysis of images is a current trend in computer vision. In most cases, only spatial resolution has been considered. However, image resolution has an additional aspect: gray level, or color, resolution. Color resolution has traditionally been considered in the area of computer graphics. By defining a suitable measure for the comparison of images, changes in resolution can be treated with the same tools as changes in color resolution. A gray tone image, for example, can be compared to a halftone image having only two colors (black and white), but of higher spatial resolution. An important application can be in pyramids, one of the most commonly used multiple (spatial) resolution schemes, where this approach provides a tool to change the color resolution as well. Increasing color resolution while reducing spatial resolution to retain more image details and prevent aliasing is an example of the possibility to find optimal combinations of resolution reduction, spatial and color, to best fit an application.

In this paper we have discussed what appears to be a superior implementation of the Algebraic Reconstruction Technique (ART). The method is based on 1) simultaneous application of the error correction terms as computed by ART for all rays in a given projection; 2) longitudinal weighting of the correction terms back-distributed along the rays; and 3) using bilinear elements for discrete approximation to the ray integrals of a continuous image. Since this implementation generates a good reconstruction in only one iteration, it also appears to have a computational advantage over the more traditional implementation of ART. Potential applications of this implementation include image reconstruction in conjunction with ray tracing for ultrasound and microwave tomography in which the curved nature of the rays leads to a non-uniform ray density across the image.


Question: Utilizing visual attention for action recognition and video description.

This paper presents a method for learning video representations through the use of multilayer Long Short Term Memory (LSTM) networks, focusing on unsupervised learning techniques. The proposed model employs an encoder LSTM to convert input video sequences into a compact, fixed-length representation. This representation can then be decoded by either a single or multiple decoder LSTMs to perform various tasks, such as input sequence reconstruction or future sequence prediction. The study explores two types of input sequences: image patches consisting of pixel values and high-level video frame representations, termed "percepts," obtained from a pre-trained convolutional neural network. The paper investigates various design options, including the conditioning of decoder LSTMs on the generated output. The model's outputs are qualitatively analyzed to assess its ability to extrapolate learned video representations both forward and backward in time. Efforts are made to visualize and interpret the learned features. The robustness of the model is tested by applying it to longer time scales and to data outside its training domain. Additionally, the effectiveness of the learned representations is evaluated by fine-tuning them for a supervised learning task: human action recognition on the UCF-101 and HMDB-51 datasets. The results show that these representations significantly enhance classification accuracy, particularly when the training data is limited. Notably, models pre-trained on unrelated datasets, such as 300 hours of YouTube videos, still contribute to improved action recognition performance.

The possibility of automatic understanding of customers’ shopping behavior and acting according to their needs is relevant in the marketing domain, attracting a lot of attention lately. In this work, we focus on the task of automatic assessment of customers’ shopping behavior, by proposing a multilevel framework. The framework is supported at low-level by different types of cameras, which are synchronized, facilitating efficient processing of information. A fish-eye camera is used for tracking, while a high-definition one serves for the action recognition task. The experiments are performed on both laboratory and real-life recordings in a supermarket. From the video recordings, we extract features related to the spatio-temporal behavior of trajectories, the dynamics and the time spent in each region of interest (ROI) in the shop and regarding the customer-products interaction patterns. Next ∗Corresponding Author. Phone: +31(0)15 2781537. Fax: +31(0)15 2787141. Email address: m.c.popa@tudelft.nl (M.C. Popa) Preprint submitted to Pattern Recognition Letters December 24, 2011 we analyze the shopping sequences using a Hidden Markov Model (HMM). We conclude that it is possible to accurately classify trajectories (93%), discriminate between different shopping related actions (91.6%), and recognize shopping behavioral types by means of the proposed reasoning model in 95% of the cases.

This paper discusses the problem of recognizing interaction-level human activities from a first-person viewpoint. The goal is to enable an observer (e.g., a robot or a wearable camera) to understand 'what activity others are performing to it' from continuous video inputs. These include friendly interactions such as 'a person hugging the observer' as well as hostile interactions like 'punching the observer' or 'throwing objects to the observer', whose videos involve a large amount of camera ego-motion caused by physical interactions. The paper investigates multi-channel kernels to integrate global and local motion information, and presents a new activity learning/recognition methodology that explicitly considers temporal structures displayed in first-person activity videos. In our experiments, we not only show classification results with segmented videos, but also confirm that our new approach is able to detect activities from continuous videos reliably.

Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.

[1] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 2007. [2] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. 2007. [3] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. CVPR, 2006. [4] L.-J. Li and L. Fei-Fei. What, where and who? classifying events by scene and object recognition. In Proc. ICCV, 2007. [5] G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In Proc. CVPR, 2012.

We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.

In this work, we evaluate the performance of the popular dense trajectories approach on first-person action recognition datasets. A person moving around with a wearable camera will actively interact with humans and objects and also passively observe others interacting. Hence, in order to represent real-world scenarios, the dataset must contain actions from first-person perspective as well as third-person perspective. For this purpose, we introduce a new dataset which contains actions from both the perspectives captured using a head-mounted camera. We employ a motion pyramidal structure for grouping the dense trajectory features. The relative strengths of motion along the trajectories are used to compute different bag-of-words descriptors and concatenated to form a single descriptor for the action. The motion pyramidal approach performs better than the baseline improved trajectory descriptors. The method achieves 96.7% on the JPL interaction dataset and 61.8% on our NUS interaction dataset. The same is used to detect actions in long video sequences and achieves average precision of 0.79 on JPL interaction dataset.

Recent years have seen tremendous progress in still-image segmentation; however the naı̈ve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of “clockwork” convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.

We propose an algorithm which combines the discriminative information from depth images as well as from 3D joint positions to achieve high action recognition accuracy. To avoid the suppression of subtle discriminative information and also to handle local occlusions, we compute a vector of many independent local features. Each feature encodes spatiotemporal variations of depth and depth gradients at a specific space-time location in the action volume. Moreover, we encode the dominant skeleton movements by computing a local 3D joint position difference histogram. For each joint, we compute a 3D space-time motion volume which we use as an importance indicator and incorporate in the feature vector for improved action discrimination. To retain only the discriminant features, we train a random decision forest (RDF). The proposed algorithm is evaluated on three standard datasets and compared with nine state-of-the-art algorithms. Experimental results show that, on the average, the proposed algorithm outperform all other algorithms in accuracy and have a processing speed of over 112 frames/second.

Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-tointerpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers etc., can understand what triggered a particular behavior. Here we explore the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network’s output (steering control). Our approach is two-stage. In the first stage, we use a visual attention model to train a convolution network endto- end from images to steering angle. The attention model highlights image regions that potentially influence the network’s output. Some of these are true influences, but some are spurious. We then apply a causal filtering step to determine which input regions actually influence the output. This produces more succinct visual explanations and more accurately exposes the network’s behavior. We demonstrate the effectiveness of our model on three datasets totaling 16 hours of driving. We first show that training with attention does not degrade the performance of the end-to-end network. Then we show that the network causally cues on a variety of features that are used by humans while driving.

Person re-identification is an important task that requires learning discriminative visual features for distinguishing different person identities. Diverse auxiliary information has been utilized to improve the visual feature learning. In this paper, we propose to exploit natural language description as additional training supervisions for effective visual features. Compared with other auxiliary information, language can describe a specific person from more compact and semantic visual aspects, thus is complementary to the pixel-level image data. Our method not only learns better global visual feature with the supervision of the overall description but also enforces semantic consistencies between local visual and linguistic features, which is achieved by building global and local image-language associations. The global image-language association is established according to the identity labels, while the local association is based upon the implicit correspondences between image regions and noun phrases. Extensive experiments demonstrate the effectiveness of employing language as training supervisions with the two association schemes. Our method achieves state-of-the-art performance without utilizing any auxiliary information during testing and shows better performance than other joint embedding methods for the image-language association.


Question: Using visual attention for action recognition and video description.

Introducing the Spatial Transformer Module for Enhanced Visual Understanding
Convolutional Neural Networks (CNNs) have proven to be remarkably powerful for image processing tasks, yet they face limitations in efficiently handling spatial transformations of input data. To address this, we propose the Spatial Transformer Module, a novel and learnable component designed to augment CNNs with the capability for spatial manipulation. This module, seamlessly integrated into existing CNN architectures, empowers networks to dynamically adjust feature maps based on the content of the images, without requiring additional training data or modifications to the optimization process. By incorporating spatial transformers, models can learn to be invariant to various spatial transformations such as translation, scaling, rotation, and even more complex deformations, leading to state-of-the-art performance on multiple benchmarks across a wide range of transformation types. This innovation significantly enhances the robustness and adaptability of CNNs to diverse visual inputs.

We describe an approach to object and scene retrieval which searches for and localizes all the occurrences of a user outlined object in a video. The object is represented by a set of viewpoint invariant region descriptors so that recognition can proceed successfully despite changes in viewpoint, illumination and partial occlusion. The temporal continuity of the video within a shot is used to track the regions in order to reject unstable regions and reduce the effects of noise in the descriptors. The analogy with text retrieval is in the implementation where matches on descriptors are pre-computed (using vector quantization), and inverted file systems and document rankings are used. The result is that retrieval is immediate, returning a ranked list of key frames/shots in the manner of Google. The method is illustrated for matching on two full length feature films.

Learning semantic attributes for person re-identification and description-based person search has gained increasing interest due to attributes' great potential as a pose and view-invariant representation. However, existing attribute-centric approaches have thus far underperformed state-of-the-art conventional approaches. This is due to their nonscalable need for extensive domain (camera) specific annotation. In this paper we present a new semantic attribute learning approach for person re-identification and search. Our model is trained on existing fashion photography datasets - either weakly or strongly labelled. It can then be transferred and adapted to provide a powerful semantic description of surveillance person detections, without requiring any surveillance domain supervision. The resulting representation is useful for both unsupervised and supervised person re-identification, achieving state-of-the-art and near state-of-the-art performance respectively. Furthermore, as a semantic representation it allows description-based person search to be integrated within the same framework.

Deep models are the defacto standard in visual decision problems due to their impressive performance on a wide array of visual tasks. However, they are frequently seen as opaque and are unable to explain their decisions. In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which supports their decisions. We propose a method which incorporates a novel explanation attention mechanism; our model is trained using textual rationals, and infers latent attention to visually ground explanations. We collect two novel datasets in domains where it is interesting and challenging to explain decisions. First, we extend the visual question answering task to not only provide an answer but also visual and natural language explanations for the answer. Second, we focus on explaining human activities in a contemporary activity recognition dataset. We extensively evaluate our model, both on the justification and pointing tasks, by comparing it to prior models and ablations using both automatic and human evaluations.

This report documents the design and implementation of a binocular, foveated active vision system as part of the Cog project at the MIT Artificial Intelligence Laboratory. The active vision system features a 3 degree of freedom mechanical platform that supports four color cameras, a motion control system, and a parallel network of digital signal processors for image processing. To demonstrate the capabilities of the system, we present results from four sample visual-motor tasks. The author receives support from a National Defense Science and Engineering Graduate Fellowship. Support for this project is provided in part by an ONR/ARPA Vision MURI Grant (No. N00014-95-1-0600).

How do humans use target-predictive contextual information to facilitate visual search? How are consistently paired scenic objects and positions learned and used to more efficiently guide search in familiar scenes? For example, humans can learn that a certain combination of objects may define a context for a kitchen and trigger a more efficient search for a typical object, such as a sink, in that context. The ARTSCENE Search model is developed to illustrate the neural mechanisms of such memory-based context learning and guidance and to explain challenging behavioral data on positive-negative, spatial-object, and local-distant cueing effects during visual search, as well as related neuroanatomical, neurophysiological, and neuroimaging data. The model proposes how global scene layout at a first glance rapidly forms a hypothesis about the target location. This hypothesis is then incrementally refined as a scene is scanned with saccadic eye movements. The model simulates the interactive dynamics of object and spatial contextual cueing and attention in the cortical What and Where streams starting from early visual areas through medial temporal lobe to prefrontal cortex. After learning, model dorsolateral prefrontal cortex (area 46) primes possible target locations in posterior parietal cortex based on goal-modulated percepts of spatial scene gist that are represented in parahippocampal cortex. Model ventral prefrontal cortex (area 47/12) primes possible target identities in inferior temporal cortex based on the history of viewed objects represented in perirhinal cortex.

Choosing clothes with complex patterns and colors is a challenging task for visually impaired people. Automatic clothing pattern recognition is also a challenging research problem due to rotation, scaling, illumination, and especially large intraclass pattern variations. We have developed a camera-based prototype system that recognizes clothing patterns in four categories (plaid, striped, patternless, and irregular) and identifies 11 clothing colors. The system integrates a camera, a microphone, a computer, and a Bluetooth earpiece for audio description of clothing patterns and colors. A camera mounted upon a pair of sunglasses is used to capture clothing images. The clothing patterns and colors are described to blind users verbally. This system can be controlled by speech input through microphone. To recognize clothing patterns, we propose a novel Radon Signature descriptor and a schema to extract statistical properties from wavelet subbands to capture global features of clothing patterns. They are combined with local features to recognize complex clothing patterns. To evaluate the effectiveness of the proposed approach, we used the CCNY Clothing Pattern dataset. Our approach achieves 92.55% recognition accuracy which significantly outperforms the state-of-the-art texture analysis methods on clothing pattern recognition. The prototype was also used by ten visually impaired participants. Most thought such a system would support more independence in their daily life but they also made suggestions for improvements.

The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision. This critical faculty allows us to understand highly complex visual scenes. Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene. Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory. We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory. We subsequently introduce a fully differentiable Short Term Attentive Working Memory model (STAWM) which uses transformational attention to learn a memory over each image it sees. The state of our HebbRosenblatt memory is embedded in STAWM as the weights space of a layer. By projecting different queries through this layer we can obtain goal-oriented latent representations for tasks including classification and visual reconstruction. Our model obtains highly competitive classification performance on MNIST and CIFAR-10. As demonstrated through the CelebA dataset, to perform reconstruction the model learns to make a sequence of updates to a canvas which constitute a parts-based representation. Classification with the self supervised representation obtained from MNIST is shown to be in line with the state of the art models (none of which use a visual attention mechanism). Finally, we show that STAWM can be trained under the dual constraints of classification and reconstruction to provide an interpretable visual sketchpad which helps open the ‘black-box’ of deep learning.

With the rapid proliferation of smartphones and tablet computers, search has moved beyond text to other modalities like images and voice. For many applications like Fashion, visual search offers a compelling interface that can capture stylistic visual elements beyond color and pattern that cannot be as easily described using text. However, extracting and matching such attributes remains an extremely challenging task due to high variability and deformability of clothing items. In this paper, we propose a fine-grained learning model and multimedia retrieval framework to address this problem. First, an attribute vocabulary is constructed using human annotations obtained on a novel fine-grained clothing dataset. This vocabulary is then used to train a fine-grained visual recognition system for clothing styles. We report benchmark recognition and retrieval results on Women's Fashion Coat Dataset and illustrate potential mobile applications for attribute-based multimedia retrieval of clothing items and image annotation.

In this paper, we propose a new approach based on distribution descriptors for action recognition in depth videos. Our local features are computed from binary patterns which incorporate the shape and motion cues for effective action recognition. Given pixel-level features, our approach estimates video local statistics in a hierarchical manner, where the distribution of pixel-level features and that of frame-level descriptors are modeled using single Gaussians. In this way, our approach constructs video descriptors directly from low-level features without resorting to codebook learning required by Bag-of-features (BoF) based approaches. In order to capture the spatial geometry and temporal order of a video, we use a spatio-temporal pyramid representation for each video. Our approach is validated on six benchmark datasets, i.e. MSRAction3D, MSRGesture3D, DHA, SKIG, UTD-MHAD and CAD-120. The experimental results show that our approach gives good performance on all the datasets. In particular, it achieves state-of-the-art accuracies on DHA, SKIG and UTD-MHAD datasets.

Traditional video understanding tasks include human action recognition and actor/object semantic segmentation. However, the combined task of providing semantic segmentation for different actor classes simultaneously with their action class remains a challenging but necessary task for many applications. In this work, we propose a new end-to-end architecture for tackling this task in videos. Our model effectively leverages multiple input modalities, contextual information, and multitask learning in the video to directly output semantic segmentations in a single unified framework. We train and benchmark our model on the Actor-Action Dataset (A2D) for joint actor-action semantic segmentation, and demonstrate state-of-the-art performance for both segmentation and detection. We also perform experiments verifying our approach improves performance for zero-shot recognition, indicating generalizability of our jointly learned feature space.


Question: Using visual attention for action recognition and video description.

This paper presents an innovative deep convolutional neural network architecture, coined Inception, which has set new benchmarks for both classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The core feature of this architecture is its efficient utilization of computational resources within the network. Through meticulous design, we managed to expand both the depth and breadth of the network without increasing the computational cost. Our design philosophy was guided by the Hebbian principle and the concept of multi-scale processing, aimed at maximizing quality. A specific version of this architecture, GoogLeNet, which comprises 22 layers, was submitted for ILSVRC14. Its performance was rigorously evaluated in terms of classification and detection capabilities, showcasing its superior quality in these domains.

Fine-grained object classification attracts increasing attention in multimedia applications. However, it is a quite challenging problem due to the subtle interclass difference and large intraclass variation. Recently, visual attention models have been applied to automatically localize the discriminative regions of an image for better capturing critical difference, which have demonstrated promising performance. Unfortunately, without consideration of the diversity in attention process, most of existing attention models perform poorly in classifying fine-grained objects. In this paper, we propose a diversified visual attention network (DVAN) to address the problem of fine-grained object classification, which substantially relieves the dependency on strongly supervised information for learning to localize discriminative regions com-pared with attention-less models. More importantly, DVAN explicitly pursues the diversity of attention and is able to gather discriminative information to the maximal extent. Multiple attention canvases are generated to extract convolutional features for attention. An LSTM recurrent unit is employed to learn the attentiveness and discrimination of attention canvases. The proposed DVAN has the ability to attend the object from coarse to fine granularity, and a dynamic internal representation for classification is built up by incrementally combining the information from different locations and scales of the image. Extensive experiments conducted on CUB-2011, Stanford Dogs, and Stanford Cars datasets have demonstrated that the pro-posed DVAN achieves competitive performance compared to the state-of-the-art approaches, without using any prior knowledge, user interaction, or external resource in training and testing.

In this paper, we propose a machine learning-based NLP system for automatically creating animated storyboards using the action descriptions of movie scripts. We focus particularly on the importance of verb semantics when generating graphics commands, and find that semantic role labelling boosts performance and is relatively robust to the effects of

We present a video-based gesture dataset and a methodology for annotating video-based gesture datasets. Our dataset consists of user-defined gestures generated by 18 participants from a previous investigation of gesture memorability. We design and use a crowd-sourced classification task to annotate the videos. The results are made available through a web-based visualization that allows researchers and designers to explore the dataset. Finally, we perform an additional descriptive analysis and quantitative modeling exercise that provide additional insights into the results of the original study. To facilitate the use of the presented methodology by other researchers we share the data, the source of the human intelligence tasks for crowdsourcing, a new taxonomy that integrates previous work, and the source code of the visualization tool.

Introduction Vision, due to its significance in surviving and socializing, is one of the most important and  extensively studied sensory functions in the human brain. In order to fully understand visual  information processing, or more specifically, visual recognition, David Marr proposed the  Tri-level Hypothesis [29], in that three levels of the system should be studied: the computational  goal of the system, the internal representation or the algorithm the system uses to achieve the  goal, and the neural substrates that implement the system. It is well-known that visual  recognition in the human brain is implemented by the ventral visual pathway [32], which  receives visual information from the retina and goes through a layered structure including V1  (also known as the primary visual cortex), V2, V4, before reaching the inferior temporal cortex  (IT). The topographic mapping between the retina and the human visual cortex follows a  log-polar transformation, in which the Cartesian coordinates of the retina are transformed to  polar coordinates (polar angle and eccentricity) in the human visual cortex. From V1 to V4, each  of the visual areas contains a complete polar angle and eccentricity map, and have a increasing  receptive field (RF) size and respond to increasing complex visual features [12,22]. In higher  level visual areas such as the ventral temporal cortex (VTC), studies using functional magnetic  resonance imaging (fMRI) and diffusion tensor imaging (DTI) have found there exists visual  category-selective regions, such as the Fusiform Face Area (FFA, [26]) for face processing, the  Parahippocampal Place Area (PPA, [13]) for scene perception, and the Lateral Occipital  Complex (LOC, [19]) for general object recognition. Grill-Spector and Malach [20] proposed that  layer-based hierarchical processing, together with the functional specialization, are the two  organizing principles of the human visual cortex. More recent studies have shown that the  central-biased face recognition pathway (FFA) and the peripheral-biased scene recognition  pathway (PPA) are both functionally and anatomically segregated by mid-fusiform sulcus (MFS),  which enables parallel fast processing of visual recognition tasks in the ventral pathway [18, 43,  21, 44].

Introduction A rapid development of eye tracking technology has been observed in recent years. Today’s eye trackers can determine the current focus point of the eye precisely while being relatively unobtrusive in their application. Also, a variety of research and commercial groups has been working on this technology, and there is a growing interest for such devices on the market. Eye tracking has great potential and it can be assumed that it will advance further and might become a widespread technology used at a large number of personal or office computer workplaces. Approaches using simple webcams for eye tracking already exist, for example webcams integrated into laptop computers by default. Thus, they allow for new kinds of applications using eye gaze data. However, not only eye tracking technology is advancing rapidly to an easily usable state. Additionally, during the past 100 years researchers gathered a considerable amount of knowledge on eye movements, why and how they occur, and what they might mean. So, today, we have the technology and knowledge for tracking and analyzing eye movements, making an excellent starting point for sophisticated interactive gaze-based applications. Naive approaches where gaze data is directly employed for interacting with the system, e. g., pressing buttons on the screen with the “blink of an eye” generally have serious problems. Because the eyes are organs used for perceiving the world and not for manipulating the world, it is hard and against human nature to control eye movements deliberately. However, a highly promising approach is just to observe eye movements of the user during his or her daily work in front of the computer, to infer user intentions based on eye movement behavior, and to provide assistance where helpful. Gaze can be seen as a proxy for the user’s attention, and eye movements are known to be usually tightly coupled with cognitive processes in the brain, so that a great deal about those processes can be observed by eye tracking. For example, by interpreting eye movements, reading behavior of the user can be detected, which most likely entails cognitive processes of understanding with regard to the currently read text. In this paper we are focusing particularly on reading behavior since reading is probably the most common activity of knowledge workers sitting in front of a computer screen. We present an algorithm for online reading detection based on eye tracking data and introduce an application for assisted and augmented reading called the eyeBook. The idea behind the eyeBook is to create an interactive and entertaining reading experience. The system observes, which text parts are currently being read by the user on the screen and generates appropriate effects such as playing sounds, presenting

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

It is very attractive to formulate vision in terms of pattern theory [26], where patterns are defined hierarchically by compositions of elementary building blocks. But applying pattern theory to real world images is currently less successful than discriminative methods such as deep networks. Deep networks, however, are black-boxes which are hard to interpret and can easily be fooled by adding occluding objects. It is natural to wonder whether by better understanding deep networks we can extract building blocks which can be used to develop pattern theoretic models. This motivates us to study the internal representations of a deep network using vehicle images from the PASCAL3D+ dataset. We use clustering algorithms to study the population activities of the features and extract a set of visual concepts which we show are visually tight and correspond to semantic parts of vehicles. To analyze this we annotate these vehicles by their semantic parts to create a new dataset, VehicleSemanticParts, and evaluate visual concepts as unsupervised part detectors. We show that visual concepts perform fairly well but are outperformed by supervised discriminative methods such as Support Vector Machines (SVM). We next give a more detailed analysis of visual concepts and how they relate to semantic parts. Following this, we use the visual concepts as building blocks for a simple pattern theoretical model, which we call compositional voting. In this model several visual concepts combine to detect semantic parts. We show that this approach is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection. Finally, we return to studying occlusion by creating an annotated dataset with occlusion, called VehicleOcclusion, and show that compositional voting outperforms even deep networks when the amount of occlusion becomes large.

This paper presents a Semantic Attribute assisted video SUMmarization framework (SASUM). Compared with traditional methods, SASUM has several innovative features. Firstly, we use a natural language processing tool to discover a set of keywords from an image and text corpora to form the semantic attributes of visual contents. Secondly, we train a deep convolution neural network to extract visual features as well as predict the semantic attributes of video segments which enables us to represent video contents with visual and semantic features simultaneously. Thirdly, we construct a temporally constrained video segment affinity matrix and use a partially near duplicate image discovery technique to cluster visually and semantically consistent video frames together. These frame clusters can then be condensed to form an informative and compact summary of the video. We will present experimental results to show the effectiveness of the semantic attributes in assisting the visual features in video summarization and our new technique achieves state-of-the-art performance.

We introduce a set of nine measures to characterize viewers' visual attention patterns for multi-screen TV. We apply our measures during an experiment involving nine screen layouts with two, three, and four TV screens, for which we report new findings on visual attention. For example, we found that viewers need an average discovery time up to 4.5 seconds to visually fixate four screens, and their perceptions of how long they watched each screen are substantially accurate, i.e., we report Pearson correlations up to .892 with measured eye tracking data. We hope our set of new measures (and the companion toolkit to compute them automatically) will benefit the community as a first step toward understanding visual attention for emerging multi-screen TV applications.

Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.


Question: Utilizing visual attention for action recognition and video description.

Leveraging Temporal Dynamics for Enhanced Video Description

Inspired by recent advancements in recurrent neural networks (RNNs) for image captioning, we extend this methodology to the more complex task of video description. Unlike static images, videos necessitate the modeling of their inherent temporal dynamics and effectively integrating this information into a coherent narrative. Our proposed framework adeptly captures both the local and global temporal structure of videos for generating accurate descriptions. Initially, we employ a spatial-temporal 3-D convolutional neural network (3-D CNN) to represent the video's short-term temporal dynamics. This 3-D CNN is trained on video action recognition tasks to create a representation that is finely tuned to human motion and actions. Subsequently, we introduce a temporal attention mechanism that extends beyond local temporal modeling by automatically selecting the most salient temporal segments. This mechanism is guided by the text-generating RNN, enabling the model to focus on relevant video segments during the description generation process. Our approach outperforms existing state-of-the-art methods in terms of BLEU and METEOR metrics on the Youtube2Text dataset. Additionally, we present promising results on a novel, larger, and more challenging dataset that pairs videos with natural language descriptions, further validating the efficacy of our method.

Vast quantities of videos are now being captured at astonishing rates, but the majority of these are not labelled. To cope with such data, we consider the task of content-based activity recognition in videos without any manually labelled examples, also known as zero-shot video recognition. To achieve this, videos are represented in terms of detected visual concepts, which are then scored as relevant or irrelevant according to their similarity with a given textual query. In this paper, we propose a more robust approach for scoring concepts in order to alleviate many of the brittleness and low precision problems of previous work. Not only do we jointly consider semantic relatedness, visual reliability, and discriminative power. To handle noise and non-linearities in the ranking scores of the selected concepts, we propose a novel pairwise order matrix approach for score aggregation. Extensive experiments on the large-scale TRECVID Multimedia Event Detection data show the superiority of our approach.

A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system.

Visual speech information plays an important role in automatic speech recognition (ASR) especially when audio is corrupted or even inaccessible. Despite the success of audio-based ASR, the problem of visual speech decoding remains widely open. This paper provides a detailed review of recent advances in this research area. In comparison with the previous survey [97] which covers the whole ASR system that uses visual speech information, we focus on the important questions asked by researchers and summarize the recent studies that attempt to answer them. In particular, there are three questions related to the extraction of visual features, concerning speaker dependency, pose variation and temporal information, respectively. Another question is about audio-visual speech fusion, considering the dynamic changes of modality reliabilities encountered in practice. In addition, the state-of-the-art on facial landmark localization is briefly introduced in this paper. Those advanced techniques can be used to improve the region-of-interest detection, but have been largely ignored when building a visual-based ASR system. We also provide details of audio-visual speech databases. Finally, we discuss the remaining challenges and offer our insights into the future research on visual speech decoding.

Real-world actions occur often in crowded, dynamic environments. This poses a difficult challenge for current approaches to video event detection because it is difficult to segment the actor from the background due to distracting motion from other objects in the scene. We propose a technique for event recognition in crowded videos that reliably identifies actions in the presence of partial occlusion and background clutter. Our approach is based on three key ideas: (1) we efficiently match the volumetric representation of an event against oversegmented spatio-temporal video volumes; (2) we augment our shape-based features using flow; (3) rather than treating an event template as an atomic entity, we separately match by parts (both in space and time), enabling robustness against occlusions and actor variability. Our experiments on human actions, such as picking up a dropped object or waving in a crowd show reliable detection with few false positives.

Microsoft Kinect's output is a multi-modal signal which gives RGB videos, depth sequences and skeleton information simultaneously. Various action recognition techniques focused on different single modalities of the signals and built their classifiers over the features extracted from one of these channels. For better recognition performance, it's desirable to fuse these multi-modal information into an integrated set of discriminative features. Most of current fusion methods merged heterogeneous features in a holistic manner and ignored the complementary properties of these modalities in finer levels. In this paper, we proposed a new hierarchical bag-of-words feature fusion technique based on multi-view structured spar-sity learning to fuse atomic features from RGB and skeletons for the task of action recognition.

Learning semantic attributes for person re-identification and description-based person search has gained increasing interest due to attributes' great potential as a pose and view-invariant representation. However, existing attribute-centric approaches have thus far underperformed state-of-the-art conventional approaches. This is due to their nonscalable need for extensive domain (camera) specific annotation. In this paper we present a new semantic attribute learning approach for person re-identification and search. Our model is trained on existing fashion photography datasets - either weakly or strongly labelled. It can then be transferred and adapted to provide a powerful semantic description of surveillance person detections, without requiring any surveillance domain supervision. The resulting representation is useful for both unsupervised and supervised person re-identification, achieving state-of-the-art and near state-of-the-art performance respectively. Furthermore, as a semantic representation it allows description-based person search to be integrated within the same framework.

The two basic phenomena that define the problem of visual attention can be illustrated in a simple example. Consider the arrays shown in each panel of Figure 1. In a typical experiment, before the arrays were presented, subjects would be asked to report letters appearing in one color (targets, here black letters), and to disregard letters in the other color (nontargets, here white letters). The array would then be briefly flashed, and the subjects, without any opportunity for eye movements, would give their report. The display mimics our. usual cluttered visual environment: It contains one or more objects that are relevant to current behavior, along with others that are irrelevant. The first basic phenomenon is limited capacity for processing information. At any given time, only a small amount of the information available on the retina can be processed and used in the control of behavior. Subjectively, giving attention to any one target leaves less available for others. In Figure 1, the probability of reporting the target letter N is much lower with two accompa nying targets (Figure la) than with none (Figure Ib). The second basic phenomenon is selectivity-the ability to filter out un wanted information. Subjectively, one is aware of attended stimuli and largely unaware of unattended ones. Correspondingly, accuracy in identifying an attended stimulus may be independent of the number of nontargets in a display (Figure la vs Ie) (see Bundesen 1990, Duncan 1980).

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

In this paper we introduce a template-based method for recognizing human actions called action MACH. Our approach is based on a maximum average correlation height (MACH) filter. A common limitation of template-based methods is their inability to generate a single template using a collection of examples. MACH is capable of capturing intra-class variability by synthesizing a single Action MACH filter for a given action class. We generalize the traditional MACH filter to video (3D spatiotemporal volume), and vector valued data. By analyzing the response of the filter in the frequency domain, we avoid the high computational cost commonly incurred in template-based approaches. Vector valued data is analyzed using the Clifford Fourier transform, a generalization of the Fourier transform intended for both scalar and vector-valued data. Finally, we perform an extensive set of experiments and compare our method with some of the most recent approaches in the field by using publicly available datasets, and two new annotated human action datasets which include actions performed in classic feature films and sports broadcast television.

How do humans use target-predictive contextual information to facilitate visual search? How are consistently paired scenic objects and positions learned and used to more efficiently guide search in familiar scenes? For example, humans can learn that a certain combination of objects may define a context for a kitchen and trigger a more efficient search for a typical object, such as a sink, in that context. The ARTSCENE Search model is developed to illustrate the neural mechanisms of such memory-based context learning and guidance and to explain challenging behavioral data on positive-negative, spatial-object, and local-distant cueing effects during visual search, as well as related neuroanatomical, neurophysiological, and neuroimaging data. The model proposes how global scene layout at a first glance rapidly forms a hypothesis about the target location. This hypothesis is then incrementally refined as a scene is scanned with saccadic eye movements. The model simulates the interactive dynamics of object and spatial contextual cueing and attention in the cortical What and Where streams starting from early visual areas through medial temporal lobe to prefrontal cortex. After learning, model dorsolateral prefrontal cortex (area 46) primes possible target locations in posterior parietal cortex based on goal-modulated percepts of spatial scene gist that are represented in parahippocampal cortex. Model ventral prefrontal cortex (area 47/12) primes possible target identities in inferior temporal cortex based on the history of viewed objects represented in perirhinal cortex.


Question: Utilizing visual attention for action recognition and video description.

This paper delves into the methodologies for transforming a recurrent neural network (RNN) into a deep RNN, exploring various strategies to achieve depth within the RNN framework. Unlike the straightforward concept of depth in feedforward neural networks, the application of depth in RNNs requires a more nuanced approach. By dissecting the RNN architecture, we identify three primary areas that can be deepened: (1) the input-to-hidden function, (2) the hidden-to-hidden transition, and (3) the hidden-to-output function. Drawing on these insights, we introduce two innovative deep RNN architectures that diverge from previous attempts to deepen RNNs through the stacking of multiple recurrent layers (Schmidhuber, 1992; El Hihi and Bengio, 1996). We also offer a fresh perspective on these deep RNNs through a novel framework centered on neural operators. The efficacy of our proposed deep RNNs is substantiated through empirical evaluation on tasks involving polyphonic music prediction and language modeling, with the results affirming the advantages of depth and demonstrating superior performance over traditional shallow RNNs.

A unified theory of visual recognition and attentional selection is developed by integrating the biased-choice model for single-stimulus recognition (Luce, 1963; Shepard, 1957) with a choice model for selection from multielement displays (Bundesen, Pedersen, & Larsen, 1984) in a race model framework. Mathematically, the theory is tractable, and it specifies the computations necessary for selection. The theory is applied to extant findings from a broad range of experimental paradigms. The findings include effects of object integrality in selective report, number and spatial position of targets in divided-attention paradigms, selection criterion and number of distracters in focused-attention paradigms, delay of selection cue in partial report, and consistent practice in search. On the whole, the quantitative fits are encouraging.

A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system.

Nowadays, almost any web site that provides means for sharing user-generated multimedia content, like Flickr, Facebook, YouTube and Vimeo, has tagging functionalities to let users annotate the material that they want to share. The tags are then used to retrieve the uploaded content, and to ease browsing and exploration of these collections, e.g. using tag clouds. However, while tagging a single image is straightforward, and sites like Flickr and Facebook allow also to tag easily portions of the uploaded photos, tagging a video sequence is more cumbersome, so that users just tend to tag the overall content of a video. Moreover, the tagging process is completely manual, and often users tend to spend as few time as possible to annotate the material, resulting in a sparse annotation of the visual content. A semi-automatic process, that helps the users to tag a video sequence would improve the quality of annotations and thus the overall user experience. While research on image tagging has received a considerable attention in the latest years, there are still very few works that address the problem of automatically assigning tags to videos, locating them temporally within the video sequence. In this paper we present a system for video tag suggestion and temporal localization based on collective knowledge and visual similarity of frames. The algorithm suggests new tags that can be associated to a given keyframe exploiting the tags associated to videos and images uploaded to social sites like YouTube and Flickr and visual features.

This paper discusses the problem of recognizing interaction-level human activities from a first-person viewpoint. The goal is to enable an observer (e.g., a robot or a wearable camera) to understand 'what activity others are performing to it' from continuous video inputs. These include friendly interactions such as 'a person hugging the observer' as well as hostile interactions like 'punching the observer' or 'throwing objects to the observer', whose videos involve a large amount of camera ego-motion caused by physical interactions. The paper investigates multi-channel kernels to integrate global and local motion information, and presents a new activity learning/recognition methodology that explicitly considers temporal structures displayed in first-person activity videos. In our experiments, we not only show classification results with segmented videos, but also confirm that our new approach is able to detect activities from continuous videos reliably.

What are the computational tasks that an executive controller for visual attention must solve? This question is posed in the context of the Selective Tuning model of attention. The range of required computations go beyond top-down bias signals or region-of-interest determinations, and must deal with overt and covert fixations, process timing and synchronization, information routing, memory, matching control to task, spatial localization, priming, and coordination of bottom-up with top-down information. During task execution, results must be monitored to ensure the expected results. This description includes the kinds of elements that are common in the control of any kind of complex machine or system. We seek a mechanistic integration of the above, in other words, algorithms that accomplish control. Such algorithms operate on representations, transforming a representation of one kind into another, which then forms the input to yet another algorithm. Cognitive Programs (CPs) are hypothesized to capture exactly such representational transformations via stepwise sequences of operations. CPs, an updated and modernized offspring of Ullman's Visual Routines, impose an algorithmic structure to the set of attentional functions and play a role in the overall shaping of attentional modulation of the visual system so that it provides its best performance. This requires that we consider the visual system as a dynamic, yet general-purpose processor tuned to the task and input of the moment. This differs dramatically from the almost universal cognitive and computational views, which regard vision as a passively observing module to which simple questions about percepts can be posed, regardless of task. Differing from Visual Routines, CPs explicitly involve the critical elements of Visual Task Executive (vTE), Visual Attention Executive (vAE), and Visual Working Memory (vWM). Cognitive Programs provide the software that directs the actions of the Selective Tuning model of visual attention.

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

We present a model of visually-grounded language learning based on stacked gated recurrent neural networks which learns to predict visual features given an image description in the form of a sequence of phonemes. The learning task resembles that faced by human language learners who need to discover both structure and meaning from noisy and ambiguous data across modalities. We show that our model indeed learns to predict features of the visual context given phonetically transcribed image descriptions, and show that it represents linguistic information in a hierarchy of levels: lower layers in the stack are comparatively more sensitive to form, whereas higher layers are more sensitive to meaning.

This paper presents a study in which driver's gaze zone is categorized using new deep learning techniques. Since the sequence of gaze zones of a driver reflects precisely what and how he behaves, it allows us infer his drowsiness, focusing or distraction by analyzing the images coming from a camera. A Haar feature based face detector is combined with a correlation filter based MOSS tracker for the face detection task to handle a tough visual environment in the car. Driving database is a big-data which was constructed using a recording setup within a compact sedan by driving around the urban area. The gaze zones consist of 9 categories depending on where a driver is looking at during driving. A convolutional neural network is trained to categorize driver's gaze zone from a given face detected image using a multi-GPU platform, and then its network parameters are transferred to a GPU within a PC running on Windows to operate in the real-time basis. Result suggests that the correct rate of gaze zone categorization reaches to 95% in average, indicating that our system outperforms the state-of-art gaze zone categorization methods based on conventional computer vision techniques.

The primary problem dealt with in this paper is the following. Given some description of a visual object, find that object in an actual photograph. Part of the solution to this problem is the specification of a descriptive scheme, and a metric on which to base the decision of "goodness" of matching or detection.

The aim of this paper is to address recognition of natural human actions in diverse and realistic video settings. This challenging but important subject has mostly been ignored in the past due to several problems one of which is the lack of realistic and annotated video datasets. Our first contribution is to address this limitation and to investigate the use of movie scripts for automatic annotation of human actions in videos. We evaluate alternative methods for action retrieval from scripts and show benefits of a text-based classifier. Using the retrieved action samples for visual learning, we next turn to the problem of action classification in video. We present a new method for video classification that builds upon and extends several recent ideas including local space-time features, space-time pyramids and multi-channel non-linear SVMs. The method is shown to improve state-of-the-art results on the standard KTH action dataset by achieving 91.8% accuracy. Given the inherent problem of noisy labels in automatic annotation, we particularly investigate and show high tolerance of our method to annotation errors in the training set. We finally apply the method to learning and classifying challenging action classes in movies and show promising results.


Question: Emotions' influence on moral decision-making.

Psychopaths Recognize Morality but Lack Empathy's Restraint

Psychopaths exhibit a distinct lack of empathy and emotional regulation, often engaging in morally questionable actions and failing to differentiate between moral and conventional transgressions. Traditionally, the discipline has posited that emotional processes are essential for moral judgment, leading to the assumption that psychopaths are incapable of understanding right from wrong. This paper challenges that view, proposing instead that psychopaths possess a normal comprehension of morality but exhibit an abnormal control over their behavior in moral contexts. To test this hypothesis, we presented a group of psychopaths with moral dilemmas, comparing their responses with those of age- and sex-matched healthy individuals and non-psychopathic delinquents. Across the board, all subjects deemed personal harms (involving direct physical contact) less permissible than impersonal harms, despite both scenarios leading to utilitarian outcomes. Notably, psychopaths' decision patterns mirrored those of the other groups, suggesting they understand the distinction between right and wrong but lack the empathetic restraint that typically prevents harmful actions. This finding refutes the notion that emotional processes are inherently required for moral judgment, indicating that psychopaths are aware of moral principles but are indifferent to them and the repercussions of their actions.

We propose that guilt proneness is a critical characteristic of leaders and find support for this hypothesis across 3 studies. Participants in the first study rated a set of guilt-prone behaviors as more indicative of leadership potential than a set of less guilt-prone behaviors. In a follow-up study, guilt-prone participants in a leaderless group task engaged in more leadership behaviors than did less guilt-prone participants. In a third, and final, study, we move to the field and analyze 360° feedback from a group of young managers working in a range of industries. The results indicate that highly guilt-prone individuals were rated as more capable leaders than less guilt-prone individuals and that a sense of responsibility for others underlies the positive relationship between guilt proneness and leadership evaluations.

In this study the effects of musical background on the emotional appraisal of film sequences was investigated. Four pairs of polar emotions defined in Plutchik’s model were used as basic emotional qualities: joy-sadness, anticipation-surprise, fear-anger, and trustdisgust. In the preliminary study eight film sequences and eight music themes were selected as the best representatives of all eight Plutchik’s emotions. In the main experiment the participant judged the emotional qualities of film-music combinations on eight seven-point scales. Half of the combinations were congruent (e.g. joyful film joyful music), and half were incongruent (e.g. joyful film sad music). Results have shown that visual information (film) had greater effects on the emotion appraisal than auditory information (music). The modulation effects of music background depend on emotional qualities. In some incongruent combinations (joysadness) the modulations in the expected directions were obtained (e.g. joyful music reduces the sadness of a sad film), in some cases (anger-fear) no modulation effects were obtained, and in some cases (trust-disgust, anticipation-surprise) the modulation effects were in an unexpected direction (e.g. trustful music increased the appraisal of disgust of a disgusting film). These results suggest that the appraisals of conjoint effects of emotions depend on the medium (film masks the music) and emotional quality (three types of modulation effects).


Question: Emotions and their influence on moral judgment.

The impact of emotions on moral judgments is nuanced, varying with the nature of the emotion and the context of the moral scenario. Although emotions are recognized as pivotal in shaping moral decisions, the specific mechanisms through which they influence such judgments remain underexplored. This study advances a fresh theoretical framework positing that the motivational orientation of emotions is a key factor in their impact on moral judgments. We experimentally induced two emotions with identical valence but distinct motivational underpinnings: anger, which is associated with approach motivation, and disgust, linked to withdrawal motivation. Additionally, we examined four categories of moral scenarios: disgust-related, impersonal, personal, and beliefs. Our hypothesis was that anger, due to its approach-related nature, would lead to more lenient moral judgments, whereas disgust, given its withdrawal association, would result in stricter judgments. Our findings supported these predictions, revealing that the induced emotions significantly influenced moral judgments in impersonal and personal scenarios, but not in disgust-related or beliefs-based scenarios. This suggests that emotions indeed have a substantial role in moral judgments, but their effects are contingent on the type of emotion elicited. Moreover, the influence of induced emotions was more pronounced in scenarios requiring active decision-making, such as personal and impersonal contexts, possibly because they necessitate action rather than abstract reasoning. Our results underscore that the motivational aspect of emotions can be a predictive factor in understanding their role in moral judgments, offering a new perspective for moral psychology by highlighting an underappreciated pathway through which emotions and moral decisions are interconnected.

Recent studies have begun to elucidate the roles played in social cognition by specific neural structures, genes, and neurotransmitter systems. Cortical regions in the temporal lobe participate in perceiving socially relevant stimuli, whereas the amygdala, right somatosensory cortices, orbitofrontal cortices, and cingulate cortices all participate in linking perception of such stimuli to motivation, emotion, and cognition. Open questions remain about the domain-specificity of social cognition, about its overlap with emotion and with communication, and about the methods best suited for its investigation.

Research has suggested that approach-related positive emotions are associated with greater left frontal brain activity and that withdrawal-related negative emotions are associated with greater right frontal brain activity. Different explanations have been proposed. One posits that frontal asymmetry is due to emotional valence (positivity/negativity), one posits that frontal asymmetry is due to motivational direction (approach/withdrawal), and one posits that frontal asymmetry is due to a combination of emotional valence and motivational direction (positive-approach/negative-withdrawal). Because research had confounded emotional valence and motivational direction, the theoretical explanation was muddled. Solely supporting the motivational direction model, recent research has revealed that anger and cognitive dissonance, emotions with negative valence and approach motivational tendencies, are related to relatively greater left frontal activity.


Question: Emotions' influence on moral judgments.

This paper delves into the evolutionary roots and functions of gossip in human social dynamics. Gossip, defined as informal talk about the social behavior of others, is shown to occupy a significant portion of human conversation, often estimated to constitute around two-thirds of dialogue content. The study traces the origins of gossip to its primal roots in social grooming practices among primates, which served as a bonding mechanism for group cohesion. It posits that gossip evolved as a sophisticated form of social grooming, allowing humans to maintain larger social networks by efficiently sharing information about group members.

The paper further explores the multifaceted role of gossip in human societies. Beyond its role in social bonding, gossip is argued to have a critical function in the regulation of social norms and the policing of free riders. By disseminating information about individuals who violate group norms or exploit group resources without contributing, gossip acts as a social control mechanism, discouraging antisocial behavior and promoting group stability.

Moreover, the paper discusses the cognitive abilities that underpin gossip. Social cognition, including theory of mind and the capacity for empathy, enables humans to understand and interpret the intentions and emotions of others, making gossip a complex and nuanced form of communication. The sophisticated cognitive machinery required for gossip suggests that it is not a trivial or frivolous activity, but rather a sophisticated social tool that has played a significant role in the evolution of human social structures.

In summary, the paper offers an evolutionary perspective on gossip, highlighting its importance as a social bonding mechanism, a tool for controlling free riders, and a reflection of advanced social cognition in humans. This perspective underscores the significance of gossip in the maintenance of social order and the evolution of complex human societies.

Previous works in computer science, as well as political and social science, have shown correlation in text between political ideologies and the moral foundations expressed within that text. Additional work has shown that policy frames, which are used by politicians to bias the public towards their stance on an issue, are also correlated with political ideology. Based on these associations, this work takes a first step towards modeling both the language and how politicians frame issues on Twitter, in order to predict the moral foundations that are used by politicians to express their stances on issues. The contributions of this work includes a dataset annotated for the moral foundations, annotation guidelines, and probabilistic graphical models which show the usefulness of jointly modeling abstract political slogans, as opposed to the unigrams of previous works, with policy frames for the prediction of the morality underlying political tweets.

Marketers often employ a variety of positive emotions to encourage consumption or promote a particular behavior (e.g., buying, donating, recycling) to benefit an organization or cause. The authors show that specific positive emotions do not universally increase prosocial behavior but, rather, encourage different types of prosocial behavior. Four studies show that whereas positive emotions (i.e., love, hope, pride, and compassion) all induce prosocial behavior toward close entities (relative to a neutral emotional state), only love induces prosocial behavior toward distant others and international organizations. Love’s effect is driven by a distinct form of broadening, characterized by extending feelings of social connection and the boundary of caring to be more inclusive of others regardless of relatedness. Love—as a trait and a momentary emotion— is unique among positive emotions in fostering connectedness that other positive emotions (hope and pride) do not and broadening behavior in a way that other connected emotions (compassion) do not. This research contributes to the broaden-and-build theory of positive emotion by demonstrating a distinct type of broadening for love and adds an important qualification to the general finding that positive emotions uniformly encourage prosocial behavior.


Question: Emotions' influence on moral judgments.

This study explores the neural mechanisms underlying affective processing abnormalities in criminal psychopaths through functional magnetic resonance imaging (fMRI). The investigation of psychopathy, a multifaceted personality disorder with unclear origins, reveals a central component of affective processing anomalies.

In an effort to elucidate the neurobiological underpinnings of these anomalies, fMRI was employed to examine the brain activity of criminal psychopaths during an affective memory task. The findings indicate that, in contrast to both criminal nonpsychopaths and noncriminal individuals, psychopathic criminals exhibited significantly reduced affect-related activity in several key limbic regions. These regions include the amygdala/hippocampal formation, parahippocampal gyrus, ventral striatum, and the anterior and posterior cingulate gyri. Additionally, the study uncovered evidence of excessive activation in the bilateral fronto-temporal cortex among psychopathic criminals when processing affective stimuli.

These results suggest a potential link between the affective abnormalities characteristic of psychopathic offenders and diminished or compromised input from limbic structures, which may contribute to their distinctive emotional and moral judgment patterns.

Humor is an important, ubiquitous phenomenon; however, seemingly disparate conditions seem to facilitate humor. We integrate these conditions by suggesting that laughter and amusement result from violations that are simultaneously seen as benign. We investigated three conditions that make a violation benign and thus humorous: (a) the presence of an alternative norm suggesting that the situation is acceptable, (b) weak commitment to the violated norm, and (c) psychological distance from the violation. We tested the benign-violation hypothesis in the domain of moral psychology, where there is a strong documented association between moral violations and negative emotions, particularly disgust. Five experimental studies show that benign moral violations tend to elicit laughter and amusement in addition to disgust. Furthermore, seeing a violation as both wrong and not wrong mediates behavioral displays of humor. Our account is consistent with evolutionary accounts of laughter, explains humor across many domains, and suggests that humor can accompany negative emotion.

Functional magnetic resonance imaging (fMRI) of the human brain was used to study whether the amygdala is activated in response to emotional stimuli, even in the absence of explicit knowledge that such stimuli were presented. Pictures of human faces bearing fearful or happy expressions were presented to 10 normal, healthy subjects by using a backward masking procedure that resulted in 8 of 10 subjects reporting that they had not seen these facial expressions. The backward masking procedure consisted of 33 msec presentations of fearful or happy facial expressions, their offset coincident with the onset of 167 msec presentations of neutral facial expressions. Although subjects reported seeing only neutral faces, blood oxygen level-dependent (BOLD) fMRI signal in the amygdala was significantly higher during viewing of masked fearful faces than during the viewing of masked happy faces. This difference was composed of significant signal increases in the amygdala to masked fearful faces as well as significant signal decreases to masked happy faces, consistent with the notion that the level of amygdala activation is affected differentially by the emotional valence of external stimuli. In addition, these facial expressions activated the sublenticular substantia innominata (SI), where signal increases were observed to both fearful and happy faces--suggesting a spatial dissociation of territories that respond to emotional valence versus salience or arousal value. This study, using fMRI in conjunction with masked stimulus presentations, represents an initial step toward determining the role of the amygdala in nonconscious processing.

A wealth of animal data implicates the amygdala in aspects of emotional processing. In recent years, functional neuroimaging and neuropsychological studies have begun to refine our understanding of the functions of the amygdala in humans. This literature offers insights into the types of stimuli that engage the amygdala and the functional consequences that result from this engagement. Specific conclusions and hypotheses include: (1) the amygdala activates during exposure to aversive stimuli from multiple sensory modalities; (2) the amygdala responds to positively valenced stimuli, but these responses are less consistent than those induced by aversive stimuli; (3) amygdala responses are modulated by the arousal level, hedonic strength or current motivational value of stimuli; (4) amygdala responses are subject to rapid habituation; (5) the temporal characteristics of amygdala responses vary across stimulus categories and subject populations; (6) emotionally valenced stimuli need not reach conscious awareness to engage amygdala processing; (7) conscious hedonic appraisals do not require amygdala activation; (8) activation of the amygdala is associated with modulation of motor readiness, autonomic functions, and cognitive processes including attention and memory; (9) amygdala activations do not conform to traditional models of the lateralization of emotion; and (10) the extent and laterality of amygdala activations are related to factors including psychiatric status, gender and personality. The strengths and weakness of these hypotheses and conclusions are discussed with reference to the animal literature.

People's desires to see themselves as moral actors can contribute to their striving for and achievement of a sense of self-completeness. The authors use self-completion theory to predict (and show) that recalling one's own (im)moral behavior leads to compensatory rather than consistent moral action as a way of completing the moral self. In three studies, people who recalled their immoral behavior reported greater participation in moral activities (Study 1), reported stronger prosocial intentions (Study 2), and showed less cheating (Study 3) than people who recalled their moral behavior. These compensatory effects were related to the moral magnitude of the recalled event, but they did not emerge when people recalled their own positive or negative nonmoral behavior (Study 2) or others' (im)moral behavior (Study 3). Thus, the authors extend self-completion theory to the moral domain and use it to integrate the research on moral cleansing (remunerative moral strivings) and moral licensing (relaxed moral strivings).