Question: Forecasting tasks from eye movements: The significance of spatial distribution, dynamics, and image characteristics.

In this study, we explore how gaze is deployed in the context of specific tasks, a departure from the traditional focus on gaze patterns during passive viewing of two-dimensional images. To this end, we designed an experiment where participants navigated a virtual environment, performing a combination of approach and avoidance tasks. Throughout the experiment, we tracked both their body position and gaze, aiming to understand how gaze is influenced by the current task and the visual characteristics of the scene. Our analysis compared the observed gaze patterns with those predicted by a random gaze allocation and a specialized "saliency model." We found a remarkable consistency in gaze distributions across participants. More importantly, the specific fixation locations on objects were significantly influenced by the ongoing task, to the extent that the task could be accurately inferred from the fixation data. This contrasts sharply with the predictions of both the random and saliency models, which failed to account for the observed task-dependent gaze behavior. Our findings underscore the importance of considering task context and visual scene dynamics when analyzing gaze patterns, highlighting the role of purposeful actions in shaping visual attention.

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

The vast majority of studies into visual processing are conducted using computer display technology. The current paper describes a new free suite of software tools designed to make this task easier, using the latest advances in hardware and software. PsychoPy is a platform-independent experimental control system written in the Python interpreted language using entirely free libraries. PsychoPy scripts are designed to be extremely easy to read and write, while retaining complete power for the user to customize the stimuli and environment. Tools are provided within the package to allow everything from stimulus presentation and response collection (from a wide range of devices) to simple data analysis such as psychometric function fitting. Most importantly, PsychoPy is highly extensible and the whole system can evolve via user contributions. If a user wants to add support for a particular stimulus, analysis or hardware device they can look at the code for existing examples, modify them and submit the modifications back into the package so that the whole community benefits.

It is now evident from anecdotal evidence and preliminary research that distractions can hinder the task of operating a vehicle, and consequently reduce driver safety. However with increasing wireless connectivity and the portability of office devices, the vehicle of the future is visualized as an extension of the static work place - i.e. an office-on-the-move, with a phone, a fax machine and a computer all within the reach of the vehicle operator. For this research a Head mounted Eye-tracking Device (HED), is used for tracking the eye movements of a driver navigating a test route in an automobible while completing various driving tasks. Issues arising from data collection of eye movements during the completion of various driving tasks as well as the analysis of this data are discussed. Methods for collecting video and scan-path data, as well as difficulties and limitations are also reported.

Pfinder is a real-time system for tracking people and interpreting their behavior. It runs at 10Hz on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to obtain a 2D representation of head and hands in a wide range of viewing conditions. Pfinder has been successfully used in a wide range of applications including wireless interfaces, video databases, and low-bandwidth coding. Index Terms —Blobs, blob tracking, real-time, person tracking, 3D person tracking, segmentation, gesture recognition, mixture model, MDL.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

Image saliency detection is very useful in many computer vision tasks while it still remains a challenging problem. In this paper, we propose a new computational saliency detection model which is implemented with a coarse to fine strategy under the Bayesian framework. First, saliency points are applied to get a coarse location of the saliency region. And then, based on the rough region, we compute a prior map for the Bayesian model to achieve the final saliency map. Experimental results on a public available dataset show the effectiveness of the proposed prior map and the strength of our saliency map compared with several previous method.

Peripheral vision provides a less faithful representation of the visual input than foveal vision. Nonetheless, we can gain a lot of information about the world from our peripheral vision, for example in order to plan eye movements. The phenomenon of crowding shows that the reduction of information available in the periphery is not merely the result of reduced resolution. Crowding refers to visual phenomena in which identification of a target stimulus is significantly impaired by the presence of nearby stimuli, or flankers. What information is available in the periphery? We propose that the visual system locally represents peripheral stimuli by the joint statistics of responses of cells sensitive to different position, phase, orientation, and scale. This "textural" representation by summary statistics predicts the subjective "jumble" of features often associated with crowding. We show that the difficulty of performing an identification task within a single pooling region using this representation of the stimuli is correlated with peripheral identification performance under conditions of crowding. Furthermore, for a simple stimulus with no flankers, this representation can be adequate to specify the stimulus with some position invariance. This provides evidence that a unified neuronal mechanism may underlie peripheral vision, ordinary pattern recognition in central vision, and texture perception. A key component of our methodology involves creating visualizations of the information available in the summary statistics of a stimulus. We call these visualizations "mongrels" and show that they are highly useful in examining how the early visual system represents the visual input. Mongrels enable one to study the "equivalence classes" of our model, i.e., the sets of stimuli that map to the same representation according to the model.

With the presence of ubiquitous devices in our daily lives, effectively capturing and managing user attention becomes a critical device requirement. While gaze-tracking is typically employed to determine the user’s focus of attention, gaze-lock detection to sense eyecontact with a device is proposed in [16]. This work proposes eye contact detection using deep neural networks, and makes the following contributions: (1) With a convolutional neural network (CNN) architecture, we achieve superior eye-contact detection performance as compared to [16] with minimal data pre-processing ; our algorithm is furthermore validated on multiple datasets, (2) Gaze-lock detection is improved by combining head pose and eye-gaze information consistent with social attention literature, and (3) We demonstrate gaze-locking on an Android mobile platform via CNN model compression.

ÐA new view-based approach to the representation and recognition of human movement is presented. The basis of the representation is a temporal templateÐa static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: The first value is a binary value indicating the presence of motion and the second value is a function of the recency of motion in a sequence. We then develop a recognition method matching temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on standard platforms. Index TermsÐMotion recognition, computer vision.

In this paper, a new dynamic facial expression recognition method is proposed. Dynamic facial expression recognition is formulated as a longitudinal groupwise registration problem. The main contributions of this method lie in the following aspects: 1) subject-specific facial feature movements of different expressions are described by a diffeomorphic growth model; 2) salient longitudinal facial expression atlas is built for each expression by a sparse groupwise image registration method, which can describe the overall facial feature changes among the whole population and can suppress the bias due to large intersubject facial variations; and 3) both the image appearance information in spatial domain and topological evolution information in temporal domain are used to guide recognition by a sparse representation method. The proposed framework has been extensively evaluated on five databases for different applications: the extended Cohn-Kanade, MMI, FERA, and AFEW databases for dynamic facial expression recognition, and UNBC-McMaster database for spontaneous pain expression monitoring. This framework is also compared with several state-of-the-art dynamic facial expression recognition methods. The experimental results demonstrate that the recognition rates of the new method are consistently higher than other methods under comparison.


Question: Forecasting tasks from eye movements: The significance of spatial distribution, dynamics, and image characteristics.

This paper introduces a computational model of visual attention designed for rapid scene analysis, drawing inspiration from the primate visual system's behavior and architecture. The model integrates a variety of multiscale image features into a unified topographical saliency map. A dynamic neural network mechanism subsequently identifies and prioritizes locations of interest based on their saliency, facilitating a detailed examination in a time-efficient manner. By focusing on salient features, the system efficiently decomposes the intricate task of scene comprehension, enabling a swift and systematic analysis of visually prominent areas.

The importance of spatial distribution, dynamics, and image features in predicting tasks from eye movements is a critical area of research. In line with this, the model described in the paper emphasizes the role of saliency in guiding visual attention. By considering the spatial distribution of features and their dynamics, the model is capable of accurately predicting where an observer's gaze will be directed next. This predictive ability is enhanced by the incorporation of image features, which provide additional context for understanding the visual scene. The model's effectiveness in predicting eye movements underscores the significance of integrating these three components in the analysis of visual attention and task prediction.

Pedestrians are important participants in urban traffic environments, and thus act as an interesting category of objects for autonomous cars. Automatic pedestrian detection is an essential task for protecting pedestrians from collision. In this thesis, we investigate and develop novel approaches by interpreting spatial and temporal characteristics of pedestrians, in three different aspects: shape, cognition and motion. The special up-right human body shape, especially the geometry of the head and shoulder area, is the most discriminative characteristic for pedestrians from other object categories. Inspired by the success of Haar-like features for detecting human faces, which also exhibit a uniform shape structure, we propose to design particular Haar-like features for pedestrians. Tailored to a pre-defined statistical pedestrian shape model, Haar-like templates with multiple modalities are designed to describe local difference of the shape structure. Cognition theories aim to explain how human visual systems process input visual signals in an accurate and fast way. By emulating the center-surround mechanism in human visual systems, we design multi-channel, multi-direction and multi-scale contrast features, and boost them to respond to the appearance of pedestrians. In this way, our detector is considered as a top-down saliency system. In the last part of this thesis, we exploit the temporal characteristics for moving pedestrians and then employ motion information for feature design, as well as for regions of interest (ROIs) selection. Motion segmentation on optical flow fields enables us to select those blobs most probably containing moving pedestrians; a combination of Histogram of Oriented Gradients (HOG) and motion self difference features further enables robust detection. We test our three approaches on image and video data captured in urban traffic scenes, which are rather challenging due to dynamic and complex backgrounds. The achieved results demonstrate that our approaches reach and surpass state-of-the-art performance, and can also be employed for other applications, such as indoor robotics or public surveillance.

We present Gazture, a light-weight gaze based real-time gesture control system on commercial tablets. Unlike existing approaches that require dedicated hardware (e.g., high resolution camera), high computation overhead (powerful CPU) or specific user behavior (keeping head steady), Gazture provides gesture recognition based on easy-to-control user gaze input with a small overhead. To achieve this goal, Gazture incorporates a two-layer structure: The first layer focuses on real-time gaze estimation with acceptable tracking accuracy while incurring a small overhead. The second layer implements a robust gesture recognition algorithm while compensating gaze estimation error. To address user posture change while using mobile device, we design a online transfer function based method to convert current eye features into corresponding eye features in reference posture, which then facilitates efficient gaze position estimation. We implement Gazture on Lenovo Tab3 8 Plus tablet with Android 6.0.1, and evaluate its performance in different scenarios. The evaluation results show that Gazture can achieve a high accuracy in gesture recognition while incurring a low overhead.

What are the computational tasks that an executive controller for visual attention must solve? This question is posed in the context of the Selective Tuning model of attention. The range of required computations go beyond top-down bias signals or region-of-interest determinations, and must deal with overt and covert fixations, process timing and synchronization, information routing, memory, matching control to task, spatial localization, priming, and coordination of bottom-up with top-down information. During task execution, results must be monitored to ensure the expected results. This description includes the kinds of elements that are common in the control of any kind of complex machine or system. We seek a mechanistic integration of the above, in other words, algorithms that accomplish control. Such algorithms operate on representations, transforming a representation of one kind into another, which then forms the input to yet another algorithm. Cognitive Programs (CPs) are hypothesized to capture exactly such representational transformations via stepwise sequences of operations. CPs, an updated and modernized offspring of Ullman's Visual Routines, impose an algorithmic structure to the set of attentional functions and play a role in the overall shaping of attentional modulation of the visual system so that it provides its best performance. This requires that we consider the visual system as a dynamic, yet general-purpose processor tuned to the task and input of the moment. This differs dramatically from the almost universal cognitive and computational views, which regard vision as a passively observing module to which simple questions about percepts can be posed, regardless of task. Differing from Visual Routines, CPs explicitly involve the critical elements of Visual Task Executive (vTE), Visual Attention Executive (vAE), and Visual Working Memory (vWM). Cognitive Programs provide the software that directs the actions of the Selective Tuning model of visual attention.

The possibility of automatic understanding of customers’ shopping behavior and acting according to their needs is relevant in the marketing domain, attracting a lot of attention lately. In this work, we focus on the task of automatic assessment of customers’ shopping behavior, by proposing a multilevel framework. The framework is supported at low-level by different types of cameras, which are synchronized, facilitating efficient processing of information. A fish-eye camera is used for tracking, while a high-definition one serves for the action recognition task. The experiments are performed on both laboratory and real-life recordings in a supermarket. From the video recordings, we extract features related to the spatio-temporal behavior of trajectories, the dynamics and the time spent in each region of interest (ROI) in the shop and regarding the customer-products interaction patterns. Next ∗Corresponding Author. Phone: +31(0)15 2781537. Fax: +31(0)15 2787141. Email address: m.c.popa@tudelft.nl (M.C. Popa) Preprint submitted to Pattern Recognition Letters December 24, 2011 we analyze the shopping sequences using a Hidden Markov Model (HMM). We conclude that it is possible to accurately classify trajectories (93%), discriminate between different shopping related actions (91.6%), and recognize shopping behavioral types by means of the proposed reasoning model in 95% of the cases.

Temporal language is often couched in spatial metaphors. English has been claimed to have two space-time metaphoric systems: the ego-moving metaphor, wherein the observer’s context progresses along the time-line towards the future, and the time-moving metaphor, wherein time is conceived of as a river or conveyor belt on which events are moving from the future to the past. In three experiments, we investigated the psychological status of these metaphors by asking subjects to carry out temporal inferences stated in terms of spatial metaphors. In Experiment 1, we found that subjects were slowed in their processing when the assertions shifted from one spatial metaphoric system to the other. In Experiment 2, we determined that this cost of shifting could not be attributed to local lexical factors. In Experiment 3, we again found this metaphor consistency effect in a naturalistic version of the study in which we asked commonsense time questions of passengers at an airport. The results of the three studies provide converging evidence that people use spatial metaphors in temporal reasoning. Implications for the status of metaphoric systems arc discussed.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

When compared to image recognition, object detection is a much more challenging task because it requires the accurate real-time localization of an object in the target image. In interaction scenarios, this pipeline can be simplified by incorporating the users’ point of regard. Wearable eye trackers can estimate the gaze direction, but lack own processing capabilities. We enable mobile gaze-aware applications by developing an open-source platform which supports mobile eye tracking based on the Pupil headset and a smartphone running Android OS. Through our platform, we offer researchers and developers a rapid prototyping environment for gaze-enabled applications. We describe the concept, our current progress, and research implications. ACM Classification

Space time cube representation is an information visualization technique where spatiotemporal data points are mapped into a cube. Information visualization researchers have previously argued that space time cube representation is beneficial in revealing complex spatiotemporal patterns in a data set to users. The argument is based on the fact that both time and spatial information are displayed simultaneously to users, an effect difficult to achieve in other representations. However, to our knowledge the actual usefulness of space time cube representation in conveying complex spatiotemporal patterns to users has not been empirically validated. To fill this gap, we report on a between-subjects experiment comparing novice users' error rates and response times when answering a set of questions using either space time cube or a baseline 2D representation. For some simple questions, the error rates were lower when using the baseline representation. For complex questions where the participants needed an overall understanding of the spatiotemporal structure of the data set, the space time cube representation resulted in on average twice as fast response times with no difference in error rates compared to the baseline. These results provide an empirical foundation for the hypothesis that space time cube representation benefits users analyzing complex spatiotemporal patterns.

Changes in neural responses based on spatial attention have been demonstrated in many areas of visual cortex, indicating that the neural correlate of attention is an enhanced response to stimuli at an attended location and reduced responses to stimuli elsewhere. Here we demonstrate non-spatial, feature-based attentional modulation of visual motion processing, and show that attention increases the gain of direction-selective neurons in visual cortical area MT without narrowing the direction-tuning curves. These findings place important constraints on the neural mechanisms of attention and we propose to unify the effects of spatial location, direction of motion and other features of the attended stimuli in a ‘feature similarity gain model’ of attention.

By using the knowledge of facial structure and temperature distribution, this paper proposes an automatic eye localization method from long wave infrared thermal images both with eyeglasses and without eyeglasses. First, with the help of support vector machine classifier, three gray-projection features are defined to determine whether a subject is with eyeglasses. For subjects with eyeglasses, the locations of valleys in the projection curve are used to perform eye localization. For subjects without eyeglasses, a facial structure consisting of 15 sub-regions is proposed to extract Haar-like features. Eight classifiers are learned from the features selected by Adaboost algorithm for left and right eye, respectively. A vote strategy is employed to find the most likely eyes. To evaluate the effectiveness of our approach, experiments are performed on NVIE and Equinox databases. The eyeglass detection results on NVIE database and Equinox database are 99.36% and 95%, respectively, which demonstrate the effectiveness and robustness of our eyeglass detection method. Eye localization results of withindatabase experiments and cross-database experiments on these two databases are very comparable with the previous results in this field, verifying the effectiveness and the generalization ability of our approach. & 2013 Elsevier Ltd. All rights reserved.


Question: Forecasting tasks from eye movements: The significance of spatial distribution, dynamics, and image characteristics.

This study delves into the characteristics of visual representations that guide ongoing performance in sensorimotor tasks, specifically focusing on how visual information from various gaze positions is integrated. To investigate this, the research employs the method of introducing task-relevant display alterations during saccadic eye movements. Participants engaged in the task of replicating a pattern of colored blocks on a computer screen, utilizing a mouse to move the blocks. Eye movements were tracked using a dual-Purkinje eye tracker, and changes to the color of the blocks within the pattern were introduced at various stages of task execution. When the saccade target's color changed during the saccade, fixation durations on the model pattern increased, with the magnitude of increase dependent on the timing of the change within the task. This finding highlights that fixations on identical visual stimuli can serve distinct purposes. The results further suggest that the visual information retained across consecutive fixations is contingent on the immediate task requirements, aligning with prior assertions that visual representations are limited and task-specific. Additional changes to blocks beyond the saccade target resulted in more pronounced increases in fixation duration, indicating the retention of some global aspect of the pattern across different fixations. Fixation durations revealed effects of display changes not captured by perceptual reports, which can be explained by considering processes operating at various levels of description and time scales. This implies that our conscious experience of the world may encapsulate events over a broader time horizon than those underlying the fine-grained workings of the perceptual-motor system.

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model's performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.

The primary problem dealt with in this paper is the following. Given some description of a visual object, find that object in an actual photograph. Part of the solution to this problem is the specification of a descriptive scheme, and a metric on which to base the decision of "goodness" of matching or detection.

Introduction Vision, due to its significance in surviving and socializing, is one of the most important and  extensively studied sensory functions in the human brain. In order to fully understand visual  information processing, or more specifically, visual recognition, David Marr proposed the  Tri-level Hypothesis [29], in that three levels of the system should be studied: the computational  goal of the system, the internal representation or the algorithm the system uses to achieve the  goal, and the neural substrates that implement the system. It is well-known that visual  recognition in the human brain is implemented by the ventral visual pathway [32], which  receives visual information from the retina and goes through a layered structure including V1  (also known as the primary visual cortex), V2, V4, before reaching the inferior temporal cortex  (IT). The topographic mapping between the retina and the human visual cortex follows a  log-polar transformation, in which the Cartesian coordinates of the retina are transformed to  polar coordinates (polar angle and eccentricity) in the human visual cortex. From V1 to V4, each  of the visual areas contains a complete polar angle and eccentricity map, and have a increasing  receptive field (RF) size and respond to increasing complex visual features [12,22]. In higher  level visual areas such as the ventral temporal cortex (VTC), studies using functional magnetic  resonance imaging (fMRI) and diffusion tensor imaging (DTI) have found there exists visual  category-selective regions, such as the Fusiform Face Area (FFA, [26]) for face processing, the  Parahippocampal Place Area (PPA, [13]) for scene perception, and the Lateral Occipital  Complex (LOC, [19]) for general object recognition. Grill-Spector and Malach [20] proposed that  layer-based hierarchical processing, together with the functional specialization, are the two  organizing principles of the human visual cortex. More recent studies have shown that the  central-biased face recognition pathway (FFA) and the peripheral-biased scene recognition  pathway (PPA) are both functionally and anatomically segregated by mid-fusiform sulcus (MFS),  which enables parallel fast processing of visual recognition tasks in the ventral pathway [18, 43,  21, 44].

Voluntarily directing visual attention to a cued position in space leads to improved processing of forthcoming visual stimuli at the attended position, and attenuated processing of competing stimuli elsewhere, due to anticipatory tuning of visual cortex activity. In EEG, recent evidence points to a determining role of modulations of posterior alpha-band activity (8-14 Hz) in such anticipatory facilitation (alpha-power decreases) versus inhibition (alpha-power increases). Yet, while such alpha-modulations are a common finding, the direction of modulation varies to a great extent across studies implying dependence on task demands. Here, we reveal opposite modulation of posterior alpha-power with early/initiation versus later/sustained processes of anticipatory attention orienting. Marked alpha-decreases were observed during shifting of attention (initial 700 ms) over occipito-parietal areas processing to-be-attended visual space, while alpha-increases dominated in the subsequent maintenance phase (>700 ms) over occipito-parietal cortex tuned to unattended positions. Notably, the presence of alpha-modulation strongly depended on individual resting alpha-power. Overall, this provides further support to an active facilitative versus inhibitory role of alpha-power decreases and increases and suggests that these attention-related changes are differentially deployed during anticipatory attention orienting to prepare versus maintain the cortex for optimal target processing.

In web search, information about times between user actions has been shown to be a good indicator of users' satisfaction with the search results. Existing work uses the mean values of the observed times, or fits probability distributions to the observed times. This implies a context-independence assumption that the time elapsed between a pair of user actions does not depend on the context, in which the first action takes place. We validate this assumption using logs of a commercial web search engine and discover that it does not always hold. For between 37% to 80% of query-result pairs, depending on the number of observations, the distributions of click dwell times have statistically significant differences in query sessions for which a given result (i) is the first item to be clicked and (ii) is not the first. To account for this context bias effect, we propose a context-aware time model (CATM). The CATM allows us (i) to predict times between user actions in contexts, in which these actions were not observed, and (ii) to compute context-independent estimates of the times by predicting them in predefined contexts. Our experimental results show that the CATM provides better means than existing methods to predict and interpret times between user actions.

Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

Results from a variety of sources, some many years old, lead ineluctably to a re-appraisal of the twin strategies of hierarchical and parallel processing used by the brain to construct an image of the visual world. Contrary to common supposition, there are at least three 'feed-forward' anatomical hierarchies that reach the primary visual cortex (V1) and the specialized visual areas outside it, in parallel. These anatomical hierarchies do not conform to the temporal order with which visual signals reach the specialized visual areas through V1. Furthermore, neither the anatomical hierarchies nor the temporal order of activation through V1 predict the perceptual hierarchies. The latter shows that we see (and become aware of) different visual attributes at different times, with colour leading form (orientation) and directional visual motion, even though signals from fast-moving, high-contrast stimuli are among the earliest to reach the visual cortex (of area V5). Parallel processing, on the other hand, is much more ubiquitous than commonly supposed but is subject to a barely noticed but fundamental aspect of brain operations, namely that different parallel systems operate asynchronously with respect to each other and reach perceptual endpoints at different times. This re-assessment leads to the conclusion that the visual brain is constituted of multiple, parallel and asynchronously operating task- and stimulus-dependent hierarchies (STDH); which of these parallel anatomical hierarchies have temporal and perceptual precedence at any given moment is stimulus and task related, and dependent on the visual brain's ability to undertake multiple operations asynchronously.

We propose a new technique for direct visual matching of images for the purposes of face recognition and image retrieval, using a probabilistic measure of similarity, based primarily on a Bayesian (MAP) analysis of image differences. The performance advantage of this probabilistic matching technique over standard Euclidean nearest-neighbor eigenface matching was demonstrated using results from DARPA’s 1996 “FERET” face recognition competition, in which this Bayesian matching algorithm was found to be the top performer. In addition, we derive a simple method of replacing costly computation of nonlinear (on-line) Bayesian similarity measures by inexpensive linear (off-line) subspace projections and simple Euclidean norms, thus resulting in a significant computational speed-up for implementation with very large databases. Appears in: Pattern Recognition, Vol. 33, No. 11, pps. 1771-1782, November, 2000. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c Mitsubishi Electric Research Laboratories, Inc., 2002 201 Broadway, Cambridge, Massachusetts 02139 Appears in: Pattern Recognition, Vol. 33, No. 11, pps. 1771-1782, November, 2000.

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

Existing research suggests that individual personality differences are correlated with a user's speed and accuracy in solving problems with different types of complex visualization systems. We extend this research by isolating factors in personality traits as well as in the visualizations that could have contributed to the observed correlation. We focus on a personality trait known as "locus of control” (LOC), which represents a person's tendency to see themselves as controlled by or in control of external events. To isolate variables of the visualization design, we control extraneous factors such as color, interaction, and labeling. We conduct a user study with four visualizations that gradually shift from a list metaphor to a containment metaphor and compare the participants' speed, accuracy, and preference with their locus of control and other personality factors. Our findings demonstrate that there is indeed a correlation between the two: participants with an internal locus of control perform more poorly with visualizations that employ a containment metaphor, while those with an external locus of control perform well with such visualizations. These results provide evidence for the externalization theory of visualization. Finally, we propose applications of these findings to adaptive visual analytics and visualization evaluation.


Question: Forecasting tasks from eye movements: The significance of spatial distribution, dynamics, and image characteristics.

This paper presents an innovative model that merges top-down and bottom-up attention mechanisms to optimize the speed of visual target detection. Traditionally, research has concentrated on either top-down or bottom-up models separately. Our proposed model, however, integrates both approaches. The bottom-up aspect evaluates the visual saliency of scene locations through feature maps extracted across various spatial scales. The top-down component, informed by accumulated statistical knowledge of the visual characteristics of the search target and background clutter, adjusts these bottom-up maps to maximize the speed of target detection. Our model has been rigorously tested on a diverse set of 750 artificial and natural scenes, with outcomes closely aligning with a significant body of human psychophysics literature on visual search. These findings suggest that our integrated model provides a credible representation of how humans effectively combine bottom-up and top-down cues to expedite target detection.

Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism — a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.

In order to control a reaching movement of the arm and body, several different computational problems must be solved. Some parallel methods that could be implemented in networks of neuron-like processors are described. Each method solves a different part of the overall task. First, a method is described for finding the torques necessary to follow a desired trajectory. The methods is more economical and more versatile than table look-up and requires very few sequential steps. Then a way of generating an internal representation of a desired trajectory is described. This method shows the trajectory one piece at a time by applying a large set of heuristic rules to a "motion blackboard" that represents the static and dynamic parameters of the state of the body at the current point in the trajectory. The computations are simplified by expressing the positions, orientations, and motions of parts of the body in terms of a single, non-accelerating, world-based frame of reference, rather than in terms of the joint-angles or an egocentric frame based on the body itself.

In this paper, we address the issue of personality and interaction style recognition from profile pictures in Facebook. We recruited volunteers among Facebook users and collected a dataset of profile pictures, labeled with gold standard self-assessed personality and interaction style labels. Then, we exploited a bag-of-visual-words technique to extract features from pictures. Finally, different machine learning approaches were used to test the effectiveness of these features in predicting personality and interaction style traits. Our good results show that this task is very promising, because profile pictures convey a lot of information about a user and are directly connected to impression formation and identity management.

Recently, the physiological signals have been analyzed more and more, especially in the context of everyday life activities such as watching video or looking at pictures. Tracking these signals gives access to the mental state of the user (interest, tiredness, stress) but also to his emotions (sadness, fright, happiness). The analysis of the reader's physiological signals during reading can provide a better understanding of the reader's feelings but also a better understanding of the documents. Our main research direction is to find the relationship between a change in the reader's physiological signal and the content of the reading. As a first step, we investigate whether it is possible to distinguish a manga (Japanese comic book) from another by analyzing the physiological signals of the reader. We use 3 different manga genres (horror, romance, comedy) and try to predict which one is read by analyzing the features extracted from the physiological signals of the reader. Our method uses the blood volume pulse, the electrodermal activity and the skin temperature of the reader while reading. We show that by using these physiological signals with a support vector machine we can retrieve which manga has been read with a 90% average accuracy.

The paper presents the setup and results of a mobile application usability test using mobile eye-tracking glasses. Quasi-real conditions of the natural environment were assured for the testing in order to avoid incidental disorders possible in natural conditions and the weaknesses of a fully lab experiment. The experiment methodology was developed and used to carry out a usability test of the mobile part of a Sale Force Automation (SFA) application in terms of its basic functionality. The paper's outlines are the following: the methodology used, arrangement and realisation of the usability testing experiment, test results and answers to the research questions. A list of general conclusions rounds off the report.

Social media platforms allow rapid information diffusion, and serve as a source of information to many of the users. Particularly, in Twitter information provided by tweets diffuses over the users through retweets. Hence, being able to predict the retweet count of a given tweet is important for understanding and controlling information diffusion on Twitter. Since the length of a tweet is limited to 140 characters, extracting relevant features to predict the retweet count is a challenging task. However, visual features of images linked in tweets may provide predictive features. In this study, we focus on predicting the expected retweet count of a tweet by using visual cues of an image linked in that tweet in addition to content and structure-based features.

A driver assistance system (DAS) should support the driver by monitoring road and vehicle events and presenting relevant and timely information to the driver. It is impossible to know what a driver is thinking, but we can monitor the driver’s gaze direction and compare it with the position of information in the driver’s viewfield to make inferences. In this way, not only do we monitor the driver’s actions, we monitor the driver’s observations as well. In this paper we present the automated detection and recognition of road signs, combined with the monitoring of the driver’s response. We present a complete system that reads speed signs in real-time, compares the driver’s gaze, and provides immediate feedback if it appears the sign has been missed by t ©

Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.

Based on life-long observations of physical, chemical, and biologic phenomena in the natural world, humans can often easily picture in their minds what an object will look like in the future. But, what about computers? In this paper, we learn computational models of object transformations from time-lapse videos. In particular, we explore the use of generative models to create depictions of objects at future times. These models explore several different prediction tasks: generating a future state given a single depiction of an object, generating a future state given two depictions of an object at different times, and generating future states recursively in a recurrent framework. We provide both qualitative and quantitative evaluations of the generated results, and also conduct a human evaluation to compare variations of our models.

A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system.


Question: Meme identification and tracking during critical incidents.

This study explores the diffusion dynamics of competing memes across interconnected networks from a network science viewpoint. We focus on a scenario involving two competing "memes," which could represent data, rumors, or other forms of information, spreading through a complex network structure. Two pivotal questions are addressed: (1) which meme is likely to dominate the information landscape? and (2) is it possible to influence the outcome of the competition between these memes? Our investigation is grounded in a dual-conceptual framework: a structural graph model (composite network) and a viral propagation model (SI1I2S). Through this lens, we develop a non-linear dynamical system and conduct an eigenvalue analysis to pinpoint the critical threshold for viral behavior. Drawing from this analysis, we introduce a novel and precise prediction tool, the EigenPredictor, capable of forecasting meme dominance. To test the efficacy of various strategies to mitigate viral spread, we employ a blend of synthetic and authentic composite networks, assessing methods that either (a) suppress both memes simultaneously or (b) target one meme for suppression while allowing the other to spread largely unchecked.

Modern social media are becoming overloaded with information because of the rapidly-expanding number of information feeds. We analyze the user-generated content in Sina Weibo, and find evidence that the spread of popular messages often follow a mechanism that differs from the spread of disease, in contrast to common belief. In this mechanism, an individual with more friends needs more repeated exposures to spread further the information. Moreover, our data suggest that for certain messages the chance of an individual to share the message is proportional to the fraction of its neighbours who shared it with him/her, which is a result of competition for attention. We model this process using a fractional susceptible infected recovered (FSIR) model, where the infection probability of a node is proportional to its fraction of infected neighbors. Our findings have dramatic implications for information contagion. For example, using the FSIR model we find that real-world social networks have a finite epidemic threshold in contrast to the zero threshold in disease epidemic models. This means that when individuals are overloaded with excess information feeds, the information either reaches out the population if it is above the critical epidemic threshold, or it would never be well received.

In this paper, we present a study on 4.4 million Twitter messages related to 24 systematically chosen real-world events. For each of the 4.4 million tweets, we first extracted sentiment scores based on the eight basic emotions according to Plutchik's wheel of emotions. Subsequently, we investigated the effects of shifts in the emotional valence on the spread of information. We found that in general OSN users tend to conform to the emotional valence of the respective real-world event. However, we also found empirical evidence that prospectively negative real-world events exhibit a significant amount of shifted emotions in the corresponding tweets (i.e. positive messages). To explain this finding, we use the theory of social connection and emotional contagion. To the best of our knowledge, this is the first study that provides empirical evidence for the undoing hypothesis in online social networks (OSNs). The undoing hypothesis postulates that positive emotions serve as an antidote during negative events.

The Twitter social network provides a constant stream of concise data, useful within both geospatial and temporal domains. Previous studies have attempted to mine and find recent news topics within the data set. Although the efforts were a step in the right direction, we believe more is needed to accomplish a useful and interesting task: using live Twitter data to automatically identify breaking news events in near real-time. In this paper, we present methods to collect data, identify breaking news topics, and display results in a geo-temporal visualization. Our goal is to provide a solution that discovers breaking news faster than traditional reporting mediums.

In the context of a social gathering, such as a cocktail party, the memorable moments are often captured by professional photographers or the participants. The latter case is generally undesirable because many participants would rather enjoy the event instead of being occupied by the tedious photo capturing task. Motivated by this scenario, we propose an automated social event photo-capture framework for which, given the multiple sensor data streams and the information from the Web as input, will output the visually appealing photos of the social event. Our proposal consists of three components: (1) social attribute extraction from both the physical space and the cyberspace; (2) social attribute fusion; and (3) active camera control. Current work is presented and we conclude with expected contributions as well as future direction.


Question: Meme identification and tracking during critical incidents.

This study introduces a novel methodology for monitoring and analyzing the dissemination and evolution of memes and topical narratives across the digital landscape, a task of significant scholarly and practical interest. Existing approaches have primarily focused on detecting long-term topic shifts or sudden surges in mentions of specific entities. However, these methodologies fall short in capturing the lifecycle of content that rapidly gains traction and subsequently diminishes within a timeframe of days, aligning with the human perception of news and events. To address this gap, we propose a system designed to trace the journey of concise, distinctive phrases as they propagate through online text, largely maintaining their integrity. By devising efficient algorithms for clustering textual permutations of these phrases, we unveil a comprehensive category of memes characterized by extensive dissemination and dynamic variation on a daily scale. Our primary case study demonstrates how this meme-tracking mechanism can offer a structured representation of the news cycle—the cyclical nature of news media that has traditionally been subject to qualitative interpretation but has lacked the precision for quantitative scrutiny. Through the systematic monitoring of 1.6 million mainstream media outlets and blogs over a three-month duration, encompassing 90 million articles, we uncover a series of distinctive and recurrent patterns within the news cycle. Notably, we observe an average delay of 2.5 hours between the peak attention given to a particular phrase in mainstream media and its corresponding peak in blogosphere discussions. This temporal lag is accompanied by distinct behavioral traits surrounding the apex of attention and a "heartbeat"-like rhythm in the transition of focus between traditional news sources and blogs. Additionally, we construct and evaluate a mathematical framework to elucidate the temporal dynamics exhibited by the system.

Text-based counseling and support systems have seen an increasing proliferation in the past decade. We present Fathom, a natural language interface to help crisis counselors on Crisis Text Line, a new 911-like crisis hotline that takes calls via text messaging rather than voice. Text messaging opens up the opportunity for software to read the messages as well as people, and to provide assistance for human counselors who give clients emotional and practical support. Crisis counseling is a tough job that requires dealing with emotionally stressed people in possibly life-critical situations, under time constraints. Fathom is a system that provides topic modeling of calls and graphical visualization of topic distributions, updated in real time. We develop a mixed-initiative paradigm to train coherent topic and word distributions and use them to power real-time visualizations aimed at reducing counselor cognitive overload. We believe Fathom to be the first real-time computational framework to assist in crisis counseling.

The status updates posted to social networks, such as Twitter and Facebook, contain a myriad of information about what people are doing and watching. During events, such as sports games, many updates are sent describing and expressing opinions about the event. In this paper, we describe an algorithm that generates a journalistic summary of an event using only status updates from Twitter as a source. Temporal cues, such as spikes in the volume of status updates, are used to identify the important moments within an event, and a sentence ranking method is used to extract relevant sentences from the corpus of status updates describing each important moment within an event. We evaluate our algorithm compared to human-generated summaries and the previous best summarization algorithm, and find that the results of our method are superior to the previous algorithm and approach the readability and grammaticality of the human-generated summaries.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.


Question: Meme identification and tracking during critical incidents.

This paper delves into the novel methodology for the extraction and tracking of memes during crisis situations. Unlike conventional approaches, this method focuses on identifying and mapping the spread of memes in real-time, particularly in the context of emergencies or critical events. Central to this methodology is the analysis of sentiment within short, informal texts, often found on social media platforms. By examining the emotional tone and intensity within these messages, the algorithm can pinpoint the emergence and evolution of memes. This approach not only captures the immediate reactions and sentiments of the public but also traces the memes' trajectory through different online communities. The methodology employs advanced natural language processing techniques to parse the informal language and abbreviations commonly used in social media communications. Through sentiment analysis, it discerns the underlying attitudes and emotions associated with crisis-related memes, providing valuable insights into public perception and response. This is crucial for understanding the collective mindset during emergencies and can aid in the formulation of more effective crisis communication strategies.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An advanced evolutionary algorithm for the economic dispatch problem with valve-point effects

**Similar Documents**: This paper introduces an innovative evolutionary algorithm designed to tackle the economic dispatch problem, particularly accounting for valve-point effects. The algorithm is a sophisticated solution that addresses the complexities of the valve-point effect, a nonlinear characteristic in thermal power plants that significantly impacts the cost of electricity generation. By optimizing the dispatch of power across different generating units, the algorithm aims to minimize the total production cost while adhering to operational constraints. The evolutionary approach enables the algorithm to explore a vast solution space efficiently, converging on the most cost-effective dispatch strategy. This is achieved through a combination of genetic operators, such as crossover and mutation, and a robust selection mechanism that favors solutions with lower costs and better feasibility. The algorithm is validated through extensive simulations on benchmark systems, demonstrating its ability to outperform conventional methods in terms of both solution quality and computational efficiency. The results highlight the algorithm's potential for practical application in power system operations, enhancing the economic and environmental sustainability of electricity generation.

In many online social systems, social ties between users play an important role in dictating their behavior. One of the ways this can happen is through social influence, the phenomenon that the actions of a user can induce his/her friends to behave in a similar way. In systems where social influence exists, ideas, modes of behavior, or new technologies can diffuse through the network like an epidemic. Therefore, identifying and understanding social influence is of tremendous interest from both analysis and design points of view.
 This is a difficult task in general, since there are factors such as homophily or unobserved confounding variables that can induce statistical correlation between the actions of friends in a social network. Distinguishing influence from these is essentially the problem of distinguishing correlation from causality, a notoriously hard statistical problem.
 In this paper we study this problem systematically. We define fairly general models that replicate the aforementioned sources of social correlation. We then propose two simple tests that can identify influence as a source of social correlation when the time series of user actions is available.
 We give a theoretical justification of one of the tests by proving that with high probability it succeeds in ruling out influence in a rather general model of social correlation. We also simulate our tests on a number of examples designed by randomly generating actions of nodes on a real social network (from Flickr) according to one of several models. Simulation results confirm that our test performs well on these data. Finally, we apply them to real tagging data on Flickr, exhibiting that while there is significant social correlation in tagging behavior on this system, this correlation cannot be attributed to social influence.

The proliferation of social media and user-generated content in the Web has opened new opportunities for detecting and disseminating information quickly. The Twitter stream is one large source of information, but the magnitude of tweets posted and the noisy nature of its content makes the harvesting of knowledge from Twitter very hard. Aiming at overcoming some of the challenges and extract some of the hidden information, this thesis proposes a system for real-time detection of news events from the Twitter stream. The first step of our approach is to let a classifier, based on an Artificial Neural Network and deep learning, detect news relevant tweets from the stream. Next, a novel streaming data clustering algorithm is applied to the detected news tweets to form news events. Finally, the events of highest interest is retrieved based on events’ sizes and rapid growth in tweet frequencies, before the news events are presented and visualized in a web user interface. We evaluate the proposed system on a large, publicly available corpus of annotated news events from Twitter. As part of the evaluation, we compare our approach with a related state-of-the-art solution. Overall, our experiments and user-based evaluation show that our approach on detecting current (real) news events delivers state-of-the-art performance.

Recently, the physiological signals have been analyzed more and more, especially in the context of everyday life activities such as watching video or looking at pictures. Tracking these signals gives access to the mental state of the user (interest, tiredness, stress) but also to his emotions (sadness, fright, happiness). The analysis of the reader's physiological signals during reading can provide a better understanding of the reader's feelings but also a better understanding of the documents. Our main research direction is to find the relationship between a change in the reader's physiological signal and the content of the reading. As a first step, we investigate whether it is possible to distinguish a manga (Japanese comic book) from another by analyzing the physiological signals of the reader. We use 3 different manga genres (horror, romance, comedy) and try to predict which one is read by analyzing the features extracted from the physiological signals of the reader. Our method uses the blood volume pulse, the electrodermal activity and the skin temperature of the reader while reading. We show that by using these physiological signals with a support vector machine we can retrieve which manga has been read with a 90% average accuracy.


Question: Meme identification and tracking during critical incidents.

This paper explores the identification and monitoring of orchestrated political campaigns on social media platforms, focusing on the phenomenon of astroturfing. Astroturfing involves the use of multiple accounts, often centrally managed, to artificially inflate the perception of public support for a particular political candidate or viewpoint. We introduce a sophisticated machine learning framework that integrates various dimensions of analysis—topological, content-based, and crowdsourced—to effectively pinpoint the onset of viral dissemination of political disinformation on Twitter. The framework is designed to capture the nuanced patterns of information diffusion, allowing for the early detection of astroturf content. Our preliminary findings indicate a remarkable level of accuracy, exceeding 96%, in identifying such manipulative content in the lead-up to the 2010 U.S. midterm elections, showcasing the framework's potential for combating misinformation in real-world scenarios.

Sentiment analysis has been widely researched in the domain of online review sites with the aim of generating summarized opinions of product users about different aspects of the products. However, there has been little work focusing on identifying the polarity of sentiments expressed by users during disaster events. Identifying sentiments expressed by users in an online social networking site can help understand the dynamics of the network, e.g., the main users’ concerns, panics, and the emotional impacts of interactions among members. Data produced through social networking sites is seen as ubiquitous, rapid and accessible, and it is believed to empower average citizens to become more situationally aware during disasters and coordinate to help themselves. In this work, we perform sentiment classification of user posts in Twitter during the Hurricane Sandy and visualize these sentiments on a geographical map centered around the hurricane. We show how users' sentiments change according not only to users’ locations, but also based on the distance from the disaster.

The modern requirements and the best practices in the field of Information Security (IS) Incident Management Process (ISIMP) are analyzed. "IS event" and "IS incident" terms, being used for ISIMP, have been defined. An approach to ISIMP development has been created. According to this approach ISIMP processes are described. As an example the «Vulnerabilities, IS events and incidents detection and notification» joint process is examined in detail.

In this work we examine the characteristics of social activity and patterns of communication on Twitter, a prominent example of the emerging class of communication systems we call "social awareness streams." We use system data and message content from over 350 Twitter users, applying human coding and quantitative analysis to provide a deeper understanding of the activity of individuals on the Twitter network. In particular, we develop a content-based categorization of the type of messages posted by Twitter users, based on which we examine users' activity. Our analysis shows two common types of user behavior in terms of the content of the posted messages, and exposes differences between users in respect to these activities.

This paper uses the case of Hurricane Sandy and reddit's topical community (subreddit) /r/sandy to examine the production and curation of news content around events on a social news site. Through qualitative analysis, we provide a coded topology of produced content and describe how types of networked gatekeeping impact the framing of a crisis situation. This study also examines, through quantitative modeling, what kind of information becomes negotiated and voted as relevant. We suggest that highly scored content shared in a social news setting focused more on human-interest media and perspective-based citizen journalism than professional news reports. We conclude by discussing how the mechanisms of social news sites conflict with the social norms and culture of reddit to produce differing expectations around news.


Question: Meme identification and tracking during critical incidents.

This paper delves into the intricate web of person-to-person recommendation networks, specifically examining a vast dataset comprising 4 million individuals who made 16 million product recommendations across half a million items. We dissect the diffusion patterns of these recommendations and the resulting cascade sizes, elucidating our observations through a straightforward stochastic model. A significant aspect of our study is the variation in user behavior across distinct communities delineated by the recommendation network. It is noteworthy that product purchases exhibit a 'long-tail' distribution, with a substantial portion of sales attributed to items with low sales volume. We further investigate the temporal evolution of the recommendation network and its efficacy from both the recommender's and recipient's perspectives. Our findings indicate that, on average, recommendations are not particularly effective in stimulating purchases and do not propagate extensively. However, we propose a model that adeptly identifies specific communities, product segments, and pricing tiers where viral marketing strategies demonstrate exceptional effectiveness, offering valuable insights for targeted marketing efforts.

Sentiment analysis has been widely researched in the domain of online review sites with the aim of generating summarized opinions of product users about different aspects of the products. However, there has been little work focusing on identifying the polarity of sentiments expressed by users during disaster events. Identifying sentiments expressed by users in an online social networking site can help understand the dynamics of the network, e.g., the main users’ concerns, panics, and the emotional impacts of interactions among members. Data produced through social networking sites is seen as ubiquitous, rapid and accessible, and it is believed to empower average citizens to become more situationally aware during disasters and coordinate to help themselves. In this work, we perform sentiment classification of user posts in Twitter during the Hurricane Sandy and visualize these sentiments on a geographical map centered around the hurricane. We show how users' sentiments change according not only to users’ locations, but also based on the distance from the disaster.

Online social media, such as Reddit, has become an important resource to share personal experiences and communicate with others. Among other personal information, some social media users communicate about mental health problems they are experiencing, with the intention of getting advice, support or empathy from other users. Here, we investigate the language of Reddit posts specific to mental health, to define linguistic characteristics that could be helpful for further applications. The latter include attempting to identify posts that need urgent attention due to their nature, e.g. when someone announces their intentions of ending their life by suicide or harming others. Our results show that there are a variety of linguistic features that are discriminative across mental health user communities and that can be further exploited in subsequent classification tasks. Furthermore, while negative sentiment is almost uniformly expressed across the entire data set, we demonstrate that there are also condition-specific vocabularies used in social media to communicate about particular disorders. Source code and related materials are available from: https: //github.com/gkotsis/ reddit-mental-health.


Question: Deep Neural Network approach for sentiment classification.

This paper introduces an innovative learning technique, known as back-propagation, designed for networks of neuron-inspired units. This procedure iteratively fine-tunes the connection weights within the network to reduce the discrepancy between the network's actual output vector and the desired output vector. Through this adjustment process, internal 'hidden' units, which are neither part of the input nor output, evolve to encode critical features of the task domain. The network's capacity to capture underlying patterns is a result of the interactions among these hidden units. This feature extraction capability sets back-propagation apart from previous, more rudimentary methods, such as the perceptron-convergence procedure.

**Question**: An Analysis of Convolutional Neural Networks for Text Classification

**Documents**: Convolutional Neural Networks for Sentence Classification
We present an effective and efficient convolutional neural network (CNN) for sentence classification. Our CNN model is based on a novel architecture that incorporates multi-perspective convolutional networks (MP-CNN) to capture different aspects of the sentence. The MP-CNN architecture consists of multiple convolutional layers, each with a different kernel size to capture information at different levels of granularity. We also use a max-over-time pooling layer to aggregate the most important features from the convolutional layers. Our model is trained end-to-end using backpropagation and stochastic gradient descent. We evaluate our model on several benchmark datasets for sentence classification and achieve state-of-the-art results. Our results demonstrate the effectiveness of using CNNs for sentence classification and the importance of multi-perspective feature extraction.

**Similar Documents**: This paper introduces a highly effective and efficient convolutional neural network (CNN) designed specifically for sentence classification tasks. The proposed CNN model is built on a groundbreaking architecture that integrates multi-perspective convolutional networks (MP-CNN) to encompass various facets of the sentence structure. The MP-CNN architecture features a series of convolutional layers, each equipped with a distinct kernel size to capture information across different scales of detail. Additionally, a max-over-time pooling layer is employed to consolidate the most salient features extracted from the convolutional layers. The model is trained in an end-to-end manner utilizing backpropagation and stochastic gradient descent. The efficacy of our model is showcased through evaluations on a variety of benchmark datasets for sentence classification, where it consistently outperforms existing state-of-the-art approaches. These findings underscore the potency of CNNs in sentence classification and emphasize the significance of multi-perspective feature extraction in achieving superior performance.

In this paper we present a detailed scheme for annotating expressions of opinions, beliefs, emotions, sentiment and speculation (private states) in the news and other discourse. We explore inter-annotator agreement for individual private state expressions, and show that these low-level annotations are useful for producing higher-level subjective sentence annotations.

In this paper, we describe a Convolutional Neural Network (CNN) approach to realtime emotion detection. We utilize data from the Extended Cohn-Kanade dataset, Japanese Female Facial Expression data set, and our own custom images in order to train this model, and apply preprocessing steps to improve performance. We re-train a LeNet and AlexNet implementation, both of which perform with above 97% accuracy. Qualitative analysis of real-time images shows that the above models perform reasonably well at classifying facial expressions, but not as well as the quantitative results would indicate.

There are numerous studies suggesting that published news stories have an important effect on the direction of the stock market, its volatility, the volume of trades, and the value of individual stocks mentioned in the news. There is even some published research suggesting that automated sentiment analysis of news documents, quarterly reports, blogs and/or twitter data can be productively used as part of a trading strategy. This paper presents just such a family of trading strategies, and then uses this application to re-examine some of the tacit assumptions behind how sentiment analyzers are generally evaluated, in spite of the contexts of their application. This discrepancy comes at a cost.

In present days, the social media and networking act as one of the key platforms for sharing information and opinions. Many people share ideas, express their view points and opinions on various topic of their interest. Social media text has rich information about the companies, their products and various services offered by them. In this research we focus exploring the association of sentiments of social media text and stock prices of a company. The tweets of several company has been extracted and performed sentiment classification using Naïve Bayes classifier and SVM classifier. To perform the classification, N-gram based feature vectors are constructed using important words of tweets. Further, the pattern of association between number of tweets which are positive or negative and stock prices has been explored. Motivated by such an association, the features related to tweets such as number of positive, negative, neutral tweets and total number of tweets are used to predict the stock market status using Support Vector Machine classifier.

State-of-the-art sequence labeling systems traditionally require large amounts of taskspecific knowledge in the form of handcrafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both wordand character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data preprocessing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks — Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both datasets — 97.55% accuracy for POS tagging and 91.21% F1 for NER.

Existing target-specific sentiment recognition methods consider only a single target per tweet, and have been shown to miss nearly half of the actual targets mentioned. We present a corpus of UK election tweets, with an average of 3.09 entities per tweet and more than one type of sentiment in half of the tweets. This requires a method for multi-target specific sentiment recognition, which we develop by using the context around a target as well as syntactic dependencies involving the target. We present results of our method on both a benchmark corpus of single targets and the multi-target election corpus, showing state-of-the art performance in both corpora and outperforming previous approaches to multi-target sentiment task as well as deep learning models for singletarget sentiment.

In this paper, we propose an approach to automatically detect sentiments on Twitter messages (tweets) that explores some characteristics of how tweets are written and meta-information of the words that compose these messages. Moreover, we leverage sources of noisy labels as our training data. These noisy labels were provided by a few sentiment detection websites over twitter data. In our experiments, we show that since our features are able to capture a more abstract representation of tweets, our solution is more effective than previous ones and also more robust regarding biased and noisy data, which is the kind of data provided by these

With the exponential growth of social media i.e. blogs and social networks, organizations and individual persons are increasingly using the number of reviews of these media for decision making about a product or service. Opinion mining detects whether the emotions of an opinion expressed by a user on Web platforms in natural language, is positive or negative. This paper presents extensive experiments to study the effectiveness of the classification of Spanish opinions in five categories: highly positive, highly negative, positive, negative and neutral, using the combination of the psychological and linguistic features of LIWC. LIWC is a text analysis software that enables the extraction of different psychological and linguistic features from natural language text. For this study, two corpora have been used, one about movies and one about technological products. Furthermore, we have conducted a comparative assessment of the performance of various classification techniques: J48, SMO and BayesNet, using precision, recall and F-measure metrics. All in all, findings have revealed that the positive and negative categories provide better results than the other categories. Finally, experiments on both corpora indicated that SMO produces better results than BayesNet and J48 algorithms, obtaining an F-measure of 90.4% and 87.2% in each domain.

Access to expressions of subjective personal posts increased with the popularity of Social Media. However, most of the work in sentiment analysis focuses on predicting only valence from text and usually targeted at a product, rather than affective states. In this paper, we introduce a new data set of 2895 Social Media posts rated by two psychologicallytrained annotators on two separate ordinal nine-point scales. These scales represent valence (or sentiment) and arousal (or intensity), which defines each post’s position on the circumplex model of affect, a well-established system for describing emotional states (Russell, 1980; Posner et al., 2005). The data set is used to train prediction models for each of the two dimensions from text which achieve high predictive accuracy – correlated at r = .65 with valence and r = .85 with arousal annotations. Our data set offers a building block to a deeper study of personal affect as expressed in social media. This can be used in applications such as mental illness detection or in automated large-scale psychological studies.

We present Sentiment Analyzer (SA) that extracts sentiment (or opinion) about a subject from online text documents. Instead of classifying the sentiment of an entire document about a subject, SA detects all references to the given subject, and determines sentiment in each of the references using natural language processing (NLP) techniques. Our sentiment analysis consists of 1) a topic specific feature term extraction, 2) sentiment extraction, and 3) (subject, sentiment) association by relationship analysis. SA utilizes two linguistic resources for the analysis: the sentiment lexicon and the sentiment pattern database. The performance of the algorithms was verified on online product review articles (“digital camera” and “music” reviews), and more general documents including general webpages and news articles.


Question: Deep Neural Network-based method for sentiment classification.

This study presents a set of experiments employing convolutional neural networks (CNNs) with pre-trained word vectors for sentence classification tasks. The results reveal that a straightforward CNN, requiring minimal hyperparameter adjustment and utilizing static word vectors, attains exceptional performance on various benchmarks. The introduction of task-specific vectors through fine-tuning yields additional improvements. Moreover, the paper introduces a minor architectural adjustment to facilitate the integration of both task-specific and static vectors. The CNN models, as outlined in this work, outperform existing state-of-the-art methods in 4 out of 7 tasks, encompassing sentiment analysis and question classification.

Mining is used to help people to extract valuable information from large amount of data. Sentiment analysis focuses on the analysis and understanding of the emotions from the text patterns. It identifies the opinion or attitude that a person has towards a topic or an object and it seeks to identify the viewpoint underlying a text span. Sentiment analysis is useful in social media monitoring to automatically characterize the overall feeling or mood of consumers as reflected in social media toward a specific brand or company and determine whether they are viewed positively or negatively on the web. This new form of analysis has been widely adopted in customer relation management especially in the context of complaint management. For automating the task of classifying a single topic textual review, document-level sentiment classification is used for expressing a positive or negative sentiment. So analyzing sentiment using Multi-theme document is very difficult and the accuracy in the classification is less. The document level classification approximately classifies the sentiment using Bag of words in Support Vector Machine (SVM) algorithm. In proposed work, a new algorithm called Sentiment Fuzzy Classification algorithm with parts of speech tags is used to improve the classification accuracy on the benchmark dataset of Movies reviews dataset.

This paper compares the performance of two recurrent neural network models learning the grammar of simple English sentences and introduces a method for creating useful negative training examples from a set of grammatical sentences. The learning task is cast as a classification problem as opposed to a prediction problem; networks are taught to classify a given sequence as grammatical or ungrammatical. The two neural network models compared are the simple architecture proposed by Elman in [1], and the Long Short-Term Memory (LSTM) network from [4]. For comparison, a Naive Bayes (NB) classifier is also trained with bigrams to accomplish this task. Only the Elman network learns the training data better than NB, and neither network generalizes better than NB on sentences longer than the training sentences. The LSTM network does not generalize as well as the Elman network on a simple regular grammar, but shows comparable ability to generalize knowledge of English Grammar.

This paper describes the SemEval 2016 shared task on Aspect Based Sentiment Analysis (ABSA), a continuation of the respective tasks of 2014 and 2015. In its third year, the task provided 19 training and 20 testing datasets for 8 languages and 7 domains, as well as a common evaluation procedure. From these datasets, 25 were for sentence-level and 14 for text-level ABSA; the latter was introduced for the first time as a subtask in SemEval. The task attracted 245 submissions from 29 teams.

Deep learning is popular as an end-to-end framework extracting the prominent features and performing the classification also. In this paper, we extensively investigate deep networks as an alternate to feature encoding technique of lowlevel descriptors for emotion recognition on the benchmark EmoDB dataset. Fusion performance with such obtained encoded features with other available features is also investigated. Highest performance to date in the literature is observed.

With people increasingly using emoticons in written text on the Web in order to express, stress, or disambiguate their sentiment, it is crucial for automated sentiment analysis tools to correctly account for such graphical cues for sentiment. We analyze how emoticons typically convey sentiment and we subsequently propose and evaluate a novel method for exploiting this with a manually created emoticon sentiment lexicon in a lexicon-based polarity classification method. We evaluate our approach on 2,080 Dutch tweets and forum messages, which all contain emoticons. We validate our findings on 10,069 English reviews of apps, some of which contain emoticons. We find that accounting for the sentiment conveyed by emoticons on a paragraph level – and, to a lesser extent, on a sentence level – significantly improves polarity classification performance. Whenever emoticons are used, their associated sentiment tends to dominate the sentiment conveyed by textual cues and forms a good proxy for the polarity of text.

In this paper, we propose a system for aspect-based sentiment analysis (ABSA) by incorporating the concepts of multi-objective optimization (MOO), distributional thesaurus (DT) and unsupervised lexical induction. The task can be thought of as a sequence of processes such as aspect term extraction, opinion target expression identification and sentiment classification. We use MOO for selecting the most relevant features, and demonstrate that classification with the resulting feature set can improve classification accuracy on many datasets. As base learning algorithms we make use of Support Vector Machines (SVM) for sentiment classification and Conditional Random Fields (CRF) for aspect term and opinion target expression extraction tasks. Distributional thesaurus and unsupervised DT prove to be effective with enhanced performance. Experiments on benchmark setups of SemEval-2014 and SemEval-2016 shared tasks show that we achieve the state of the art on aspect-based sentiment analysis for several languages.

*This work is supported by a grant from National Science Foundation (IIS-1111092) and a strategic research grant from City University of Hong Kong (7002770) *The work was done when the first author was visiting University of Illinois at Chicago. Abstract • The Web has seen a tremendous rise in social media. • People’s experiences and opinions expressed in Social Media (e.g., Twitter, Facebook) contain rich social signals (e.g., about consumer sentiment, stock market, economic conditions, etc.). • The goal of this paper is to exploit public sentiment for stock market prediction. • Topic based sentiment is an important factor in stock prediction because it reflects public opinions about current issues/topics. • We employ a Bayesian non-parametric topic-based sentiment time series approach learned from streaming tweets to predict the S&P100 Stock Index.

Emoticons (e.g., :) and :( ) have been widely used in sentiment analysis and other NLP tasks as features to machine learning algorithms or as entries of sentiment lexicons. In this paper, we argue that while emoticons are strong and common signals of sentiment expression on social media the relationship between emoticons and sentiment polarity are not always clear. Thus, any algorithm that deals with sentiment polarity should take emoticons into account but extreme caution should be exercised in which emoticons to depend on. First, to demonstrate the prevalence of emoticons on social media, we analyzed the frequency of emoticons in a large recent Twitter data set. Then we carried out four analyses to examine the relationship between emoticons and sentiment polarity as well as the contexts in which emoticons are used. The first analysis surveyed a group of participants for their perceived sentiment polarity of the most frequent emoticons. The second analysis examined clustering of words and emoticons to better understand the meaning conveyed by the emoticons. The third analysis compared the sentiment polarity of microblog posts before and after emoticons were removed from the text. The last analysis tested the hypothesis that removing emoticons from text hurts sentiment classification by training two models with and without emoticons in the text, respectively. The results confirms the arguments that: 1) a few emoticons are strong and reliable signals of sentiment polarity and one should take advantage of them in any sentiment analysis; 2) a large group of the emoticons conveys complicated sentiment hence they should be treated with extreme caution.

The volume of discussion about a product in weblogs has recently been shown to correlate with the product’s financial performance. In this paper, we study whether applying sentiment analysis methods to weblog data results in better correlation than volume only, in the domain of movies. Our main finding is that positive sentiment is indeed a better predictor for movie success when applied to a limited context around references to the movie in weblogs, posted prior to its release. If my film makes one more person miserable, I’ve done my job.

The Internet is frequently used as a medium for exchange of information and opinions, as well as propaganda dissemination. In this study the use of sentiment analysis methodologies is proposed for classification of Web forum opinions in multiple languages. The utility of stylistic and syntactic features is evaluated for sentiment classification of English and Arabic content. Specific feature extraction components are integrated to account for the linguistic characteristics of Arabic. The entropy weighted genetic algorithm (EWGA) is also developed, which is a hybridized genetic algorithm that incorporates the information-gain heuristic for feature selection. EWGA is designed to improve performance and get a better assessment of key features. The proposed features and techniques are evaluated on a benchmark movie review dataset and U.S. and Middle Eastern Web forum postings. The experimental results using EWGA with SVM indicate high performance levels, with accuracies of over 91% on the benchmark dataset as well as the U.S. and Middle Eastern forums. Stylistic features significantly enhanced performance across all testbeds while EWGA also outperformed other feature selection methods, indicating the utility of these features and techniques for document-level classification of sentiments.