Question: Estimating absolute head pose from wide-angle overhead cameras.

The Carnegie Mellon University Pose, Illumination, and Expression (CMU PIE) Dataset
During the period from October 2000 to December 2000, we amassed a comprehensive dataset comprising over 40,000 facial images of 68 individuals. This dataset, known as the CMU Pose, Illumination, and Expression (PIE) database, was gathered using the advanced CMU 3D Room imaging facility. Each subject was photographed across a diverse range of 13 distinct poses, under varied lighting conditions totaling 43, and across four different expressions. The purpose of this paper is to provide a detailed account of the imaging equipment utilized, the methodology employed during the data collection phase, the systematic organization of the database, potential applications of the dataset, and instructions on how to access the database for research purposes. The CMU PIE dataset serves as a valuable resource for researchers in the fields of computer vision, machine learning, and facial recognition, offering a rich and diverse set of facial images that can be used to train and test algorithms for various applications, including absolute head pose estimation from overhead wide-angle cameras.

This paper addresses the task of estimating the 6D-pose of a known 3D object from a single RGB-D image. Most modern approaches solve this task in three steps: i) compute local features, ii) generate a pool of pose-hypotheses, iii) select and refine a pose from the pool. This work focuses on the second step. While all existing approaches generate the hypotheses pool via local reasoning, e.g. RANSAC or Hough-Voting, we are the first to show that global reasoning is beneficial at this stage. In particular, we formulate a novel fully-connected Conditional Random Field (CRF) that outputs a very small number of pose-hypotheses. Despite the potential functions of the CRF being non-Gaussian, we give a new, efficient two-step optimization procedure, with some guarantees for optimality. We utilize our global hypotheses generation procedure to produce results that exceed state-of-the-art for the challenging Occluded Object Dataset.

There have been increasing demands for developing microaerial vehicles with vision-based autonomy for search and rescue missions in complex environments. In particular, the monocular visual–inertial system (VINS), which consists of only an inertial measurement unit (IMU) and a camera, forms a great lightweight sensor suite due to its low weight and small footprint. In this paper, we address two challenges for rapid deployment of monocular VINS: 1) the initialization problem and 2) the calibration problem. We propose a methodology that is able to initialize velocity, gravity, visual scale, and camera–IMU extrinsic calibration on the fly. Our approach operates in natural environments and does not use any artificial markers. It also does not require any prior knowledge about the mechanical configuration of the system. It is a significant step toward plug-and-play and highly customizable visual navigation for mobile robots. We show through online experiments that our method leads to accurate calibration of camera–IMU transformation, with errors less than 0.02 m in translation and 1° in rotation. We compare out method with a state-of-the-art marker-based offline calibration method and show superior results. We also demonstrate the performance of the proposed approach in large-scale indoor and outdoor experiments.

This paper proposes the design of a custom mirror-based light field camera adapter that is cheap, simple in construction, and accessible. Mirrors of different shape and orientation reflect the scene into an upwards-facing camera to create an array of virtual cameras with overlapping field of view at specified depths, and deliver video frame rate light fields. We describe the design, construction, decoding and calibration processes of our mirrorbased light field camera adapter in preparation for an open-source release to benefit the robotic vision community. The latest report, computer-aided design models, diagrams and code can be obtained from the following repository: https://bitbucket.org/acrv/mirrorcam.

We present a system that transforms a monocular video of a soccer game into a moving 3D reconstruction, in which the players and field can be rendered interactively with a 3D viewer or through an Augmented Reality device. At the heart of our paper is an approach to estimate the depth map of each player, using a CNN that is trained on 3D player data extracted from soccer video games. We compare with state of the art body pose and depth estimation techniques, and show results on both synthetic ground truth benchmarks, and real YouTube soccer footage.

With the increasing rise of professionalism in sport, athletes, teams, and coaches are looking to technology to monitor performance in both games and training in order to find a competitive advantage. The use of inertial sensors has been proposed as a cost effective and adaptable measurement device for monitoring wheelchair kinematics; however, the outcomes are dependent on the reliability of the processing algorithms. Though there are a variety of algorithms that have been proposed to monitor wheelchair propulsion in court sports, they all have limitations. Through experimental testing, we have shown the Attitude and Heading Reference System (AHRS)-based algorithm to be a suitable and reliable candidate algorithm for estimating velocity, distance, and approximating trajectory. The proposed algorithm is computationally inexpensive, agnostic of wheel camber, not sensitive to sensor placement, and can be embedded for real-time implementations. The research is conducted under Griffith University Ethics (GU Ref No: 2016/294).

In this paper, we describe how to establish an embedded framework for real-time top-view people counting. The development of our system consists of two parts, i.e. establishing an embedded signal processing platform and designing a people counting algorithm for the embedded system. For the hardware platform construction, we use Kinect as the camera and exploit NVIDIA Jetson TK1 board as the embedded processing platform. We describe how to build a channel to make Kinect for windows version 2.0 communicate with Jetson TK1. Based on the embedded system, we adapt a water filling based scheme for top-view people counting, which integrates head detection based on water drop, people tracking and counting. Gaussian Mixture Model is used to construct and update the background model. The moving people in each video frame are extracted using background subtraction method. Additionally, the water filling algorithm is used to segment head area as Region Of Interest(ROI). Tracking and counting people are performed by calculating the distance of ROI center point before and after the frame. The whole framework is flexible and practical for real-time application.

The concept of active tracking is presented to simulate the characteristics of human vision in intelligent visual surveillance. The Pan/Tilt/Zoom (PTZ) camera is generally used for active tracking. In this paper, we present a novel and effective approach for active object tracking with a PTZ camera, and construct a near real-time system for indoor and outdoor scenes. The tracking algorithm of our system is based on the feature matching, with the PID control to drive the camera. The feature extracted from moving people is described as a region covariance matrix which combines the spatial and statistical properties of the targets (e.g. coordinates, color, and gradient). Results from indoor and outdoor experiments demonstrate the effectiveness and accuracy of our approach.

This paper describes a general method for estimating the nominal relationship and expected error (covariance) between coordinate frames representing the relative locations of objects. The frames may be known only indirectly through a series of spatial relationships, each with its associated error, arisingfrom diverse causes, including positioning errors, measurement errors, or tolerances in part dimensions. This estimation method can be used to answer such questions as whether a camera attached, to a robot is likely to have a particular reference object in its field of view. The calculated estimates agree well with those/rom an independent Monte Carlo simulation. The method makes it possible to decide in advance whether an uncertain relationship is known accurately enough for some task and, i f not, how much of an improvement in locational knowledge a proposed sensor will provide. The method presented can be generalized to six degrees of freedom and provides a practical means of estimating the relationships (position and orientation) among objects, as well as estimating the uncertainty associated with the relationships.

An estimation algorithm is developed for determining pitch and roll angles (tilt) of a rigid body fixed at a pivot point using multiple accelerometers. The estimate is independent of the rigid body dynamics; the method is applicable both in static conditions and for any dynamic motion of the body. No dynamic model is required for the estimator; only the mounting positions of the sensors need to be known. The proposed estimator is the optimal linear estimate in a least-squares sense if knowledge of the system dynamics is not used. The estimate may be used as a basis for further filtering and fusion techniques, such as sensor fusion with rate gyro data. The estimation algorithm is applied to the problem of state estimation for the Balancing Cube, a rigid structure that can actively balance on its corners. Experimental results are provided.

This work addresses the challenging problem of unconstrained 3D human pose estimation (HPE) from a novel perspective. Existing approaches struggle to operate in realistic applications, mainly due to their scene-dependent priors, such as background segmentation and multi-camera network, which restrict their use in unconstrained environments. We therfore present a framework which applies action detection and 2D pose estimation techniques to infer 3D poses in an unconstrained video. Action detection offers spatiotemporal priors to 3D human pose estimation by both recognising and localising actions in space-time. Instead of holistic features, e.g. silhouettes, we leverage the flexibility of deformable part model to detect 2D body parts as a feature to estimate 3D poses. A new unconstrained pose dataset has been collected to justify the feasibility of our method, which demonstrated promising results, significantly outperforming the relevant state-of-the-arts.


Question: Estimating absolute head pose using overhead wide-angle cameras.

This paper introduces an innovative framework for overcoming the challenges of object recognition under changing pose and illumination conditions. Traditionally, appearance-based recognition methods have struggled with the variability introduced by these factors. However, the proposed approach requires significantly fewer training images to construct a comprehensive representation that effectively models pose and illumination changes. Specifically, the method can reconstruct a 3D surface and an albedo map from as little as three images of an object in a static pose, captured under slightly varying but unspecified lighting conditions. These reconstructed features are subsequently utilized to synthesize a large number of images with extensive variations in pose and illumination, thereby creating a robust representation that enhances object recognition capabilities. The effectiveness of the method has been evaluated in the context of face recognition, using a subset of the Yale Face Database B, which comprises 4050 images of 10 distinct faces under diverse pose and lighting conditions. This database was specifically compiled to test the performance of the generative methods presented. Comparative analysis reveals that the proposed methods outperform conventional techniques in terms of recognition accuracy.

This paper proposes a method for detecting moving obstacles on roads, by using a vehicle mounted monocular camera. To detect various moving obstacles, such as vehicles and pedestrians, the ego-motion of the vehicle is initially estimated from images captured by the camera. There are two problems in ego-motion estimation. Firstly, a typical road scene contains moving obstacles. This causes false estimation of the ego-motion. Secondly, roads possess fewer features, when compared to the number associated with background structures. This reduces the accuracy of ego-motion estimation. In our approach, the ego-motion is estimated from the correspondences of dispersed feature points extracted from various regions other than those that contain moving obstacles. After estimating the ego-motion, any moving obstacles are detected by tracking the feature points over consecutive frames. In our experiments, it has been shown that the proposed method is able to detect moving obstacles

Positioning is a fundamental issue in mobile robot applications, and it can be achieved in multiple ways. Among these methods, triangulation based on angle measurements is widely used, robust, accurate, and flexible. This paper presents BeAMS, which is a new active beacon-based angle measurement system used for mobile robot positioning. BeAMS introduces several major innovations. One innovation is the use of a unique unsynchronized channel with on-off keying modulated infrared signals to measure angles and to identify the beacons. We also introduce a new mechanism to measure angles: Our system detects a beacon when it enters and leaves an angular window. We show that the estimator resulting from the center of this angular window provides an unbiased estimate of the beacon angle. A theoretical framework for a thorough performance analysis of BeAMS is provided. We establish the upper bound of the variance and validate this bound through experiments and simulations; the overall error measure of BeAMS is lower than 0.24° for an acquisition rate of 10 Hz. In conclusion, BeAMS is a low-power, flexible, and robust solution for angle measurement and a reliable component for robot positioning.

We present an overall framework of services for indoor navigation, which includes Indoor Mapping, Indoor Positioning, Path Planning, and En-route Assistance. Within such framework we focus on an augmented reality (AR) solution for en-route assistance. AR assists the user walking in a multi-floor building by displaying a directional arrow under a camera view, thus freeing the user from knowing his/her position. Our AR solution relies on geomagnetic positioning and north-oriented space coordinates transformation. Therefore, it can work without infrastructure and without relying on GPS. The AR visual interface and the integration with magnetic positioning is the main novelty of our solution, which has been validated by experiments and shows a good performance.

In immersive virtual environments (IVEs), users can control their virtual viewpoint by moving their tracked head and walking through the real world. Usually, movements in the real world are mapped one-to-one to virtual camera motions. With redirection techniques, the virtual camera is manipulated by applying gains to user motion so that the virtual world moves differently than the real world. Thus, users can walk through large-scale IVEs while physically remaining in a reasonably small workspace. In psychophysical experiments with a two-alternative forced-choice task, we have quantified how much humans can unknowingly be redirected on physical paths that are different from the visually perceived paths. We tested 12 subjects in three different experiments: (E1) discrimination between virtual and physical rotations, (E2) discrimination between virtual and physical straightforward movements, and (E3) discrimination of path curvature. In experiment E1, subjects performed rotations with different gains, and then had to choose whether the visually perceived rotation was smaller or greater than the physical rotation. In experiment E2, subjects chose whether the physical walk was shorter or longer than the visually perceived scaled travel distance. In experiment E3, subjects estimate the path curvature when walking a curved path in the real world while the visual display shows a straight path in the virtual world. Our results show that users can be turned physically about 49 percent more or 20 percent less than the perceived virtual rotation, distances can be downscaled by 14 percent and upscaled by 26 percent, and users can be redirected on a circular arc with a radius greater than 22 m while they believe that they are walking straight.

A novel method is proposed for the calibration of a camera - 3D lidar pair without the use of any special calibration pattern or point correspondences. The proposed method has no specific assumption about the data source: plain depth information is expected from the lidar scan and a simple perspective camera is used for the 2D images. The calibration is solved as a 2D-3D registration problem using a minimum of one (for extrinsic) or two (for intrinsic-extrinsic) planar regions visible in both cameras. The registration is then traced back to the solution of a non-linear system of equations which directly provides the calibration parameters between the bases of the two sensors. The method has been tested on a large set of synthetic lidar-camera image pairs as well as on real data acquired in outdoor environment.

Multi-view structure from motion (SfM) estimates the position and orientation of pictures in a common 3D coordinate frame. When views are treated incrementally, this external calibration can be subject to drift, contrary to global methods that distribute residual errors evenly. We propose a new global calibration approach based on the fusion of relative motions between image pairs. We improve an existing method for robustly computing global rotations. We present an efficient a contrario trifocal tensor estimation method, from which stable and precise translation directions can be extracted. We also define an efficient translation registration method that recovers accurate camera positions. These components are combined into an original SfM pipeline. Our experiments show that, on most datasets, it outperforms in accuracy other existing incremental and global pipelines. It also achieves strikingly good running times: it is about 20 times faster than the other global method we could compare to, and as fast as the best incremental method. More importantly, it features better scalability properties.

Recent advances with Convolutional Networks (ConvNets) have shifted the bottleneck for many computer vision tasks to annotated data collection. In this paper, we present a geometry-driven approach to automatically collect annotations for human pose prediction tasks. Starting from a generic ConvNet for 2D human pose, and assuming a multi-view setup, we describe an automatic way to collect accurate 3D human pose annotations. We capitalize on constraints offered by the 3D geometry of the camera setup and the 3D structure of the human body to probabilistically combine per view 2D ConvNet predictions into a globally optimal 3D pose. This 3D pose is used as the basis for harvesting annotations. The benefit of the annotations produced automatically with our approach is demonstrated in two challenging settings: (i) fine-tuning a generic ConvNet-based 2D pose predictor to capture the discriminative aspects of a subjects appearance (i.e.,personalization), and (ii) training a ConvNet from scratch for single view 3D human pose prediction without leveraging 3D pose groundtruth. The proposed multi-view pose estimator achieves state-of-the-art results on standard benchmarks, demonstrating the effectiveness of our method in exploiting the available multi-view information.

The goal of our work is to provide a fast and accurate method to estimate the camera motion from RGB-D images. Our approach registers two consecutive RGB-D frames directly upon each other by minimizing the photometric error. We estimate the camera motion using non-linear minimization in combination with a coarse-to-fine scheme. To allow for noise and outliers in the image data, we propose to use a robust error function that reduces the influence of large residuals. Furthermore, our formulation allows for the inclusion of a motion model which can be based on prior knowledge, temporal filtering, or additional sensors like an IMU. Our method is attractive for robots with limited computational resources as it runs in real-time on a single CPU core and has a small, constant memory footprint. In an extensive set of experiments carried out both on a benchmark dataset and synthetic data, we demonstrate that our approach is more accurate and robust than previous methods. We provide our software under an open source license.

We explore the feasibility of measuring learner engagement and classifying the engagement level based on machine learning applied on data from 2D/3D camera sensors and eye trackers in a 1:1 learning setting. Our results are based on nine pilot sessions held in a local high school where we recorded features related to student engagement while consuming educational content. We label the collected data as Engaged or NotEngaged while observing videos of the students and their screens. Based on the collected data, perceptual user features (e.g., body posture, facial points, and gaze) are extracted. We use feature selection and classification methods to produce classifiers that can detect whether a student is engaged or not. Accuracies of up to 85-95% are achieved on the collected dataset. We believe our work pioneers in the successful classification of student engagement based on perceptual user features in a 1:1 authentic learning setting.

The need to generate new views of a 3D object from a single real image arises in several fields, including graphics and°object recognition. While the traditional approach relies on the use of 3D models, we have recently introduced [11, 6, 5] techniques that are applicable under restricted conditions but simpler. The approach exploits image transformations that are specific to the relevant object class and learnable from example views of other "prototypical" objects of the same class. In this paper, we introduce such a new technique by extending the notion of linear class first proposed by Poggio and Vetter [12]. For linear object classes it is shown that linear transformations can be learned exactly from a basis set of 2D prototypical views. We demonstrate the approach on artificial objects and then show preliminary evidence that the technique can effectively "rotate" high-resolution face images from a single 2D view.


Question: Absolute determination of head pose from overhead wide-angle camera systems.

This paper describes a method for automatically detecting and tracking moving vehicles within monocular image sequences captured in road traffic scenes by a stationary camera. To effectively utilize the inherent knowledge about the shape and motion characteristics of vehicles in such environments, a parameterized vehicle model is employed in an intraframe matching procedure. Additionally, a recursive estimator based on a motion model is integrated for motion estimation, enhancing the tracking accuracy. The process is further refined by an interpretation cycle that supports the intraframe matching with a state Maximum A Posteriori (MAP) update step, improving the robustness of the tracking. The generation of initial model hypotheses is facilitated by an image segmentation component that clusters coherently moving image features, identifying potential representations of moving vehicles. By incorporating an illumination model, the system can account for shadow edges of the vehicles during the matching process, a critical feature for tracking under complex lighting conditions. This comprehensive approach, combining various techniques, has allowed for the successful tracking of vehicles over extended periods (exceeding 400 frames) in real-world road traffic scenes, even under challenging illumination scenarios. The paper concludes with a presentation of results from diverse real-world traffic scenarios and outlines ongoing issues and directions for future research.

We present a new technique for calibrating ultrawide fisheye lens cameras by imposing the constraint that collinear points be rectified to be collinear, parallel lines to be parallel, and orthogonal lines to be orthogonal. Exploiting the fact that line fitting reduces to an eigenvalue problem in 3D, we do a rigorous perturbation analysis to obtain a practical calibration procedure. Doing experiments, we point out that spurious solutions exist if collinearity and parallelism alone are imposed. Our technique has many desirable properties. For example, no metric information is required about the reference pattern or the camera position, and separate stripe patterns can be displayed on a video screen to generate a virtual grid, eliminating the grid point extraction processing.

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

In this work, we describe a novel method for creating 3D models of persons freely moving in front of a consumer depth sensor and we show how they can be used for long-term person re-identification. For overcoming the problem of the different poses a person can assume, we exploit the information provided by skeletal tracking algorithms for warping every point cloud frame to a standard pose in real time. Then, the warped point clouds are merged together to compose the model. Re-identification is performed by matching body shapes in terms of whole point clouds warped to a standard pose with the described method. We compare this technique with a classification method based on a descriptor of skeleton features and with a mixed approach which exploits both skeleton and shape features. We report experiments on two datasets we acquired for RGB-D re-identification which use different skeletal tracking algorithms and which are made publicly available to foster research in this new research branch.

A novel scene reconstruction technique is presented, different from previous approaches in its ability to cope with large changes in visibility and its modeling of intrinsic scene color and texture information. The method avoids image correspondence problems by working in a discretized scene space whose voxels are traversed in a fixed visibility ordering. This strategy takes full account of occlusions and allows the input cameras to be far apart and widely distributed about the environment. The algorithm identifies a special set of invariant voxels which together form a spatial and photometric reconstruction of the scene, fully consistent with the input images. The approach is evaluated with images from both inward-facing and outward-facing cameras.

While intelligent environments are often cited as a reason for doing work on visual person-tracking, really making an intelligent environment exposes many realworld problems in visual tracking that must be solved to make the technology practical. In the context of our EasyLiving project in intelligent environments, we created a practical person-tracking system that solves most of the real-world problems. It uses two sets of color stereo cameras for tracking multiple people during live demonstrations in a living room. The stereo images are used for locating people, and the color images are used for maintaining their identities. The system runs quickly enough to make the room feel responsive, and it tracks multiple people standing, walking, sitting, occluding, and entering and leaving the

In this paper, we present a flexible new technique for single viewpoint omnidirectional camera calibration. The proposed method only requires the camera to observe a planar pattern shown at a few different orientations. Either the camera or the planar pattern can be freely moved. No a priori knowledge of the motion is required, nor a specific model of the omnidirectional sensor. The only assumption is that the image projection function can be described by a Taylor series expansion whose coefficients are estimated by solving a two-step least-squares linear minimization problem. To test the proposed technique, we calibrated a panoramic camera having a field of view greater than 200 in the vertical direction, and we obtained very good results. To investigate the accuracy of the calibration, we also used the estimated omni-camera model in a structure from motion experiment. We obtained a 3D metric reconstruction of a scene from two highly distorted omnidirectional images by using image correspondences only. Compared with classical techniques, which rely on a specific parametric model of the omnidirectional camera, the proposed procedure is independent of the sensor, easy to use, and flexible.

There have been increasing demands for developing robotic system combining camera and inertial measurement unit in navigation task, due to their low-cost, lightweight and complementary properties. In this paper, we present a Visual Inertial Odometry (VIO) system which can utilize sparse depth to estimate 6D pose in GPS-denied and unstructured environments. The system is based on Multi-State Constraint Kalman Filter (MSCKF), which benefits from low computation load when compared to optimization-based method, especially on resource-constrained platform. Features are enhanced with depth information forming 3D landmark position measurements in space, which reduces uncertainty of position estimate. And we derivate measurement model to access compatibility with both 2D and 3D measurements. In experiments, we evaluate the performance of the system in different in-flight scenarios, both cluttered room and industry environment. The results suggest that the estimator is consistent, substantially improves the accuracy compared with original monocular-based MSKCF and achieves competitive accuracy with other research.

In this work we propose a new CNN+LSTM architecture for camera pose regression for indoor and outdoor scenes. CNNs allow us to learn suitable feature representations for localization that are robust against motion blur and illumination changes. We make use of LSTM units on the CNN output in spatial coordinates in order to capture contextual information. This substantially enlarges the receptive field of each pixel leading to drastic improvements in localization performance. We provide extensive quantitative comparison of CNN-based vs SIFT-based localization methods, showing the weaknesses and strengths of each. Furthermore, we present a new large-scale indoor dataset with accurate ground truth from a laser scanner. Experimental results on both indoor and outdoor public datasets show our method outperforms existing deep architectures, and can localize images in hard conditions, e.g., in the presence of mostly textureless surfaces.

This paper presents a tracking system for ego-motion estimation which fuses vision and inertial measurements using EKF and UKF (Extended and Unscented Kalman Filters), where a comparison of their performance has been done. It also considers the multi-rate nature of the sensors: inertial sensing is sampled at a fast sampling frequency while the sampling frequency of vision is lower. the proposed approach uses a constant linear acceleration model and constant angular velocity model based on quaternions, which yields a non-linear model for states and a linear model in measurement equations. Results show that a significant improvement is obtained on the estimation when fusing both measurements with respect to just vision or just inertial measurements. It is also shown that the proposed system can estimate fast-motions even when vision system fails. Moreover, a study of the influence of the noise covariance is also performed, which aims to select their appropriate values at the tuning process. The setup is an end-effector mounted camera, which allow us to pre-define basic rotational and translational motions for validating results. KEY WORDS—vision and inertial, multi-rate systems, sensor fusion, tracking

Indoor localization has long been a goal of pervasive computing research. In this paper, we explore the possibility of determining user's location based on the camera images received from a smart phone. In our system, the smart phone is worn by the user as a pendant and images are periodically captured and transmitted over GPRS to a web server. The web server returns the location of the user by comparing the received images with images stored in a database. We tested our system inside the Computer Science department building. Preliminary results show that user's location can be determined correctly with more than 80% probability of success. As opposed to earlier solutions for indoor localization, this approach does not have any infrastructure requirements. The only cost is that of building an image database.


Question: Estimating absolute head pose from wide-angle overhead cameras.

This document introduces a cutting-edge object detection framework specifically tailored for identifying pedestrians within cluttered, static images. This task presents significant challenges due to the high variability in human appearance regarding size, shape, color, and texture. Unlike conventional methods, the proposed system does not depend on preconceived models or motion cues; instead, it learns from a dataset of examples. The core of the detection mechanism is the innovative concept of the wavelet template, which characterizes the object's shape through a selective set of wavelet coefficients from the image. This method is inherently robust to changes in color and texture, enabling it to accurately define a broad and intricate class of objects, including humans. The paper demonstrates the efficacy of the wavelet template for object detection by highlighting its invariant properties and computational efficiency, which together make it a powerful tool for tackling this complex problem.

This work explores the feasibility of using sensors embedded in Google Glass, a head-mounted wearable device, to measure physiological signals of the wearer. In particular, we develop new methods to use Glass's accelerometer, gyroscope, and camera to extract pulse and respiratory rates of 12 participants during a controlled experiment. We show it is possible to achieve a mean absolute error of 0.83 beats per minute (STD: 2.02) for heart rate and 1.18 breaths per minute (STD: 2.04) for respiration rate when considering different combinations of sensors. These results included testing across sitting, supine, and standing still postures before and after physical exercise.

In this paper, we first review the approaches to recover 3D shape and related movements of a human and then we present an easy and reliable approach to recover a 3D model using just one image or monocular video sequence. A simplification of the perspective camera model is required, due to the absence of stereo view. The human figure is reconstructed in a skeleton form and to improve the visual quality, a pre-defined human model is also fitted to the recovered 3D data.

In this paper, we address the estimation, control, navigation and mapping problems to achieve autonomous inspection of penstocks and tunnels using aerial vehicles with on-board sensing and computation. Penstocks and tunnels have the shape of a generalized cylinder. They are generally dark and featureless. State estimation is challenging because range sensors do not yield adequate information and cameras do not work in the dark. We show that the six degrees of freedom (DOF) pose and velocity can be estimated by fusing information from an inertial measurement unit (IMU), a lidar and a set of cameras. This letter discusses in detail the range-based estimation part while leaving the details of vision component to our earlier work. The proposed algorithm relies only on a model of the generalized cylinder and is robust to changes in shape of the tunnel. The approach is validated through real experiments showing autonomous and shared control, state estimation and environment mapping in the penstock at Center Hill Dam, TN. To our knowledge, this is the first time autonomous navigation and mapping has been achieved in a penstock without any external infrastructure such GPS or external cameras.

SLAM is a very popular research stream in computer vision and robotics nowadays. For more effective SLAM implementation it is necessary to have reliable information about the environment, also the data should be aligned and scaled according to the real world coordinate system. Monocular SLAM research is an attractive sub-stream, because of the low equipment cost, size and weight. In this paper we present a way to build a conversion from LSD-SLAM coordinate space to the real world coordinates using a true metric scale with IMU sensor data implementation. The causes of differences between the real and calculated spaces are explained and the possibility of conversions between the spaces is proved. Additionally, a closed-form solution for inter space transformation calculation is presented. The synthetic method of generating high level accurate and well controlled input data for the LSD-SLAM algorithm is presented. Finally, the reconstructed 3D environment representation is delivered as an output of the implemented conversion.

Pose estimation of object is one of the key problems for the automatic-grasping task of robotics. In this paper, we present a new vision-based robotic grasping system, which can not only recognize different objects but also estimate their poses by using a deep learning model, finally grasp them and move to a predefined destination. The deep learning model demonstrates strong power in learning hierarchical features which greatly facilitates the recognition mission. We apply the Max-pooling Convolutional Neural Network (MPCNN), one of the most popular deep learning models, in this system, and assign different poses of objects as different classes in MPCNN. Besides, a new object detection method is also presented to overcome the disadvantage of the deep learning model. We have built a database comprised of 5 objects with different poses and illuminations for experimental performance evaluation. The experimental results demonstrate that our system can achieve high accuracy on object recognition as well as pose estimation. And the vision-based robotic system can grasp objects successfully regardless of different poses and illuminations.

We propose a method for human pose estimation based on Deep Neural Networks (DNNs). The pose estimation is formulated as a DNN-based regression problem towards body joints. We present a cascade of such DNN regres- sors which results in high precision pose estimates. The approach has the advantage of reasoning about pose in a holistic fashion and has a simple but yet powerful formula- tion which capitalizes on recent advances in Deep Learn- ing. We present a detailed empirical analysis with state-of- art or better performance on four academic benchmarks of diverse real-world images.

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. RGB-D cameras rely on either structured light patterns combined with stereo sensing [6,10] or time-of-flight laser sensing [1] to generate depth estimates that can be associated with RGB pixels. Very soon, small, high-quality RGB-D cameras developed for computer gaming and home entertainment applications will become available at cost below $100. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. The robotics and computer vision communities have developed a variety of techniques for 3D mapping based on laser range scans [8, 11], stereo cameras [7], monocular cameras [3], and unsorted collections of photos [4]. While RGB-D cameras provide the opportunity to build 3D maps of unprecedented richness, they have drawbacks that make their application to 3D mapping difficult: They provide depth only up to a limited distance (typically less than 5m), depth values are much noisier than those provided by laser scanners, and their field of view (∼ 60◦) is far more constrained than that of specialized cameras or laser scanners typically used for 3D mapping (∼ 180◦). In our work, we use a camera developed by PrimeSense [10]. The key insights of this investigation are: first, that existing frame matching techniques are not sufficient to provide robust visual odometry with these cameras; second, that a tight integration of depth and color information can yield robust frame matching and loop closure detection; third, that building on best practice techniques in SLAM and computer graphics makes it possible to build and visualize accurate and extremely rich 3D maps with such cameras; and, fourth, that it will be feasible to build complete robot navigation and interaction systems solely based on cheap depth cameras.

This paper presents a camera tracking system for the spatial stabilization of Augmented Reality (AR) media. Our approach integrates both artificial landmarks (fiducials) and natural features for camera tracking. Artificial landmarks are used for system initialization and computation of initial camera pose. Robust and extendible tracking is achieved by dynamically calibrating the 3D positions of a priori uncalibrated natural features. Analysis and experimental results demonstrate the effectiveness of this approach for presenting stabilized AR media in long camera-motion video sequences.

We explore the feasibility of measuring learner engagement and classifying the engagement level based on machine learning applied on data from 2D/3D camera sensors and eye trackers in a 1:1 learning setting. Our results are based on nine pilot sessions held in a local high school where we recorded features related to student engagement while consuming educational content. We label the collected data as Engaged or NotEngaged while observing videos of the students and their screens. Based on the collected data, perceptual user features (e.g., body posture, facial points, and gaze) are extracted. We use feature selection and classification methods to produce classifiers that can detect whether a student is engaged or not. Accuracies of up to 85-95% are achieved on the collected dataset. We believe our work pioneers in the successful classification of student engagement based on perceptual user features in a 1:1 authentic learning setting.

For applications such as augmented reality, autonomous driving, self-localization/camera pose estimation and scene parsing are crucial technologies. In this paper, we propose a unified framework to tackle these two problems simultaneously. The uniqueness of our design is a sensor fusion scheme which integrates camera videos, motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robustness and efficiency of the system. Specifically, we first have an initial coarse camera pose obtained from consumer-grade GPS/IMU, based on which a label map can be rendered from the 3D semantic map. Then, the rendered label map and the RGB image are jointly fed into a pose CNN, yielding a corrected camera pose. In addition, to incorporate temporal information, a multi-layer recurrent neural network (RNN) is further deployed improve the pose accuracy. Finally, based on the pose from RNN, we render a new label map, which is fed together with the RGB image into a segment CNN which produces perpixel semantic label. In order to validate our approach, we build a dataset with registered 3D point clouds and video camera images. Both the point clouds and the images are semantically-labeled. Each video frame has ground truth pose from highly accurate motion sensors. We show that practically, pose estimation solely relying on images like PoseNet [25] may fail due to street view confusion, and it is important to fuse multiple sensors. Finally, various ablation studies are performed, which demonstrate the effectiveness of the proposed system. In particular, we show that scene parsing and pose estimation are mutually beneficial to achieve a more robust and accurate system.


Question: Estimating absolute head pose from wide-angle overhead cameras.

This paper presents an enhanced methodology for the fitting of parameterized three-dimensional models to images, a critical component in model-based recognition and motion tracking systems. The proposed approach significantly expands upon existing parameter-solving techniques, enabling the accurate fitting of 3-D models to 2-D image features for objects with complex curved surfaces and an arbitrary number of internal parameters, which may represent articulations, variable dimensions, or surface deformations. To address the inherent inaccuracies in image measurements, the method incorporates advanced numerical stabilization strategies, ensuring the derivation of meaningful solutions even in scenarios with fewer matches than unknown parameters. A key aspect of this methodology is the utilization of the Levenberg-Marquardt algorithm, which guarantees the convergence of the solution, making it particularly robust and reliable. These innovations broaden the applicability of model-based vision techniques to a wider range of problems than previously feasible, as demonstrated through the successful tracking of curved, parameterized objects in motion. This work has been published in the IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 13, Issue 5 (May 1991), on pages 441-450, showcasing its contribution to the field of computer vision.

In this paper, we present a patch-based regression framework for addressing the human age and head pose estimation problems. Firstly, each image is encoded as an ensemble of orderless coordinate patches, the global distribution of which is described by Gaussian mixture models (GMM), and then each image is further expressed as a specific distribution model by Maximum a Posteriori adaptation from the global GMM. Then the patch-kernel is designed for characterizing the Kullback-Leibler divergence between the derived models for any two images, and its discriminating power is further enhanced by a weak learning process, called inter-modality similarity synchronization. Finally, kernel regression is employed for ultimate human age or head pose estimation. These three stages are complementary to each other, and jointly minimize the regression error. The effectiveness of this regression framework is validated by three experiments: 1) on the YAMAHA aging database, our solution brings a more than 50% reduction in age estimation error compared with the best reported results; 2) on the FG-NET aging database, our solution based on raw image features performs even better than the state-of-the-art algorithms which require fine face alignment for extracting warped appearance features; and 3) on the CHIL head pose database, our solution significantly outperforms the best one reported in the CLEAR07 evaluation.

We present an overall framework of services for indoor navigation, which includes Indoor Mapping, Indoor Positioning, Path Planning, and En-route Assistance. Within such framework we focus on an augmented reality (AR) solution for en-route assistance. AR assists the user walking in a multi-floor building by displaying a directional arrow under a camera view, thus freeing the user from knowing his/her position. Our AR solution relies on geomagnetic positioning and north-oriented space coordinates transformation. Therefore, it can work without infrastructure and without relying on GPS. The AR visual interface and the integration with magnetic positioning is the main novelty of our solution, which has been validated by experiments and shows a good performance.

Multiple cameras are increasingly prevalent on robotic and human-driven vehicles. These cameras come in a variety of wide-angle, fish-eye, and catadioptric models. Furthermore, wheel odometry is generally available on the vehicles on which the cameras are mounted. For robustness, vision applications tend to use wheel odometry as a strong prior for camera pose estimation, and in these cases, an accurate extrinsic calibration is required in addition to an accurate intrinsic calibration. To date, there is no known work on automatic intrinsic calibration of generic cameras, and more importantly, automatic extrinsic calibration of a rig with multiple generic cameras and odometry. We propose an easy-to-use automated pipeline that handles both intrinsic and extrinsic calibration; we do not assume that there are overlapping fields of view. At the begining, we run an intrinsic calibration for each generic camera. The intrinsic calibration is automatic and requires a chessboard. Subsequently, we run an extrinsic calibration which finds all camera-odometry transforms. The extrinsic calibration is unsupervised, uses natural features, and only requires the vehicle to be driven around for a short time. The intrinsic parameters are optimized in a final bundle adjustment step in the extrinsic calibration. In addition, the pipeline produces a globally-consistent sparse map of landmarks which can be used for visual localization. The pipeline is publicly available as a standalone C++ package.

The Walking-In-Place interaction technique was introduced to navigate infinitely in 3D virtual worlds by walking in place in the real world. The technique has been initially developed for users standing in immersive setups and was built upon sophisticated visual displays and tracking equipments.
 In this paper, we propose to revisit the whole pipeline of the Walking-In-Place technique to match a larger set of configurations and apply it notably to the context of desktop Virtual Reality. With our novel "Shake-Your-Head" technique, the user is left with the possibility to sit down, and to use small screens and standard input devices such as a basic webcam for tracking. The locomotion simulation can compute various motions such as turning, jumping and crawling, using as sole input the head movements of the user. We also introduce the use of additional visual feedback based on camera motions to enhance the walking sensations.
 An experiment was conducted to compare our technique with classical input devices used for navigating in desktop VR. Interestingly, the results showed that our technique could even allow faster navigations when sitting, after a short learning. Our technique was also perceived as more fun and increasing presence, and was generally more appreciated for VR navigation.

We present Gazture, a light-weight gaze based real-time gesture control system on commercial tablets. Unlike existing approaches that require dedicated hardware (e.g., high resolution camera), high computation overhead (powerful CPU) or specific user behavior (keeping head steady), Gazture provides gesture recognition based on easy-to-control user gaze input with a small overhead. To achieve this goal, Gazture incorporates a two-layer structure: The first layer focuses on real-time gaze estimation with acceptable tracking accuracy while incurring a small overhead. The second layer implements a robust gesture recognition algorithm while compensating gaze estimation error. To address user posture change while using mobile device, we design a online transfer function based method to convert current eye features into corresponding eye features in reference posture, which then facilitates efficient gaze position estimation. We implement Gazture on Lenovo Tab3 8 Plus tablet with Android 6.0.1, and evaluate its performance in different scenarios. The evaluation results show that Gazture can achieve a high accuracy in gesture recognition while incurring a low overhead.

A novel method is proposed for the calibration of a camera - 3D lidar pair without the use of any special calibration pattern or point correspondences. The proposed method has no specific assumption about the data source: plain depth information is expected from the lidar scan and a simple perspective camera is used for the 2D images. The calibration is solved as a 2D-3D registration problem using a minimum of one (for extrinsic) or two (for intrinsic-extrinsic) planar regions visible in both cameras. The registration is then traced back to the solution of a non-linear system of equations which directly provides the calibration parameters between the bases of the two sensors. The method has been tested on a large set of synthetic lidar-camera image pairs as well as on real data acquired in outdoor environment.

We propose a novel algorithm for uncalibrated photometric stereo. While most of previous methods rely on various assumptions on scene properties, we exploit constraints in lighting configurations. We first derive an ambiguous reconstruction by requiring lights to lie on a view centered cone. This reconstruction is upgraded to Euclidean by constraints derived from lights of equal intensity and multiple view geometry. Compared to previous methods, our algorithm deals with more general data and achieves high accuracy. Another advantage of our method is that we can model weak perspective effects of lighting, while previous methods often assume orthographical illumination. We use both synthetic and real data to evaluate our algorithm. We further build a hardware prototype to demonstrate our approach.

A novel gaze estimation method based on a novel aspherical model of the cornea is proposed in this paper. The model is a surface of revolution about the optical axis of the eye. The calculation method is explained on the basis of the model. A prototype system for estimating the point of gaze (POG) has been developed using this method. The proposed method has been found to be more accurate than the gaze estimation method based on a spherical model of the cornea.

We present a color calibration pipeline for large camera arrays. We assume static lighting conditions for each camera, such as studio lighting or a stationary array outdoors. We also assume we can place a planar calibration target so it is visible from every camera. Our goal is uniform camera color responses, not absolute color accuracy, so we match the cameras to each other instead of to a color standard. We first iteratively adjust the color channel gains and offsets for each camera to make their responses as similar as possible. This step white balances the cameras, and for studio applications, ensures that the range of intensities in the scene are mapped to the usable output range of the cameras. Residual errors are then calibrated in post-processing. We present results calibrating an array of 100 CMOS image sensors in different physical configurations, including closely or widely spaced cameras with overlapping fields of views, and tightly packed cameras with non-overlapping fields of view. The process is entirely automatic, and the camera configuration runs in less than five minutes on the 100 camera array. CR Categories: I.4.1 [Image Processing and Computer Vision]: Digitization and Image Capture.Camera Calibration;

In this paper we have proposed and designed FindingHuMo (Finding Human Motion), a real-time user tracking system for Smart Environments. FindingHuMo can perform device-free tracking of multiple (unknown and variable number of) users in the Hallway Environments, just from non-invasive and anonymous (not user specific) binary motion sensor data stream. The significance of our designed system are as follows: (a) fast tracking of individual targets from binary motion data stream from a static wireless sensor network in the infrastructure. This needs to resolve unreliable node sequences, system noise and path ambiguity, (b) Scaling for multi-user tracking where user motion trajectories may crossover with each other in all possible ways. This needs to resolve path ambiguity to isolate overlapping trajectories, FindingHumo applies the following techniques on the collected motion data stream: (i) a proposed motion data driven adaptive order Hidden Markov Model with Viterbi decoding (called Adaptive-HMM), and then (ii) an innovative path disambiguation algorithm (called CPDA). Using this methodology the system accurately detects and isolates motion trajectories of individual users. The system performance is illustrated with results from real-time system deployment experience in a Smart Environment.


Question: The impact of violent video games on young people and the consequences for public policy.

This comprehensive review examines the scientific literature on the effects of violent video games on various aspects of behavior and cognition. The accumulated research indicates that exposure to violent video games can lead to an increase in aggressive behavior among children and young adults. This finding is supported by both experimental and nonexperimental studies conducted in diverse settings, including laboratories and real-world environments, and is observed across genders. The review also highlights that playing violent video games can elevate physiological arousal and foster aggression-related thoughts and emotions. Additionally, it suggests that such exposure may diminish prosocial behavior, which refers to actions that benefit others. These findings have significant implications for understanding the potential impact of violent video games on youth development and may inform public policy discussions regarding media violence and its effects.

In recent years many parents, advocates and policy makers have expressed concerns regarding the potential negative impact of social media use. Some studies have indicated that social media use may be tied to negative mental health outcomes, including suicidality, loneliness and decreased empathy. Other studies have not found evidence for harm, or have indicated that social media use may be beneficial for some individuals. The current correlational study examined 467 young adults for their time spent using social media, importance of social media in their lives and tendency to engage in vaguebooking (posting unclear but alarming sounding posts to get attention). Outcomes considered included general mental health symptoms, suicidal ideation, loneliness, social anxiety and decreased empathy. Results indicated that social media use was not predictive of impaired mental health functioning. However, vaguebooking was predictive of suicidal ideation, suggesting this particular behavior could be a warning sign for serious issues. Overall, results from this study suggest that, with the exception of vaguebooking, concerns regarding social media use may be misplaced.

Digital games have caught the attention of scholars across a variety of disciplines. Today, scholars from fields as diverse as media design, literature, computer science, education, and theatre studies have together contributed to the understanding of this emergent medium and its phenomenon as a learning tool The unsatisfying experience of directly adopting game formats into educational contexts has motivated researchers to study successful (commercial) digital games, in particular to examine the play activities of digital games and the principles underlying the design of digital games in order to develop concepts and strategies for rich virtual learning contexts. In this review, provided is a brief account of academic arguments regarding digital games and learning.

Demographic change and the aging population push health and welfare system to its limits. Increased physical fitness and increased awareness for health issues will help elderly to live independently for longer and will thereby reduce the costs in the health care system. Exergames seem to be a promising solution for promoting physical fitness. Still, there is little evidence under what conditions Exergames will be accepted and used by elderly. To investigate promoting and hindering factors we conducted a user study with a prototype of an Exergame. We contrasted young vs. elderly players and investigated the role of gamer types, personality factors and technical expertise on the performance within the game and changes in the attitude towards individual health after the game. Surprisingly, performance within the game is not affected by performance motivation but by gamer type. More importantly, a universal positive effect on perceived pain is detected after the Exergame

Despite increasing evidence that shows action video game play improves perceptual and cognitive skills, the mechanisms of transfer are not well-understood. In line with previous work, we suggest that transfer is dependent upon common demands between the game and transfer task. In the current study, participants played one of four action games with varying speed, visual, and attentional demands for 20 h. We examined whether training enhanced performance for attentional blink, selective attention, attending to multiple items, visual search and auditory detection. Non-gamers who played the game (Modern Combat) with the highest demands showed transfer to tasks of attentional blink and attending to multiple items. The game (MGS Touch) with fewer attentional demands also decreased attentional blink, but to a lesser degree. Other games failed to show transfer, despite having many action game characteristics but at a reduced intensity. The results support the common demands hypothesis.


Question: The impact of violent video games on young people and the consequences for public policy.

Two investigations explored the impact of violent video games on aggression-related aspects in controlled settings and in real-life scenarios. The first study revealed a positive correlation between real-world engagement in violent video games and aggressive behavior, as well as delinquency. This relationship was more pronounced among individuals with a naturally aggressive disposition and among males. Moreover, academic performance was found to be inversely related to the total time spent on video games. In the second study, experimental exposure to a graphically violent video game was shown to escalate aggressive thoughts and actions. Across both studies, males exhibited a more adversarial worldview compared to females. The findings from these two studies align with the predictions of the General Affective Aggression Model, which posits that violent video game exposure can lead to an increase in aggressive behavior in both the immediate aftermath (e.g., laboratory aggression) and over extended periods (e.g., delinquency). This research underscores the significance of considering individual characteristics and gender when evaluating the effects of violent video games on aggression.


Question: The impact of violent video games on young people and the implications for public policy.

This study investigated the impact of video game practice on spatial abilities among young girls and boys. Prior to the intervention, spatial skills, assessed through two computerized spatial skills subtests, revealed a notable advantage in boys over girls. Participants were then divided into two groups: one practiced an action video game, Marble Madness, while the other engaged in a word game, Conjecture. The results indicated that video game practice significantly outperformed the word game in enhancing spatial performance post-intervention, with no significant difference based on gender. Notably, video game practice was particularly effective for those with initially weaker spatial skills. These findings suggest that video games may serve as an effective tool in minimizing individual disparities in spatial abilities, including those linked to gender differences. This research underscores the potential of video games to level the playing field in spatial skill development among youth.

As violence in movies has harmful influence on children, in this paper, we propose an algorithm to detect violent scene in movies. Under our definition of violence, the task of violent scene detection is decomposed into action scene detection and bloody frame detection. While previous approaches addressed on shot level of video structure only, our approach works on more semantic-complete scene structure of video. The input video (digital movie) is first segmented into several scenes. Based on the filmmaking characteristics of action scene, some features of the scene are extracted to feed into the support vector machine for classification. Finally, the face, blood and motion information are integrated to determine whether the action scene has violent content. Experimental results show that the proposed approach works reasonably well in detecting most of the violent scenes. Compared with related work, our approach is computationally simple yet effective.


Question: The impact of violent video games on young people and the implications for public policy.

The Impact of Violent Video Games on Youth and Public Policy Considerations

The discussion around violent video games and their effects on youth has been a topic of significant debate. This paper delves into the psychological mechanisms and public policy implications surrounding this issue. Much of the research on self-esteem has concentrated on the trait level, overlooking the dynamic aspects of self-worth that fluctuate in response to external events. William James, over a century ago, posited that self-esteem is a variable that adjusts according to successes and failures in domains deemed important by the individual. Building on James' theory, we propose a model of self-esteem that emphasizes the contingencies on which it is based. This model can elucidate the role of self-esteem in emotional, cognitive, and behavioral self-regulation, as well as its connection to social issues. It can also help to reconcile the debates around the nature and function of self-esteem, and explain paradoxes in related fields, such as why marginalized groups do not always exhibit low self-esteem and why self-esteem does not necessarily decrease with age. Furthermore, this perspective opens up questions about the acquisition and change of self-worth contingencies, their role as a strength or weakness, and the possibility of noncontingent self-esteem. In the context of violent video games, these considerations are crucial for understanding their impact on youth and formulating effective public policies.


Question: The impact of violent video games on young people and the implications for public policy.

This paper presents a cognitive-affective system theory of personality aimed at resolving the apparent contradiction between the stability of personality traits and the variability of behavior in different situations. The theory posits that individuals exhibit distinct differences in two key areas: the accessibility of cognitive-affective mediating components, including cognitive encodings, expectancies, beliefs, emotional states, and goals; and the relational organization through which these components interact with one another and with situational psychological characteristics. This framework explains individual variability in behavior across situations in a systematic manner (for example, if condition A is present, then behavior X is likely, whereas if condition B is present, behavior Y is more probable), as well as overall average behavioral levels. Both these patterns are considered essential expressions of the same underlying personality system. From this perspective, situations, personality traits, behavioral dynamics, and structural components are reinterpreted to better understand the complex interplay between personality and situational factors.

In the context of violent video games and their impact on youth, this theory can provide insights into how individual differences in cognitive-affective systems might influence reactions to violent content. For instance, a youth with a highly accessible aggression-related cognitive-affective component may exhibit heightened aggression after playing violent video games, whereas another youth with less accessible aggression-related components might not show significant changes in behavior. This variability in response can be understood as a manifestation of the individual's underlying personality system, where situational factors (such as exposure to violent video games) interact with pre-existing cognitive-affective structures to produce observable behavioral outcomes. The theory also suggests that average levels of aggression in youth can be indicative of more stable personality traits that are expressed differently across various situations, including video game play. Thus, understanding the dynamics of cognitive-affective systems can contribute to more nuanced public policy implications regarding the effects of violent video games on youth, emphasizing the importance of individual differences in shaping behavioral responses.

In recent years many parents, advocates and policy makers have expressed concerns regarding the potential negative impact of social media use. Some studies have indicated that social media use may be tied to negative mental health outcomes, including suicidality, loneliness and decreased empathy. Other studies have not found evidence for harm, or have indicated that social media use may be beneficial for some individuals. The current correlational study examined 467 young adults for their time spent using social media, importance of social media in their lives and tendency to engage in vaguebooking (posting unclear but alarming sounding posts to get attention). Outcomes considered included general mental health symptoms, suicidal ideation, loneliness, social anxiety and decreased empathy. Results indicated that social media use was not predictive of impaired mental health functioning. However, vaguebooking was predictive of suicidal ideation, suggesting this particular behavior could be a warning sign for serious issues. Overall, results from this study suggest that, with the exception of vaguebooking, concerns regarding social media use may be misplaced.

Many laboratory studies of aggression use a measure known as the modified Taylor Competitive Reaction Time Test (TCRTT), for which validation studies are lacking. Using sound blasts administered by the participant against a fictional human opponent, the TCRTT also allows for multiple methods of measuring aggression. The validity of the TCRTT was tested in 53 college student participants. Participants took a self-report measure of aggressiveness as well as neuropsychological measures of frontal lobe functioning predictive of aggression. Results were not supportive of the TCRTT’s validity and indicated concerns regarding the use of the TCRTT as a measure of aggression. Results suggest that labaratory studies of media violence using the TCRTT are of questionable validity.


Question: SarcasmBot: An open-source module for generating sarcasm in chatbots.

This paper chronicles the development of a significant linguistic resource: the Penn Treebank, an extensive annotated corpus of American English comprising over 4.5 million words. During the initial three-year phase of the Penn Treebank Project (1989-1992), the corpus was meticulously annotated for part-of-speech (POS) information. Moreover, a substantial portion of the corpus—over half—received skeletal syntactic structure annotations. The Penn Treebank stands as a testament to the University of Pennsylvania's Department of Computer and Information Science's commitment to advancing linguistic research. It is accessible through the ScholarlyCommons platform at the following address: http://repository.upenn.edu/cis_reports/237. The collaborative effort behind the Penn Treebank involved key contributors Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz from the University of Pennsylvania's School of Engineering and Applied Science, Department of Computer and Information Science, located in Philadelphia, PA 19104-6389. The Penn Treebank's comprehensive annotations have since become an indispensable resource for researchers in the fields of computational linguistics and natural language processing.

Past works on personality detection has shown that psycho-linguistic features, frequency based analysis at lexical level , emotive words and other lexical clues such as number of first person or sec ond person words carry major role to identify personality associated with the te xt. In this work, we propose a new architecture for the same task using common sen se knowledge with associated sentiment polarity and affective labels. To extract the common sense knowledge with sentiment polarity scores and affect iv labels we used Senticnet which is one of the most useful resources for opinion mining and sentiment analysis. In particular, we combined comm n sense knowledge based features with phycho-linguistic features and freque ncy based features and later the features were employed in supervised classifier . We designed five SMO based supervised classifiers for five personality t raits. We observe that the use of common sense knowledge with affective and sentim ent information enhances the accuracy of the existing frameworks which use o nly psycho-linguistic features and frequency based analysis at lexical le ve .

he aim of the paper is fourfold: (a) show why humor scholars should study irony, (b) explore the need for considering interindividual differences in healthy adults’ irony performance, (c) stress the necessity for developing tools assessing habitual differences in irony performance, and (d) indicate future directions for joint irony and humor research and outline possible applications. Verbal irony is often employed with a benevolent humorous intent by speakers, but can also serve as a means of disparagement humor. In both cases, encoding and decoding activities entailing irony need to be considered in the context of the psychology of humor. We argue that verbal irony performance can be considered a phenomenon native to the realm of humor and individual differences. We point out that research has widely neglected the meaningfulness of variance in irony performance within experimental groups when looking at determinants of irony detection and production. Based on theoretical considerations and previous empirical findings we show that this variance can be easily related to individual-differences variables such as the sense of humor, dispositions toward laughter and ridicule (e.g., gelotophobia), and general mental ability. Furthermore, we hypothesize that there is an enduring trait determining irony performance we will label the sense of irony. The sense of irony possibly goes along with inclinations toward specific affective and cognitive processing patterns when dealing with verbal irony. As an application, novel irony performance tests can help to study psychological and neurophysiological correlates of irony performance more feasibly, that is, in nonclinical groups. DOI: https://doi.org/10.1037/tps0000054 Posted at the Zurich Open Repository and Archive, University of Zurich ZORA URL: https://doi.org/10.5167/uzh-124094 Accepted Version Originally published at: Bruntsch, Richard; Hofmann, Jennifer; Ruch, Willibald (2016). Virgin soil in irony research: Personality, humor, and the “sense of irony”. Translational Issues in Psychological Science, 2(1):25-34. DOI: https://doi.org/10.1037/tps0000054 VIRGIN SOIL IN IRONY RESEARCH 1 Running head: VIRGIN SOIL IN IRONY RESEARCH Virgin soil in irony research: Personality, humor, and the “sense of irony”

CHAPTER 18 Lexicons for Sentiment and Affect Extraction " [W]e write, not with the fingers, but with the whole person. The nerve which controls the pen winds itself about every fibre of our being, threads the heart, pierces the liver. " Virginia Woolf, Orlando " She runs the gamut of emotions from A to B. " Dorothy Parker, reviewing Hepburn's performance in Little Women In this chapter we turn to tools for interpreting affective meaning, extending our affective study of sentiment analysis in Chapter 7. We use the word 'affective', following the tradition in affective computing (Picard, 1995) to mean emotion, sentiment, personality , mood, and attitudes. Affective meaning is closely related to subjectivity, subjectivity the study of a speaker or writer's evaluations, opinions, emotions, and speculations (Wiebe et al., 1999). How should affective meaning be defined? One influential typology of affec-tive states comes from Scherer (2000), who defines each class of affective states by factors like its cognition realization and time course: Emotion: Relatively brief episode of response to the evaluation of an external or internal event as being of major significance. Mood: Diffuse affect state, most pronounced as change in subjective feeling, of low intensity but relatively long duration, often without apparent cause. Interpersonal stance: Affective stance taken toward another person in a specific interaction, colouring the interpersonal exchange in that situation. Figure 18.1 The Scherer typology of affective states, with descriptions from Scherer (2000).

This research focuses on natural, robot-led, human-robot interaction that enables a robot to discover what drinks a barman can prepare through continuous application of speech recognition, understanding and generation. Speech was recognised using Google Cloud’s speech to text API, understood by matching either the object or main verb of a sentence against a list of key words and, finally, generated using templates with variable parts. The difficulty lies in the large quantity of key words, as they are based on the properties of the ordered drinks. The results show that having the aforementioned interaction works well to some extent, i.e. the naturalness of the interaction was ranked 5.5 on average. Furthermore, the obtained precision when identifying the unavailable drinks was 0.625 and the obtained recall was 1, resulting in an F1 measure of 0.769.

Sentiment analysis using deep learning models is a leading subject of interest in Natural Language Processing that is as powerful as it is opaque. Current state-ofthe-art models can produce accurate predictions, but they provide little insight as to why the model predicted this sentiment. Businesses relying on these models might be less likely to act on insight given the lack of evidence for predictions. These people would be more likely to trust such predictions if a brief explanation of the outcome is provided. Recent work by Lei et al [4]. has set forth a framework for a multi-aspect sentiment analysis concurrently providing text rationalization with each prediction. This framework sets forth a two-part approach, which summarizes a review and predicts a sentiment. In this paper, we explore the performance of this framework, seeking to recreate and improve upon it in TensorFlow.

Telling stories about our daily lives is one of the most ubiquitous, consequential and seamless ways in which we socialize. Current narrative generation methods mostly require specification of a priori knowledge or comprehensive domain models, which are not generalizable across contexts. Hence, such approaches do not lend themselves well to new and unpredictable domains of observation and interaction, in which social stories usually occur. In this paper, we describe a methodology for categorizing event descriptions as being socially interesting. The event sequences are drawn from crowd-sourced Plot Graphs. The models include low-level natural language and higher-level features. The results from classification and regression tasks look promising overall, indicating that general metrics of social interestingness of stories could be modeled for sociable agents.

We investigate the accuracy of a set of surface patterns in identifying ironic sentences in comments submitted by users to an on-line newspaper. The initial focus is on identifying irony in sentences containing positive predicates since these sentences are more exposed to irony, making their true polarity harder to recognize. We show that it is possible to find ironic sentences with relatively high precision (from 45% to 85%) by exploring certain oral or gestural clues in user comments, such as emoticons, onomatopoeic expressions for laughter, heavy punctuation marks, quotation marks and positive interjections. We also demonstrate that clues based on deeper linguistic information are relatively inefficient in capturing irony in user-generated content, which points to the need for exploring additional types of oral clues.

We present a new approach to evaluate computational models for the task of text understanding by the means of out-of-context error detection. Through the novel design of our automated modification process, existing large-scale data sources can be adopted for a vast number of text understanding tasks. The data is thereby altered on a semantic level, allowing models to be tested against a challenging set of modified text passages that require to comprise a broader narrative discourse. Our newly introduced task targets actual real-world problems of transcription and translation systems by inserting authentic out-of-context errors. The automated modification process is applied to the 2016 TEDTalk corpus. Entirely automating the process allows the adoption of complete datasets at low cost, facilitating supervised learning procedures and deeper networks to be trained and tested. To evaluate the quality of the modification algorithm a language model and a supervised binary classification model are trained and tested on the altered dataset. A human baseline evaluation is examined to compare the results with human performance. The outcome of the evaluation task indicates the difficulty to detect semantic errors for machine-learning algorithms and humans, showing that the errors cannot be identified when limited to a single sentence.

There is high demand for automated tools that assign polarity to microblog content such as tweets (Twitter posts), but this is challenging due to the terseness and informality of tweets in addition to the wide variety and rapid evolution of language in Twitter. It is thus impractical to use standard supervised machine learning techniques dependent on annotated training examples. We do without such annotations by using label propagation to incorporate labels from a maximum entropy classifier trained on noisy labels and knowledge about word types encoded in a lexicon, in combination with the Twitter follower graph. Results on polarity classification for several datasets show that our label propagation approach rivals a model supervised with in-domain annotated tweets, and it outperforms the noisily supervised classifier it exploits as well as a lexicon-based polarity ratio classifier.

This study presents a novel approach to automatic emotion recognition from text. First, emotion generation rules (EGRs) are manually deduced from psychology to represent the conditions for generating emotion. Based on the EGRs, the emotional state of each sentence can be represented as a sequence of semantic labels (SLs) and attributes (ATTs); SLs are defined as the domain-independent features, while ATTs are domain-dependent. The emotion association rules (EARs) represented by SLs and ATTs for each emotion are automatically derived from the sentences in an emotional text corpus using the a priori algorithm. Finally, a separable mixture model (SMM) is adopted to estimate the similarity between an input sentence and the EARs of each emotional state. Since some features defined in this approach are domain-dependent, a dialog system focusing on the students' daily expressions is constructed, and only three emotional states, happy, unhappy, and neutral, are considered for performance evaluation. According to the results of the experiments, given the domain corpus, the proposed approach is promising, and easily ported into other domains.