Question: EmoBGM: Assessing sound emotion for generating slideshows with fitting BGM.

This paper explores the use of machine learning for Music Emotion Identification (MEI) from lyrics, aiming to enhance content-based music retrieval systems. With the proliferation of extensive online music databases, a significant gap exists in automated classification methods that can effectively categorize music by emotional content. While some services like Allmusic.com have attempted to address this by employing human experts to classify songs into emotional categories, our work seeks to automate this process using machine learning algorithms. We employ a psychological model of emotion, expanded to encompass 23 distinct categories, to train our models. The results from analyzing song lyrics for specific emotional content are promising, yielding classification models that are not only comprehensible to humans but also align with common intuitions about the emotional nuances of music. This research on lyric-based MEI is part of a broader effort to integrate multiple classifiers, such as acoustic features and lyrical text, for a comprehensive approach to music emotion recognition. Our findings could significantly improve the user experience in music discovery and personalization, particularly in applications like creating emotionally congruent background music for slideshows or other multimedia presentations.

Robots have the potential to facilitate future therapies for children on the autism spectrum. However, existing robots are limited in their ability to automatically perceive and respond to human affect, which is necessary for establishing and maintaining engaging interactions. Their inference challenge is made even harder by the fact that many individuals with autism have atypical and unusually diverse styles of expressing their affective-cognitive states. To tackle the heterogeneity in children with autism, we used the latest advances in deep learning to formulate a personalized machine learning (ML) framework for automatic perception of the children’s affective states and engagement during robot-assisted autism therapy. Instead of using the traditional one-size-fits-all ML approach, we personalized our framework to each child using their contextual information (demographics and behavioral assessment scores) and individual characteristics. We evaluated this framework on a multimodal (audio, video, and autonomic physiology) data set of 35 children (ages 3 to 13) with autism, from two cultures (Asia and Europe), and achieved an average agreement (intraclass correlation) of ~60% with human experts in the estimation of affect and engagement, also outperforming nonpersonalized ML solutions. These results demonstrate the feasibility of robot perception of affect and engagement in children with autism and have implications for the design of future autism therapies.

We argue that an evaluation of system behavior at the level of the music is required to usefully address the fundamental problems of music genre recognition (MGR), and indeed other tasks of music information retrieval, such as autotagging. A recent review of works in MGR since 1995 shows that most (82 %) measure the capacity of a system to recognize genre by its classification accuracy. After reviewing evaluation in MGR, we show that neither classification accuracy, nor recall and precision, nor confusion tables, necessarily reflect the capacity of a system to recognize genre in musical signals. Hence, such figures of merit cannot be used to reliably rank, promote or discount the genre recognition performance of MGR systems if genre recognition (rather than identification by irrelevant confounding factors) is the objective. This motivates the development of a richer experimental toolbox for evaluating any system designed to intelligently extract information from music signals.

Depression is a major mental health disorder that is rapidly affecting lives worldwide. Depression not only impacts emotional but also physical and psychological state of the person. Its symptoms include lack of interest in daily activities, feeling low, anxiety, frustration, loss of weight and even feeling of selfhatred. This report describes work done by us for Audio Visual Emotion Challenge (AVEC) 2017 during our second year BTech summer internship. With the increase in demand to detect depression automatically with the help of machine learning algorithms, we present our multimodal feature extraction and decision level fusion approach for the same. Features are extracted by processing on the provided Distress Analysis Interview Corpus-Wizard of Oz (DAIC-WOZ) database. Gaussian Mixture Model (GMM) clustering and Fisher vector approach were applied on the visual data; statistical descriptors on gaze, pose; low level audio features and head pose and text features were also extracted. Classification is done on fused as well as independent features using Support Vector Machine (SVM) and neural networks. The results obtained were able to cross the provided baseline on validation data set by 17% on audio features and 24.5% on video features. Keywords—AVEC 2017, SVM, Depression, Neural network, RMSE, MAE, fusion, speech processing.

We present a data-processing pipeline based on sparse feature learning and describe its applications to music annotation and retrieval. Content-based music annotation and retrieval systems process audio starting with features. While commonly used features, such as MFCC, are handcrafted to extract characteristics of the audio in a succinct way, there is increasing interest in learning features automatically from data using unsupervised algorithms. We describe a systemic approach applying feature-learning algorithms to music data, in particular, focusing on a highdimensional sparse-feature representation. Our experiments show that, using only a linear classifier, the newly learned features produce results on the CAL500 dataset comparable to state-of-the-art music annotation and retrieval systems.

We report on the construction of a real-time computer system capable of distinguishing speech signals from music signals over a wide range of digital audio input. We have examined 13 features intended to measure conceptually distinct properties of speech and/or music signals, and combined them in several multidimensional classification frameworks. We provide extensive data on system performance and the cross-validated training/test setup used to evaluate the system. For the datasets currently in use, the best classifier classifies with 5.8% error on a frame-by-frame basis, and 1.4% error when integrating long (2.4 second) segments of sound.

Computer animated agents and robots bring a social dimension to human computer interaction and force us to think in new ways about how computers could be used in daily life. Face to face communication is a real-time process operating at a a time scale in the order of 40 milliseconds. The level of uncertainty at this time scale is considerable, making it necessary for humans and machines to rely on sensory rich perceptual primitives rather than slow symbolic inference processes. In this paper we present progress on one such perceptual primitive. The system automatically detects frontal faces in the video stream and codes them with respect to 7 dimensions in real time: neutral, anger, disgust, fear, joy, sadness, surprise. The face finder employs a cascade of feature detectors trained with boosting techniques [15, 2]. The expression recognizer receives image patches located by the face detector. A Gabor representation of the patch is formed and then processed by a bank of SVM classifiers. A novel combination of Adaboost and SVM's enhances performance. The system was tested on the Cohn-Kanade dataset of posed facial expressions [6]. The generalization performance to new subjects for a 7- way forced choice correct. Most interestingly the outputs of the classifier change smoothly as a function of time, providing a potentially valuable representation to code facial expression dynamics in a fully automatic and unobtrusive manner. The system has been deployed on a wide variety of platforms including Sony's Aibo pet robot, ATR's RoboVie, and CU animator, and is currently being evaluated for applications including automatic reading tutors, assessment of human-robot interaction.

While many lexica annotated with words polarity are available for sentiment analysis, very few tackle the harder task of emotion analysis and are usually quite limited in coverage. In this paper, we present a novel approach for extracting – in a totally automated way – a highcoverage and high-precision lexicon of roughly 37 thousand terms annotated with emotion scores, called DepecheMood. Our approach exploits in an original way ‘crowd-sourced’ affective annotation implicitly provided by readers of news articles from rappler.com. By providing new state-of-the-art performances in unsupervised settings for regression and classification tasks, even using a naı̈ve approach, our experiments show the beneficial impact of harvesting social media data for affective lexicon building.

The aim of this paper is to investigate the feasibility of an audio-based context recognition system. Here, context recognition refers to the automatic classification of the context or an environment around a device. A system is developed and compared to the accuracy of human listeners in the same task. Particular emphasis is placed on the computational complexity of the methods, since the application is of particular interest in resource-constrained portable devices. Simplistic low-dimensional feature vectors are evaluated against more standard spectral features. Using discriminative training, competitive recognition accuracies are achieved with very low-order hidden Markov models (1-3 Gaussian components). Slight improvement in recognition accuracy is observed when linear data-driven feature transformations are applied to mel-cepstral features. The recognition rate of the system as a function of the test sequence length appears to converge only after about 30 to 60 s. Some degree of accuracy can be achieved even with less than 1-s test sequence lengths. The average reaction time of the human listeners was 14 s, i.e., somewhat smaller, but of the same order as that of the system. The average recognition accuracy of the system was 58% against 69%, obtained in the listening tests in recognizing between 24 everyday contexts. The accuracies in recognizing six high-level classes were 82% for the system and 88% for the subjects.

As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological processes, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals. That men are machines (whatever else they may be) has long been suspected; but not till our generation have men fairly felt in concrete just what wonderful psycho-neuro-physical mechanisms they are. William James (1893; 1842 – 1910) A. Fred, J. Filipe, and H. Gamboa (Eds.): BIOSTEC 2009, CCIS 52, pp. 21–47, 2010. c © Springer-Verlag Berlin Heidelberg 2010 22 E.L. van den Broek et al.

This paper proposed a multimodal fusion between brain and peripheral signals for emotion detection. The input signals were electroencephalogram, galvanic skin resistance, temperature, blood pressure and respiration, which can reflect the influence of emotion on the central nervous system and autonomic nervous system respectively. The acquisition protocol is based on a subset of pictures which correspond to three specific areas of valance-arousal emotional space (positively excited, negatively excited, and calm). The features extracted from input signals, and to improve the results, correlation dimension as a strong nonlinear feature is used for brain signals. The performance of the Quadratic Discriminant Classifier has been evaluated on different feature sets: peripheral signals, EEG's, and both. In comparison among the results of different feature sets, EEG signals seem to perform better than other physiological signals, and the results confirm the interest of using brain signals as peripherals in emotion assessment. According to the improvement in EEG results compare in each raw of the table, it seems that nonlinear features would lead to better understanding of how emotional activities work.


Question: EmoBGM: Assessing sound emotion for generating fitting BGM in slideshows.

This paper introduces a cutting-edge methodology for sentiment analysis from multimodal content, focusing on the vast amount of video data shared daily on social media platforms like Facebook and YouTube. With the exponential growth of internet-based information, effectively managing and extracting valuable knowledge from this sea of data has become increasingly challenging. Our proposed approach to multimodal sentiment analysis innovatively harnesses audio, visual, and textual cues as information sources to extract sentiments from web videos. We employ both feature-level and decision-level fusion techniques to integrate the affective data derived from these diverse modalities. A comprehensive evaluation against existing methods in the field highlights the distinctiveness and superiority of our methodology. Initial comparative tests using the YouTube dataset reveal that our multimodal system achieves a remarkable accuracy of approximately 80%, surpassing all leading systems by over 20%. Copyright & 2015 Elsevier B.V. All rights are reserved.

Preference elicitation is a challenging fundamental problem when designing recommender systems. In the present work we propose a content-based technique to automatically generate a semantic representation of the user’s musical preferences directly from audio. Starting from an explicit set of music tracks provided by the user as evidence of his/her preferences, we infer high-level semantic descriptors for each track obtaining a user model. To prove the benefits of our proposal, we present two applications of our technique. In the first one, we consider three approaches to music recommendation, two of them based on a semantic music similarity measure, and one based on a semantic probabilistic model. In the second application, we address the visualization of the user’s musical preferences by creating a humanoid cartoon-like character – the Musical Avatar – automatically inferred from the semantic representation. We conducted a preliminary evaluation of the proposed technique in the context of these applications with 12 subjects. The results are promising: the recommendations were positively evaluated and close to those coming from state-ofthe-art metadata-based systems, and the subjects judged the generated visualizations to capture their core preferences. Finally, we highlight the advantages of the proposed semantic user model for enhancing the user interfaces of information filtering systems. 2012 Elsevier Ltd. All rights reserved.

Building a text corpus suitable to be used in corpus-based speech synthesis is a time-consuming process that usually requires some human intervention to select the desired phonetic content and the necessary variety of prosodic contexts. If an emotional text-to-speech (TTS) system is desired, the complexity of the corpus generation process increases. This paper presents a study aiming to validate or reject the use of a semantically neutral text corpus for the recording of both neutral and emotional (acted) speech. The use of this kind of texts would eliminate the need to include semantically emotional texts into the corpus. The study has been performed for Basque language. It has been made by performing subjective and objective comparisons between the prosodic characteristics of recorded emotional speech using both semantically neutral and emotional texts. At the same time, the performed experiments allow for an evaluation of the capability of prosody to carry emotional information in Basque language. Prosody manipulation is the most common processing tool used in concatenative TTS. Experiments of automatic recognition of the emotions considered in this paper (the "Big Six emotions") show that prosody is an important emotional indicator, but cannot be the only manipulated parameter in an emotional TTS system-at least not for all the emotions. Resynthesis experiments transferring prosody from emotional to neutral speech have also been performed. They corroborate the results and support the use of a neutral-semantic-content text in databases for emotional speech synthesis

The ability to recognize emotions is an important part of building intelligent computers. Emotionally-aware systems would have a rich context from which to make appropriate decisions about how to interact with the user or adapt their system response. There are two main problems with current system approaches for identifying emotions that limit their applicability: they can be invasive and can require costly equipment. Our solution is to determine user emotion by analyzing the rhythm of their typing patterns on a standard keyboard. We conducted a field study where we collected participants' keystrokes and their emotional states via self-reports. From this data, we extracted keystroke features, and created classifiers for 15 emotional states. Our top results include 2-level classifiers for confidence, hesitance, nervousness, relaxation, sadness, and tiredness with accuracies ranging from 77 to 88%. In addition, we show promise for anger and excitement, with accuracies of 84%.

Automatic emotion recognition is an interdisciplinary research field which deals with the algorithmic detection of human affect, e.g. anger or sadness, from a variety of sources, such as speech or facial gestures. Apart from the obvious usage for industry applications in human-robot interaction, acquiring the emotional state of a person automatically also is of great potential for the health domain, especially in psychology and psychiatry. Here, evaluation of human emotion is often done using oral feedback or questionnaires during doctor-patient sessions. However, this can be perceived as intrusive by the patient. Furthermore, the evaluation can only be done in a noncontinuous manner, e.g. once a week during therapy sessions. In contrast, using automatic emotion detection, the affect state of a person can be evaluated in a continuous non-intrusive manner, for example to detect early on-sets of depression. An additional benefit of automatic emotion recognition is the objectivity of such an approach, which is not influenced by the perception of the patient and the doctor. To reach the goal of objectivity, it is important, that the source of the emotion is not easily manipulable, e.g. as in the speech modality. To circumvent this caveat, novel approaches in emotion detection research the potential of using physiological measures, such as galvanic skin sensors or pulse meters. In this paper we outline a way of detecting emotion from brain waves, i.e., EEG data. While EEG allows for a continuous, real-time automatic emotion recognition, it furthermore has the charm of measuring the affect close to the point of emergence: the brain. Using EEG data for emotion detection is nevertheless a challenging task: Which features, EEG channel locations and frequency bands are best suited for is an issue of ongoing research. In this paper we evaluate the use of state of the art feature extraction, feature selection and classification algorithms for EEG emotion classification using data from the de facto standard dataset, DEAP. Moreover, we present results that help choose methods to enhance classification performance while simultaneously reducing computational complexity.

Using recent insights from Cognitive, Affective and Social Neuroscience this paper addresses how affective states in social interactions can be used through social media to analyze and support behaviour for a certain lifestyle. A computational model is provided integrating mechanisms for the impact of one’s emotions on behaviour, and for the impact of emotions of others on one’s own emotion. The model is used to reason about and assess the state of a user with regard to a lifestyle goal (such as exercising frequently), based on extracted information about emotions exchanged in social interaction. Support is provided by proposing ways to affecting these social interactions, thereby indirectly influencing the impact of the emotions of others. An ambient intelligent system incorporating this has been implemented for the social medium Twitter.

Speech-driven facial motion synthesis is a well explored research topic. However, little has been done to model expressive visual behavior during speech. We address this issue using a machine learning approach that relies on a database of speech-related high-fidelity facial motions. From this training set, we derive a generative model of expressive facial motion that incorporates emotion control, while maintaining accurate lip-synching. The emotional content of the input speech can be manually specified by the user or automatically extracted from the audio signal using a Support Vector Machine classifier.

o mimic the quality of everyday human communication, future computer interfaces must combine the benefits of high visual fidelity with conversational intelligence and—above all—the ability to modulate the emotions of their users. From a computer graphics or visual perspective, achieving this goal requires creating synthetic digital humans that have photorealistic faces capable of expressing the finest shades of emotion. From a perceptual point of view, the same system must also read users' emotional reactions and adapt the behavior of the digital human accordingly. Our research focuses on implementing this concept to create a face-to-face system called the Virtual Human Interface (VHI). Photorealistic virtual humans— in contrast to nonrealistic virtual characters—make a more effective communication interface between computers and humans because they can stimulate declarative and procedural memory in the human brain. Declarative memory is our memory of information and events. Procedural memory, on the other hand, is the memory of how to do something, such as riding a bike. Most computer interfaces primarily access users' declarative memory; the person learns information about a particular topic. The VHI system, however, engages the user through entertaining emotional responses that help engrave knowledge in the user's mind and even modify the user's behavior through procedural memory. During the past several decades, researchers have conducted countless studies on agents and human animation. Early work mainly addressed low-polygon virtual humans that users could animate in real time. 1 Many of these animation systems initially addressed purposes other than the needs of human–computer interaction. However, the requirements of facial ani-mation—and especially speech synthesis—demand a different underlying architecture that can effectively model how real faces move and change. 2-4 As a result, research began to focus on creating autonomous agents that could exhibit rich personalities and interact in virtual worlds inhabited by other characters. To provide the illusion of a lifelike character, researchers have developed detailed emotional and personality models that can control the animation channels as a function of the virtual human's personality, mood, and emotions. 7-9 The real-time interaction with these virtual characters poses an extra set of technical challenges in terms of the speed, computational power, and visual quality required to make the user believe that he or she is interacting with a living creature. To achieve this goal, researchers eventually replaced precrafted animated actions with intelligent behavior modules that could control speech, locomotion, gaze, blinks, gestures (including various postures), and interaction with the …

Emotion recognition from speech plays an important role in developing affective and intelligent systems. This study investigates sentence-level emotion recognition. We propose to use a two-step approach to leverage information from subsentence segments for sentence level decision. First we use a segment level emotion classifier to generate predictions for segments within a sentence. A second component combines the predictions from these segments to obtain a sentence level decision. We evaluate different segment units (words, phrases, time-based segments) and different decision combination methods (majority vote, average of probabilities, and a Gaussian Mixture Model (GMM)). Our experimental results on two different data sets show that our proposed method significantly outperforms the standard sentence-based classification approach. In addition, we find that using time-based segments achieves the best performance, and thus no speech recognition or alignment is needed when using our method, which is important to develop language independent emotion recognition systems.

The advent of the Social Web has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. In pace with a global deluge of videos from billions of computers, smartphones, tablets, university projectors and security cameras, the amount of multimodal content on the Web has been growing exponentially, and with that comes the need for decoding such information into useful knowledge. In this paper, a multimodal affective data analysis framework is proposed to extract user opinion and emotions from video content. In particular, multiple kernel learning is used to combine visual, audio and textual modalities. The proposed framework outperforms the state-of-the-art model in multimodal sentiment analysis research with a margin of 10-13% and 3-5% accuracy on polarity detection and emotion recognition, respectively. The paper also proposes an extensive study on decision-level fusion.

Although it is typically expected that using a large amount of labeled training data would lead to improve performance in deep learning, it is generally difficult to obtain such DataBase (DB). In competitions such as the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge Task 1, participants are constrained to use a relatively small DB as a rule, which is similar to the aforementioned issue. To improve Acoustic Scene Classification (ASC) performance without employing additional DB, this paper proposes to use Generative Adversarial Networks (GAN) based method for generating additional training DB. Since it is not clear whether every sample generated by GAN would have equal impact in classification performance, this paper proposes to use Support Vector Machine (SVM) hyper plane for each class as reference for selecting samples, which have class discriminative information. Based on the crossvalidated experiments on development DB, the usage of the generated features could improve ASC performance.


Question: A probability-based model for object search with 6-DOF pose prediction.

This paper details the design, implementation, and testing of HERB (Home Exploring Robotic Butler), an advanced autonomous mobile manipulator designed to perform a variety of manipulation tasks within a domestic setting. HERB is equipped with a suite of innovative algorithms that enable it to search for objects effectively, navigate through cluttered and dynamic indoor environments, accurately recognize and register objects in high-clutter scenarios using vision, manipulate doors and other constrained objects using caging grasps, plan and execute grasps in clutter, and perform manipulation tasks on pose and torque constraint manifolds. The development team, comprising researchers from Intel Research Pittsburgh and The Robotics Institute at Carnegie Mellon University, have conducted extensive real-world testing, showcasing the seamless integration and operation of these algorithms on HERB, demonstrating its capabilities in a challenging home environment. The authors, including S.S. Srinivasa, D. Ferguson, C.J. Helfrich, D. Berenson, A. Collet, R. Diankov, G. Gallagher, G. Hollinger, J. Kuffner, and M.V. Weghe, have contributed significantly to the project, each bringing their expertise in robotics, manipulation, and autonomous systems to create a robotic system capable of navigating and manipulating objects in a complex and dynamic setting. The results from these tests underscore HERB's potential for real-world applications in home automation and assistance.

When compared to image recognition, object detection is a much more challenging task because it requires the accurate real-time localization of an object in the target image. In interaction scenarios, this pipeline can be simplified by incorporating the users’ point of regard. Wearable eye trackers can estimate the gaze direction, but lack own processing capabilities. We enable mobile gaze-aware applications by developing an open-source platform which supports mobile eye tracking based on the Pupil headset and a smartphone running Android OS. Through our platform, we offer researchers and developers a rapid prototyping environment for gaze-enabled applications. We describe the concept, our current progress, and research implications. ACM Classification

In this paper, we present MO-SLAM, a novel visual SLAM system that is capable of detecting duplicate objects in the scene during run-time without requiring an offline training stage to pre-populate a database of objects. Instead, we propose a novel method to detect landmarks that belong to duplicate objects. Further, we show how landmarks belonging to duplicate objects can be converted to first-order entities which generate additional constraints for optimizing the map. We evaluate the performance of MO-SLAM with extensive experiments on both synthetic and real data, where the experimental results verify the capabilities of MO-SLAM in detecting duplicate objects and using these constraints to improve the accuracy of the map.

An important aspect of human perception is anticipation and anticipating which activities will a human do next (and how to do them) in useful for many applications, for example, anticipation enables an assistive robot to plan ahead for reactive responses in the human environments. In this work, we present a constructive approach for generating various possible future human activities by reasoning about the rich spatial-temporal relations through object affordances. We represent each possible future using an anticipatory temporal conditional random field (ATCRF) where we sample the nodes and edges corresponding to future object trajectories and human poses from a generative model. We then represent the distribution over the potential futures using a set of constructed ATCRF particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, for new subjects (not seen in the training set), we obtain an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of 75.4%, 69.2% and 58.1% for an anticipation time of 1, 3 and 10 seconds respectively. 1

This paper presents a new algorithm for mobile robot localization, called Monte Carlo Localization (MCL). MCL is a version of Markov localization, a family of probabilistic approaches that have recently been applied with great practical success. However, previous approaches were either computationally cumbersome (such as grid-based approaches that represent the state space by high-resolution 3D grids), or had to resort to extremely coarse-grained resolutions. Our approach is computationally efficient while retaining the ability to represent (almost) arbitrary distributions. MCL applies sampling-based methods for approximating probability distributions, in a way that places computation “where needed.” The number of samples is adapted on-line, thereby invoking large sample sets only when necessary. Empirical results illustrate that MCL yields improved accuracy while requiring an order of magnitude less computation when compared to previous approaches. It is also much easier to implement.

In this paper, we address the problem of extreme head pose estimation from intensity images, in a monocular setup. We introduce a novel fusion pipeline to integrate into a dedicated Kalman Filter the pose estimated from a tracking scheme in the prediction stage and the pose estimated from a detection scheme in the correction stage. To that end, the measurement covariance of the Kalman Filter is updated in every frame. The tracking scheme is performed using a set of keypoints extracted in the area of the head along with a simple 3D geometric model. The detection scheme, on the other hand, relies on the alignment of facial landmarks in each frame combined with 3D features extracted on a head mesh. The head pose in each scheme is estimated by minimizing the reprojection error from the 3D-2D correspondences. By combining both frameworks, we extend the applicability of head pose estimation from facial landmarks to cases where these features are no longer visible. We compared the proposed method to other related approaches, showing that it can achieve state-of-the-art performance. We also demonstrate that our approach is suitable for cases with extreme head rotations and (self-) occlusions, besides being suitable for real time applications.

In this paper we describe a trainable object detector and its instantiations for detecting faces and cars at any size, location, and pose. To cope with variation in object orientation, the detector uses multiple classifiers, each spanning a different range of orientation. Each of these classifiers determines whether the object is present at a specified size within a fixed-size image window. To find the object at any location and size, these classifiers scan the image exhaustively. Each classifier is based on the statistics of localized parts. Each part is a transform from a subset of wavelet coefficients to a discrete set of values. Such parts are designed to capture various combinations of locality in space, frequency, and orientation. In building each classifier, we gathered the class-conditional statistics of these part values from representative samples of object and non-object images. We trained each classifier to minimize classification error on the training set by using Adaboost with Confidence-Weighted Predictions (Shapire and Singer, 1999). In detection, each classifier computes the part values within the image window and looks up their associated class-conditional probabilities. The classifier then makes a decision by applying a likelihood ratio test. For efficiency, the classifier evaluates this likelihood ratio in stages. At each stage, the classifier compares the partial likelihood ratio to a threshold and makes a decision about whether to cease evaluation—labeling the input as non-object—or to continue further evaluation. The detector orders these stages of evaluation from a low-resolution to a high-resolution search of the image. Our trainable object detector achieves reliable and efficient detection of human faces and passenger cars with out-of-plane rotation.

This paper develops a probabilistic anticipation algorithm for dynamic objects observed by an autonomous robot in an urban environment. Predictive Gaussian mixture models are used due to their ability to probabilistically capture continuous and discrete obstacle decisions and behaviors; the predictive system uses the probabilistic output (state estimate and covariance) of a tracking system and map of the environment to compute the probability distribution over future obstacle states for a specified anticipation horizon. A Gaussian splitting method is proposed based on the sigma-point transform and the nonlinear dynamics function, which enables increased accuracy as the number of mixands grows. An approach to caching elements of this optimal splitting method is proposed, in order to enable real-time implementation. Simulation results and evaluations on data from the research community demonstrate that the proposed algorithm can accurately anticipate the probability distributions over future states of nonlinear systems.

2D and 3D computer vision systems are frequently being used in automated production to detect and determine the position of objects. Accuracy is important in the production industry, and computer vision systems require structured environments to function optimally. For 2D vision systems, a change in surfaces, lighting and viewpoint angles can reduce the accuracy of a method, maybe even to a degree that it will be erroneous, while for 3D vision systems, the accuracy mainly depends on the 3D laser sensors. Commercially available 3D cameras lack the precision found in high-grade 3D laser scanners, and are therefore not suited for accurate measurements in industrial use. In this paper, we show that it is possible to identify and locate objects using a combination of 2D and 3D cameras. A rough estimate of the object pose is first found using a commercially available 3D camera. Then, a robotic arm with an eye-in-hand 2D camera is used to determine the pose accurately. We show that this increases the accuracy to < 1 mm and < 1◦. This was demonstrated in a real industrial assembly task where high accuracy is required.

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their cooccurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art.


Question: A probability-based scheme for searching objects with 6-degree-of-freedom position and orientation assessment.

This document discusses an enhanced stereo vision system for robots, designed to overcome the limitations of passive stereo vision, particularly the issue of dropouts in low-texture areas. By integrating a powerful texture projector with the stereo system, the authors aim to mitigate these dropouts, thereby improving the system's overall performance. The paper presents a methodical approach to developing an effective stereo projector system, starting with the identification of optimal patterns for projection under ideal conditions. It then delves into the impact of system blur and phase noise on these patterns, ensuring that the system remains robust in real-world scenarios. To demonstrate the practicality of their solution, the authors have engineered a compact projector capable of delivering high-quality performance up to a distance of 3 meters in indoor settings. Experimental results from the implemented system showcase remarkable depth precision and resolution, particularly within the 1.5-meter range, highlighting the system's potential for enhancing robotic perception in texture-limited environments.

**Question**: An investigation into the use of swarm intelligence for optimizing the performance of wind turbine arrays

**Documents**: Particle swarm optimization of a multi-layer perceptron for wind speed prediction
Wind speed prediction is crucial for the efficient operation of wind power plants. This paper presents the use of particle swarm optimization (PSO) for the optimization of a multi-layer perceptron (MLP) neural network for wind speed prediction. The PSO algorithm is used to optimize the weights and biases of the MLP network. The performance of the PSO-optimized MLP network is compared with that of a backpropagation-trained MLP network. The results show that the PSO-optimized MLP network has a better performance in terms of prediction accuracy and generalization ability.

**Similar Documents**: This work explores the application of particle swarm optimization (PSO) in enhancing the predictive capabilities of a multi-layer perceptron (MLP) neural network for wind speed forecasting, a critical component in the effective management of wind power facilities. The PSO algorithm is employed to fine-tune the MLP network's weights and biases, aiming to improve its prediction accuracy and generalization capacity. Comparative analyses between the MLP network optimized through PSO and one trained using backpropagation reveal that the PSO-enhanced MLP network outperforms the latter in both prediction precision and adaptability, underscoring the potential of PSO in optimizing neural networks for wind speed prediction tasks.

The idea described in this paper is to use the built-in cameras of consumer mobile phones as sensors for 2-dimensional visual codes. Such codes can be attached to physical objects in order to retrieve object-related information and functionality. They are also suitable for display on electronic screens. The proposed visual code system allows the simultaneous detection of multiple codes, introduces a position-independent coordinate system, and provides the phone’s orientation as a parameter. The ability to detect objects in the user’s vicinity offers a natural way of interaction and strengthens the role of mobile phones in a large number of application scenarios. We describe the hardware requirements, the design of a suitable visual code, a lightweight recognition algorithm, and present some example applications.

This paper describes a mapping system that acquires 3D object models of man-made indoor environments such as kitchens. The system segments and geometrically reconstructs cabinets with doors, tables, drawers, and shelves, objects that are important for robots retrieving and manipulating objects in these environments. The system also acquires models of objects of daily use such glasses, plates, and ingredients. The models enable the recognition of the objects in cluttered scenes and the classification of newly encountered objects. Key technical contributions include (1) a robust, accurate, and efficient algorithm for constructing complete object models from 3D point clouds constituting partial object views, (2) feature-based recognition procedures for cabinets, tables, and other task-relevant furniture objects, and (3) automatic inference of object instance and class signatures for objects of daily use that enable robots to reliably recognize the objects in cluttered and real task contexts. We present results from the sensor-based mapping of a real kitchen.

Recently, global features aggregated from local convolutional features of the convolutional neural network have shown to be much more effective in comparison with hand-crafted features for image retrieval. However, the global feature might not effectively capture the relevance between the query object and reference images in the object instance search task, especially when the query object is relatively small and there exist multiple types of objects in reference images. Moreover, the object instance search requires to localize the object in the reference image, which may not be achieved through global representations. In this paper, we propose a Fuzzy Objects Matching (FOM) framework to effectively and efficiently capture the relevance between the query object and reference images in the dataset. In the proposed FOM scheme, object proposals are utilized to detect the potential regions of the query object in reference images. To achieve high search efficiency, we factorize the feature matrix of all the object proposals from one reference image into the product of a set of fuzzy objects and sparse codes. In addition, we refine the feature of the generated fuzzy objects according to its neighborhood in the feature space to generate more robust representation. The experimental results demonstrate that the proposed FOM framework significantly outperforms the state-of-theart methods in precision with less memory and computational cost on three public datasets. The task of object instance search, is to retrieve all the images containing a specific object query and localize the query object in the reference images. It has received a sustained attention over the last decade, leading to many object instance search systems (Meng et al. 2010; Jiang, Meng, and Yuan 2012; Jiang et al. 2015; Tolias, Avrithis, and Jégou 2013; Tao et al. 2014; Razavian et al. 2014a; 2014b; Tolias, Sicre, and Jégou 2016; Meng et al. 2016; Bhattacharjee et al. 2016b; 2016a; Mohedano et al. 2016; Cao et al. 2016; Wu et al. 2016). Since the query object only occupies a small part of an image, the global representation may not be effective to capture the relevance between the query object with reference image. Therefore, the relevance between the query object and one reference image is not determined by the overall similarity between the query and the reference image. Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ...

Online 3D reconstruction of real-world scenes has been attracting increasing interests from both the academia and industry, especially with the consumer-level depth cameras becoming widely available. Recent most online reconstruction systems take live depth data from a moving Kinect camera and incrementally fuse them to a single high-quality 3D model in real time. Although most real-world scenes have static environment, the daily objects in a scene often move dynamically, which are non-trivial to reconstruct especially when the camera is also not still. To solve this problem, we propose a single depth camera-based real-time approach for simultaneous reconstruction of dynamic object and static environment, and provide solutions for its key issues. In particular, we first introduce a robust optimization scheme which takes advantage of raycasted maps to segment moving object and background from the live depth map. The corresponding depth data are then fused to the volumes, respectively. These volumes are raycasted to extract views of the implicit surface which can be used as a consistent reference frame for the next iteration of segmentation and tracking. Particularly, in order to handle fast motion of dynamic object and handheld camera in the fusion stage, we propose a sequential 6D pose prediction method which largely increases the registration robustness and avoids registration failures occurred in conventional methods. Experimental results show that our approach can reconstruct moving object as well as static environment with rich details, and outperform conventional methods in multiple aspects.

The combination of on-board sensors measurements with different statistical characteristics can be employed in robotics for localization and control, especially in GPS-denied environments. In particular, most aerial vehicles are packaged with low cost sensors, important for aerial robotics, such as camera, a gyroscope, and an accelerometer. In this work, we develop a visual inertial odometry system based on the Unscented Kalman Filter (UKF) acting on the Lie group SE(3), such to obtain an unique, singularity-free representation of a rigid body pose. We model this pose with the Lie group SE(3) and model the noise on the corresponding Lie algebra. Moreover, we extend the concepts used in the standard UKF formulation, such as state uncertainty and modeling, to correctly incorporate elements that do not belong to an Euclidean space such as the Lie group members. In this analysis, we use the parallel transport, which requires us to explicitly consider SE(3) as representing rigid bodies though the use of the affine connection. We present experimental results to show the effectiveness of the proposed approach for state estimation of a quadrotor platform.

Magnetic anomaly detection (MAD) is an important problem in applications ranging from geological surveillance to military reconnaissance. MAD sensors detect local disturbances in the magnetic field, which can be used to detect the existence of and to estimate the position of buried, hidden, or submerged objects, such as ore deposits or mines. These sensors may experience false positive and false negative detections and, without prior knowledge of the targets, can only determine proximity to a target. The uncertainty in the sensors, coupled with a lack of knowledge of even the existence of targets, makes the estimation and control problems challenging. We utilize a hierarchical decomposition of the environment, coupled with an estimation algorithm based on random finite sets, to determine the number of and the locations of targets in the environment. The small team of robots follow the gradient of mutual information between the estimated set of targets and the future measurements, locally maximizing the rate of information gain. We present experimental results of a team of quadrotor micro aerial vehicles discovering and localizing an unknown number of permanent magnets.

We present a Bag of Words-based active object categorization technique implemented and tested on a humanoid robot. The robot is trained to categorize objects that are handed to it by a human operator. The robot uses hand and head motions to actively acquire a number of different views. A view planning scheme using entropy minimization reduces the number of views needed to achieve a valid decision. Categorization results are significantly improved by active elimination of background features using robot arm motion. Our experiments cover both, categorization when the object is handed to the robot in a fixed pose at training and testing, and object pose independent categorization. Results on a 4-class object database demonstrate the classification efficiency, a significant gain from multi-view compared to single-view classification, and the advantage of view planning. We conclude that humanoid robotic systems can be successfully applied to actively categorize objects a task with many potential applications ranging from edutainment to active surveillance.

Several randomized path planners have been proposed during the last few years. Their attractiveness stems from their applicability to virtually any type of robots, and their empirically observed success. In this paper we attempt to present a unifying view of these planners and to theoretically explain their success. First, we introduce a general planning scheme that consists of randomly sampling the robot’s configuration space. We then describe two previously developed planners as instances of planners based on this scheme, but applying very different sampling strategies. These planners are probabilistically complete: if a path exists, they will find one with high probability, if we let them run long enough. Next, for one of the planners, we analyze the relation between the probability of failure and the running time. Under assumptions characterizing the “goodness” of the robot’s free space, we show that the running time only grows as the absolute value of the logarithm of the probability of failure that we are willing to tolerate. We also show that it increases at a reasonable rate as the space goodness degrades. In the last section we suggest directions for future research.

In this paper we propose a new method for detecting multiple specific 3D objects in real time. We start from the template-based approach based on the LINE2D/LINEMOD representation introduced recently by Hinterstoisser et al., yet extend it in two ways. First, we propose to learn the templates in a discriminative fashion. We show that this can be done online during the collection of the example images, in just a few milliseconds, and has a big impact on the accuracy of the detector. Second, we propose a scheme based on cascades that speeds up detection. Since detection of an object is fast, new objects can be added with very low cost, making our approach scale well. In our experiments, we easily handle 10-30 3D objects at frame rates above 10fps using a single CPU core. We outperform the state-of-the-art both in terms of speed as well as in terms of accuracy, as validated on 3 different datasets. This holds both when using monocular color images (with LINE2D) and when using RGBD images (with LINEMOD). Moreover, we propose a challenging new dataset made of 12 objects, for future competing methods on monocular color images.

In this paper we describe a trainable object detector and its instantiations for detecting faces and cars at any size, location, and pose. To cope with variation in object orientation, the detector uses multiple classifiers, each spanning a different range of orientation. Each of these classifiers determines whether the object is present at a specified size within a fixed-size image window. To find the object at any location and size, these classifiers scan the image exhaustively. Each classifier is based on the statistics of localized parts. Each part is a transform from a subset of wavelet coefficients to a discrete set of values. Such parts are designed to capture various combinations of locality in space, frequency, and orientation. In building each classifier, we gathered the class-conditional statistics of these part values from representative samples of object and non-object images. We trained each classifier to minimize classification error on the training set by using Adaboost with Confidence-Weighted Predictions (Shapire and Singer, 1999). In detection, each classifier computes the part values within the image window and looks up their associated class-conditional probabilities. The classifier then makes a decision by applying a likelihood ratio test. For efficiency, the classifier evaluates this likelihood ratio in stages. At each stage, the classifier compares the partial likelihood ratio to a threshold and makes a decision about whether to cease evaluation—labeling the input as non-object—or to continue further evaluation. The detector orders these stages of evaluation from a low-resolution to a high-resolution search of the image. Our trainable object detector achieves reliable and efficient detection of human faces and passenger cars with out-of-plane rotation.


Question: A probability-based model for locating objects with 6-DOF orientation and position estimation.

This paper introduces a real-time methodology for the estimation and tracking of a rigid object's 3D pose, designed specifically for visual servo systems utilizing natural landmarks. Existing advanced techniques for 3D pose recognition in natural environments are often incompatible with real-time servo systems due to their inherent latency. Here, we demonstrate that a real-time 3D pose estimation solution can be achieved by integrating a rapid tracking algorithm, such as the Kanade-Lucas-Tomasi (KLT) feature tracker, with a method for determining the 3D coordinates of tracking points on the object at the onset of Scale-Invariant Feature Transform (SIFT) based tracking point initialization. This is made possible under the assumption that a 3D geometric model of the object, along with its SIFT descriptors, is pre-known. By continuously tracking points with KLT, automatically eliminating outlier tracking points, and periodically reinitializing tracking points using SIFT when performance degrades, the 3D pose of the object can be accurately estimated and tracked in real-time. This methodology is adaptable to both monocular and stereoscopic camera setups for 3D pose estimation and tracking. Monocular systems offer higher frame rates with approximately 1 ms for local pose estimation, while stereoscopic systems provide more precise pose data, albeit with a slightly increased local pose estimation time of about 16 ms. Extensive experimental results confirm the efficacy of the proposed approach, delivering real-time performance for 3D object pose estimation and tracking.

Millimeter-wave (mm-wave) signals and large antenna arrays are considered enabling technologies for future 5G networks. While their benefits for achieving high-data rate communications are well-known, their potential advantages for accurate positioning are largely undiscovered. We derive the Cramér-Rao bound (CRB) on position and rotation angle estimation uncertainty from mm-wave signals from a single transmitter, in the presence of scatterers. We also present a novel two-stage algorithm for position and rotation angle estimation that attains the CRB for average to high signal-to-noise ratio. The algorithm is based on multiple measurement vectors matching pursuit for coarse estimation, followed by a refinement stage based on the space-alternating generalized expectation maximization algorithm. We find that accurate position and rotation angle estimation is possible using signals from a single transmitter, in either line-of-sight, non-line-of-sight, or obstructed-line-of-sight conditions.

Real-time simultaneous tracking of hands manipulating and interacting with external objects has many potential applications in augmented reality, tangible computing, and wearable computing. However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately. Many previous approaches resort to complex multi-camera setups to remedy the occlusion problem and often employ expensive segmentation and optimization steps which makes real-time tracking impossible. In this paper, we propose a real-time solution that uses a single commodity RGB-D camera. The core of our approach is a 3D articulated Gaussian mixture alignment strategy tailored to handobject tracking that allows fast pose optimization. The alignment energy uses novel regularizers to address occlusions and hand-object contacts. For added robustness, we guide the optimization with discriminative part classification of the hand and segmentation of the object. We conducted extensive experiments on several existing datasets and introduce a new annotated hand-object dataset. Quantitative and qualitative results show the key advantages of our method: speed, accuracy, and robustness.

A location and tracking system becomes very important to our future world of pervasive computing, where information is all around us. Location is one of the most needed information for emerging and future applications. Since the public use of GPS satellite is allowed, several state-of-the-art devices become part of our life, e.g. a car navigator and a mobile phone with a built-in GPS receiver. However, location information for indoor environments is still very limited as mentioned in paper. Several techniques are proposed to get location information in buildings such as using a radio signal triangulation. Using Radio Frequency Identification (RFID) tags is a new way of giving location information to users. Due to its passive communication circuit, RFID tags can be embedded almost anywhere without an energy source. The tags stores location information and give it to any reader that is within a proximity range which can be up to 7-10 centimeters. In this paper RFID-based system for navigation in a building for blind people or visually impaired

Positioning within a local area refers to technology whereby each node is self-aware of its position. Based on empirical study, this paper proposes an enhancement to the path loss model in the indoor environment for improved accuracy in the relationship between distance and received signal strength. We further demonstrate the potential of our model for the WiFi positioning system, where the mean errors in the distance estimation are 2.3 m and 2.9 m for line of sight and non line of sight environments, respectively.

This paper presents a new approach to the cooperative localization problem, namely distributed multi-robot localization. A group of M robots is viewed as a single system composed of robots that carry, in general, di erent sensors and have di erent positioning capabilities. A single Kalman lter is formulated to estimate the position and orientation of all the members of the group. This centralized schema is capable of fusing information provided by the sensors distributed on the individual robots while accommodating independencies and interdependencies among the collected data. In order to allow for distributed processing, the equations of the centralized Kalman lter are treated so that this lter can be decomposed into M modi ed Kalman lters each running on a separate robot. The distributed localization algorithm is applied to a group of 3 robots and the improvement in localization accuracy is presented.

This paper presents an approach to textureless object detection and tracking of the 3D pose. Our detection and tracking schemes are coherently integrated in a particle filtering framework on the special Euclidean group, SE(3), in which the visual tracking problem is tackled by maintaining multiple hypotheses of the object pose. For textureless object detection, an efficient chamfer matching is employed so that a set of coarse pose hypotheses is estimated from the matching between 2D edge templates of an object and a query image. Particles are then initialized from the coarse pose hypotheses by randomly drawing based on costs of the matching. To ensure the initialized particles are at or close to the global optimum, an annealing process is performed after the initialization. While a standard edge-based tracking is employed after the annealed initialization, we employ a refinement process to establish improved correspondences between projected edge points from the object model and edge points from an input image. Comparative results for several image sequences with clutter are shown to validate the effectiveness of our approach.

Multi-object tracking is still a challenging task in computer vision. We propose a robust approach to realize multi-object tracking using multi-camera networks. Detection algorithms are utilized to detect object regions with confidence scores for initialization of individual particle filters. Since data association is the key issue in Tracking-by-Detection mechanism, we present an efficient greedy matching algorithm considering multiple judgments based on likelihood functions. Furthermore, tracking in single cameras is realized by a greedy matching method. Afterwards, 3D geometry positions are obtained from the triangulation relationship between cameras. Corresponding objects are tracked in multiple cameras to take the advantages of multi-camera based tracking. Our algorithm performs online and does not need any information about the scene, no restrictions of enter-and-exit zones, no assumption of areas where objects are moving on and can be extended to any class of object tracking. Experimental results show the benefits of using multiple cameras by the higher accuracy and precision rates.

Probabilistic point-set registration methods have been gaining more attention for their robustness to noise, outliers and occlusions. However, these methods tend to be much slower than the popular iterative closest point (ICP) algorithms, which severely limits their usability. In this paper, we contribute a novel probabilistic registration method that achieves state-of-the-art robustness as well as substantially faster computational performance than modern ICP implementations. This is achieved using a rigorous yet computationally-efficient probabilistic formulation. Pointset registration is cast as a maximum likelihood estimation and solved using the EM algorithm. We show that with a simple augmentation, the E step can be formulated as a filtering problem, allowing us to leverage advances in efficient Gaussian filtering methods. We also propose a customized permutohedral filter [1] for improved efficiency while retaining sufficient accuracy for our task. Additionally, we present a simple and efficient twist parameterization that generalizes our method to the registration of articulated and deformable objects. For articulated objects, the complexity of our method is almost independent of the Degrees Of Freedom (DOFs), which makes it highly efficient even for high DOF systems. The results demonstrate the proposed method consistently outperforms many competitive baselines on a variety of registration tasks. The video demo and source code are available on our project page.

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. RGB-D cameras rely on either structured light patterns combined with stereo sensing [6,10] or time-of-flight laser sensing [1] to generate depth estimates that can be associated with RGB pixels. Very soon, small, high-quality RGB-D cameras developed for computer gaming and home entertainment applications will become available at cost below $100. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. The robotics and computer vision communities have developed a variety of techniques for 3D mapping based on laser range scans [8, 11], stereo cameras [7], monocular cameras [3], and unsorted collections of photos [4]. While RGB-D cameras provide the opportunity to build 3D maps of unprecedented richness, they have drawbacks that make their application to 3D mapping difficult: They provide depth only up to a limited distance (typically less than 5m), depth values are much noisier than those provided by laser scanners, and their field of view (∼ 60◦) is far more constrained than that of specialized cameras or laser scanners typically used for 3D mapping (∼ 180◦). In our work, we use a camera developed by PrimeSense [10]. The key insights of this investigation are: first, that existing frame matching techniques are not sufficient to provide robust visual odometry with these cameras; second, that a tight integration of depth and color information can yield robust frame matching and loop closure detection; third, that building on best practice techniques in SLAM and computer graphics makes it possible to build and visualize accurate and extremely rich 3D maps with such cameras; and, fourth, that it will be feasible to build complete robot navigation and interaction systems solely based on cheap depth cameras.

Both simultaneous localization and mapping (SLAM) and detection and tracking of moving objects (DTMO) play key roles in robotics and automation. For certain constrained environments, SLAM and DTMO are becoming solved problems. But for robots working outdoors, and at high speeds, SLAM and DTMO are still incomplete. In earlier works, SLAM and DTMO are treated as two separate problems. In fact, they can be complementary to one another. In this paper, we present a new method to integrate SLAM and DTMO to solve both problems simultaneously for both indoor and outdoor applications. The results of experiments carried out with CMU Navlab8 and Navlab11 vehicles with the maximum speed of 45 mph in crowded urban and suburban areas verify the described work.


Question: A probability-based model for locating objects with 6-DOF position and orientation determination.

This paper outlines a novel object recognition system that employs a distinctive set of local image features, designed to be invariant to various image transformations. These features, akin to the functionalities of neurons in the primate visual system's inferior temporal cortex, are robust to changes in image scale, translation, rotation, and exhibit partial invariance to alterations in illumination and affine or 3D projection. The system detects these features efficiently through a multi-stage filtering process, pinpointing stable points within the scale space. To accommodate local geometric deformations, image keys are generated by encoding blurred image gradients across multiple orientation planes and scales. These keys serve as inputs to a nearest-neighbor indexing technique, facilitating the identification of potential object matches. Each match undergoes final verification through the computation of a low-residual least-squares solution for the unknown model parameters. The system's efficacy is demonstrated through experiments on cluttered, partially occluded images, with computational times consistently under 2 seconds, showcasing its capability for robust object recognition under challenging conditions.

The operation of industrial Automated Guided Vehicles (AGV) today requires designated infrastructure and readily available maps for their localization. In logistics, high effort and investment is necessary to enable the introduction of AGVs. Within the SICK AG coordinated EU-funded research project PAN-Robots we aim to reduce the installation time and costs dramatically by semi-automated plant exploration and localization based on natural landmarks. In this paper, we present our current mapping and localization results based on measurement data acquired at the site of our project partner Coca-Cola Iberian Partners in Bilbao, Spain. We evaluate our solution in terms of accuracy of the map, i.e. comparing landmark position estimates with a ground truth map of millimeter accuracy. The localization results are shown based on artificial landmarks as well as natural landmarks (gridmaps) based on the same graph based optimization solution.

When we look at a picture, our prior knowledge about the world allows us to resolve some of the ambiguities that are inherent to monocular vision, and thereby infer 3d information about the scene. We also recognize different objects, decide on their orientations, and identify how they are connected to their environment. Focusing on the problem of autonomous 3d reconstruction of indoor scenes, in this paper we present a dynamic Bayesian network model capable of resolving some of these ambiguities and recovering 3d information for many images. Our model assumes a "floorwall" geometry on the scene and is trained to recognize the floor-wall boundary in each column of the image. When the image is produced under perspective geometry, we show that this model can be used for 3d reconstruction from a single image. To our knowledge, this was the first monocular approach to automatically recover 3d reconstructions from single indoor images.

This paper describes the simultaneous localization and mapping (SLAM) problem and the essential methods for solving the SLAM problem and summarizes key implementations and demonstrations of the method. While there are still many practical issues to overcome, especially in more complex outdoor environments, the general SLAM method is now a well understood and established part of robotics. Another part of the tutorial summarized more recent works in addressing some of the remaining issues in SLAM, including computation, feature representation, and data association

This paper investigated various indoor positioning techniques and presented a comprehensive study about their advantages and disadvantages. Infrared, Ultrasonic and RF technologies are used in different indoor positioning systems. RFID positioning systems based on RSSI technology are the most recent developments. Positioning accuracy was greatly improved by using integrated RFID technologies.

Simultaneous tracking of multiple persons in real-world environments is an active research field and several approaches have been proposed, based on a variety of features and algorithms. Recently, there has been a growing interest in organizing systematic evaluations to compare the various techniques. Unfortunately, the lack of common metrics for measuring the performance of multiple object trackers still makes it hard to compare their results. In this work, we introduce two intuitive and general metrics to allow for objective comparison of tracker characteristics, focusing on their precision in estimating object locations, their accuracy in recognizing object configurations and their ability to consistently label objects over time. These metrics have been extensively used in two large-scale international evaluations, the 2006 and 2007 CLEAR evaluations, to measure and compare the performance of multiple object trackers for a wide variety of tracking tasks. Selected performance results are presented and the advantages and drawbacks of the presented metrics are discussed based on the experience gained during the evaluations.

The widespread use of vision-based video surveillance systems has inspired many research efforts on people localization. One of the current main trends in this field is based on probabilistic occupancy map (POM) obtained from multiple camera views. Although the POM-based approaches are robust against noisy foregrounds and can achieve great localization accuracy, they require high computation complexity. In this paper, two enhancement schemes are proposed to improve the efficiency of the POM-based people localization: (i) quick screening of potential people locations, and (ii) timely termination of iterations for occupancy probability estimation. Experimental results show that the proposed approach achieves up to 7.25 times speed-up compared to the standard POM-based approach, while delivering comparable people localization accuracy.

Probabilistic point-set registration methods have been gaining more attention for their robustness to noise, outliers and occlusions. However, these methods tend to be much slower than the popular iterative closest point (ICP) algorithms, which severely limits their usability. In this paper, we contribute a novel probabilistic registration method that achieves state-of-the-art robustness as well as substantially faster computational performance than modern ICP implementations. This is achieved using a rigorous yet computationally-efficient probabilistic formulation. Pointset registration is cast as a maximum likelihood estimation and solved using the EM algorithm. We show that with a simple augmentation, the E step can be formulated as a filtering problem, allowing us to leverage advances in efficient Gaussian filtering methods. We also propose a customized permutohedral filter [1] for improved efficiency while retaining sufficient accuracy for our task. Additionally, we present a simple and efficient twist parameterization that generalizes our method to the registration of articulated and deformable objects. For articulated objects, the complexity of our method is almost independent of the Degrees Of Freedom (DOFs), which makes it highly efficient even for high DOF systems. The results demonstrate the proposed method consistently outperforms many competitive baselines on a variety of registration tasks. The video demo and source code are available on our project page.

We introduce a new dataset, Human3.6M, of 3.6 Million accurate 3D Human poses, acquired by recording the performance of 5 female and 6 male subjects, under 4 different viewpoints, for training realistic human sensing systems and for evaluating the next generation of human pose estimation models and algorithms. Besides increasing the size of the datasets in the current state-of-the-art by several orders of magnitude, we also aim to complement such datasets with a diverse set of motions and poses encountered as part of typical human activities (taking photos, talking on the phone, posing, greeting, eating, etc.), with additional synchronized image, human motion capture, and time of flight (depth) data, and with accurate 3D body scans of all the subject actors involved. We also provide controlled mixed reality evaluation scenarios where 3D human models are animated using motion capture and inserted using correct 3D geometry, in complex real environments, viewed with moving cameras, and under occlusion. Finally, we provide a set of large-scale statistical models and detailed evaluation baselines for the dataset illustrating its diversity and the scope for improvement by future work in the research community. Our experiments show that our best large-scale model can leverage our full training set to obtain a 20% improvement in performance compared to a training set of the scale of the largest existing public dataset for this problem. Yet the potential for improvement by leveraging higher capacity, more complex models with our large dataset, is substantially vaster and should stimulate future research. The dataset together with code for the associated large-scale learning models, features, visualization tools, as well as the evaluation server, is available online at http://vision.imar.ro/human3.6m.

Mobile robot localization is the problem of determining a robot's pose from sensor data. Monte Carlo Localization is a family of algorithms for localization based on particle filters, which are approximate Bayes filters that use random samples for posterior estimation. Recently, they have been applied with great success for robot localization. Unfortunately, regular particle filters perform poorly in certain situations. Mixture-MCL, the algorithm described here, overcomes these problems by using a "dual" sampler, integrating two complimentary ways of generating samples in the estimation. To apply this algorithm for mobile robot localization, a kd-tree is learned from data that permits fast dual sampling. Systematic empirical results obtained using data collected in crowded public places illustrate superior performance, robustness, and efficiency, when compared to other state-of-the-art localization algorithms.

RFIDs are emerging as a vital component of the Internet of Things. In 2012, billions of RFIDs have been deployed to locate equipment, track drugs, tag retail goods, etc. Current RFID systems, however, can only identify whether a tagged object is within radio range (which could be up to tens of meters), but cannot pinpoint its exact location. Past proposals for addressing this limitation rely on a line-of-sight model and hence perform poorly when faced with multipath effects or non-line-of-sight, which are typical in real-world deployments. This paper introduces the first fine-grained RFID positioning system that is robust to multipath and non-line-of-sight scenarios. Unlike past work, which considers multipath as detrimental, our design exploits multipath to accurately locate RFIDs. The intuition underlying our design is that nearby RFIDs experience a similar multipath environment (e.g., reflectors in the environment) and thus exhibit similar multipath profiles. We capture and extract these multipath profiles by using a synthetic aperture radar (SAR) created via antenna motion. We then adapt dynamic time warping (DTW) techniques to pinpoint a tag's location. We built a prototype of our design using USRP software radios. Results from a deployment of 200 commercial RFIDs in our university library demonstrate that the new design can locate misplaced books with a median accuracy of 11~cm.


Question: A probability-based scheme for object search incorporating 6-DOF pose estimation.

This paper explores the intricacies of algorithmic complexity within the realm of coding theory, beginning with a comprehensive survey of the pertinent issues. It then delves into a significant finding: the computation of the minimum distance for a binary linear code is not only NP-hard, but the associated decision problem is also NP-complete. This discovery substantiates a conjecture first posited by Bedekamp, McEliece, and van Tilborg in 1978. The paper further elaborates on the implications of this breakthrough, examining how it relates to and impacts other areas within coding theory.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An Improved Genetic Algorithm for Economic Dispatch with Valve-Point Effect
In this paper, an improved genetic algorithm (IGA) is presented for solving the economic dispatch problem with valve-point effect. The IGA incorporates several enhancements over the traditional GA, such as adaptive mutation rate, tournament selection, and elitist strategy, to improve the search efficiency and convergence speed. The proposed IGA is tested on standard test systems with valve-point loading effects, and the results are compared with those obtained by other algorithms from the literature. The comparative analysis demonstrates the superior performance of the IGA in terms of solution quality and computational efficiency.

**Similar Documents**: This paper introduces an Enhanced Genetic Algorithm (EGA) designed specifically to address the economic dispatch problem in the presence of valve-point loading effects. The EGA features several innovative improvements over conventional genetic algorithms, including an adaptive mutation rate, tournament selection, and an elitist strategy, aimed at enhancing search efficacy and accelerating convergence. The performance of the EGA is evaluated on established benchmark systems with valve-point characteristics, and the outcomes are contrasted with results from alternative algorithms reported in previous studies. Comparative evaluations highlight the EGA's exceptional performance in delivering high-quality solutions with greater computational efficiency.

We propose an approach to estimate 3D human pose in real world units from a single RGBD image and show that it exceeds performance of monocular 3D pose estimation approaches from color as well as pose estimation exclusively from depth. Our approach builds on robust human keypoint detectors for color images and incorporates depth for lifting into 3D. We combine the system with our learning from demonstration framework to instruct a service robot without the need of markers. Experiments in real world settings demonstrate that our approach enables a PR2 robot to imitate manipulation actions observed from a human teacher.

We propose DeepHand to estimate the 3D pose of a hand using depth data from commercial 3D sensors. We discriminatively train convolutional neural networks to output a low dimensional activation feature given a depth map. This activation feature vector is representative of the global or local joint angle parameters of a hand pose. We efficiently identify 'spatial' nearest neighbors to the activation feature, from a database of features corresponding to synthetic depth maps, and store some 'temporal' neighbors from previous frames. Our matrix completion algorithm uses these 'spatio-temporal' activation features and the corresponding known pose parameter values to estimate the unknown pose parameters of the input feature vector. Our database of activation features supplements large viewpoint coverage and our hierarchical estimation of pose parameters is robust to occlusions. We show that our approach compares favorably to state-of-the-art methods while achieving real time performance (≈ 32 FPS) on a standard computer.

We describe a method for human pose estimation in static images based on a novel representation of part models. Notably, we do not use articulated limb parts, but rather capture orientation with a mixture of templates for each part. We describe a general, flexible mixture model for capturing contextual co-occurrence relations between parts, augmenting standard spring models that encode spatial relations. We show that such relations can capture notions of local rigidity. When co-occurrence and spatial relations are tree-structured, our model can be efficiently optimized with dynamic programming. We present experimental results on standard benchmarks for pose estimation that indicate our approach is the state-of-the-art system for pose estimation, outperforming past work by 50% while being orders of magnitude faster.

Many object classes are primarily defined by their functions. However, this fact has been left largely unexploited by visual object categorization or detection systems. We propose a method to learn an affordance detector. It identifies locations in the 3d space which “support” the particular function. Our novel approach “imagines” an actor performing an action typical for the target object class, instead of relying purely on the visual object appearance. So, function is handled as a cue complementary to appearance, rather than being a consideration after appearance-based detection. Experimental results are given for the functional category “sitting”. Such affordance is tested on a 3d representation of the scene, as can be realistically obtained through SfM or depth cameras. In contrast to appearance-based object detectors, affordance detection requires only very few training examples and generalizes very well to other sittable objects like benches or sofas when trained on a few chairs.

In this project, we jointly solve the problem of indoor object recognition and scene classification. The joint problem is modeled by a Conditional Random Field (CRF). Specifically, we implement unary potential on scene appearance, unary potential on object appearance, and binary potential on geometry between objects. The two appearance potentials are obtained by training two neural networks respectively and extracting features from them. Binary potential on object-object geometry relationship is inferred by projecting one object onto the other one’s surface and perceiving the overlap. Other potentials include object cuboid geometry, binary potential on scene-object concurrence, and binary potential on object-object concurrence. Experiments on the challenging NYU-RGBD v2 dataset show that the approach of jointly solving object detection and scene classification problems by integrating several feature representations achieves better performance than vanilla classification. Meanwhile, improving the potentials on scene appearance and object-object geometry relationship achieve fairly satisfactory improvements compared with the performance in [12]. We achieve object recognition precision of 0.3859 and scene classification accuracy of 0.6208 among 21 classes of objects and 13 classes of scenes. Both improve obviously comparing to 0.3829 and 0.6086 in baseline [12]. This result is also comparable to the state-of-art method implemented on the identical dataset [18].

Hand pose estimation has matured rapidly in recent years. The introduction of commodity depth sensors and a multitude of practical applications have spurred new advances. We provide an extensive analysis of the state-of-the-art, focusing on hand pose estimation from a single depth frame. To do so, we have implemented a considerable number of systems, and have released software and evaluation code. We summarize important conclusions here: (1) Coarse pose estimation appears viable for scenes with isolated hands. However, high precision pose estimation [required for immersive virtual reality and cluttered scenes (where hands may be interacting with nearby objects and surfaces) remain a challenge. To spur further progress we introduce a challenging new dataset with diverse, cluttered scenes. (2) Many methods evaluate themselves with disparate criteria, making comparisons difficult. We define a consistent evaluation criteria, rigorously motivated by human experiments. (3) We introduce a simple nearest-neighbor baseline that outperforms most existing systems. This implies that most systems do not generalize beyond their training sets. This also reinforces the under-appreciated point that training data is as important as the model itself. We conclude with directions for future progress.

Robust, accurate pose estimation and mapping at real-time in six dimensions is a primary need of mobile robots, in particular flying Micro Aerial Vehicles (MAVs), which still perform their impressive maneuvers mostly in controlled environments. This work presents a visual-inertial sensor unit aimed at effortless deployment on robots in order to equip them with robust real-time Simultaneous Localization and Mapping (SLAM) capabilities, and to facilitate research on this important topic at a low entry barrier. Up to four cameras are interfaced through a modern ARM-FPGA system, along with an Inertial Measurement Unit (IMU) providing high-quality rate gyro and accelerometer measurements, calibrated and hardware-synchronized with the images. This facilitates a tight fusion of visual and inertial cues that leads to a level of robustness and accuracy which is difficult to achieve with purely visual SLAM systems. In addition to raw data, the sensor head provides FPGA-pre-processed data such as visual keypoints, reducing the computational complexity of SLAM algorithms significantly and enabling employment on resource-constrained platforms. Sensor selection, hardware and firmware design, as well as intrinsic and extrinsic calibration are addressed in this work. Results from a tightly coupled reference visual-inertial motion estimation framework demonstrate the capabilities of the presented system.

Image variability due to changes in pose and illumination can seriously impair object recognition. This paper presents appearance-based methods which, unlike previous appearance-based approaches, require only a small set of training images to generate a rich representation that models this variability. Speci cally, from as few as three images of an object in xed pose seen under slightly varying but unknown lighting, a surface and an albedo map are reconstructed. These are then used to generate synthetic images with large variations in pose and illumination and thus build a representation useful for object recognition. Our methods have been tested within the domain of face recognition on a subset of the Yale Face Database B containing 4050 images of 10 faces seen under variable pose and illumination. This database was speci cally gathered for testing these generative methods. Their performance is shown to exceed that of popular existing methods.

Accurate 3-D human body pose tracking from a monocular video stream is important for a number of applications. We describe a novel hierarchical approach for tracking human pose that uses edge-based features during the coarse stage and later other features for global optimization. At first, humans are detected by motion and tracked by fitting an ellipse in the image. Then, body components are found using edge features and used to estimate the 2D positions of the body joints accurately. This helps to bootstrap the estimation of 3D pose using a sampling-based search method in the last stage. We present experiment results with sequences of different realistic scenes to illustrate the performance of the method.

We take a new approach to computing dense scene flow between a pair of consecutive RGB-D frames. We exploit the availability of depth data by seeking correspondences with respect to patches specified not as the pixels inside square windows, but as the 3D points that are the inliers of spheres in world space. Our primary contribution is to show that by reasoning in terms of such patches under 6 DoF rigid body motions in 3D, we succeed in obtaining compelling results at displacements large and small without relying on either of two simplifying assumptions that pervade much of the earlier literature: brightness constancy or local surface planarity. As a consequence of our approach, our output is a dense field of 3D rigid body motions, in contrast to the 3D translations that are the norm in scene flow. Reasoning in our manner additionally allows us to carry out occlusion handling using a 6 DoF consistency check for the flow computed in both directions and a patchwise silhouette check to help reason about alignments in occlusion areas, and to promote smoothness of the flow fields using an intuitive local rigidity prior. We carry out our optimization in two steps, obtaining a first correspondence field using an adaptation of PatchMatch, and subsequently using alpha-expansion to jointly handle occlusions and perform regularization. We show attractive flow results on challenging synthetic and real-world scenes that push the practical limits of the aforementioned assumptions.