Question: Tablets and humanoid robots as interactive tools for facilitating language instruction.

This paper delves into the multifaceted concept of embodiment within the realm of cognitive science, exploring its significance and various interpretations. While the embodiment of humans is widely accepted, the criteria for deeming an artificial construct as embodied are more contentious. The paper outlines and contrasts five distinct perspectives on embodiment: (1) Structural coupling, emphasizing the interplay between an agent and its environment; (2) Historical embodiment, which considers the embodiment as a consequence of a cumulative history of structural coupling experiences; (3) Physical embodiment, focusing on the physical attributes of an entity; (4) Organismoid embodiment, which pertains to bodies that mimic living organisms, such as humanoid robots; and (5) Organismic embodiment, exclusive to autopoietic, living systems. These definitions provide a nuanced understanding of embodiment, crucial for evaluating whether robots and other artificial entities can be considered as embodied in the same sense as living beings.

**Question**: The Effect of Social Media on Teenagers

**Documents**: The Impact of Social Networking Sites on College Students
Social networking sites (SNSs) have become an integral part of modern communication and interaction, particularly among younger generations. This study explores the effects of social networking sites on college students, focusing on the impact on their academic performance, social behavior, and mental health. Data was collected through surveys and interviews with a sample of college students from diverse backgrounds. The findings indicate that while SNSs can enhance social connections and provide educational resources, excessive use can lead to decreased academic performance, increased feelings of loneliness, and heightened anxiety. The study also reveals that students who use SNSs for academic purposes tend to have better academic outcomes compared to those who primarily use them for socializing. The results highlight the need for educational institutions to address the role of SNSs in student life and to develop guidelines for healthy social media use.

**Similar Documents**: This research investigates the influence of social networking sites (SNSs) on the lives of college students, specifically examining their academic performance, social dynamics, and psychological well-being. SNSs have transformed the way young people communicate and engage with one another, prompting a closer look at their multifaceted effects. Through a combination of surveys and interviews with a diverse group of college students, the study uncovers the dual nature of SNSs. On one hand, they can facilitate social interactions and serve as valuable educational tools. On the other hand, overreliance on these platforms can result in diminished academic focus, heightened feelings of isolation, and increased anxiety

Humour is a multi-disciplinary field of research. People have been working on humour in many fields of research like psychology, philosophy and linguistics, sociology and literature. Especially in the context of computer science (or Artificial Intelligence) humour research aims at modeling humour in a computationally tractable way. Having computational models of humour allows interface designers to have the computer generate and interpret humour when interacting with users. There are many situations in human-human interaction where humour plays an important role in keeping the conversation going. Making use of the so-called CASA paradigm (Computers Are Social Actors) we may expect that a similar role can be played in human-computer interaction. In this report we survey current humour research with the aim to identify useful theories that can be applied in the human-computer interaction context. We focus on the following subjects: humour theories, related humour research, linguistic aspects of humour, computational aspects of humour, applications and resources. 1 University of Twente, Center of Telematics and Information Technology, Technical Report CTIT-02-34, September 2002, 24 pp.

The Programming by Demonstration (PbD) technique aims at teaching a robot to accomplish a task by learning from a human demonstration. In a manipulation context, recognizing the demonstrator's hand gestures, specifically when and how objects are grasped, plays a significant role. Here, a system is presented that uses both hand shape and contact-point information obtained from a data glove and tactile sensors to recognize continuous human-grasp sequences. The sensor fusion, grasp classification, and task segmentation are made by a hidden Markov model recognizer. Twelve different grasp types from a general, task-independent taxonomy are recognized. An accuracy of up to 95% could be achieved for a multiple-user system.

We are interested in how the concept of affordances can affect our view to autonomous robot control, and how the results obtained from autonomous robotics can be reflected back upon the discussion and studies on the concept of affordances. In this paper, we studied how a mobile robot, equipped with a 3D laser scanner, can learn to perceive the traversability affordance and use it to wander in a room tilled with spheres, cylinders and boxes. The results showed that after learning, the robot can wander around avoiding contact with non-traversable objects (i.e. boxes, upright cylinders, or lying cylinders in certain orientation), but moving over traversable objects (such as spheres, and lying cylinders in a rollable orientation with respect to the robot) rolling them out of its way. We have shown that for each action approximately 1% of the perceptual features were relevant to determine whether it is afforded or not and that these relevant features are positioned in certain regions of the range image. The experiments are conducted both using a physics-based simulator and on a real robot.

In recent years, the humanoid robot is received great attention, and gradually develop to households and personal service field. The prominent progress of cloud computing, big data, and machine learning fields provides a strong support for the research of the robot. With affective interaction ability of robot has a broad market space and research value. In this paper, we propose a cloud-assisted humanoid robotics for affective interaction system architecture, and introduce the essential composition, design and implementation of related components. Finally, through an actual robot emotional interaction test platform, validating the feasibility and extendibility of proposed architecture.

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.

This article describes the software architecture of an auto nomous, interactive tour-guide robot. It presents a modular and distributed software archi te ture, which integrates localization, mapping, collision avoidance, planning, and vari ous modules concerned with user interaction and Web-based telepresence. At its heart, the s oftware approach relies on probabilistic computation, on-line learning, and any-time alg orithms. It enables robots to operate safely, reliably, and at high speeds in highly dynamic environments, and does not require any modifications of the environment to aid the robot ’s peration. Special emphasis is placed on the design of interactive capabilities that appeal to people’s intuition. The interface provides new means for human-robot interaction w ith crowds of people in public places, and it also provides people all around the world with the ability to establish a “virtual telepresence” using the Web. To illustrate our approac h, results are reported obtained in mid-1997, when our robot “RHINO” was deployed for a period of six days in a densely populated museum. The empirical results demonstrate relia bl operation in public environments. The robot successfully raised the museum’s atten dance by more than 50%. In addition, thousands of people all over the world controlled the robot through the Web. We conjecture that these innovations transcend to a much large r range of application domains for service robots.

Facial expressions play a significant role in human dialogue. As a result, there has been considerable work done on the recognition of emotional expressions and the application of this research will be beneficial in improving human-machine dialogue. One can imagine the improvements to computer interfaces, automated clinical (psychological) research or even interactions between humans and autonomous robots.

The study presented in this paper explored people's perceptions and attitudes towards the idea of a future robot companion for the home. A human-centred approach was adopted using questionnaires and human-robot interaction trials to derive data from 28 adults. Results indicated that a large proportion of participants were in favour of a robot companion and saw the potential role as being an assistant, machine or servant. Few wanted a robot companion to be a friend. Household tasks were preferred to child/animal care tasks. Humanlike communication was desirable for a robot companion, whereas humanlike behaviour and appearance were less essential. Results are discussed in relation to future research directions for the development of robot companions.

We present a Bag of Words-based active object categorization technique implemented and tested on a humanoid robot. The robot is trained to categorize objects that are handed to it by a human operator. The robot uses hand and head motions to actively acquire a number of different views. A view planning scheme using entropy minimization reduces the number of views needed to achieve a valid decision. Categorization results are significantly improved by active elimination of background features using robot arm motion. Our experiments cover both, categorization when the object is handed to the robot in a fixed pose at training and testing, and object pose independent categorization. Results on a 4-class object database demonstrate the classification efficiency, a significant gain from multi-view compared to single-view classification, and the advantage of view planning. We conclude that humanoid robotic systems can be successfully applied to actively categorize objects a task with many potential applications ranging from edutainment to active surveillance.

1. ABSTRACT Animated agents based either on real video, cartoon-style drawings or even model-based 3D graphics offer great promise for computer-based presentations as they make presentations more lively and appealing and allow for the emulation of conversation styles known from human-human communication. In this paper, we describe a life-like interface agent which presents multimedia material to the user following the directives of a script. The overall behavior of the presentation agent is partly determined by such a script, and partly by the agent's self-behavior. In our approach, the agent's behavior is defined in a declarative specification language. Behavior specifications are used to automatically generate a control module for an agent display system. The first part of the paper describes the generation process which involves AI planning and a two-step compilation. Since the manual creation of presentation scripts is tedious and error-prone, we also address the automated generation of presentation scripts which may be forwarded to the interface agent. The second part of the paper presents an approach for multimedia presentation design which combines hierarchical planning with temporal reasoning.


Question: ModDrop: An adaptive method for multi-modal gesture recognition.

This paper explores the recognition of natural human actions within varied and realistic video contexts, a topic that has historically received limited attention due to challenges including the scarcity of annotated, realistic video datasets. Our initial contribution involves addressing this gap by investigating the use of movie scripts for the automatic annotation of human actions in videos. We assess different techniques for extracting action instances from scripts and demonstrate the effectiveness of a text-based classifier. Utilizing the retrieved action samples for visual learning, we then address the task of action classification in video sequences. We introduce a novel method for video classification that builds upon and expands ideas such as local space-time features, space-time pyramids, and multi-channel non-linear SVMs. This method achieves state-of-the-art results on the widely used KTH action dataset, with an accuracy rate of 91.8%. Recognizing the inherent issue of noisy labels in automatically annotated datasets, we specifically examine and illustrate the robustness of our method to annotation inaccuracies in the training data. Ultimately, we apply this method to the learning and classification of complex action classes within movie scenes, yielding encouraging outcomes.

Domain adaptation is an important emerging topic in computer vision. In this paper, we present one of the first studies of domain shift in the context of object recognition. We introduce a method that adapts object models acquired in a particular visual domain to new imaging conditions by learning a transformation that minimizes the effect of domain-induced changes in the feature distribution. The transformation is learned in a supervised manner and can be applied to categories for which there are no labeled examples in the new domain. While we focus our evaluation on object recognition tasks, the transform-based adaptation technique we develop is general and could be applied to non-image data. Another contribution is a new multi-domain object database, freely available for download. We experimentally demonstrate the ability of our method to improve recognition on categories with few or no target domain labels and moderate to large changes in the imaging conditions.

In this work, we evaluate the feasibility of tracking-based interaction using a mobile phone's or tablet's camera in order to create and edit 3D objects in augmented reality applications. We present a feasibility study investigating if and how gestures made with your finger can be used to create such objects. A revised interface design is evaluated in a user study with 24 subjects that reveals a high usability and entertainment value, but also identifies issues such as ergonomic discomfort and imprecise input for complex tasks. Hence, our results suggest huge potential for this type of interaction in the entertainment, edutainment, and leisure domain, but limited usefulness for serious applications.

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

This paper introduces a novel human activity monitoring system combining Inertial Measurement Units (IMU) and Mechanomyographic (MMG) muscle sensing technology. While other work has recognised and implemented systems for combined muscle activity and motion recording, they have focused on muscle activity through EMG sensors, which have limited use outside controlled environments. MMG is a low frequency vibration emitted by skeletal muscle whose measurement does not require gel or adhesive electrodes, potentially offering much more efficient implementation for pervasive use. We have developed a combined IMU and MMG sensor the fusion of which provides a new dimension in human activity monitoring and bridges the gap between human dynamics and muscle activity. Results show that synergy between motion and muscle detection is not only viable, but straightforward, inexpensive, and can be packaged in a lightweight wearable package.

Short-range continuous-wave Doppler radar sensors have been mainly used for noncontact detection of various motions. In this paper, we investigate the feasibility to implement the function of a remote mouse, an input device of a computer, by recognizing human gestures based on a dual-channel Doppler radar sensor. Direct conversion architecture, symmetric subcarrier modulation, and bandpass sampling techniques are used to obtain a cost-effective solution. An arcsine algorithm and a motion imaging algorithm are proposed to linearly reconstruct the hand and finger motions in a 2-D plane from the demodulated Doppler phase shifts. Experimental results verified the effectiveness of the proposed architecture and algorithms. Different from the frequency-domain “micro-Doppler” approach, the proposed remote gesture recognition based on linear motion reconstruction is able to recognize definitive signatures for the corresponding motions, exhibiting promising potential in practical applications of human-computer interaction.

Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

Enabling touch gesture sensing on all surfaces of the mobile device, not limited to the touchscreen area, leads to new user interaction experiences. In this paper, we propose VSkin, a system that supports fine-grained gesture-sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds, i.e., sounds propagating through the structure of the device, and the air-borne sounds, i.e., sounds propagating through the air, to sense finger tapping and movements. By measuring both the amplitude and the phase of each path of sound signals, VSkin detects tapping events with an accuracy of 99.65% and captures finger movements with an accuracy of 3.59mm.

A 3-Finger Adaptive Robot Gripper is an advanced robotic research that provides a robotic hand-like capabilities due to its flexibility and versatility. However, the grasping performance has to be analyzed and monitored based on the motor encoder, motor current, and force feedback so that the finger position and grasping force can be effectively controlled. This paper provides an open-loop grasping analysis for a 3-Finger Adaptive Robot Gripper. A series of grasping tests has been conducted to demonstrate the robot capabilities and functionalities. Different stiffness levels of the grasped objects have been chosen to demonstrate the grasping ability. In the experiment, a Modbus RTU protocol and Matlab/Simulink are used as communication and control platform. A specially modified interlink FSR sensor is proposed where a special plastic cover has been developed to enhance the sensor sensitivity. The Arduino IO Package is employed to interface the sensor and Matlab/Simulink. The results show that the significant relationships between finger position, motor current, and force sensor are found and the results can be used for a proper grasping performance.

In this paper, we propose an acceleration-based human activity recognition method using popular deep architecture, Convolution Neural Network (CNN). In particular, we construct a CNN model and modify the convolution kernel to adapt the characteristics of tri-axial acceleration signals. Also, for comparison, we use some widely used methods to accomplish the recognition task on the same dataset. The large dataset we constructed consists of 31688 samples from eight typical activities. The experiment results show that the CNN works well, which can reach an average accuracy of 93.8% without any feature extraction methods.

Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality.
 The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.


Question: ModDrop: An adaptive system for multi-modal gesture identification.

This paper introduces DeepFace, a novel approach to face recognition that significantly narrows the gap to human-level performance in face verification tasks. The traditional face recognition pipeline involves four key steps: detection, alignment, representation, and classification. We focus on refining the alignment and representation stages by integrating explicit 3D face modeling, which enables us to apply a piecewise affine transformation. This transformation enhances the accuracy of face representation, which is then derived from a deep neural network consisting of nine layers. Unlike conventional convolutional layers, our network utilizes more than 120 million parameters across several locally connected layers without weight sharing. To train this complex model, we utilized the largest facial dataset to date, featuring over four million facial images from more than 4,000 distinct identities. The representations learned through this process, when combined with precise model-based alignment and a vast facial database, display exceptional generalization capabilities, even in unconstrained environments, with a simple classifier. Our method achieves a remarkable 97.35% accuracy on the Labeled Faces in the Wild (LFW) dataset, marking a 27% reduction in error compared to the previous state of the art, and closely mirroring human-level performance.

**Question**: A Comparative Study on Text Classification Techniques

**Documents**: A Comparative Study of Machine Learning Algorithms for Classification of Credit Card Fraud
Credit card fraud is a serious problem for financial institutions, costing billions of dollars annually. In this paper, we present a comparative study of machine learning algorithms for the classification of credit card fraud. We evaluate several popular algorithms, including logistic regression, decision trees, random forests, gradient boosting, and support vector machines. We use a large dataset of credit card transactions, with over 284,000 records, to train and test our models. The dataset contains a highly imbalanced class distribution, with only 0.172% of fraudulent transactions. We address this issue by applying oversampling techniques such as SMOTE and ADASYN to balance the dataset. We then evaluate the performance of the models using various metrics such as accuracy, precision, recall, and F1-score. Our results show that gradient boosting and random forests outperform the other algorithms in terms of accuracy and F1-score. However, logistic regression and decision trees have higher precision and lower false positive rates. We also find that the oversampling techniques improve the performance of all algorithms, but they have a greater impact on the algorithms that are sensitive to class imbalance, such as logistic regression and decision trees

In this paper we present a novel wearable navigation system along with implicit HCI (iHCI) model, where interaction with technology is dissolved into a day-today activity. In this type of HCI model a computer takes the input and tries to output an action that is a proactive anticipation of next action of a user. Usually, in urban areas people use voice assisted navigation systems or navigation guidelines displayed on a mobile phone. Some navigation systems are already installed on car dashboard, which needs explicit attention in order to make driving decisions. A navigation system using haptic perception to guide a user throughout a journey is the key contribution of this paper. It does not ask for explicit user attention and demonstrates the indolent form of technological interaction. This wearable device is an index finger sleeve, which consists of vibrator modules, Bluetooth communication module and Microcontroller Unit (MCU). A working prototype has been built and tested.

In this article, we consider the design of a human gesture recognition system based on pattern recognition of signatures from a portable smart radar sensor. Powered by AAA batteries, the smart radar sensor operates in the 2.4 GHz industrial, scientific and medical (ISM) band. We analyzed the feature space using principle components and application-specific time and frequency domain features extracted from radar signals for two different sets of gestures. We illustrate that a nearest neighbor based classifier can achieve greater than 95% accuracy for multi class classification using 10 fold cross validation when features are extracted based on magnitude differences and Doppler shifts as compared to features extracted through orthogonal transformations. The reported results illustrate the potential of intelligent radars integrated with a pattern recognition system for high accuracy smart home and health monitoring purposes.

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

Demand for interactive wearable devices is rapidly increasing with the development of smart devices. To accurately utilize wearable devices for remote robot controls, limited data should be analyzed and utilized efficiently. For example, the motions by a wearable device, called Myo device, can be estimated by measuring its orientation, and calculating a Bayesian probability based on these orientation data. Given that Myo device can measure various types of data, the accuracy of its motion estimation can be increased by utilizing these additional types of data. This paper proposes a motion estimation method based on weighted Bayesian probability and concurrently measured data, orientations and electromyograms (EMG). The most probable motion among estimated is treated as a final estimated motion. Thus, recognition accuracy can be improved when compared to the traditional methods that employ only a single type of data. In our experiments, seven subjects perform five predefined motions. When orientation is measured by the traditional methods, the sum of the motion estimation errors is 37.3%; likewise, when only EMG data are used, the error in motion estimation by the proposed method was also 37.3%. The proposed combined method has an error of 25%. Therefore, the proposed method reduces motion estimation errors by 12%.

Both recognizing human behavior and understanding a user's mobility from sensor data are critical issues in ubiquitous computing systems. As a kind of user behavior, the transportation modes, such as walking, driving, etc., that a user takes, can enrich the user's mobility with informative knowledge and provide pervasive computing systems with more context information. In this paper, we propose an approach based on supervised learning to infer people's motion modes from their GPS logs. The contribution of this work lies in the following two aspects. On one hand, we identify a set of sophisticated features, which are more robust to traffic condition than those other researchers ever used. On the other hand, we propose a graph-based post-processing algorithm to further improve the inference performance. This algorithm considers both the commonsense constraint of real world and typical user behavior based on location in a probabilistic manner. Using the GPS logs collected by 65 people over a period of 10 months, we evaluated our approach via a set of experiments. As a result, based on the change point-based segmentation method and Decision Tree-based inference model, the new features brought an eight percent improvement in inference accuracy over previous result, and the graph-based post-processing achieve a further four percent enhancement.

Reliable prediction of affective states in real world scenarios is very challenging and a significant amount of ongoing research is targeted towards improvement of existing systems. Major problems include the unreliability of labels, variations of the same affective states amongst different persons and in different modalities as well as the presence of sensor noise in the signals. This work presents a framework for adaptive fusion of input modalities incorporating variable degrees of certainty on different levels. Using a strategy that starts with ensembles of weak learners, gradually, level by level, the discriminative power of the system is improved by adaptively weighting favorable decisions, while concurrently dismissing unfavorable ones. For the final decision fusion the proposed system leverages a trained Kalman filter. Besides its ability to deal with missing and uncertain values, in its nature, the Kalman filter is a time series predictor and thus a suitable choice to match input signals to a reference time series in the form of ground truth labels. In the case of affect recognition, the proposed system exhibits superior performance in comparison to competing systems on the analysed dataset.

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

We propose an intuitive user interaction system with virtual objects by directly using user's hands through a physics simulation. This system aims to provide a more direct way to manipulate and interact with virtual objects and to induce more realistic reactions of the virtual objects in the interaction than the existing methods. The key contribution of this paper is to model the deformations of the user's hands in two ways: global deformation by the hand poses and the local deformation by contacts with objects. By the two deformation models, objects can be manipulated with better stability and robustness without any predefinition of interaction methods depending on the object shapes, by solely using a physics simulation. The experiment shows that the proposed method enables a user to intuitively manipulate virtual object accurately.

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.

Short-range continuous-wave Doppler radar sensors have been mainly used for noncontact detection of various motions. In this paper, we investigate the feasibility to implement the function of a remote mouse, an input device of a computer, by recognizing human gestures based on a dual-channel Doppler radar sensor. Direct conversion architecture, symmetric subcarrier modulation, and bandpass sampling techniques are used to obtain a cost-effective solution. An arcsine algorithm and a motion imaging algorithm are proposed to linearly reconstruct the hand and finger motions in a 2-D plane from the demodulated Doppler phase shifts. Experimental results verified the effectiveness of the proposed architecture and algorithms. Different from the frequency-domain “micro-Doppler” approach, the proposed remote gesture recognition based on linear motion reconstruction is able to recognize definitive signatures for the corresponding motions, exhibiting promising potential in practical applications of human-computer interaction.


Question: ModDrop: An adaptive method for multi-modal gesture recognition.

This paper presents a comprehensive framework for multi-modal gesture recognition, integrating diverse data sources like depth video, articulated body pose, and speech. Our methodology uniquely dissects each gesture into two components: large-scale body movement and localized, nuanced actions like hand gestures. The principle of multi-scale learning is extended to the time domain, recognizing gestures as a sequence of distinct motion pulses or dynamic postures. Each data modality undergoes individual processing in brief spatio-temporal segments, with features either meticulously selected or automatically learned to maximize their discriminatory power. Subsequently, a Recurrent Neural Network (RNN) is deployed to capture long-term temporal relationships, integrate the various modalities, and execute the final gesture classification. Our empirical evaluation on the 2013 Multimodal Gesture Recognition Challenge dataset has confirmed that leveraging multiple modalities at different spatial and temporal scales significantly boosts recognition accuracy. This approach enables the system to mitigate the shortcomings of individual classifiers and handle noise across separate channels more effectively, thereby enhancing overall performance.

Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality.
 The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.

It usually takes a fusion of image processing and machine learning algorithms in order to build a fully-functioning computer vision system for hand gesture recognition. Fortunately, the complexity of developing such a system could be alleviated by treating the system as a collection of multiple sub-systems working together, in such a way that they can be dealt with in isolation. Machine learning need to feed on thousands of exemplars (e.g. images, features) to automatically establish some recognisable patterns for all possible classes (e.g. hand gestures) that applies to the problem domain. A good number of exemplars helps, but it is also important to note that the efficacy of these exemplars depends on the variability of illumination conditions, hand postures, angles of rotation, scaling and on the number of volunteers from whom the hand gesture images were taken. These exemplars are usually subjected to image processing first, to reduce the presence of noise and extract the important features from the images. These features serve as inputs to the machine learning system. Different sub-systems are integrated together to form a complete computer vision system for gesture recognition. The main contribution of this work is on the production of the exemplars. We discuss how a dataset of standard American Sign Language (ASL) hand gestures containing 2425 images from 5 individuals, with variations in lighting conditions and hand postures is generated with the aid of image processing techniques. A minor contribution is given in the form of a specific feature extraction method called moment invariants, for which the computation method and the values are furnished with the dataset.

Cross-modal retrieval aims to enable flexible retrieval experience across different modalities (e.g., texts vs. images). The core of cross-modal retrieval research is to learn a common subspace where the items of different modalities can be directly compared to each other. In this paper, we present a novel Adversarial Cross-Modal Retrieval (ACMR) method, which seeks an effective common subspace based on adversarial learning. Adversarial learning is implemented as an interplay between two processes. The first process, a feature projector, tries to generate a modality-invariant representation in the common subspace and to confuse the other process, modality classifier, which tries to discriminate between different modalities based on the generated representation. We further impose triplet constraints on the feature projector in order to minimize the gap among the representations of all items from different modalities with same semantic labels, while maximizing the distances among semantically different images and texts. Through the joint exploitation of the above, the underlying cross-modal semantic structure of multimedia data is better preserved when this data is projected into the common subspace. Comprehensive experimental results on four widely used benchmark datasets show that the proposed ACMR method is superior in learning effective subspace representation and that it significantly outperforms the state-of-the-art cross-modal retrieval methods.

This paper presents a trimodal biometric recognition system based on iris, palmprint and fingerprint. Wavelet transform and Gabor filter are used to extract features in different scales and orientations from iris and palmprint. Minutiae extraction and alignment is used for fingerprint matching. Six different fusion algorithms including score-based, rank-based and decision-based methods are used to combine the results of three modules. We also propose a new rank-based fusion algorithm Maximum Inverse Rank (MIR) which is robust with respect to variations in scores and also bad ranking from a module. CASIA datasets for iris, palmprint and fingerprint are used in this study. The experiments show the effectiveness of our fusion method and our trimodal biometric recognition system in comparison to existing multi-modal recognition systems.

Interacting with smartwatches poses new challenges. Although capable of displaying complex content, their extremely small screens poorly match many of the touchscreen interaction techniques dominant on larger mobile devices. Addressing this problem, this paper presents beating gestures, a novel form of input based on pairs of simultaneous or rapidly sequential and overlapping screen taps made by the index and middle finger of one hand. Distinguished simply by their temporal sequence and relative left/right position these gestures are designed explicitly for the very small screens (approx. 40mm square) of smartwatches and to operate without interfering with regular single touch input. This paper presents the design of beating gestures and a rigorous empirical study that characterizes how users perform them -- in a mean of 355ms and with an error rate of 5.5%. We also derive thresholds for reliably distinguishing between simultaneous (under 30ms) and sequential (under 400ms) pairs of screen touches or releases. We then present five interface designs and evaluate them in a qualitative study in which users report valuing the speed and ready availability of beating gestures.

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

ÐA new method for the representation, recognition, and interpretation of parameterized gesture is presented. By parameterized gesture we mean gestures that exhibit a systematic spatial variation; one example is a point gesture where the relevant parameter is the two-dimensional direction. Our approach is to extend the standard hidden Markov model method of gesture recognition by including a global parametric variation in the output probabilities of the HMM states. Using a linear model of dependence, we formulate an expectation-maximization (EM) method for training the parametric HMM. During testing, a similar EM algorithm simultaneously maximizes the output likelihood of the PHMM for the given sequence and estimates the quantifying parameters. Using visually derived and directly measured three-dimensional hand position measurements as input, we present results that demonstrate the recognition superiority of the PHMM over standard HMM techniques, as well as greater robustness in parameter estimation with respect to noise in the input features. Last, we extend the PHMM to handle arbitrary smooth (nonlinear) dependencies. The nonlinear formulation requires the use of a generalized expectation-maximization (GEM) algorithm for both training and the simultaneous recognition of the gesture and estimation of the value of the parameter. We present results on a pointing gesture, where the nonlinear approach permits the natural spherical coordinate parameterization of pointing direction. Index TermsÐGesture recognition, hidden Markov models, expectation-maximization algorithm, time-series modeling, computer vision.

The confluence of decades of computer science and computer engineering research in multimodal interaction (e.g., speech and gesture recognition), machine learning (e.g., classification and feature extraction), software (e.g., web browsers, distributed agents), electronics (e.g., energy-efficient microprocessors, head-mounted displays), design methodology in user-centered design, and rapid prototyping have enabled a new class of computers—wearable computers. The lecture takes the viewpoint of a potential designer or researcher in wearable computing. Designing wearable computers requires attention to many different factors because of the computer’s closeness to the body and its use while performing other tasks. For the purposes of discussion, we have created the UCAMP framework, which consists of the following factors: user, corporal, attention, manipulation, and perception. Each of these factors and their importance is described. A number of example prototypes developed by the authors, as well as by other researchers, are used to illustrate these concepts. Wearable computers have established their first foothold in several application domains, such as vehicle and aircraft maintenance and manufacturing, inspection, language translation, and other areas. The lecture continues by describing the next step in the evolution of wearable computers, namely, context awareness. Context-aware computing takes into account a user’s state and surroundings, and the mobile computer modifies its behavior based on this information. A user’s context can be quite rich, consisting of attributes such as physical location, physiological state, personal history, daily behavioral patterns, and so forth. If a human assistant were given such context, he or she would make decisions in a proactive fashion, anticipating user needs, and acting as a proactive assistant. The goal is to enable mobile computers to play an analogous role, exploiting context information to significantly reduce demands on human attention. Context-aware intelligent agents can deliver relevant information when a user needs that information. These data make possible many exciting new applications, such as augmented reality, context-aware collaboration, and augmented manufacturing. The combined studies and research reported in this lecture suggest a number of useful guidelines for designing wearable computing devices. Also included with the guidelines is a list of questions that designers should consider when beginning to design a wearable computer. The research directions section emphasizes remaining challenges and trends in the areas of user interface, modalities of interaction, and wearable cognitive augmentation. Finally, we summarize the most important challenges and conclude with a projection of future directions in wearable computing. KEyWoRDS wearable computers, wearability design, user interface, manipulation devices, interaction modalities, context-aware computing, cognitive augmentation vi

Recognizing how objects interact with each other is a crucial task in visual recognition. If we define the context of the interaction to be the objects involved, then most current methods can be categorized as either: (i) training a single classifier on the combination of the interaction and its context; or (ii) aiming to recognize the interaction independently of its explicit context. Both methods suffer limitations: the former scales poorly with the number of combinations and fails to generalize to unseen combinations, while the latter often leads to poor interaction recognition performance due to the difficulty of designing a contextindependent interaction classifier.,,To mitigate those drawbacks, this paper proposes an alternative, context-aware interaction recognition framework. The key to our method is to explicitly construct an interaction classifier which combines the context, and the interaction. The context is encoded via word2vec into a semantic space, and is used to derive a classification result for the interaction. The proposed method still builds one classifier for one interaction (as per type (ii) above), but the classifier built is adaptive to context via weights which are context dependent. The benefit of using the semantic space is that it naturally leads to zero-shot generalizations in which semantically similar contexts (subject-object pairs) can be recognized as suitable contexts for an interaction, even if they were not observed in the training set. Our method also scales with the number of interaction-context pairs since our model parameters do not increase with the number of interactions. Thus our method avoids the limitation of both approaches. We demonstrate experimentally that the proposed framework leads to improved performance for all investigated interaction representations and datasets.

Enabling touch gesture sensing on all surfaces of the mobile device, not limited to the touchscreen area, leads to new user interaction experiences. In this paper, we propose VSkin, a system that supports fine-grained gesture-sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds, i.e., sounds propagating through the structure of the device, and the air-borne sounds, i.e., sounds propagating through the air, to sense finger tapping and movements. By measuring both the amplitude and the phase of each path of sound signals, VSkin detects tapping events with an accuracy of 99.65% and captures finger movements with an accuracy of 3.59mm.


Question: ModDrop: An adaptive system for multi-modal gesture identification.

This paper outlines an unsupervised learning technique designed to construct a hierarchical framework of sparse feature detectors that are robust against minor shifts and distortions. The feature extraction model is composed of multiple convolution filters, followed by a pooling layer that calculates the maximum output of each filter within contiguous windows, and subsequently applies a sigmoid non-linearity on a point-by-point basis. The subsequent level of features, characterized by increased size and invariance, is attained through the application of the identical algorithm on patches derived from the features of the preceding level. When a supervised classifier is trained on these features, it achieves an error rate of 0.64% on the MNIST dataset and an average recognition rate of 54% on the Caltech 101 dataset, using only 30 training samples per category. Despite the resemblance to convolutional networks, the layer-wise unsupervised training process effectively mitigates the overfitting issues common to purely supervised learning methodologies, thereby delivering commendable performance with a significantly reduced number of labeled training instances.

In this paper, we introduce an unsupervised learning approach for building a hierarchy of sparse feature detectors that exhibit invariance to small transformations and distortions. The proposed feature extractor architecture consists of multiple convolution filters, followed by a pooling layer that computes the maximum value of each filter's output within overlapping windows, and a point-wise sigmoid non-linearity. A higher level of features, with larger size and enhanced invariance, is achieved by training the same algorithm on patches of features extracted from the initial level. When a supervised classifier is trained using these features, it demonstrates an error rate of 0.64% on the MNIST dataset and an average recognition rate of 54% on the Caltech 101 dataset, with only 30 labeled examples per category. While the structure of the resulting architecture mirrors that of convolutional networks, the layer-wise unsupervised training process circumvents the overfitting challenges typically associated with supervised learning, leading to superior performance with a minimal amount of labeled training data.

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

Nowadays, the convergence of devices, electronic computing, and massive media produces huge volumes of information, which demands the need for faster and more efficient interaction between users and information. How to make information access manageable, efficient, and easy becomes the major challenge for Human-Computer Interaction (HCI) researchers. The different types of computing devices, such as PDAs (personal digital assistants), tablet PCs, desktops, game consoles, and the next generation phones, provide many different modalities for information access. This makes it possible to dynamically adapt application user interfaces to the changing context. However, as applications go more and more pervasive, these devices show theirs limited input/output capacity caused by small visual displays, use of hands to operate buttons and the lack of an alphanumeric keyboard and mouse (Gu & Gilbert, 2004). Voice User Interface (VUI) systems are capable of, besides recognizing the voice of their users, to understand voice commands, and to provide responses to them, usually, in real time. The state-of-the-art in speech technology already allows the development of automatic systems designed to work in real conditions. VUI is perhaps the most critical factor in the success of any automated speech recognition (ASR) system, determining whether the user experience will be satisfying or frustrating, or even whether the customer will remain one. This chapter describes a practical methodology for creating an effective VUI design. The methodology is scientifically based on principles in linguistics, psychology, and language technology (Cohen et al. 2004; San-Segundo et al., 2005). Given the limited input/output capabilities of mobile devices, speech presents an excellent way to enter and retrieve information either alone or in combination with other modalities. Furthermore, people with disabilities should be provided with a wide range of alternative interaction modalities other than the traditional screen-mouse based desktop computing devices. Whether the disability is temporary or permanent, people with reading difficulty, visual impairment, and/or any difficulty using a keyboard, or mouse can rely on speech as an alternate approach for information access. Source: User Interfaces, Book edited by: Rita Mátrai, ISBN 978-953-307-084-1, pp. 270, May 2010, INTECH, Croatia, downloaded from SCIYO.COM

A major difficulty in writing Single Display Groupware (co-present collaborative) applications is getting input from multiple devices. We introduce MID, a Java package that addresses this problem and offers an architecture to access advanced events through Java. In this paper, we describe the features, architecture and limitations of MID. We also briefly describe an application that uses MID to get input from multiple mice: KidPad.

Our goal is to automate the understanding of natural hand-object manipulation by developing computer visionbased techniques. Our hypothesis is that it is necessary to model the grasp types of hands and the attributes of manipulated objects in order to accurately recognize manipulation actions. Specifically, we focus on recognizing hand grasp types, object attributes and actions from a single image within an unified model. First, we explore the contextual relationship between grasp types and object attributes, and show how that context can be used to boost the recognition of both grasp types and object attributes. Second, we propose to model actions with grasp types and object attributes based on the hypothesis that grasp types and object attributes contain complementary information for characterizing different actions. Our proposed action model outperforms traditional appearance-based models which are not designed to take into account semantic constraints such as grasp types or object attributes. Experiment results on public egocentric activities datasets strongly support our hypothesis.

This paper describes a depth image based real-time skeleton fitting algorithm for the hand, using an object recognition by parts approach, and the use of this hand modeler in an American Sign Language (ASL) digit recognition application. In particular, we created a realistic 3D hand model that represents the hand with 21 different parts. Random decision forests (RDF) are trained on synthetic depth images generated by animating the hand model, which are then used to perform per pixel classification and assign each pixel to a hand part. The classification results are fed into a local mode finding algorithm to estimate the joint locations for the hand skeleton. The system can process depth images retrieved from Kinect in real-time at 30 fps. As an application of the system, we also describe a support vector machine (SVM) based recognition module for the ten digits of ASL based on our method, which attains a recognition rate of 99.9% on live depth images in real-time1.

This paper presents a new design of a wireless sensor glove developed for American Sign Language finger spelling gesture recognition. Five contact sensors are installed on the glove, in addition to five flex sensors on the fingers and a 3D accelerometer on the back of the hand. Each pair of flex and contact sensors are combined into the same input channel on the BSN node in order to save the number of channels and the installation area. After which, the signal is analyzed and separated back into flex and contact features by software. With electrical contacts and wirings made of conductive fabric and threads, the glove design has become thinner and more flexible. For validation, ASL finger spelling gesture recognition experiments have been performed on signals collected from six speech-impaired subjects and a normal subject. With the new sensor glove design, the experimental results have shown a significant increase in classification accuracy.

Hand pose estimation has matured rapidly in recent years. The introduction of commodity depth sensors and a multitude of practical applications have spurred new advances. We provide an extensive analysis of the state-of-the-art, focusing on hand pose estimation from a single depth frame. To do so, we have implemented a considerable number of systems, and have released software and evaluation code. We summarize important conclusions here: (1) Coarse pose estimation appears viable for scenes with isolated hands. However, high precision pose estimation [required for immersive virtual reality and cluttered scenes (where hands may be interacting with nearby objects and surfaces) remain a challenge. To spur further progress we introduce a challenging new dataset with diverse, cluttered scenes. (2) Many methods evaluate themselves with disparate criteria, making comparisons difficult. We define a consistent evaluation criteria, rigorously motivated by human experiments. (3) We introduce a simple nearest-neighbor baseline that outperforms most existing systems. This implies that most systems do not generalize beyond their training sets. This also reinforces the under-appreciated point that training data is as important as the model itself. We conclude with directions for future progress.

Activity recognition fits within the bigger framework of context awareness. In this paper, we report on our efforts to recognize user activity from accelerometer data. Activity recognition is formulated as a classification problem. Performance of base-level classifiers and meta-level classifiers is compared. Plurality Voting is found to perform consistently well across different settings.

Body-worn sensors in general and accelerometers in particular have been widely used in order to detect human movements and activities. The execution of each type of movement by each particular individual generates sequences of time series of sensed data from which specific movement related patterns can be assessed. Several machine learning algorithms have been used over windowed segments of sensed data in order to detect such patterns in activity recognition based on intermediate features (either hand-crafted or automatically learned from data). The underlying assumption is that the computed features will capture statistical differences that can properly classify different movements and activities after a training phase based on sensed data. In order to achieve high accuracy and recall rates (and guarantee the generalization of the system to new users), the training data have to contain enough information to characterize all possible ways of executing the activity or movement to be detected. This could imply large amounts of data and a complex and time-consuming training phase, which has been shown to be even more relevant when automatically learning the optimal features to be used. In this paper, we present a novel generative model that is able to generate sequences of time series for characterizing a particular movement based on the time elasticity properties of the sensed data. The model is used to train a stack of auto-encoders in order to learn the particular features able to detect human movements. The results of movement detection using a newly generated database with information on five users performing six different movements are presented. The generalization of results using an existing database is also presented in the paper. The results show that the proposed mechanism is able to obtain acceptable recognition rates (F = 0.77) even in the case of using different people executing a different sequence of movements and using different hardware.

A novel data glove embedded low cost MEMS inertial and magnetic measurement units, is proposed for fingers motion capture. Each unit consists of a tri-axial gyroscope, a tri-axial accelerometer and a tri-axial magnetometer. The sensor board and processor board are compactly designed, which are small enough to fit the size of our fingers. The data glove is equipped with fifteen units to measure each joint angle of the fingers. Then the calibration approach is put up to improve the accuracy of measurements by both offline and online procedures, and a fast estimation method is used to determine the three orientations of fifteen units simultaneously. The proposed algorithm is easy to be implemented, and more precise and efficient measurements can be obtained as compared with existing methods. The fingers motion capture experiments are implemented to acquire the characteristics of the fingers and teleoperate the robotic hands, which prove the effectiveness of the data glove.


Question: ModDrop: An adaptive approach for multi-modal gesture recognition.

This paper presents an efficient model-based technique for real-time 3D tracking of hand articulations using Kinect sensor data. The problem of tracking the intricate movements of human hands is both theoretically intriguing and practically significant, with a wide range of applications. Previous efforts have primarily focused on markerless visual data, leading to two main categories of approaches: model-based and appearance-based techniques. Our proposed method falls under the model-based category (Fig. 1), where we define the problem as an optimization task aimed at minimizing the disparity between the hypothesized 3D hand model instances' structure and appearance, and the actual visual observations captured by the Kinect. To solve this optimization problem, we employ a modified version of the Particle Swarm Optimization (PSO) algorithm, a well-established stochastic optimization method. To enhance computational efficiency, particularly in the most resource-intensive stages of the process, we leverage GPU computing. This results in a system capable of tracking hand articulations at a high rate of 15Hz, showcasing its suitability for real-time applications.

We address the problem of structure learning of human motion in order to recognize actions from a continuous monocular motion sequence of an arbitrary person from an arbitrary viewpoint. Human motion sequences are represented by multivariate time series in the joint-trajectories space. Under this structured time series framework, we first propose Kernelized Temporal Cut (KTC), an extension of previous works on change-point detection by incorporating Hilbert space embedding of distributions, to handle the nonparametric and high dimensionality issues of human motions. Experimental results demonstrate the effectiveness of our approach, which yields realtime segmentation, and produces high action segmentation accuracy. Second, a spatio-temporal manifold framework is proposed to model the latent structure of time series data. Then an efficient spatio-temporal alignment algorithm Dynamic Manifold Warping (DMW) is proposed for multivariate time series to calculate motion similarity between action sequences (segments). Furthermore, by combining the temporal segmentation algorithm and the alignment algorithm, online human action recognition can be performed by associating a few labeled examples from motion capture data. The results on human motion capture data and 3D depth sensor data demonstrate the effectiveness of the proposed approach in automatically segmenting and recognizing motion sequences, and its ability to handle noisy and partially occluded data, in the transfer learning module.

We describe the design and evaluation of a geometric interaction technique for bare-hand mid-air virtual pottery. We model the shaping of a pot as a gradual and progressive convergence of the potprofile to the shape of the user’s hand represented as a point-cloud (PCL). Our pottery-inspired application served as a platform for systematically revealing how users use their hands to express the intent of deformation during a pot shaping process. Through our approach, we address two specific problems: (a) determining start and end of deformation without explicit clutching and declutching, and (b) identifying user’s intent by characterizing grasp and motion of the hand on the pot. We evaluated our approach’s performance in terms of intent classification, users’ behavior, and users’ perception of controllability. We found that the expressive capability of hand articulation can be effectively harnessed for controllable shaping by organizing the deformation process in broad classes of intended operations such as pulling, pushing and fairing. After minimal practice with the pottery application, users could figure out their own strategy for reaching, grasping and deforming the pot. Further, the use of PCL as mid-air input allows for using common physical objects as tools for pot deformation. Users particularly enjoyed this aspect of our method for shaping pots.

Cross-modal retrieval aims to enable flexible retrieval experience across different modalities (e.g., texts vs. images). The core of cross-modal retrieval research is to learn a common subspace where the items of different modalities can be directly compared to each other. In this paper, we present a novel Adversarial Cross-Modal Retrieval (ACMR) method, which seeks an effective common subspace based on adversarial learning. Adversarial learning is implemented as an interplay between two processes. The first process, a feature projector, tries to generate a modality-invariant representation in the common subspace and to confuse the other process, modality classifier, which tries to discriminate between different modalities based on the generated representation. We further impose triplet constraints on the feature projector in order to minimize the gap among the representations of all items from different modalities with same semantic labels, while maximizing the distances among semantically different images and texts. Through the joint exploitation of the above, the underlying cross-modal semantic structure of multimedia data is better preserved when this data is projected into the common subspace. Comprehensive experimental results on four widely used benchmark datasets show that the proposed ACMR method is superior in learning effective subspace representation and that it significantly outperforms the state-of-the-art cross-modal retrieval methods.

The ubiquity of smartphones together with their ever-growing computing, networking and sensing powers have been changing the landscape of people’s daily life. Among others, activity recoginition, which takes the raw sensor reading as inputs and predicts a user’s motion activity, has become an active research area in recent years. It is the core building block in many high-impact applications, ranging from health and fitness monitoring, personal biometric signature, urban computing, assistive technology and elder-care, to indoor localization and navigation, etc. This paper presents a comprehensive survey of the recent advances in activity recognition with smartphones’ sensors. We start with the basic concepts such as sensors, activity types, etc. We review the core data mining techniques behind the main stream activity recognition algorithms, analyze their major challenges and introduce a variety of real applications enabled by activity recognition. Professor: Dr. Hanghang Tong Signature: Date: 06-12-2014 Professor: Dr. Ping Ji Signature: Date: 06-12-2014 Professor: Dr. Ted Brown Signature: Date: 06-12-2014 Advisor: Dr. Hanghang Tong

This paper presents a deep learning architecture for detecting the palm and fingertip positions of stable grasps directly from partial object views. The architecture is trained using RGBD image patches of fingertip and palm positions from grasps computed on complete object models using a grasping simulator. At runtime, the architecture is able to estimate grasp quality metrics without the need to explicitly calculate the given metric. This ability is useful as the exact calculation of these quality functions is impossible from an incomplete view of a novel object without any tactile feedback. This architecture for grasp quality prediction provides a framework for generalizing grasp experience from known to novel objects.

ÐA new method for the representation, recognition, and interpretation of parameterized gesture is presented. By parameterized gesture we mean gestures that exhibit a systematic spatial variation; one example is a point gesture where the relevant parameter is the two-dimensional direction. Our approach is to extend the standard hidden Markov model method of gesture recognition by including a global parametric variation in the output probabilities of the HMM states. Using a linear model of dependence, we formulate an expectation-maximization (EM) method for training the parametric HMM. During testing, a similar EM algorithm simultaneously maximizes the output likelihood of the PHMM for the given sequence and estimates the quantifying parameters. Using visually derived and directly measured three-dimensional hand position measurements as input, we present results that demonstrate the recognition superiority of the PHMM over standard HMM techniques, as well as greater robustness in parameter estimation with respect to noise in the input features. Last, we extend the PHMM to handle arbitrary smooth (nonlinear) dependencies. The nonlinear formulation requires the use of a generalized expectation-maximization (GEM) algorithm for both training and the simultaneous recognition of the gesture and estimation of the value of the parameter. We present results on a pointing gesture, where the nonlinear approach permits the natural spherical coordinate parameterization of pointing direction. Index TermsÐGesture recognition, hidden Markov models, expectation-maximization algorithm, time-series modeling, computer vision.

Recent advances in sensing technology have enabled a new generation of tabletop displays that can sense multiple points of input from several users simultaneously. However, apart from a few demonstration techniques [17], current user interfaces do not take advantage of this increased input bandwidth. We present a variety of multifinger and whole hand gestural interaction techniques for these displays that leverage and extend the types of actions that people perform when interacting on real physical tabletops. Apart from gestural input techniques, we also explore interaction and visualization techniques for supporting shared spaces, awareness, and privacy. These techniques are demonstrated within a prototype room furniture layout application, called RoomPlanner.

Interactions within virtual environments (VE) often require manipulation of 3D virtual objects. For this purpose, several research have been carried out focusing on mouse, touch and gestural input. While existing mid-air gestures in immersive VE (IVE) can offer natural manipulations, mimicking interactions with physical objects, they cannot achieve the same levels of precision attained with mouse-based applications for computer-aided design (CAD). Following prevailing trends in previous research, such as degrees-offreedom (DOF) separation, virtual widgets and scaled user motion, we intend to explore techniques for IVEs that combine the naturalness of mid-air gestures and the precision found in traditional CAD systems. In this document we survey and discuss the state-of-the-art in 3D object manipulation. With challenges identified, we present a research proposal and corresponding work plan.

Action recognition in videos remains a challenging problem in the machine learning community. Particularly challenging is the differing degree of intra-class variation between actions: While background information is enough to distinguish certain classes, many others are abstract and require fine-grained knowledge for discrimination. To approach this problem, in this work we evaluate different modalities on the recently published Moments in Time dataset, a collection of one million videos of short length.

Many multimedia applications can benefit from techniques for adapting existing classifiers to data with different distributions. One example is cross-domain video concept detection which aims to adapt concept classifiers across various video domains. In this paper, we explore two key problems for classifier adaptation: (1) how to transform existing classifier(s) into an effective classifier for a new dataset that only has a limited number of labeled examples, and (2) how to select the best existing classifier(s) for adaptation. For the first problem, we propose Adaptive Support Vector Machines (A-SVMs) as a general method to adapt one or more existing classifiers of any type to the new dataset. It aims to learn the "delta function" between the original and adapted classifier using an objective function similar to SVMs. For the second problem, we estimate the performance of each existing classifier on the sparsely-labeled new dataset by analyzing its score distribution and other meta features, and select the classifiers with the best estimated performance. The proposed method outperforms several baseline and competing methods in terms of classification accuracy and efficiency in cross-domain concept detection in the TRECVID corpus.


Question: Jack the Reader - A framework for machine reading comprehension.

This paper presents an innovative approach to the link prediction problem in statistical relational learning, a crucial task for automatically deciphering the structure of extensive knowledge bases. Our method employs complex valued embeddings in a latent factorization framework, distinguishing itself from previous studies that typically use real-valued embeddings. Complex embeddings offer a significant advantage in modeling a wide range of binary relations, including symmetric and antisymmetric relations, by leveraging the compositional properties of complex numbers. Contrary to complex models like Neural Tensor Networks and Holographic Embeddings, our method is remarkably straightforward, relying solely on the Hermitian dot product, which is essentially the complex equivalent of the standard dot product used for real vectors. This simplicity does not compromise performance or efficiency; our complex embeddings approach remains linear in both space and time complexity, making it scalable to large datasets. Moreover, it consistently surpasses alternative methods in standard link prediction benchmarks, demonstrating its superiority in handling complex relational structures.

Generating an article automatically with computer program is a challenging task in artificial intelligence and natural language processing. In this paper, we target at essay generation, which takes as input a topic word in mind and generates an organized article under the theme of the topic. We follow the idea of text planning (Reiter and Dale, 1997) and develop an essay generation framework. The framework consists of three components, including topic understanding, sentence extraction and sentence reordering. For each component, we studied several statistical algorithms and empirically compared between them in terms of qualitative or quantitative analysis. Although we run experiments on Chinese corpus, the method is language independent and can be easily adapted to other language. We lay out the remaining challenges and suggest avenues for future research.

Many applications of Machine Learning (ML) involve interactions with humans. Humans may provide input to a learning algorithm (in the form of labels, demonstrations, corrections, rankings or evaluations) while observing its outputs (in the form of feedback, predictions or executions). Although humans are an integral part of the learning process, traditional ML systems used in these applications are agnostic to the fact that inputs/outputs are from/for humans.
 However, a growing community of researchers at the intersection of ML and human-computer interaction are making interaction with humans a central part of developing ML systems. These efforts include applying interaction design principles to ML systems, using human-subject testing to evaluate ML systems and inspire new methods, and changing the input and output channels of ML systems to better leverage human capabilities. With this Interactive Machine Learning (IML) workshop at IUI 2013 we aim to bring this community together to share ideas, get up-to-date on recent advances, progress towards a common framework and terminology for the field, and discuss the open questions and challenges of IML.

In this paper we develop a story generator that leverages knowledge inherent in corpora without requiring extensive manual involvement. A key feature in our approach is the reliance on a story planner which we acquire automatically by recording events, their participants, and their precedence relationships in a training corpus. Contrary to previous work our system does not follow a generate-and-rank architecture. Instead, we employ evolutionary search techniques to explore the space of possible stories which we argue are well suited to the story generation task. Experiments on generating simple children’s stories show that our system outperforms previous data-driven approaches.

This paper describes the Ephyra question answering engine, a modular and extensible framework that allows to integrate multiple approaches to question answering in one system. Our framework can be adapted to languages other than English by replacing language-specific components. It supports the two major approaches to question answering, knowledge annotation and knowledge mining. Ephyra uses the web as a data resource, but could also work with smaller corpora. In addition, we propose a novel approach to question interpretation which abstracts from the original formulation of the question. Text patterns are used to interpret a question and to extract answers from text snippets. Our system automatically learns the patterns for answer extraction, using question-answer pairs as training data. Experimental results revealed the potential of this approach.

IBM Research has over 200 people working on Unstructured Information Management (UIM) technologies with a strong focus on Natural Language Processing (NLP). These researchers are engaged in activities ranging from natural language dialog, information retrieval, topic-tracking, named-entity detection, document classification and machine translation to bioinformatics and open-domain question answering. An analysis of these activities strongly suggested that improving the organization’s ability to quickly discover each other's results and rapidly combine different technologies and approaches would accelerate scientific advance. Furthermore, the ability to reuse and combine results through a common architecture and a robust software framework would accelerate the transfer of research results in NLP into IBM’s product platforms. Market analyses indicating a growing need to process unstructured information, specifically multilingual, natural language text, coupled with IBM Research’s investment in NLP, led to the development of middleware architecture for processing unstructured information dubbed UIMA. At the heart of UIMA are powerful search capabilities and a data-driven framework for the development, composition and distributed deployment of analysis engines. In this paper we give a general introduction to UIMA focusing on the design points of its analysis engine architecture and we discuss how UIMA is helping to accelerate research and technology transfer. 1 Motivation and Objectives We characterize a UIM application as a software system that analyzes large volumes of unstructured information while consulting structured information sources to discover, organize and present knowledge relevant to the application’s end-user. We define structured information as information whose intended meaning is unambiguous and explicitly represented in the structure or format of the data. The canonical example of structured information is a relational database table. We define unstructured information as information whose intended meaning is only loosely implied by its form. The canonical example is a natural language document; other examples include voice, audio, image and video. To appear in a special issue of the Journal of Natural Language Engineering 2004 2 Ferrucci and Lally Do Not Distirubte without written permission from authors Market indicators suggest a rapid increase in the use of text and voice analytics to a growing class of commercially viable applications (Roush 2003). National security interests are driving applications to involve an even broader set of UIM technologies including image and video analytics. Emerging and commercially important UIM application areas include life sciences, e-commerce, technical support, advanced search, and national and business intelligence. In six major labs spread out over the globe, IBM Research has over 200 people working on Unstructured Information Management (UIM) technologies with a primary focus on Natural Language Processing (NLP). These researchers are engaged in activities ranging from natural language dialog, information retrieval, topic-tracking, named-entity detection, document classification and machine translation to bioinformatics and open-domain question answering. Each group is developing different technical and engineering approaches to process unstructured information in pursuit of their specific research agenda in the UIM space. While IBM Research’s independent UIM technologies fare well in scientific venues, accelerated development of integrated, robust applications is becoming increasingly important to IBM’s commercial interests. Furthermore the rapid integration of different alogorthims and techniques is also a means to advance scientific results. For example, experiments in statistical name-entity detection or machine translation may benefit from using a deep parser as a feature generator. Facilitating the rapid integration of an independently produced deep parser with statistical algorithm implementations can accelerate the research cycle time as well as the time to product. Some of the challenges that face a large research organization in efficiently leveraging and integrating its (and third-party) UIM assets include: • Organizational Structure. IBM Research is largely organized in three to five person teams geographically dispersed among different labs throughout world. This structure makes it more difficult to jointly develop and reuse technologies. • Skills Alignment. There are inefficiencies in having PhDs trained and specialized in, for example, computational linguistics or statistical machine learning, develop and apply the requisite systems expertise to deliver their work into product architectures and/or make it easily usable by other researchers. To appear in a special issue of the Journal of Natural Language Engineering 2004 3 Ferrucci and Lally Do Not Distirubte without written permission from authors • Development Inefficiencies. Research results are delivered in a myriad of interfaces and technologies. Reuse and integration often requires too much effort dedicated to technical adaptation. Too often work is replicated. • Speed to product. Software Products introduce their own architectures, interfaces and implementation technologies. Technology transfer from the lab to the products often becomes a significant engineering or rewrite effort for researchers. A common framework and engineering discipline could help address the challenges listed above. The Unstructured Information Management Architecture (UIMA) project was initiated within IBM Research based on the premise that a common software architecture for developing, composing and delivering UIM technologies, if adopted by the organization, would facilitate technology reuse and integration, enable quicker scientific experimentation and speed the path to product. IBM’s Unstructured Information Management Architecture (UIMA) high-level objectives are two fold: 1) Accelerate scientific advances by enabling the rapid combination of UIM technologies (e.g., natural language processing, audio and video analysis, information retrieval, etc.). 2) Accelerate transfer of UIM technologies to product by providing an architecture and associated framework implementations that promote reuse and support flexible deployment options. 2 The Unstructured Information Management Architecture (UIMA) UIMA is a software architecture for developing UIM applications. The UIMA high-level architecture, illustrated in Figure 1, defines the roles, interfaces and communications of large-grained components essential for UIM applications. These include components capable of analyzing unstructured artifacts, integrating and accessing structured sources and storing, indexing and searching for artifacts based on discovered semantic content resulting from analyses. A primary design point for UIMA is that it should admit the implementation of middleware frameworks that provide a component-based infrastructure for developing UIM applications suitable for vastly different deployment environments. These should range from lightweight and embeddable implementations to highly scalable implementations that are meant to exploit clusters of machines and provide high throughput, high availability serviced-based offerings. While the primary focus of current UIMA framework implementations is squarely on natural language To appear in a special issue of the Journal of Natural Language Engineering 2004 4 Ferrucci and Lally Do Not Distirubte without written permission from authors UIM Application Analysis Collection Level Analysis Document, Collection & Metadata Store Knowledge Source Access Collection Processing Manager (CPM) Knowledge & Data Bases Unstructured Information (Text) Analysis Engines Crawlers Analysis Engine Directory Acquisition Unstructured Information Analysis Structured Information Access Semantic Search Engine Indices Indices Documents Collections Metadata Knowledge Source Adapter Directory Access

Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees. In this paper, we propose structural embedding of syntactic trees (SEST), an algorithm framework to utilize structured information and encode them into vector representations that can boost the performance of algorithms for the machine comprehension. We evaluate our approach using a state-of-the-art neural attention model on the SQuAD dataset. Experimental results demonstrate that our model can accurately identify the syntactic boundaries of the sentences and extract answers that are syntactically coherent over the baseline methods.

QA systems have been making steady advances in the challenging elementary science exam domain. In this work, we develop an explanation-based analysis of knowledge and inference requirements, which supports a fine-grained characterization of the challenges. In particular, we model the requirements based on appropriate sources of evidence to be used for the QA task. We create requirements by first identifying suitable sentences in a knowledge base that support the correct answer, then use these to build explanations, filling in any necessary missing information. These explanations are used to create a fine-grained categorization of the requirements. Using these requirements, we compare a retrieval and an inference solver on 212 questions. The analysis validates the gains of the inference solver, demonstrating that it answers more questions requiring complex inference, while also providing insights into the relative strengths of the solvers and knowledge sources. We release the annotated questions and explanations as a resource with broad utility for science exam QA, including determining knowledge base construction targets, as well as supporting information aggregation in automated inference.

While much health data is available online, patients who are not technically astute may be unable to access it because they may not know the relevant resources, they may be reluctant to confront an unfamiliar interface, and they may not know how to compose an answer from information provided by multiple heterogeneous resources. We describe ongoing research in using natural English text queries and automated deduction to obtain answers based on multiple structured data sources in a specific subject domain. Each English query is transformed using natural language technology into an unambiguous logical form; this is submitted to a theorem prover that operates over an axiomatic theory of the subject domain. Symbols in the theory are linked to relations in external databases known to the system. An answer is obtained from the proof, along with an English language explanation of how the answer was obtained. Answers need not be present explicitly in any of the databases, but rather may be deduced or computed from the information they provide. Although English is highly ambiguous, the natural language technology is informed by subject domain knowledge, so that readings of the query that are syntactically plausible but semantically impossible are discarded. When a question is still ambiguous, the system can interrogate the patient to determine what meaning was intended. Additional queries can clarify earlier ones or ask questions referring to previously computed answers. We describe a prototype system, Quadri, which answers questions about HIV treatment using the Stanford HIV Drug Resistance Database and other resources. Natural language processing is provided by PARC’s Bridge, and the deductive mechanism is SRI’s SNARK theorem prover. We discuss some of the problems that must be faced to make this approach work, and some of our solutions.

Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this thesis work, we explore how can we teach machines to ask clarification questions when faced with uncertainty, a goal of increasing importance in today’s automated society. We do a preliminary study using data from StackExchange, a plentiful online resource where people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We build neural network models inspired by the idea of the expected value of perfect information: a good question is one whose expected answer is going to be most useful. To build generalizable systems, we propose two future research directions: a template-based model and a sequence-to-sequence based neural generative model.

In this paper, we present a system that automatically generates questions from natural language text using discourse connectives. We explore the usefulness of the discourse connectives for Question Generation (QG) that looks at the problem beyond sentence level. Our work divides the QG task into content selection and question formation. Content selection consists of finding the relevant part in text to frame question from while question formation involves sense disambiguation of the discourse connectives, identification of question type and applying syntactic transformations on the content. The system is evaluated manually for syntactic and semantic correctness.